Semi-open trace based simulation for reliable evaluation of job throughput and user productivity

New scheduling algorithms are first evaluated using simulation. In these simulations, the workload has a huge influence on the measured performance of the simulated system. Therefore, it is customary to use workload traces recorded previously from real systems. Such open-system simulations preserve all the jobs' properties. However, preserving the jobs' arrival times actually destroys the logic of the user's workflow, especially dependencies and think times between successive jobs. Furthermore, performance in such simulations is measured by the average wait time and slowdown, under the fixed load and throughput conditions dictated by the trace. Therefore, it is impossible to evaluate the system's effect on throughput and productivity. As an alternative we propose semi-open trace based simulations that include dynamic user activity and internal feedback from the system to the users. In these simulations, like in a real system, users adjust their job-submittal behavior in response to system performance. As a result, the simulations produce different loads and throughputs for different scheduling algorithms or parametrizations. We implemented such a simulation for evaluating the schedulers of parallel job systems. We also developed a novel user-aware scheduler designed specifically to increase users' productivity. While conventional simulations cannot measure this scheduler's influence reliably, and would suggest it is useless, our simulation evaluates it realistically and shows its beneficial effect on the users' productivity and the system's throughput.