Reducing performance evaluation sensitivity and variability by input shaking

Dan Tsafrir*, Keren Ouaknine, Dror G. Feitelson

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

17 Scopus citations

Abstract

Simulations sometimes lead to observed sensitivity to configuration parameters as well as inconsistent performance results. The question is then what is the true effect and what is a coincidental artifact of the evaluation. The shaking methodology answers this by executing multiple simulations under small perturbations to the input workload, and calculating the average performance result; if the effect persists we can be more confident that it is real, whereas if it disappears it was an artifact. We present several examples where the sensitivity that appears in results based on a single evaluation is eliminated or considerably reduced by the shaking methodology. While our examples come from evaluations of scheduling algorithms for supercomputers, we believe the method has wider applicability.

Original languageAmerican English
Title of host publicationProceedings of MASCOTS'07 15th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems
Pages231-237
Number of pages7
DOIs
StatePublished - 2007
Event15th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, MASCOTS'07 - Istanbul, Turkey
Duration: 24 Oct 200726 Oct 2007

Publication series

NameIEEE International Workshop on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems - Proceedings

Conference

Conference15th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, MASCOTS'07
Country/TerritoryTurkey
CityIstanbul
Period24/10/0726/10/07

Keywords

  • Instability
  • Performance evaluation
  • Sensitivity
  • Simulation
  • Variability
  • Workload perturbations
  • Workload trace

Fingerprint

Dive into the research topics of 'Reducing performance evaluation sensitivity and variability by input shaking'. Together they form a unique fingerprint.

Cite this