The workload on parallel supercomputers: Modelling the characteristics of rigid jobs

Uri Lublin; Dror G. Feitelson

doi:10.1016/S0743-7315(03)00108-4

The workload on parallel supercomputers: Modelling the characteristics of rigid jobs

Uri Lublin, Dror G. Feitelson^*

^*Corresponding author for this work

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Contribution to journal › Article › peer-review

327 Scopus citations

Abstract

The analysis of workloads is important for understanding how systems are used. In addition, workload models are needed as input for the evaluation of new system designs, and for the comparison of system designs. This is especially important in costly large-scale parallel systems. Luckily, workload data are available in the form of accounting logs. Using such logs from three different sites, we analyze and model the job-level workloads with an emphasis on those aspects that are universal to all sites. As many distributions turn out to span a large range, we typically first apply a logarithmic transformation to the data, and then fit it to a novel hyper-Gamma distribution or one of its special cases. This is a generalization of distributions proposed previously, and leads to good goodness-of-fit scores. The parameters for the distribution are found using the iterative EM algorithm. The results of the analysis have been codified in a modeling program that creates a synthetic workload based on the results of the analysis.

Original language	American English
Pages (from-to)	1105-1122
Number of pages	18
Journal	Journal of Parallel and Distributed Computing
Volume	63
Issue number	11
DOIs	https://doi.org/10.1016/S0743-7315(03)00108-4
State	Published - Nov 2003

Bibliographical note

Funding Information:
This research was supported in part by the Israel Science Foundation (Grant No. 219/99). The workload logs on which it is based are available on-line from the Parallel Workloads Archive [19] . The workload log from the SDSC Paragon was graciously provided by Reagan Moore and Allen Downey. The workload log from the KTH SP2 was graciously provided by Lars Malinowsky. The workload log from the LANL CM-5 was graciously provided by Curt Canada. Many thanks to them for making the data available and for their help with background information and interpretation. Thanks also to Prof. Ya'acov Ritov for statistics advice.

Keywords

Arrival pattern
Parallel jobs
Runtime distribution
Size distribution
Workload model

Access to Document

10.1016/S0743-7315(03)00108-4

Cite this

@article{1d8de7aff9b24969ae61b3970fdba335,

title = "The workload on parallel supercomputers: Modelling the characteristics of rigid jobs",

abstract = "The analysis of workloads is important for understanding how systems are used. In addition, workload models are needed as input for the evaluation of new system designs, and for the comparison of system designs. This is especially important in costly large-scale parallel systems. Luckily, workload data are available in the form of accounting logs. Using such logs from three different sites, we analyze and model the job-level workloads with an emphasis on those aspects that are universal to all sites. As many distributions turn out to span a large range, we typically first apply a logarithmic transformation to the data, and then fit it to a novel hyper-Gamma distribution or one of its special cases. This is a generalization of distributions proposed previously, and leads to good goodness-of-fit scores. The parameters for the distribution are found using the iterative EM algorithm. The results of the analysis have been codified in a modeling program that creates a synthetic workload based on the results of the analysis.",

keywords = "Arrival pattern, Parallel jobs, Runtime distribution, Size distribution, Workload model",

author = "Uri Lublin and Feitelson, {Dror G.}",

note = "Funding Information: This research was supported in part by the Israel Science Foundation (Grant No. 219/99). The workload logs on which it is based are available on-line from the Parallel Workloads Archive [19] . The workload log from the SDSC Paragon was graciously provided by Reagan Moore and Allen Downey. The workload log from the KTH SP2 was graciously provided by Lars Malinowsky. The workload log from the LANL CM-5 was graciously provided by Curt Canada. Many thanks to them for making the data available and for their help with background information and interpretation. Thanks also to Prof. Ya'acov Ritov for statistics advice. ",

year = "2003",

month = nov,

doi = "10.1016/S0743-7315(03)00108-4",

language = "American English",

volume = "63",

pages = "1105--1122",

journal = "Journal of Parallel and Distributed Computing",

issn = "0743-7315",

publisher = "Academic Press Inc.",

number = "11",

}

TY - JOUR

T1 - The workload on parallel supercomputers

T2 - Modelling the characteristics of rigid jobs

AU - Lublin, Uri

AU - Feitelson, Dror G.

N1 - Funding Information: This research was supported in part by the Israel Science Foundation (Grant No. 219/99). The workload logs on which it is based are available on-line from the Parallel Workloads Archive [19] . The workload log from the SDSC Paragon was graciously provided by Reagan Moore and Allen Downey. The workload log from the KTH SP2 was graciously provided by Lars Malinowsky. The workload log from the LANL CM-5 was graciously provided by Curt Canada. Many thanks to them for making the data available and for their help with background information and interpretation. Thanks also to Prof. Ya'acov Ritov for statistics advice.

PY - 2003/11

Y1 - 2003/11

N2 - The analysis of workloads is important for understanding how systems are used. In addition, workload models are needed as input for the evaluation of new system designs, and for the comparison of system designs. This is especially important in costly large-scale parallel systems. Luckily, workload data are available in the form of accounting logs. Using such logs from three different sites, we analyze and model the job-level workloads with an emphasis on those aspects that are universal to all sites. As many distributions turn out to span a large range, we typically first apply a logarithmic transformation to the data, and then fit it to a novel hyper-Gamma distribution or one of its special cases. This is a generalization of distributions proposed previously, and leads to good goodness-of-fit scores. The parameters for the distribution are found using the iterative EM algorithm. The results of the analysis have been codified in a modeling program that creates a synthetic workload based on the results of the analysis.

AB - The analysis of workloads is important for understanding how systems are used. In addition, workload models are needed as input for the evaluation of new system designs, and for the comparison of system designs. This is especially important in costly large-scale parallel systems. Luckily, workload data are available in the form of accounting logs. Using such logs from three different sites, we analyze and model the job-level workloads with an emphasis on those aspects that are universal to all sites. As many distributions turn out to span a large range, we typically first apply a logarithmic transformation to the data, and then fit it to a novel hyper-Gamma distribution or one of its special cases. This is a generalization of distributions proposed previously, and leads to good goodness-of-fit scores. The parameters for the distribution are found using the iterative EM algorithm. The results of the analysis have been codified in a modeling program that creates a synthetic workload based on the results of the analysis.

KW - Arrival pattern

KW - Parallel jobs

KW - Runtime distribution

KW - Size distribution

KW - Workload model

UR - http://www.scopus.com/inward/record.url?scp=0345446547&partnerID=8YFLogxK

U2 - 10.1016/S0743-7315(03)00108-4

DO - 10.1016/S0743-7315(03)00108-4

M3 - Article

AN - SCOPUS:0345446547

SN - 0743-7315

VL - 63

SP - 1105

EP - 1122

JO - Journal of Parallel and Distributed Computing

JF - Journal of Parallel and Distributed Computing

IS - 11

ER -

The workload on parallel supercomputers: Modelling the characteristics of rigid jobs

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this