Abstract
Workflows are used to orchestrate data-intensive applications in many different scientific domains. Workflow applications typically communicate data between processing steps using intermediate files. When tasks are distributed, these files are either transferred from one computational node to another, or accessed through a shared storage system. As a result, the efficient management of data is a key factor in achieving good performance for workflow applications in distributed environments. In this paper we investigate some of the ways in which data can be managed for workflows in the cloud. We ran experiments using three typical workflow applications on Amazon's EC2 cloud computing platform. We discuss the various storage and file systems we used, describe the issues and problems we encountered deploying them on EC2, and analyze the resulting performance and cost of the workflows.
| Original language | English |
|---|---|
| Pages (from-to) | 5-21 |
| Number of pages | 17 |
| Journal | Journal of Grid Computing |
| Volume | 10 |
| Issue number | 1 |
| DOIs | |
| State | Published - Mar 2012 |
| Externally published | Yes |
Bibliographical note
Funding Information:Acknowledgements This work was supported by the National Science Foundation under the IntelData (IIS-0905032) and Pegasus (OCI-0722019) grants. This research made use of Montage, funded by the National Aeronautics and Space Administration’s Earth Science Technology Office, Computation Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology.
Keywords
- Cloud computing
- Scientific workflows