Communication lower bounds and optimal algorithms for numerical linear algebra

G. Ballard, E. Carson, J. Demmel, M. Hoemmen, N. Knight, O. Schwartz

Research output: Contribution to journalArticlepeer-review

71 Scopus citations

Abstract

The traditional metric for the efficiency of a numerical algorithm has been the number of arithmetic operations it performs. Technological trends have long been reducing the time to perform an arithmetic operation, so it is no longer the bottleneck in many algorithms; rather, communication, or moving data, is the bottleneck. This motivates us to seek algorithms that move as little data as possible, either between levels of a memory hierarchy or between parallel processors over a network. In this paper we summarize recent progress in three aspects of this problem. First we describe lower bounds on communication. Some of these generalize known lower bounds for dense classical (O(n 3)) matrix multiplication to all direct methods of linear algebra, to sequential and parallel algorithms, and to dense and sparse matrices. We also present lower bounds for Strassen-like algorithms, and for iterative methods, in particular Krylov subspace methods applied to sparse matrices. Second, we compare these lower bounds to widely used versions of these algorithms, and note that these widely used algorithms usually communicate asymptotically more than is necessary. Third, we identify or invent new algorithms for most linear algebra problems that do attain these lower bounds, and demonstrate large speed-ups in theory and practice.

Original languageAmerican English
Pages (from-to)1-155
Number of pages155
JournalActa Numerica
Volume23
DOIs
StatePublished - May 2014
Externally publishedYes

Fingerprint

Dive into the research topics of 'Communication lower bounds and optimal algorithms for numerical linear algebra'. Together they form a unique fingerprint.

Cite this