Efficient Methods for Natural Language Processing: A Survey

Marcos Treviso; Ji Ung Lee; Tianchu Ji; Betty van Aken; Qingqing Cao; Manuel R. Ciosici; Michael Hassid; Kenneth Heafield; Sara Hooker; Colin Raffel; Pedro H. Martins; Andre F.T. Martins; Jessica Zosa Forde; Peter Milder; Edwin Simpson; Noam Slonim; Jesse Dodge; Emma Strubell; Niranjan Balasubramanian; Leon Derczynski; Iryna Gurevych; Roy Schwartz

doi:10.1162/tacl_a_00577

Efficient Methods for Natural Language Processing: A Survey

Marcos Treviso^*
, Ji Ung Lee
, Tianchu Ji
, Betty van Aken
, Qingqing Cao
, Manuel R. Ciosici
, Michael Hassid
, Kenneth Heafield
, Sara Hooker
, Colin Raffel
, Pedro H. Martins
, Andre F.T. Martins
, Jessica Zosa Forde
, Peter Milder
, Edwin Simpson
, Noam Slonim
, Jesse Dodge
, Emma Strubell
, Niranjan Balasubramanian
, Leon Derczynski

Iryna Gurevych, Roy Schwartz

^*Corresponding author for this work

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Contribution to journal › Article › peer-review

78 Scopus citations

Abstract

Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve per¬formance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This mo¬tivates research into efficient methods that require fewer resources to achieve similar re¬sults. This survey synthesizes and relates cur¬rent methods and findings in efficient NLP. We aim to provide both guidance for con-ducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

Original language	English
Pages (from-to)	826-860
Number of pages	35
Journal	Transactions of the Association for Computational Linguistics
Volume	11
DOIs	https://doi.org/10.1162/tacl_a_00577
State	Published - 2023

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Access to Document

10.1162/tacl_a_00577

Cite this

Treviso, M., Lee, J. U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., Martins, P. H., Martins, A. F. T., Forde, J. Z., Milder, P., Simpson, E., Slonim, N., Dodge, J., Strubell, E., Balasubramanian, N., ... Schwartz, R. (2023). Efficient Methods for Natural Language Processing: A Survey. Transactions of the Association for Computational Linguistics, 11, 826-860. https://doi.org/10.1162/tacl_a_00577

@article{9727bada3ffd4d918fa249b5f4da926c,

title = "Efficient Methods for Natural Language Processing: A Survey",

abstract = "Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve per¬formance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This mo¬tivates research into efficient methods that require fewer resources to achieve similar re¬sults. This survey synthesizes and relates cur¬rent methods and findings in efficient NLP. We aim to provide both guidance for con-ducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.",

author = "Marcos Treviso and Lee, \{Ji Ung\} and Tianchu Ji and \{van Aken\}, Betty and Qingqing Cao and Ciosici, \{Manuel R.\} and Michael Hassid and Kenneth Heafield and Sara Hooker and Colin Raffel and Martins, \{Pedro H.\} and Martins, \{Andre F.T.\} and Forde, \{Jessica Zosa\} and Peter Milder and Edwin Simpson and Noam Slonim and Jesse Dodge and Emma Strubell and Niranjan Balasubramanian and Leon Derczynski and Iryna Gurevych and Roy Schwartz",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.",

year = "2023",

doi = "10.1162/tacl\_a\_00577",

language = "אנגלית",

volume = "11",

pages = "826--860",

journal = "Transactions of the Association for Computational Linguistics",

issn = "2307-387X",

publisher = "MIT Press",

}

Treviso, M, Lee, JU, Ji, T, van Aken, B, Cao, Q, Ciosici, MR, Hassid, M, Heafield, K, Hooker, S, Raffel, C, Martins, PH, Martins, AFT, Forde, JZ, Milder, P, Simpson, E, Slonim, N, Dodge, J, Strubell, E, Balasubramanian, N, Derczynski, L, Gurevych, I & Schwartz, R 2023, 'Efficient Methods for Natural Language Processing: A Survey', Transactions of the Association for Computational Linguistics, vol. 11, pp. 826-860. https://doi.org/10.1162/tacl_a_00577

TY - JOUR

T1 - Efficient Methods for Natural Language Processing

T2 - A Survey

AU - Treviso, Marcos

AU - Lee, Ji Ung

AU - Ji, Tianchu

AU - van Aken, Betty

AU - Cao, Qingqing

AU - Ciosici, Manuel R.

AU - Hassid, Michael

AU - Heafield, Kenneth

AU - Hooker, Sara

AU - Raffel, Colin

AU - Martins, Pedro H.

AU - Martins, Andre F.T.

AU - Forde, Jessica Zosa

AU - Milder, Peter

AU - Simpson, Edwin

AU - Slonim, Noam

AU - Dodge, Jesse

AU - Strubell, Emma

AU - Balasubramanian, Niranjan

AU - Derczynski, Leon

AU - Gurevych, Iryna

AU - Schwartz, Roy

PY - 2023

Y1 - 2023

N2 - Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve per¬formance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This mo¬tivates research into efficient methods that require fewer resources to achieve similar re¬sults. This survey synthesizes and relates cur¬rent methods and findings in efficient NLP. We aim to provide both guidance for con-ducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

AB - Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve per¬formance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This mo¬tivates research into efficient methods that require fewer resources to achieve similar re¬sults. This survey synthesizes and relates cur¬rent methods and findings in efficient NLP. We aim to provide both guidance for con-ducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.

UR - https://www.scopus.com/pages/publications/85168613194

U2 - 10.1162/tacl_a_00577

DO - 10.1162/tacl_a_00577

M3 - ???researchoutput.researchoutputtypes.contributiontojournal.article???

AN - SCOPUS:85168613194

SN - 2307-387X

VL - 11

SP - 826

EP - 860

JO - Transactions of the Association for Computational Linguistics

JF - Transactions of the Association for Computational Linguistics

ER -

Efficient Methods for Natural Language Processing: A Survey

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this