LSTMs Exploit Linguistic Attributes of Data

Nelson F. Liu; Omer Levy; Roy Schwartz; Chenhao Tan; Noah A. Smith

doi:10.18653/v1/w18-3024

LSTMs Exploit Linguistic Attributes of Data

Nelson F. Liu, Omer Levy, Roy Schwartz, Chenhao Tan, Noah A. Smith

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. We investigate how the properties of natural language data affect an LSTM's ability to learn a nonlinguistic task: recalling elements from its input. We find that models trained on natural language data are able to recall tokens from much longer sequences than models trained on non-language sequential data. Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training data on the learnability of LSTMs remains an open question.

Original language	American English
Title of host publication	ACL 2018 - Representation Learning for NLP, Proceedings of the 3rd Workshop
Publisher	Association for Computational Linguistics (ACL)
Pages	180-186
Number of pages	7
ISBN (Electronic)	9781948087438
DOIs	https://doi.org/10.18653/v1/w18-3024
State	Published - 2018
Externally published	Yes
Event	3rd Workshop on Representation Learning for NLP, RepL4NLP 2018 at the 56th Annual Meeting of the Association for Computational Linguistics ACL 2018 - Melbourne, Australia Duration: 20 Jul 2018 → …

Publication series

Name	Proceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)	0736-587X

Conference

Conference	3rd Workshop on Representation Learning for NLP, RepL4NLP 2018 at the 56th Annual Meeting of the Association for Computational Linguistics ACL 2018
Country/Territory	Australia
City	Melbourne
Period	20/07/18 → …

Bibliographical note

Funding Information:
We thank the ARK as well as the anonymous reviewers for their valuable feedback. NL is supported by a Washington Research Foundation Fellowship and a Barry M. Goldwater Scholarship. This work was supported in part by a hardware gift from NVIDIA Corporation and a UW High Performance Computing Club Cloud Credit Award.

Publisher Copyright:
© 2018 Association for Computational Linguistics.

Access to Document

10.18653/v1/w18-3024

https://aclanthology.org/W18-3024/

Cite this

Liu, N. F., Levy, O., Schwartz, R., Tan, C., & Smith, N. A. (2018). LSTMs Exploit Linguistic Attributes of Data. In ACL 2018 - Representation Learning for NLP, Proceedings of the 3rd Workshop (pp. 180-186). (Proceedings of the Annual Meeting of the Association for Computational Linguistics). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/w18-3024

@inproceedings{6ec37c0102e047c892167395e4543802,

title = "LSTMs Exploit Linguistic Attributes of Data",

abstract = "While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. We investigate how the properties of natural language data affect an LSTM's ability to learn a nonlinguistic task: recalling elements from its input. We find that models trained on natural language data are able to recall tokens from much longer sequences than models trained on non-language sequential data. Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training data on the learnability of LSTMs remains an open question.",

author = "Liu, {Nelson F.} and Omer Levy and Roy Schwartz and Chenhao Tan and Smith, {Noah A.}",

note = "Funding Information: We thank the ARK as well as the anonymous reviewers for their valuable feedback. NL is supported by a Washington Research Foundation Fellowship and a Barry M. Goldwater Scholarship. This work was supported in part by a hardware gift from NVIDIA Corporation and a UW High Performance Computing Club Cloud Credit Award. Publisher Copyright: {\textcopyright} 2018 Association for Computational Linguistics.; 3rd Workshop on Representation Learning for NLP, RepL4NLP 2018 at the 56th Annual Meeting of the Association for Computational Linguistics ACL 2018 ; Conference date: 20-07-2018",

year = "2018",

doi = "10.18653/v1/w18-3024",

language = "American English",

series = "Proceedings of the Annual Meeting of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

pages = "180--186",

booktitle = "ACL 2018 - Representation Learning for NLP, Proceedings of the 3rd Workshop",

address = "United States",

}

Liu, NF, Levy, O, Schwartz, R, Tan, C & Smith, NA 2018, LSTMs Exploit Linguistic Attributes of Data. in ACL 2018 - Representation Learning for NLP, Proceedings of the 3rd Workshop. Proceedings of the Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics (ACL), pp. 180-186, 3rd Workshop on Representation Learning for NLP, RepL4NLP 2018 at the 56th Annual Meeting of the Association for Computational Linguistics ACL 2018, Melbourne, Australia, 20/07/18. https://doi.org/10.18653/v1/w18-3024

LSTMs Exploit Linguistic Attributes of Data. / Liu, Nelson F.; Levy, Omer; Schwartz, Roy et al.
ACL 2018 - Representation Learning for NLP, Proceedings of the 3rd Workshop. Association for Computational Linguistics (ACL), 2018. p. 180-186 (Proceedings of the Annual Meeting of the Association for Computational Linguistics).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - LSTMs Exploit Linguistic Attributes of Data

AU - Liu, Nelson F.

AU - Levy, Omer

AU - Schwartz, Roy

AU - Tan, Chenhao

AU - Smith, Noah A.

N1 - Funding Information: We thank the ARK as well as the anonymous reviewers for their valuable feedback. NL is supported by a Washington Research Foundation Fellowship and a Barry M. Goldwater Scholarship. This work was supported in part by a hardware gift from NVIDIA Corporation and a UW High Performance Computing Club Cloud Credit Award. Publisher Copyright: © 2018 Association for Computational Linguistics.

PY - 2018

Y1 - 2018

N2 - While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. We investigate how the properties of natural language data affect an LSTM's ability to learn a nonlinguistic task: recalling elements from its input. We find that models trained on natural language data are able to recall tokens from much longer sequences than models trained on non-language sequential data. Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training data on the learnability of LSTMs remains an open question.

AB - While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. We investigate how the properties of natural language data affect an LSTM's ability to learn a nonlinguistic task: recalling elements from its input. We find that models trained on natural language data are able to recall tokens from much longer sequences than models trained on non-language sequential data. Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training data on the learnability of LSTMs remains an open question.

UR - http://www.scopus.com/inward/record.url?scp=85122037839&partnerID=8YFLogxK

U2 - 10.18653/v1/w18-3024

DO - 10.18653/v1/w18-3024

M3 - Conference contribution

T3 - Proceedings of the Annual Meeting of the Association for Computational Linguistics

SP - 180

EP - 186

BT - ACL 2018 - Representation Learning for NLP, Proceedings of the 3rd Workshop

PB - Association for Computational Linguistics (ACL)

T2 - 3rd Workshop on Representation Learning for NLP, RepL4NLP 2018 at the 56th Annual Meeting of the Association for Computational Linguistics ACL 2018

Y2 - 20 July 2018

ER -

LSTMs Exploit Linguistic Attributes of Data

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this