LSTMs Exploit Linguistic Attributes of Data

Nelson F. Liu, Omer Levy, Roy Schwartz, Chenhao Tan, Noah A. Smith

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

8 Scopus citations

Abstract

While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data. We investigate how the properties of natural language data affect an LSTM's ability to learn a nonlinguistic task: recalling elements from its input. We find that models trained on natural language data are able to recall tokens from much longer sequences than models trained on non-language sequential data. Furthermore, we show that the LSTM learns to solve the memorization task by explicitly using a subset of its neurons to count timesteps in the input. We hypothesize that the patterns and structure in natural language data enable LSTMs to learn by providing approximate ways of reducing loss, but understanding the effect of different training data on the learnability of LSTMs remains an open question.

Original languageAmerican English
Title of host publicationACL 2018 - Representation Learning for NLP, Proceedings of the 3rd Workshop
PublisherAssociation for Computational Linguistics (ACL)
Pages180-186
Number of pages7
ISBN (Electronic)9781948087438
StatePublished - 2018
Externally publishedYes
Event3rd Workshop on Representation Learning for NLP, RepL4NLP 2018 at the 56th Annual Meeting of the Association for Computational Linguistics ACL 2018 - Melbourne, Australia
Duration: 20 Jul 2018 → …

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X

Conference

Conference3rd Workshop on Representation Learning for NLP, RepL4NLP 2018 at the 56th Annual Meeting of the Association for Computational Linguistics ACL 2018
Country/TerritoryAustralia
CityMelbourne
Period20/07/18 → …

Bibliographical note

Publisher Copyright:
© 2018 Association for Computational Linguistics.

Fingerprint

Dive into the research topics of 'LSTMs Exploit Linguistic Attributes of Data'. Together they form a unique fingerprint.

Cite this