Analysis of sentence embedding models using prediction tasks in natural language processing

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, Yoav Goldberg

Research output: Contribution to journalArticlepeer-review

14 Scopus citations

Abstract

The tremendous success of word embeddings in improving the ability of computers to perform natural language tasks has shifted the research on language representation from word representation to focus on sentence representation. This shift introduced a plethora of methods for learning vector representations of sentences, many of them based on compositional methods over word embeddings. These vectors are used as features for subsequent machine learning tasks or for pretraining in the context of deep learning. However, not much is known about the properties that are encoded in these sentence representations and about the language information they encapsulate. Recent studies analyze the encoded representations and the kind of information they capture. In this paper, we analyze results from a previous study on the ability of models to encode basic properties such as content, order, and length. Our analysis led to new insights, such as the effect of word frequency or word distance on the ability to encode content and order.

Original languageAmerican English
Article number8030297
JournalIBM Journal of Research and Development
Volume61
Issue number4
DOIs
StatePublished - 1 Jul 2017
Externally publishedYes

Bibliographical note

Publisher Copyright:
© 2017 IBM.

Fingerprint

Dive into the research topics of 'Analysis of sentence embedding models using prediction tasks in natural language processing'. Together they form a unique fingerprint.

Cite this