Fine-grained analysis of sentence embeddings using auxiliary prediction tasks

Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, Yoav Goldberg

Research output: Contribution to conferencePaperpeer-review

223 Scopus citations

Abstract

There is a lot of research interest in encoding variable length sentences into fixed length vectors, in a way that preserves the sentence meanings. Two common methods include representations based on averaging word vectors, and representations based on the hidden states of recurrent neural networks such as LSTMs. The sentence vectors are used as features for subsequent machine learning tasks or for pre-training in the context of deep learning. However, not much is known about the properties that are encoded in these sentence representations and about the language information they capture. We propose a framework that facilitates better understanding of the encoded representations. We define prediction tasks around isolated aspects of sentence structure (namely sentence length, word content, and word order), and score representations by the ability to train a classifier to solve each prediction task when using the representation as input. We demonstrate the potential contribution of the approach by analyzing different sentence representation mechanisms. The analysis sheds light on the relative strengths of different sentence embedding methods with respect to these low level prediction tasks, and on the effect of the encoded vector's dimensionality on the resulting representations.

Original languageEnglish
StatePublished - 2017
Externally publishedYes
Event5th International Conference on Learning Representations, ICLR 2017 - Toulon, France
Duration: 24 Apr 201726 Apr 2017

Conference

Conference5th International Conference on Learning Representations, ICLR 2017
Country/TerritoryFrance
CityToulon
Period24/04/1726/04/17

Bibliographical note

Publisher Copyright:
© ICLR 2019 - Conference Track Proceedings. All rights reserved.

Fingerprint

Dive into the research topics of 'Fine-grained analysis of sentence embeddings using auxiliary prediction tasks'. Together they form a unique fingerprint.

Cite this