On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation

Gal Patel, Leshem Choshen, Omri Abend

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

We present a methodology that explores how sentence structure is reflected in neural representations of machine translation systems. We demonstrate our model-agnostic approach with the Transformer English-German translation model. We analyze neuron-level correlation of activations between paraphrases while discussing the methodology challenges and the need for confound analysis to isolate the effects of shallow cues. We find that similarity between activation patterns can be mostly accounted for by similarity in word choice and sentence length. Following that, we manipulate neuron activations to control the syntactic form of the output. We show this intervention to be somewhat successful, indicating that deep models capture sentence-structure distinctions, despite finding no such indication at the neuron level. To conduct our experiments, we develop a semi-automatic method to generate meaning-preserving minimal pair paraphrases (active-passive voice and adverbial clause-noun phrase) and compile a corpus of such pairs.

Original languageAmerican English
Title of host publicationCoNLL 2022 - 26th Conference on Computational Natural Language Learning, Proceedings of the Conference
PublisherAssociation for Computational Linguistics (ACL)
Pages194-212
Number of pages19
ISBN (Electronic)9781959429074
StatePublished - 2022
Event26th Conference on Computational Natural Language Learning, CoNLL 2022 collocated and co-organized with EMNLP 2022 - Abu Dhabi, United Arab Emirates
Duration: 7 Dec 20228 Dec 2022

Publication series

NameCoNLL 2022 - 26th Conference on Computational Natural Language Learning, Proceedings of the Conference

Conference

Conference26th Conference on Computational Natural Language Learning, CoNLL 2022 collocated and co-organized with EMNLP 2022
Country/TerritoryUnited Arab Emirates
CityAbu Dhabi
Period7/12/228/12/22

Bibliographical note

Publisher Copyright:
©2022 Association for Computational Linguistics.

Fingerprint

Dive into the research topics of 'On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation'. Together they form a unique fingerprint.

Cite this