The learnability consequences of Zipfian distributions in language

Ori Lavi-Rotbain*, Inbal Arnon

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

10 Scopus citations

Abstract

While the languages of the world differ in many respects, they share certain commonalties, which can provide insight on our shared cognition. Here, we explore the learnability consequences of one of the striking commonalities between languages. Across languages, word frequencies follow a Zipfian distribution, showing a power law relation between a word's frequency and its rank. While their source in language has been studied extensively, less work has explored the learnability consequences of such distributions for language learners. We propose that the greater predictability of words in this distribution (relative to less skewed distributions) can facilitate word segmentation, a crucial aspect of early language acquisition. To explore this, we quantify word predictability using unigram entropy, assess it across languages using naturalistic corpora of child-directed speech and then ask whether similar unigram predictability facilitates word segmentation in the lab. We find similar unigram entropy in child-directed speech across 15 languages. We then use an auditory word segmentation task to show that the unigram predictability levels found in natural language are uniquely facilitative for word segmentation for both children and adults. These findings illustrate the facilitative impact of skewed input distributions on learning and raise questions about the possible role of cognitive pressures in the prevalence of Zipfian distributions in language.

Original languageAmerican English
Article number105038
JournalCognition
Volume223
DOIs
StatePublished - Jun 2022

Bibliographical note

Publisher Copyright:
© 2022

Keywords

  • Distributional learning
  • Information theory
  • Language acquisition
  • Word segmentation
  • Zipf's law

Fingerprint

Dive into the research topics of 'The learnability consequences of Zipfian distributions in language'. Together they form a unique fingerprint.

Cite this