Speech resynthesis from discrete disentangled self-supervised representations

Adam Polyak*, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

58 Scopus citations

Abstract

We propose using self-supervised discrete representations for the task of speech resynthesis. To generate disentangled representation, we separately extract low-bitrate representations for speech content, prosodic information, and speaker identity. This allows to synthesize speech in a controllable manner. We analyze various state-of-the-art, self-supervised representation learning methods and shed light on the advantages of each method while considering reconstruction quality and disentanglement properties. Specifically, we evaluate the F0 reconstruction, speaker identification performance (for both resynthesis and voice conversion), recordings' intelligibility, and overall quality using subjective human evaluation. Lastly, we demonstrate how these representations can be used for an ultra-lightweight speech codec. Using the obtained representations, we can get to a rate of 365 bits per second while providing better speech quality than the baseline methods. Audio samples can be found under the following link: resynthesis-ssl.github.io.

Original languageAmerican English
Title of host publication22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021
PublisherInternational Speech Communication Association
Pages3531-3535
Number of pages5
ISBN (Electronic)9781713836902
DOIs
StatePublished - 2021
Externally publishedYes
Event22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021 - Brno, Czech Republic
Duration: 30 Aug 20213 Sep 2021

Publication series

NameProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Volume5
ISSN (Print)2308-457X
ISSN (Electronic)1990-9772

Conference

Conference22nd Annual Conference of the International Speech Communication Association, INTERSPEECH 2021
Country/TerritoryCzech Republic
CityBrno
Period30/08/213/09/21

Bibliographical note

Publisher Copyright:
© 2021 ISCA

Keywords

  • Self-supervised learning
  • Speech codec
  • Speech generation
  • Speech resynthesis

Fingerprint

Dive into the research topics of 'Speech resynthesis from discrete disentangled self-supervised representations'. Together they form a unique fingerprint.

Cite this