Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units

Gallil Maimon; Yossi Adi

doi:10.18653/v1/2023.findings-emnlp.541

Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units

Gallil Maimon, Yossi Adi

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people's unique speaking style (prosody). The proposed approach uses a pretrained, self-supervised model for encoding speech to discrete units, which makes it simple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at https://pages.cs.huji.ac.il/adiyoss-lab/dissc/.

Original language	English
Title of host publication	Findings of the Association for Computational Linguistics
Subtitle of host publication	EMNLP 2023
Publisher	Association for Computational Linguistics (ACL)
Pages	8048-8061
Number of pages	14
ISBN (Electronic)	9798891760615
DOIs	https://doi.org/10.18653/v1/2023.findings-emnlp.541
State	Published - 2023
Event	2023 Findings of the Association for Computational Linguistics: EMNLP 2023 - Singapore, Singapore Duration: 6 Dec 2023 → 10 Dec 2023

Publication series

Name	Findings of the Association for Computational Linguistics: EMNLP 2023

Conference

Conference	2023 Findings of the Association for Computational Linguistics: EMNLP 2023
Country/Territory	Singapore
City	Singapore
Period	6/12/23 → 10/12/23

Bibliographical note

Publisher Copyright:
© 2023 Association for Computational Linguistics.

Access to Document

10.18653/v1/2023.findings-emnlp.541

Cite this

Maimon, G., & Adi, Y. (2023). Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units. In Findings of the Association for Computational Linguistics: EMNLP 2023 (pp. 8048-8061). (Findings of the Association for Computational Linguistics: EMNLP 2023). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2023.findings-emnlp.541

@inproceedings{3a896e02906048c2b48527f3d02e16fe,

title = "Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units",

abstract = "We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people's unique speaking style (prosody). The proposed approach uses a pretrained, self-supervised model for encoding speech to discrete units, which makes it simple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at https://pages.cs.huji.ac.il/adiyoss-lab/dissc/.",

author = "Gallil Maimon and Yossi Adi",

note = "Publisher Copyright: {\textcopyright} 2023 Association for Computational Linguistics.; 2023 Findings of the Association for Computational Linguistics: EMNLP 2023 ; Conference date: 06-12-2023 Through 10-12-2023",

year = "2023",

doi = "10.18653/v1/2023.findings-emnlp.541",

language = "אנגלית",

series = "Findings of the Association for Computational Linguistics: EMNLP 2023",

publisher = "Association for Computational Linguistics (ACL)",

pages = "8048--8061",

booktitle = "Findings of the Association for Computational Linguistics",

address = "ארצות הברית",

}

Maimon, G & Adi, Y 2023, Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units. in Findings of the Association for Computational Linguistics: EMNLP 2023. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics (ACL), pp. 8048-8061, 2023 Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, Singapore, 6/12/23. https://doi.org/10.18653/v1/2023.findings-emnlp.541

Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units. / Maimon, Gallil; Adi, Yossi.
Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics (ACL), 2023. p. 8048-8061 (Findings of the Association for Computational Linguistics: EMNLP 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units

AU - Maimon, Gallil

AU - Adi, Yossi

PY - 2023

Y1 - 2023

N2 - We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people's unique speaking style (prosody). The proposed approach uses a pretrained, self-supervised model for encoding speech to discrete units, which makes it simple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at https://pages.cs.huji.ac.il/adiyoss-lab/dissc/.

AB - We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people's unique speaking style (prosody). The proposed approach uses a pretrained, self-supervised model for encoding speech to discrete units, which makes it simple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at https://pages.cs.huji.ac.il/adiyoss-lab/dissc/.

UR - http://www.scopus.com/inward/record.url?scp=85183296428&partnerID=8YFLogxK

U2 - 10.18653/v1/2023.findings-emnlp.541

DO - 10.18653/v1/2023.findings-emnlp.541

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85183296428

T3 - Findings of the Association for Computational Linguistics: EMNLP 2023

SP - 8048

EP - 8061

BT - Findings of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2023 Findings of the Association for Computational Linguistics: EMNLP 2023

Y2 - 6 December 2023 through 10 December 2023

ER -

Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units

Abstract

Publication series

Conference

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this