We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We tackle the problem by first applying a self-supervised discrete speech encoder on the target speech and then training a sequence-to-sequence speech-to-unit translation (S2UT) model to predict the discrete representations of the target speech. When target text transcripts are available, we design a joint speech and text training framework that enables the model to generate dual modality output (speech and text) simultaneously in the same inference pass. Experiments on the Fisher Spanish-English dataset show that the proposed framework yields improvement of 6.7 BLEU compared with a baseline direct S2ST model that predicts spectrogram features. When trained without any text transcripts, our model performance is comparable to models that predict spectrograms and are trained with text supervision, showing the potential of our system for translation between unwritten languages.
|Original language||American English|
|Title of host publication||ACL 2022 - 60th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)|
|Editors||Smaranda Muresan, Preslav Nakov, Aline Villavicencio|
|Publisher||Association for Computational Linguistics (ACL)|
|Number of pages||13|
|State||Published - 2022|
|Event||60th Annual Meeting of the Association for Computational Linguistics, ACL 2022 - Dublin, Ireland|
Duration: 22 May 2022 → 27 May 2022
|Name||Proceedings of the Annual Meeting of the Association for Computational Linguistics|
|Conference||60th Annual Meeting of the Association for Computational Linguistics, ACL 2022|
|Period||22/05/22 → 27/05/22|
Bibliographical notePublisher Copyright:
© 2022 Association for Computational Linguistics.