Vid2speech: Speech reconstruction from silent video

Ariel Ephrat; Shmuel Peleg

doi:10.1109/ICASSP.2017.7953127

Vid2speech: Speech reconstruction from silent video

Ariel Ephrat, Shmuel Peleg

The Rachel and Selim Benin School of Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

91 Scopus citations

Abstract

Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.

Original language	English
Title of host publication	2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	5095-5099
Number of pages	5
ISBN (Electronic)	9781509041176
DOIs	https://doi.org/10.1109/ICASSP.2017.7953127
State	Published - 16 Jun 2017
Event	2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - New Orleans, United States Duration: 5 Mar 2017 → 9 Mar 2017

Publication series

Name	ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
ISSN (Print)	1520-6149

Conference

Conference	2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017
Country/Territory	United States
City	New Orleans
Period	5/03/17 → 9/03/17

Bibliographical note

Publisher Copyright:
© 2017 IEEE.

Keywords

articulatory-to-acoustic mapping
neural networks
speech intelligibility
Speechreading
visual speech processing

Access to Document

10.1109/ICASSP.2017.7953127

Cite this

Ephrat, A., & Peleg, S. (2017). Vid2speech: Speech reconstruction from silent video. In 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings (pp. 5095-5099). Article 7953127 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICASSP.2017.7953127

@inproceedings{a4cbf44ffbed4178a925c5abf837da34,

title = "Vid2speech: Speech reconstruction from silent video",

abstract = "Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.",

keywords = "articulatory-to-acoustic mapping, neural networks, speech intelligibility, Speechreading, visual speech processing",

author = "Ariel Ephrat and Shmuel Peleg",

note = "Publisher Copyright: {\textcopyright} 2017 IEEE.; 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 ; Conference date: 05-03-2017 Through 09-03-2017",

year = "2017",

month = jun,

day = "16",

doi = "10.1109/ICASSP.2017.7953127",

language = "אנגלית",

series = "ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "5095--5099",

booktitle = "2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings",

address = "ארצות הברית",

}

Ephrat, A & Peleg, S 2017, Vid2speech: Speech reconstruction from silent video. in 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings., 7953127, ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings, Institute of Electrical and Electronics Engineers Inc., pp. 5095-5099, 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017, New Orleans, United States, 5/03/17. https://doi.org/10.1109/ICASSP.2017.7953127

Vid2speech: Speech reconstruction from silent video. / Ephrat, Ariel; Peleg, Shmuel.
2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2017. p. 5095-5099 7953127 (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Vid2speech

T2 - 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017

AU - Ephrat, Ariel

AU - Peleg, Shmuel

PY - 2017/6/16

Y1 - 2017/6/16

N2 - Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.

AB - Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.

KW - articulatory-to-acoustic mapping

KW - neural networks

KW - speech intelligibility

KW - Speechreading

KW - visual speech processing

UR - http://www.scopus.com/inward/record.url?scp=85023780229&partnerID=8YFLogxK

U2 - 10.1109/ICASSP.2017.7953127

DO - 10.1109/ICASSP.2017.7953127

M3 - ???researchoutput.researchoutputtypes.contributiontobookanthology.conference???

AN - SCOPUS:85023780229

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 5095

EP - 5099

BT - 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 5 March 2017 through 9 March 2017

ER -

Ephrat A, Peleg S. Vid2speech: Speech reconstruction from silent video. In 2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2017. p. 5095-5099. 7953127. (ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings). doi: 10.1109/ICASSP.2017.7953127

Vid2speech: Speech reconstruction from silent video

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this