A study of continuous space word and sentence representations applied to ASR error detection

Sahar Ghannay (1) , Yannick Estève (1) , Nathalie Camelin (1)

1 LIUM - Laboratoire d'Informatique de l'Université du Mans

This paper presents a study of continuous word representations applied to automatic detection of speech recognition errors. A neural network architecture is proposed, which is well suited to handle continuous word representations, like word embeddings. We explore the use of several types of word representations: simple and combined linguistic embeddings, and acoustic ones associated to prosodic features, extracted from the audio signal. To compensate certain phenomena highlighted by the analysis of the error average span, we propose to model the errors at the sentence level through the use of sentence embeddings. An approach to build continuous sentence representations dedicated to ASR error detection is also proposed and compared to the Doc2vec approach. Experiments are performed on automatic transcriptions generated by the LIUM ASR system applied to the French ETAPE corpus. They show that the combination of linguistic embeddings, acoustic embeddings, prosodic features, and sentence embeddings in addition to more classical features yields very competitive results. Particularly, these results show the complementarity of acoustic embeddings and prosodic information, and show that the proposed sentence em-beddings dedicated to ASR error detection achieve better results than generic sentence embeddings.

Mots clés: ASR error detection; neural networks; prosodic features; acoustic embeddings; linguistic embeddings; sentence embeddings

Domaines: Base de données [cs.DB]; Intelligence artificielle [cs.AI]; Apprentissage [cs.LG]; Réseau de neurones [cs.NE]; Traitement du texte et du document; Informatique et langage [cs.CL]

Sahar Ghannay, Yannick Estève, Nathalie Camelin. A study of continuous space word and sentence representations applied to ASR error detection. Speech Communication, 2020. ⟨hal-02501943⟩