Can our eyes trick our ears? An electrophysiological study of the effects of facial cues on the processing foreign-accented and native-accented speech.
In our increasingly globalized and interconnected world, there is a growing number of speakers of English as a second language and thus an increased likelihood to interact with foreign-accented speakers in our everyday conversations. Moreover, we are increasingly likely to encounter people from different ethnic backgrounds speaking English, who may or may not have a foreign accent. Research has found that listening to foreign-accented speech appears to be more effortful than listening to non-accented speech. One question that remained underexplored is how individuals listening to a speaker exploit this speaker's facial features as a visual cue to determine whether this speaker is likely to have an accent or not, and to what extent these visual cues affect the comprehension of foreign-accented speech. First studies on this topic indicate that listeners tend to use visual cues and information about bilingual speakers' identity to determine the language they are likely to encounter, but the exact neurocognitive mechanisms by which visual and auditory signals interact in the context of foreign-accented speech remained unexplored. Integrating theoretical perspectives and empirical knowledge on how foreign-accented speech and visual cues that reflect speaker identity affect speech comprehension, this study examined how listeners exploit visual cues (facial features regarding ethnicity) to comprehend foreign-accented and non-accented speech. Specifically, in a series of three experiments I sought to identify the neurocognitive mechanisms involved in the processing of congruent and incongruent visual and auditory signals (congruent: Caucasian face paired with non-accented speech and Asian face paired with Chinese-accented speech; incongruent: Asian face paired with non-accented speech and Caucasian face paired with Chinese-accented speech). To examine the impact of variability in listeners' prior experience with foreign-accented speech and (in)congruencies between accent and ethnicity, three different populations were studied: monolingual speakers of English (Experiment 1), Asian-American speakers for whom English is their native language (Experiment 2) and Chinese-English bilinguals who are Chinese dominant and have a Chinese accent in English (Experiment 3).