Audio-visual integration in speech processing and learning
Principal investigator: Yue Wang
Collaborators: Dawn Behne (Norwegian University of Science and Technology, Norway), Joy Hirsch (Columbia University, USA),
Research assistants: Nicole Carter, Angela Cooper, Chad Danyluc, Angela Feehan, Haisheng Jiang, Nina Leung, Lulu Liang, David Mesiha, Inha Park, Lindsay Shaw, Rebecca Simms, Jung-Yueh Tu, Xianghua Wu, William Zhang
This project investigates the integration of auditory and visual (AV) information in the processing and learning of nonnative speech. Our goal is to explore the role of linguistic experience and the neural mechanisms involved in multimodal nonnative speech learning.
Specifically, this study examines (1) how nonnative speakers rely on auditory, visual, or integrated AV information to process nonnative speech, and if nonnative AV processing patterns differ from the native patterns; and (2) if enhancement (such as training) of auditory and/or visual information can facilitate learning, and (3) how learning-induced AV integration is instantiated in the brain. The testing techniques include behavioural testing and functional neuroimaging (fMRI, EEG).
The processing and learning of Pitch in Speech
Principal investigator: Yue Wang
Collaborators: Dawn Behne (Norwegian University of Science and Technology, Norway), Qi Dong (Beijing Normal University, China)
Research assistants: Angela Cooper, Jung-Yueh Tu
Using neuroimaging (fMRI, EEG) and behavioral testing methods, this project addresses how linguistic pitch is processed in the brain, and how its neural organization may be affected by linguistic and non-linguistic experience and learning.
The goal of this study is to investigate the extent to which neural processing in second language (L2) learning is influenced by linguistic experience, or reflects a human hardwired ability to process general physical properties.
We explore (1) if L2 prosodic processing ( e.g. tone, intonation) is determined by linguistic functions or by the temporal frame in which pitch occurs; (2) whether different types of learning input (linguistic or acoustic) alter L2 learners' patterns of prosodic processing. Additionally, we will compare adults' and children's pitch learning patterns, to address the question of age of L2 acquisition on prosodic learning, and the associated learning-induced brain plasticity.
Linguistic and mathematical processing in native and second languages
Principal investigator: Yue Wang
Collaborators: Joy Hirsch (Columbia University),
Qi Dong (Beijing Normal University, China), Patricia Kuhl (University of Washington)
Using functional magnetic resonance imaging (fMRI), this study examined how mathematical problems and phonetic contrasts are processed in one's native language (L1) and in a second language (L2).
The question addressed is whether mathematical operations are language-dependent. In particular, we examine (1) whether language-related brain areas are used when completing mathematical tasks, and how brain activation patterns differ in mathematical and phonetic processing, and (2) whether L2 users employ the same or alternative neurocircuitry to perform mathematical operations in L1 and L2, and how that differs from phonetic processing in the two languages.
Perceived nativeness of temporal adjustments in speech
Principal investigator: Yue Wang
Collaborators: Dawn Behne (Norwegian University of Science and Technology, Norway), Qi Dong (Beijing Normal University, China)
Native Mandarin Chinese speakers' productions of English consonant-vowel (CV) syllables have shown syllable-internal temporal adjustments in the direction of native (English)-like CVs (Wang & Behne 2004). This research investigates whether these temporal adjustments affect perceived nativeness.
