Welcome to the Language and Brain Lab

The LAB lab focuses on the study of language and speech, including its perception, production, and acquisition, as well as cognitive and neural processing. We conduct behavioral, electro-physiological, and neuro-imaging research with both adults and children, across a variety of languages.

Latest Research Publications

Cross-Modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers

Hannah B, Wang Y, Jongman A, Sereno JA, Cao J and Nie Y (2017). Cross-Modal Association between Auditory and Visuospatial Information in Mandarin Tone Perception in Noise by Native and Non-native Perceivers. Front. Psychol. 8:2051. doi: 10.3389/fpsyg.2017.02051 PDF


Speech perception involves multiple input modalities. Research has indicated that perceivers establish cross-modal associations between auditory and visuospatial events to aid perception. Such intermodal relations can be particularly beneficial for speech development and learning, where infants and non-native perceivers need additional resources to acquire and process new sounds. This study examines how facial articulatory cues and co-speech hand gestures mimicking pitch contours in space affect non-native Mandarin tone perception. Native English as well as Mandarin perceivers identified tones embedded in noise with either congruent or incongruent Auditory-Facial (AF) and Auditory-FacialGestural (AFG) inputs. Native Mandarin results showed the expected ceiling-level performance in the congruent AF and AFG conditions. In the incongruent conditions, while AF identification was primarily auditory-based, AFG identification was partially based on gestures, demonstrating the use of gestures as valid cues in tone identification. The English perceivers’ performance was poor in the congruent AF condition, but improved significantly in AFG. While the incongruent AF identification showed some reliance on facial information, incongruent AFG identification relied more on gestural than auditory-facial information. These results indicate positive effects of facial and especially gestural input on non-native tone perception, suggesting that cross-modal (visuospatial) resources can be recruited to aid auditory perception when phonetic demands are high. The current findings may inform patterns of tone acquisition and development, suggesting how multi-modal speech enhancement principles may be applied to facilitate speech learning.

Acoustic characteristics of clearly spoken English tense and lax vowels.

Leung, K., Jongman, A., Wang, Y. and Sereno, J.A. (2016). Acoustic characteristics of clearly spoken English tense and lax vowels. Journal of the Acoustical Society of America, 140, 45-58.PDF


Clearly produced vowels exhibit longer duration and more extreme spectral properties than plain, conversational vowel. These features also characterize tense relative to lax vowels. This study explored the interaction of clear-speech and tensity effects by comparing clear and plain productions of three English tense-lax vowel pairs (/i-ɪ/, /ɑ-ʌ/, /u-ʊ/ in /kVd/ words). Both temporal and spectral acoustic features were examined, including vowel duration, vowel-to-word duration ratio, formant frequency and dynamic spectral characteristics. Results revealed that the tense-lax vowel difference was generally enhanced in clear relative to plain speech, but clear-speech modifications for tense and lax vowels showed a trade-off in the use of temporal and spectral cues. While plain-to-clear vowel lengthening was greater for tense than lax vowels, clear-speech modifications in spectral change was larger for lax than tense vowels. Moreover, peripheral tense vowels showed more consistent clear-speech modifications in the temporal than spectral domain. Presumably, articulatory constraints limit the spectral variation of these extreme vowels, so clear-speech modifications resort to temporal features and reserve the primary spectral features for tensity contrasts. These findings suggest that clear-speech and tensity interactions involve compensatory modifications in different acoustic domains.

Thai rate-varied vowel length perception and the impact of musical experience.

Cooper, A., Wang, Y., and Ashley, R. (2016 in press). Thai rate-varied vowel length perception and the impact of musical experience. Language and Speech. DOI: 10.1177/0023830916642489. PDF


Musical experience has been demonstrated to play a significant role in the perception of nonnative speech contrasts. The present study examined whether or not musical experience facilitated the normalization of speaking rate in the perception of non-native phonemic vowel length contrasts. Native English musicians and non-musicians (as well as native Thai control listeners) completed identification and AX (same–different) discrimination tasks with Thai vowels contrasting in phonemic length at three speaking rates. Results revealed facilitative effects of musical experience in the perception of Thai vowel length categories. Specifically, the English musicians patterned similarly to the native Thai listeners, demonstrating higher accuracy at identifying and discriminating between-category vowel length distinctions than at discriminating within-category durational differences due to speaking rate variations. The English musicians also outperformed non-musicians at between-category vowel length discriminations across speaking rates, indicating musicians’ superiority in perceiving categorical phonemic length differences. These results suggest that musicians’ attunement to rhythmic and temporal information in music transferred to facilitating their ability to normalize contextual quantitative variations (due to speaking rate) and perceive non-native temporal phonemic contrasts.

Effects of acoustic and linguistic experience on Japanese pitch accent processing.

Wu, X., Kawase, S. and Wang, Y. (2016). Effects of acoustic and linguistic experience on Japanese pitch accent processing. Bilingualism: Language and Cognition, 1-16. doi:10.1017/S1366728916000559. PDF


This study investigated the effects of L2 learning experience in relation to L1 background on hemispheric processing of Japanese pitch accent. Native Mandarin Chinese (tonal L1) and English (non-tonal L1) learners of Japanese were tested using dichotic listening. These listener groups were compared with those recruited in Wu, Tu & Wang (2012), including native Mandarin and English listeners without Japanese experience and native Japanese listeners. Results revealed an overall right-hemisphere preference across groups, suggesting acoustically oriented processing. Individual pitch accent patterns also revealed pattern-specific laterality differences, further reflecting acoustic-level processing. However, listener group differences indicated L1 effects, with the Chinese but not English listeners approximating the Japanese patterns. Furthermore, English learners but not naïve listeners exhibited a shift towards the native direction, revealing effects of L2 learning. These findings imply integrated effects of acoustic and linguistic aspects on Japanese pitch accent processing as a function of L1 and L2 experience.

Effects of musical and linguistic experience on categorization of lexical and melodic tones.

Chang, D., Hedberg, N. and Wang, Y. (2016). Effects of musical and linguistic experience on categorization of lexical and melodic tones. Journal of the Acoustical Society of America, 139 (5), 2432-2447. PDF


This study investigated the categorization of Mandarin lexical tones and music melodic tones by listeners differing in linguistic and musical experience (English musicians, English nonmusicians, and Mandarin non-musicians). Linguistic tonal continua were created from the Mandarin rising to level, and falling to level tones. Melodic continua were created by varying the note D under the context of C and E. The tasks involved tone discrimination and identification. Results revealed that musical training facilitated Mandarin tone categorization, with English musicians’ tone identification approximating native Mandarin patterns, being more categorical than English non-musicians’. However, English musicians showed higher discrimination accuracy than Mandarin listeners but not English non-musicians. This suggests that musical experience was not advantageous in discriminating linguistic tonal variations, which requires listeners to ignore subtle physical differences in order to make categorical judgments. Similarly, Mandarin tone experience affected melodic tone identification, with Mandarin non-musicians approximating English musicians, showing more categorical patterns than English non-musicians. In contrast, Mandarin non-musicians’ melodic discrimination was the poorest among the three groups, indicating that their experience with linguistic tone categorization may have decreased their sensitivity to fine-grained pitch variations. These results demonstrate bi-directional transfer of pitch proficiency between speech and music as a function of experience.

Examining visible articulatory features in clear and conversational speech.

Tang, L., Hannah, B., Jongman, A., Sereno, J., Wang, Y. and Hamarneh, G. (2015). Examining visible articulatory features in clear and conversational speech. Speech Communication, 75, 1-13. PDF


This study investigated the relationship between clearly produced and plain citation form speech styles and motion of visible articulators. Using state-of-the-art computer-vision and image processing techniques, we examined both front and side view videos of speakers’ faces while they recited six English words (keyed, kid, cod, cud, cooed, could) containing various vowels differing in visible articulatory features (e.g., lip spreading, lip rounding, jaw displacement), and extracted measurements corresponding to the lip and jaw movements. We compared these measurements in clear and plain speech produced by 18 native English speakers. Based on statistical analyses, we found significant effects of speech style as well as speaker gender and saliency of visual speech cues. Compared to plain speech, we found in clear speech longer duration, greater vertical lip stretch and jaw displacement across vowels, greater horizontal lip stretch for front unrounded vowels, and greater degree of lip rounding and protrusion for rounded vowels. Additionally, greater plain-to-clear speech modifications were found for male speakers than female speakers. These articulatory movement data demonstrate that speakers modify their speech productions in response to communicative needs in different speech contexts. These results also establish the feasibility of utilizing novel computerized facial detection techniques to measure articulatory movements.