Research Projects

Evaluating the Performance of Transformer-based Language Models for Neuroatypical Language

Difficulties with social aspects of language are among the hallmarks of autism spectrum disorder (ASD). These communication differences are thought to contribute to the challenges that adults with ASD experience when seeking employment, underscoring the need for interventions that focus on improving areas of weakness in pragmatic and social language. In this paper, we describe a transformer-based framework for identifying linguistic features associated with social aspects of communication using a corpus of conversations between adults with and without ASD and neurotypical conversational partners produced while engaging in collaborative tasks. While our framework yields strong accuracy overall, performance is significantly worse for the language of participants with ASD, suggesting that they use a more diverse set of strategies for some social linguistic functions. These results, while showing promise for the development of automated language analysis tools to support targeted language interventions for ASD, also reveal weaknesses in the ability of large contextualized language models to model neuroatypical language.

Duanchen Liu, Zoey Liu, Qingyun Yang, Yujing Huang, and Emily Prud’hommeaux. 2022. Evaluating the Performance of Transformer-based Language Models for Neuroatypical Language. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3412–3419, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. PDF

Automated Analysis of Pragmatic Language Development in Autism

Autism spectrum disorder (ASD) is a neurodevelopmental condition associated with life-long deficits in communication that can impact both personal and professional well-being. Although the linguistic features associated with these deficits are routinely observed in clinical settings, they are difficult to quantify. For our research, we're collecting a growing dataset of conversations between high-functioning adults with ASD and their neurotypical conversational partners as they complete several collaborative tasks. We compare the linguistic characteristics of the two groups using both manually annotated features and computationally predicted features extracted from the conversations.

Christine Yang, Duanchen Liu, Qingyun Yang, Zoey Liu and Emily Prud'hommeaux. 2021. Predicting pragmatic discourse features in the language of adults with autism spectrum disorder. To appear in Proceedings of the Association for Computational Linguistics Student Research Workshop (ACL IJCNLP-SRW). PDF

Distinctive Features of Pragmatic Expression in Adults with ASD

Background: Autism spectrum disorder (ASD) is characterized by impaired pragmatic expression, leading to challenges establishing relationships, maintaining satisfactory employment, and achieving independence. Although pragmatic language in children with ASD is well studied, there has been little research on pragmatics in the discourse of adults with ASD and their interlocutors.

Objectives: We investigate the degree of politeness, uncertainty, and informativeness of utterances in conversation between adults with ASD or with typical development (TD) and a neurotypical conversational partner while completing a collaborative task, with the joint goals of identifying distinctive pragmatic features of ASD and determining how interlocutors adapt their own pragmatic expression.

Methods: Twenty-two adult experimental participants (ASD n=14; TD n=8) engaged in a collaborative navigation task with a neurotypical conversational partner (n=11). In addition to the general eligibility criteria (PIQ>=80; VIQ>=80; age>18y; no history of speech, language, auditory, or hearing difficulty), participants with ASD met criteria for diagnosis on the ADOS. Each experimental participant (EP) completed a map navigation task with a conversational partner (CP) in which each was provided with a map of the same area containing slight differences in landmarks, labels, and obstacles. The EP provided verbal directions to the CP to lead the CP from their indicated location to the EP’s indicated location on the map. Using coding guidelines from previous work (Danescu-Niculescu-Mizil, 2013; Vincze, 2015; Pavlick and Tetrault, 2016), two annotators independently rated each utterance on 3-point ordinal scale for politeness (Krippendorff’s 𝜶=0.57), uncertainty (𝜶=0.75), and information content (𝜶=0.90), and the ratings were averaged.

Results: Utterances produced by EPs with ASD were significantly more polite (ASD=2.01, TD=1.95, t=4.358, p=0.00001, Cohen's d=0.17) and less informative (ASD=1.61, TD=1.72, t=-3.846, p=0.00012, d=0.15) than those produced by EPs with TD. Utterances produced by CPs of EPs with ASD were significantly more uncertain (CP:ASD=1.41, CP:TD=1.21, t=4.812, p=0.0000016, d=0.20) and more informative (CP:ASD=1.65, CP:TD=1.56, t=2.510 p=0.012, d=0.11) than those produced by conversational partners of EPs with TD. Group difference at the participant level followed similar trends but did not reach significance due to limited sample size. Correlations between the pragmatic features and time required for task completion were calculated, revealing a negative correlation between experimental participant informativeness and time to complete the task (Pearson’s r=-0.318).

Conclusions: Adults with ASD exhibited pragmatic language patterns distinct from those of their TD peers when engaging in a collaborative task with a neurotypical conversational partner, which influenced both the pragmatic expression of their conversational partners and extrinsic measures of task success. In particular, we note that conversational partners’ language increased in uncertainty in response to the less informative utterances of their ASD interlocutors. These results provide insight into the ways in which language differences observed in children with ASD might manifest in adulthood, which in turn may prove useful for clinicians developing interventions to improve personal and professional outcomes for adults with ASD.

Yang, C., Liu, D., Canfield, A., Hoffkins, C., Aldrich, J., Farash, S., Silverman, L., and Prud’hommeaux, E. 2021. Distinctive Features of Pragmatic Expression in Adults with ASD. Annual Conference of the International Society for Autism Research (INSAR-2021), virtual.

Authorship attribution often relies upon the writing style, vocabulary, and topics of interest among different writers, however these factors are more difficult to ascertain when analyzing shorter documents. This project experiments with different methods of authorship attribution on shorter texts, using lines of dialogue from TV shows to try and predict the speaker of each line. We explore two sets of features, one consisting of manually selected linguistic features, and the other consisting of sentence embeddings generated from the Google News word2vec model. Of the three algorithms tested, we found that using sentence embeddings with a logistic regression algorithm generally yielded the highest accuracy for each show. We also observed some interesting differences in how the show's genre affected the accuracy of the predictions.