Computer scientist with a focus on artificial intelligence and language technologies
PhD Candidate
Radboud University
Nijmegen, Netherlands
I am a PhD candidate in Behavioural Science Institute at Radboud University. In my research, I focus on developing and evaluating intelligent conversational agents that support health counselling by using natural language processing methods and technologies, including large language models, as part of the Look Who’s Talking project.
I obtained my Master’s degree from Radboud University in Artificial Intelligence, and my Bachelor’s degree from Muğla Sıtkı Koçman University in Computer Engineering. Previously, I have worked as an AI engineer and computational linguist at FloodTags where I have developed information systems that process textual data to collect critical information about natural disaster events in real-time from online (social and mainstream) media.
To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation
Erkan Basar, Iris Hendrickx, Emiel Krahmer, Gert-Jan de Bruijn, and Tibor Bosse
In Proceedings of the 1st Human-Centered Large Language Modeling Workshop, 2024
Motivational Interviewing is a counselling style that requires skillful usage of reflective listening and engaging in conversations about sensitive and personal subjects. In this paper, we investigate to what extent we can use generative large language models in motivational interviewing chatbots to generate precise and variable reflections on user responses. We conduct a two-step human evaluation where we first independently assess the generated reflections based on four criteria essential to health counseling; appropriateness, specificity, naturalness, and engagement. In the second step, we compare the overall quality of generated and human-authored reflections via a ranking evaluation. We use GPT-4, BLOOM, and FLAN-T5 models to generate motivational interviewing reflections, based on real conversational data collected via chatbots designed to provide support for smoking cessation and sexual health. We discover that GPT-4 can produce reflections of a quality comparable to human-authored reflections. Finally, we conclude that large language models have the potential to enhance and expand reflections in predetermined health counseling chatbots, but a comprehensive manual review is advised.
@inproceedings{basar2024extent,author={Basar, Erkan and Hendrickx, Iris and Krahmer, Emiel and {de Bruijn}, Gert{-}Jan and Bosse, Tibor},title={To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation},booktitle={Proceedings of the 1st Human-Centered Large Language Modeling Workshop},year={2024},publisher={Association for Computational Linguistics},pages={41--52},doi={10.18653/v1/2024.hucllm-1.4},}
Effectiveness and user experience of a smoking cessation chatbot: A mixed-methods study comparing motivational interviewing and confrontational counseling
Background: Cigarette smoking poses a major public health risk. Chatbots may serve as an accessible and useful tool to promote cessation due to their high accessibility and potential in facilitating long-term personalized interactions. To increase effectiveness and acceptability, there remains a need to identify and evaluate counseling strategies for these chatbots, an aspect that has not been comprehensively addressed in previous research.
Objective: This study aims to identify effective counseling strategies for such chatbots to support smoking cessation. In addition, we sought to gain insights into smokers’ expectations of and experiences with the chatbot.
Methods: This mixed methods study incorporated a web-based experiment and semistructured interviews. Smokers (N=229) interacted with either a motivational interviewing (MI)–style (n=112, 48.9%) or a confrontational counseling–style (n=117, 51.1%) chatbot. Both cessation-related (ie, intention to quit and self-efficacy) and user experience–related outcomes (ie, engagement, therapeutic alliance, perceived empathy, and interaction satisfaction) were assessed. Semistructured interviews were conducted with 16 participants, 8 (50%) from each condition, and data were analyzed using thematic analysis.
Results: Results from a multivariate ANOVA showed that participants had a significantly higher overall rating for the MI (vs confrontational counseling) chatbot. Follow-up discriminant analysis revealed that the better perception of the MI chatbot was mostly explained by the user experience–related outcomes, with cessation-related outcomes playing a lesser role. Exploratory analyses indicated that smokers in both conditions reported increased intention to quit and self-efficacy after the chatbot interaction. Interview findings illustrated several constructs (eg, affective attitude and engagement) explaining people’s previous expectations and timely and retrospective experience with the chatbot.
Conclusions: The results confirmed that chatbots are a promising tool in motivating smoking cessation and the use of MI can improve user experience. We did not find extra support for MI to motivate cessation and have discussed possible reasons. Smokers expressed both relational and instrumental needs in the quitting process. Implications for future research and practice are discussed.
@article{he2024roby,author={He, Linwei and Basar, Erkan and Krahmer, Emiel and Wiers, Reinout and Antheunis, Marjolijn},title={Effectiveness and user experience of a smoking cessation chatbot: A mixed-methods study comparing motivational interviewing and confrontational counseling},journal={Journal of Medical Internet Research},year={2024},volume={26},pages={e53134},doi={10.2196/53134},}
HyLECA: A Framework for Developing Hybrid Long-Term Engaging Controlled Conversational Agents
Erkan Basar, Divyaa Balaji, Linwei He, Iris Hendrickx, Emiel Krahmer, Gert-Jan de Bruijn, and Tibor Bosse
In Proceedings of the 5th ACM Conference on Conversational User Interfaces (CUI), 2023
We present HyLECA, an open-source framework designed for the development of long-term engaging controlled conversational agents. HyLECA’s dialogue manager employs a hybrid architecture, combining rule-based methods for controlled dialogue flows with retrieval-based and generation-based approaches to enhance the utterance variability and flexibility. The motivation behind HyLECA lies in enhancing user engagement and enjoyment in task-oriented chatbots by leveraging the natural language generation capabilities of open-domain large language models within the confines of predetermined dialogue flows. Moreover, we discuss the technical capabilities, potential applications, relevance, and adaptability of the system. Lastly, we report preliminary findings from integrating state-of-the-art large language models in simulating a conversation centred on smoking cessation.
@inproceedings{basar2023hyleca,author={Basar, Erkan and Balaji, Divyaa and He, Linwei and Hendrickx, Iris and Krahmer, Emiel and {de Bruijn}, Gert{-}Jan and Bosse, Tibor},title={Hy{LECA}: A Framework for Developing Hybrid Long-Term Engaging Controlled Conversational Agents},booktitle={Proceedings of the 5th ACM Conference on Conversational User Interfaces (CUI)},year={2023},publisher={Association for Computing Machinery},articleno={56},series={CUI '23},doi={10.1145/3571884.3604404},}
