Publication and Poster at HCLLM at ACL2024

Our paper, where we investigate to what extent we can use generative large language models in motivational interviewing chatbots to generate precise and variable reflections, is accepted to be published in the proceedings of the Human-Centered Large Language Modelling Workshop within the 62nd Annual Meeting of the Association for Computational Linguistics. The publication also gained a spot in the workshop for a poster presentation, which is going to take place on August 15, 2024.

Read the paper at https://aclanthology.org/2024.hucllm-1.4.pdf

Title

To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation

Abstract

Motivational Interviewing is a counselling style that requires skillful usage of reflective listening and engaging in conversations about sensitive and personal subjects. In this paper, we investigate to what extent we can use generative large language models in motivational interviewing chatbots to generate precise and variable reflections on user responses. We conduct a two-step human evaluation where we first independently assess the generated reflections based on four criteria essential to health counseling; appropriateness, specificity, naturalness, and engagement. In the second step, we compare the overall quality of generated and human-authored reflections via a ranking evaluation. We use GPT-4, BLOOM, and FLAN-T5 models to generate motivational interviewing reflections, based on real conversational data collected via chatbots designed to provide support for smoking cessation and sexual health. We discover that GPT-4 can produce reflections of a quality comparable to human-authored reflections. Finally, we conclude that large language models have the potential to enhance and expand reflections in predetermined health counseling chatbots, but a comprehensive manual review is advised.

Reference

APA
BibTeX

Basar, E., Hendrickx, I., Krahmer, E., de Bruijn, G., & Bosse, T. (2024). To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation. In Proceedings of the 1st Human-Centered Large Language Modeling Workshop, pages 41-52. Association for Computational Linguistics.

@inproceedings{basar2024extent,
  author = {Basar, Erkan and Hendrickx, Iris and Krahmer, Emiel and {de Bruijn}, Gert{-}Jan and Bosse, Tibor},
  title = {To What Extent Are Large Language Models Capable of Generating Substantial Reflections for Motivational Interviewing Counseling Chatbots? A Human Evaluation},
  booktitle = {Proceedings of the 1st Human-Centered Large Language Modeling Workshop},
  year = {2024},
  publisher = {Association for Computational Linguistics},
  pages = {41--52},
  doi = {10.18653/v1/2024.hucllm-1.4}
}

Images