Human-computer interaction guidelines for complex augmentative and alternative communication systems with Large Language Model integration

Abstract

Some individuals with complex communication needs rely on Augmentative and Alternative Communication (AAC) devices for daily interactions. However, AAC users often struggle to communicate during real-time conversations, considering the high amount of physical and mental effort it takes to compose their messages. While users can prepare for conversations by adding new words and phrases to their communication devices, pre-programmed messages may not always cover everything that is necessary to express a person's intentions during a real-time conversation. To address this issue, researchers are exploring the use of artificial intelligence (AI), particularly large language models (LLMs), to improve AAC systems. However, it is crucial to carefully analyze the benefits and drawbacks before rushing to create new technological interventions. This study examines the existing contributions and lists opportunities and challenges for applying LLMs to AAC software, collaboratively with users. Additionally, the study addresses the usage of Blissymbolics, a language with grammar and vocabulary but no phonology, which plays an essential role in assisting individuals with congenital conditions in developing their literacy skills. Two secondary research studies were conducted to systematically and critically map the relevant literature on both LLMs and Blissymbolics uses for AAC. The primary research followed a co-design approach and engaged AAC users and familiar listeners who shared their expectations, concerns, and ideas about future LLM integration into AAC devices. The data collected from the primary and secondary research was used to generate comprehensive human-computer interaction guidelines for developing robust AAC systems that use AI language models. Moreover, the opportunities, challenges and risks involved are also discussed. Furthermore, this work applies the guidelines proposed and showcases their adaptability by developing and detailing a conceptual AAC system that uses LLMs.