Poster Y2 - GlyphAI: Bridging the Communication Gap for Patients with Low English Proficiency

Introduction: Patients with Low English Proficiency (LEP) experience disproportionately negative outcomes, most commonly due to communication errors. Existing methods for the communication have been reported as ineffective for LEP patients due to lack of accessibility, adaptability, and understandability. By leveraging visual learning, images generated by Large Language Models (LLMs) have the potential to bridge this gap in communication. However, due to the nondeterminism of LLMs, it is important to understand what types of prompts, when fed into an LLM-based image generator, yield more accurate results. The purpose of this study is to evaluate the images generated for medication instructions using 4 different prompt schemes. By stratifying these comparisons by syntactic components, the results from this study help elucidate whether different prompt schemes are better for different types of words.

Materials and

Methods: We curated a dataset of 50 medication instruction phrases from discharge documents provided by Johns Hopkins Bayview Clinic. These phrases were then broken into five syntactic components: actions, quantities, units, frequencies, and durations. Four different methods of feeding prompts into the Dall-E image generator were used to produce images.
Group 1 - Full sentences: Entire medication instructions, like “Take two pills in the morning and at night for 7 days,” were used as prompts.
Group 2 - Direct word feeding into LLM-Based image generator: Individual syntactic elements such as “2,” “pills,” “take,” “every day,” and “1 month” were entered into Dall-E.
Group 3 - Definition-based prompts: GPT-4 provided definitions for each syntactic component which were then used as prompts.
Group 4 - LLM-made prompts: GPT-4 generated concise prompts for each syntactic component, formatted as “concrete icon: [ANSWER],” which were then used to create images.
We utilized the ALIGN image-to-text alignment model to evaluate the performance of each group.

Results, Conclusions, and Discussions: Direct-word feeding prompts (Group 2) outperformed other prompt engineering strategies in the majority of our tests. When stratified by syntactic component, group 2 yielded images with the highest accuracy for durations, units, and actions. For frequencies, groups 2 and 3 tied for yielding images with the highest accuracy. For quantities, group 4 yielded images with the highest accuracy.
Prompt engineering is important for improving the reliability of LLM outputs. Reliable LLM output images may have the potential to bridge and overcome existing communication gaps in healthcare through visual learning. Our results are an important first step towards improving health outcomes and reducing the cost burden associated with hospital visits for LEP patients.

Acknowledgements (Optional):