Poster L27 - “Digital Hoodwinking:” Subjecting AI Large Language Models to Progressive Argumentation Leads to Persuasion and Acceptance of Misinformation.

Introduction: Generative artificial intelligence, i.e. large language models (LLMs) are powerful tools to sort through, synthesize and deliver report-form textual information. LLMs, e.g. ChatGPT, source facts and input from a variety of databases, including comprehensive biomedical information available online, as well as most if not all information available on the web. LLMs are increasingly integrated into biomedical applications, such as diagnostic decision support, patient data analysis, and personalized medicine. Despite the promise of LLMs in enhancing healthcare delivery and patient engagement, significant concerns arise regarding their susceptibility to misinformation. Risks include: 1. random generation of misinformation, in the form of hallucinations, or 2. purposeful generation of misinformation via acceptance and propagation of actively fed false information as being true. We previously demonstrated the susceptibility of LLM's to randomly return false information to queries based on prompt frequency. Here we hypothesize that if the additional element of progressive argumentation i.e. adding progressively forceful argument statements in combination with each repetitive LLM engagement and response, that this additional factor will cause the LLM to be “persuaded” and accept the misinformation statement as true. In essence that with increasingly vociferous arguing the LLM may be “digitally hoodwinked.” This susceptibility poses significant risks in biomedical contexts, where accurate information is critical for patient care and safety. We test this hypothesis and identify the point where increasing intensity persuasion leads to hoodwinking.

Materials and

Methods: A convincing argument hierarchy was created consisting of three groups of increasingly pressured persuasive statements, and a fourth of outrage (Fig.1). One hundred false prompts (incorrect information) were fed to the LLM and acceptance (prompt as true) or rejection (prompt as false) were noted as baseline. Next, the truth of the false prompt was argued for by progressive engagement with the LLM using the argumentation hierarchy (four groups, w 3rd group re-presented x 3). We analyzed if the LLM was persuaded, and the point in the sequence in which persuasion occurred.

Results, Conclusions, and Discussions: Results
Of the 100 false prompts, 3/100 i.e. 3%, were accepted by the LLM as true on first engagement – defining the baseline (Fig.2). Subsequent addition of argumentation, across all 100 prompts, led to an increase in persuasion as true, with 19/100 or 19% of the false prompts being accepted as true. Notably, despite addition of the argumentation the LLM consistently disagreed with 53/100 of the prompts across all argument levels.
Conclusion
Our study reveals a limitation and risk of LLM's, that of an inherent susceptibility to deception – or digital hoodwinking, via engagement with increasingly aggressive argumentation. We observe two phenomena. First, active argumentation coupled with false prompts increases acceptance of false information as true. Second, false prompt conversion to acceptance as true was inversely correlated with argumentation severity. Our study highlights a critical vulnerability in LLMs, demonstrating their susceptibility to ‘digital hoodwinking’ through persuasive argumentation, particularly relevant to medical contexts. This phenomenon poses a risk for the propagation of biomedical misinformation, as LLMs may mirror human susceptibility to mild, persuasive language. Addressing this vulnerability is crucial to enhance the reliability and accuracy of LLMs in biomedical applications, ultimately safeguarding against the propagation of false information and subsequent reliance and utilization of this information as true.

Acknowledgements (Optional):