Skip to main content
2024 BMES Annual Meeting Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Back

Bioinformatics, Computational and Systems Biology

Bioinformatics, Computational and Systems Biology - Poster Session C

Poster C5 - Assessing ChatGPT as a tool in Agent-Based Modeling

Friday, October 25, 2024
10:00 AM – 11:00 AM EST
Location: Exhibit Hall E, F & G

Introduction:  Agent-Based Models (ABMs) are being increasingly used in the biomedical engineering community and beyond to simulate complex biological systems. ABMs are traditionally constructed through a labor-intensive, manual process wherein a researcher follows a sequence of steps, including: designing the model, researching the literature to develop a rule set, implementing the rule set through code (commonly in an ABM platform), running simulations to gather data, analyzing the data, and validating the model (often by comparison to experimental data). Ultimately, the researcher uses their model to predict how the biological system will respond to perturbations and to deduce the key mechanisms that govern the biological system.

The multi-step process required to develop an ABM is time-consuming and requires extensive knowledge of both the biological system and code. Leveraging generative AI may streamline the specification and application of ABMs to investigate increasingly complex biological systems. Since its inception in 2022, OpenAI’s ChatGPT has garnered global attention due to its proficiency in generating natural language and creating functioning code. ChatGPT has increasingly blurred the lines between human and machine; however, to our knowledge ChatGPT has not yet been deployed in the ABM field. The goal of this study is to evaluate how ChatGPT can be leveraged to accelerate and enhance each step of the ABM building process. We used ChatGPT to develop an ABM of sprouting angiogenesis and compared it to a manually constructed ABM of sprouting angiogenesis developed by a student researcher.

Materials and

Methods:  The model building process was distilled into the standardized sequence of ABM building steps described above. The student and ChatGPT both followed the sequence, excluding model validation, to create a model of sprouting angiogenesis from an organoid into a porous hydrogel loaded with pro-angiogenic growth factors. The user prompted ChatGPT (version ChatGPT-4o) through the predefined ABM building steps, engaging in an iterative dialogue to clarify questions. ChatGPT was first asked to identify a set of qualitative rules for sprouting angiogenesis and then it was asked to refine its rules with numerical values. ChatGPT was then asked to translate those rules into code for the user to implement in NetLogo (version 6.4.0). ChatGPT’s rule-set and associated sources were manually analyzed for accuracy by the user.

The student started by creating a 2-dimensional schematic of the biological frame of reference at the initial timestep, an intermediate timestep, and the final timestep. The student then conducted a literature search of organoids, capillary lattices, and experimental procedures to inform the model’s visual design and to establish initial conditions. After, the student conducted a literature search to develop a ruleset, looking to ABM papers for implementation strategies. The student implemented the model in NetLogo and used previous organoid studies to calibrate the model and to tune parameters for accuracy.

Both models developed by the student and ChatGPT were run for ten replicates. Screenshots of the model outputs from the final timesteps (Figure 1A, 1B) and mean vessel density for both models (Figure 1C) were compared.

Results, Conclusions, and Discussions: There were significant differences in model approach and accuracy when comparing the ABMs developed by ChatGPT and the student. ChatGPT generated four rules and had ten parameters: however, it only incorporated seven parameters into code. It excluded parameters for matrix metalloproteinase secretion rate, cell-cell interaction force, and cell-matrix adhesion strength. Additionally, 0% of the ruleset developed by ChatGPT accurately correlated to the provided source. ChatGPT mistook what appeared to be unrelated numbers reported in literature for parameters needed in the ABM. Furthermore, ChatGPT only referenced texts over ten years old, cited nonexistent sources, and fabricated data.

The student created seven parameters and three rules: vessel growth, branching, and contact inhibition. Despite having fewer rules and parameters, the student’s rules for branching and proliferation incorporated dynamic stochastic parameters derived from recent literature, most of which was published within the past ten years.

Both the student and ChatGPT encountered similar errors in their code regarding object relationships within the NetLogo environment. The student’s code elicited more run-time errors, whereas ChatGPT repeatedly assigned variables the same names as preexisting NetLogo functions. Even after manual debugging, ChatGPT’s code did not develop vascular structures and endothelial cells exhibited excessive proliferation into the hydrogel scaffold (Figure 1). Additionally, ChatGPT did not account for cell senescence throughout the model time course, resulting in a dense capillary mesh (Figure 1B, 1C).

Although ChatGPT’s rules and process resembled the student’s, its structure and implementation was fundamentally different. ChatGPT was unable to convert its knowledge of sprouting angiogenesis and hydrogels into an ABM more efficiently than the student. Additionally, ChatGPT’s predilection towards data falsification raises concerns about ethical data use. Interestingly, ChatGPT did not reference ABM manuscripts unless explicitly asked.

Despite its flaws, guiding ChatGPT through the standardized ABM building steps provides the user with another perspective in the modeling process, highlighting potential shortcomings in their own model rules and logic. Future work aims to explore aspects of the ABM building process where ChatGPT could be optimized and leveraged for experienced and novice modelers and determine how models developed by other generative AIs compare to ChatGPT’s.

Acknowledgements (Optional):  Harrison Undergraduate Research Award, and NIH R01 HL155143 and NIH R01 GM140008 to S.M.P.