Poster M3 - Establishing a hybrid parallel agent-based model of diffusive transport

Introduction: Mathematical modeling of complex biological systems requires substantial computational resources that result in a high overall time to solution and an upper limit for the tractable domain size. This work focuses on addressing such limitations in the popular package BioFVM, an established 3D diffusion solver for partial differential equations governing substrate diffusion [1]. BioFVM employs OpenMP (Open Multi-Processing), a shared-memory paradigm that achieves effective parallelization within a multi-core processor. On the supercomputer Anvil with AMD EPYC 7763 processor, the current version can simulate a cellular cubic domain with a side length of up to 3100-micron consisting of 29 million voxels. However, larger systems such as tissue or organs are intractable for BioFVM as it exceeds the capacity of the OpenMP paradigm. To overcome this limitation, we introduced a multi-node parallelization scheme in BioFVM to enhance the temporal and spatial scope that can be effectively captured. Specifically, we implemented MPI (Message Passing Interface) parallelization to enable parallelization on multiple processors with separate memory spaces [2]. The adapted version provided fine control for message passing to coordinate communication between processors, hence achieving high scalability via distributing resources needed by a large-domain-size problem to a cluster or supercomputer. Our hybrid implementation could achieve significant speedup and simulate a previously intractable domain with side length of 6000-micron and 216 million voxels using eight computing nodes on Anvil. Our enhanced model can be extended to simulate complex biological microenvironments with heterogeneous sources of substrate diffusion, providing valuable insights for disease mechanisms and therapeutics design.

Materials and

Methods: To apply the finite volume method in BioFVM's microenvironment model, the rectangular domain underwent discretization into smaller cubic units known as voxels. Each voxel was standardized to a size of 10 microns based on previous convergence studies. To parallelize the model with different tasks on multiple processors, the whole domain was decomposed along the x direction and assigned to each MPI rank (Figure 1). Calculating the substrate density of each voxel required data from adjacent voxels within the von Neumann neighborhood, where each voxel had six adjacent voxels in 3D. With the domain segmented across separate processors, inter-process communication occurred through a shared data area known as the halo region. To facilitate this communication, an additional layer of voxels was added in neighboring ranks to form the halo region. This construction of the microenvironment, including the halo region, was replicated for each rank.

The parallelization scheme was verified to have the exact solution accuracy as the serial implementation, as shown on a cubic domain with 500 randomly placed point sources and point sinks (Figure 2). To analyze model performance, metrics including total simulation time, maximum simulation loop time, inter-rank communication time, and mega floating-point operations per second (MFLOP/s) were measured. The model was developed with C++ and computing resources on Anvil from Purdue University and Stampede3 from Texas Advanced Computing Center. Strong scaling runs were set up to examine the effectiveness of resource utilization with an increasing number of processing elements with a fixed problem size to examine model scalability.

Results, Conclusions, and Discussions: The parallelization scheme was verified to be accurate as compared to substrate density solutions in the original code. The hybrid implementation could achieve a faster overall computation than the original OpenMP-only scheme (Figure 3). With the introduced parallelization scheme, we can decompose the domain and, therefore, fit a problem size to multiple nodes’ memory previously intractable to the model on one memory device. Although the original model could simulate a tissue with a side length of 3100 microns and 29 million voxels at most, our current model enabled the simulation of an eight-fold larger tissue with 216 million voxels with eight nodes. In addition, the parallelized exact solver achieved better performance and accuracy than the implicit solver using the Thomas algorithm. The original implementation employed the implicit solver to approximate the solution and gain a faster computation time [3, 4]. Our enhancement could greatly prolong the temporal domain captured in simulation without compromising the solution accuracy as the implicit solver.

Moreover, our hybrid parallelization scheme scaled well (Figure 4). On the Stampede3 system, a 58-fold speedup was achieved. While on the Anvil system, it was less optimal with a 12-fold speed-up (Figure 5). Stampede3 outperformed Anvil due to proper task distribution among processors, faster communication between processors and memory, and overall better memory utilization [5, 6]. These results underscore the critical role multi-node parallelization scheme plays in extending the domain size of this model.

In conclusion, we shifted from parallelization targeting a single node to a hybrid scheme that leverages inter and intra-node parallelization by implementing finer-grained control of message passing. The hybrid approach effectively lowers computation time to solution and enhances the physical scale over which 3D diffusion can be modeled. Its robust framework opens the potential to model various biological problems, including but not limited to systematic response to the microenvironment in organs, synergistic effects of multiple cell secretion sources, and long-term effects of less-frequent physiological events. By continually refining and extending our computational methodologies, we aim to deepen our understanding of complex biological systems and ultimately contribute to biomedical research and healthcare advancements.

Acknowledgements (Optional): This research uses computational resources from the NSF ACCESS Award MDE230012. Cyrus Tanade's work was supported by the National Science Foundation Graduate Research Fellowship under Grant No. NSF GRFP DGE 1644868. The work of Amanda Randles and William Ladd were supported by the National Institute On Aging of the NIH under Award Number DP1AG082343. The content does not necessarily represent the official views of the NSF or NIH.

References:
[1] Ghaffarizadeh, Ahmadreza, Samuel H. Friedman, and Paul Macklin. "BioFVM: an efficient, parallelized diffusive transport solver for 3-D biological simulations." Bioinformatics 32.8 (2016): 1256-1258.
[2] Jin, Haoqiang, et al. "High performance computing using MPI and OpenMP on multi-core parallel systems." Parallel Computing 37.9 (2011): 562-575.
[3] Yanenko, Nikolai Nikolaevich. "Economical implicit schemes (method of fractional steps)." Doklady Akademii Nauk. Vol. 134. No. 5. Russian Academy of Sciences, 1960.
[4] Thomas L.H. (1949). Elliptic problems in linear difference equations over a network In: Watson Sci. Comput. Lab Report. Columbia University, New York.
[5] "Stampede3 - TACC HPC Documentation." TACC HPC Documentation, Texas Advanced Computing Center, docs.tacc.utexas.edu/hpc/stampede3/. Accessed May 15th, 2024.
[6] "System Architecture." RCAC Knowledge Base, Purdue University, www.rcac.purdue.edu/knowledge/anvil/architecture. Accessed May 15th, 2024.