Poster L11 - Halo Replay Burst: Enabling Low-Cost Data Analysis of Large-Scale Cellular Simulations

Introduction: The field of in silico cellular modeling has made notable strides in both the volume and characterization of simulated cells [1,2]. The availability of modern, high performance systems enables researchers to conduct simulations at much larger scales, leading to more accurate representations of physiological phenomena. As such, understanding the dynamic interactions of cells and their environment requires more than static images spaced over longer periods of time. Although computational capabilities have grown exponentially, data analysis remains a less explored space, leading to intractable I/O costs at scale. To address this issue, in situ-based methodologies, in which data is analyzed whilst in memory, have continued to grow. Seminal frameworks are likely to encounter significant challenges when scaling up to handle increasingly larger/finer workflows, due to the overhead introduced by the tight coupling of analysis and simulation [3]. Even with innovations in the space, frameworks in which data is handed off to separate "workers" via MPI communication to decouple the tasks of visualization and simulation (in transit) [4], conventional frameworks can quickly balloon in cost when considering the desired frequency of analysis. We present Halo Replay Burst, a halo-based in situ framework that enables us to spawn concurrent simulation "windows" on separate OpenMP threads for exploring fluid-structure interactions/simulations, utilizing shared memory pools to circumvent timely MPI movement costs. These small "bursts" of simulation allow the user to perform data analysis during a period of interest and capture images at higher frequencies, the first step in enabling longer periods of analysis.

Materials and

Methods: To evaluate the effectiveness of our approach, we implement and assess the framework within the context of the massively parallel fluid solver HARVEY [5,6]. HARVEY utilizes stenciling, a technique characterized by spatial decomposition of a domain. Algorithmic updates within "halo regions" necessitate perpetual exchange of fluid and cell data within overlapping spatial boundaries. Within HARVEY, n-dimensional buffers are initialized in anticipation of communication for IBM-related operations. Separate buffers exist for each array of data being passed and are dynamically resized to fit the volume of data to be passed and received.

The traditional in transit workflow is characterized by passing data to a separate resource to perform analysis, allowing undisturbed simulation progression. Our framework differs from the standard approach, using HARVEY's stencil structure to avoid a full data transfer by co-opting exchanged halos and caching them during each timestep in memory into "halo buffers" (Figure 4).

We spawn "bursts" of simulations on secondary OpenMP threads. Checkpoints (to provide the correct "first step") and these halo buffers enable us to maintain a spatially and temporally (for the timeframe specified apriori) complete simulation copy. Because halo data can be preloaded from these buffers, processes on the secondary thread are independent outside of any enforced collective operations like visualization. Time spent on data analysis can be recovered due to this circumvention of point-to-point communication, a cost exemplified when working with heavy cellular data.

Results, Conclusions, and Discussions: To evaluate the limitations of data analysis for large-scale simulations and the effectiveness of our framework, we focus on both runtime and data management costs, costs that drive user choice in in situ frameworks. We focus our attention on a relevant case, the movement of red blood cells (RBCs) through complex, branching geometries (Figure 5). The movement of cells when approaching a bifurcation is a characteristic of importance in FSI simulations. How and when cells approach, deform, and adhere at the branching requires detailed, granular data analysis at specific regions of interest.

We first compare the data costs across frameworks. Traditional methods, such as checkpointing or in transit methods, incur data storage or data movement costs, dictated by the size of entire simulation states. In our case, this can cause terabytes of data to be moved and can cause significant slowdowns. We compare this to the cost required to simply write halo buffers to memory for a Plinko geometry at varying grid resolutions (Figure 1). The size of our halos fractions of the overall simulation size, reducing costs.

We also analyze the overhead introduced to our simulation with the introduction of our framework. We measure these costs in a set of increasing-size cubes filled with red blood cells (Figure 2). Across increasingly larger inputs, we maintain a percentage of overhead that is dismissible when writing the buffers. Introducing a resource split to accommodate a secondary burst of simulation does increase overall runtimes, albeit at manageable rates.

Reducing costs while allowing users to focus on regions and periods of interest is critical to understanding the dynamic interactions of cells within their domain (Figure 3). In this example, we can focus on the deformation and movement of RBCs as they approach a branching point. Providing a framework that boasts low data movement and runtime costs is critical for such a scenario. Maintaining two concurrent, spatially similar simulations for small temporal windows enables researchers to perform accurate analysis, analysis that would otherwise be intractable at scale. Frameworks flexible in both configuration and deployment are necessary as computational biology continues to grow.

Acknowledgements (Optional): The work of Amanda Randles and Ayman Yousef was supported by the National Institute On Aging of the NIH under Award Number DP1AG082343. The content does not necessarily represent the official views of the NIH.

[1] L. Grinberg, J. A. Insley, V. Morozov, M. E. Papka, G. E. Karniadakis, D. Fedosov,
and K. Kumaran, “A new computational paradigm in multiscale simulations:
Application to brain blood flow,” in Proceedings of 2011 International Conference
for High Performance Computing, Networking, Storage and Analysis, pp. 1–5,
2011.
[2] J. Ames, D. F. Puleri, P. Balogh, J. Gounley, E. W. Draeger, and A. Randles,
“Multi-gpu immersed boundary method hemodynamics simulations,” Journal of
computational science, vol. 44, p. 101153, 2020.
[3] J. Ahrens, S. Jourdain, P. O’Leary, J. Patchett, D. H. Rogers, and M. Petersen,
“An image-based approach to extreme scale in situ visualization and analysis,”
in SC’14: Proceedings of the International Conference for High Performance
Computing, Networking, Storage and Analysis, pp. 424–434, IEEE, 2014.
[4] J. Kress, M. Larsen, J. Choi, M. Kim, M. Wolf, N. Podhorszki, S. Klasky, H. Childs,
and D. Pugmire, “Opportunities for cost savings with in-transit visualization,” in
High Performance Computing (P. Sadayappan, B. L. Chamberlain, G. Juckeland,
and H. Ltaief, eds.), (Cham), pp. 146–165, Springer International Publishing,
2020.
[5] A. Randles and E. Kaxiras, “Parallel in time approximation of the lattice boltzmann
method for laminar flows,” Journal of Computational Physics, vol. 270, pp. 577–
586, 2014.
[6] A. P. Randles, V. Kale, J. Hammond, W. Gropp, and E. Kaxiras, “Performance
analysis of the lattice boltzmann model beyond navier-stokes,” in 2013 IEEE 27th
International Symposium on Parallel and Distributed Processing, pp. 1063–1074,
IEEE, 2013.
3