Poster T9 - Deep Reinforcement Learning Approaches to Predict Phenotypic Adaptation and Enzalutamide Resistance in Prostate Cancer

Introduction: Resistance to second-generation androgen receptor antagonists like Enzalutamide (Enza) commonly complicates treatment in patients with advanced prostate cancer (PCa). Researchers have previously linked mutations to resistant cancer cells that arise from long-term androgen inhibition. However, phenotypic changes in sensitive cells leading to short-term persistence remain less understood. Recent experiments have accurately tracked growth dynamics of both sensitive and resistant cell types under various temporally variable drug dosing schedules. We developed and applied a stochastic decision-making model to explore the dynamics between Enza-resistant and Enza-sensitive cells. This model assumes that both cell types can dynamically adjust their phenotypes based on a memory of previous environmental encounters with a time-varying drug treatment policy. We then proposed a reinforcement learning framework to comprehend patient-specific tumor dynamics and to formulate an optimal dosing policy. We further validated the efficacy of our methods using recent experimental data. This research highlights the utility of deep reinforcement learning in devising personalized adaptive cancer therapies.We further validated the efficacy of our methods using recent experimental data. This research highlights the utility of deep reinforcement learning in devising personalized adaptive cancer therapies.

Materials and

Methods: We employ the Soft Actor-Critic (SAC) algorithm within a Deep Reinforcement Learning (DRL) framework to optimize Adaptive Therapy (AT) for cancer. This model simulates the interaction between populations of tumor cells that are sensitive or resistant to therapy, based on the differential equation we defined our environment with. In the SAC approach, the system processes the state of the tumor, including size, as input and outputs continuous actions that correspond to varying levels of therapy intensity. These actions are derived from a policy that is modeled as a probability distribution over the possible actions, encouraging decision-making that is probabilistically informed. Uniquely, the SAC incorporates entropy in the policy optimization to not only maximize expected returns but also to promote exploratory actions, thereby enhancing the robustness and adaptability of therapeutic strategies.

The training protocol of the SAC utilizes outcomes from numerous trials to stabilize results and account for the inherent variability of the stochastic policy. The objective function is meticulously designed to prolong the time to tumor progression, while also favoring intermittent treatment schedules that may delay the emergence of resistance. This approach employs advanced machine learning techniques to dynamically adjust treatment schedules in cancer therapy. By adapting to the evolving dynamics of tumor cell populations, this method aims to surpass the efficacy of standard continuous treatment regimens, offering a more tailored and potentially more effective therapeutic strategy.

Results, Conclusions, and Discussions: In our study, we developed a time-varied, mixed-effect, generalized Lotka–Volterra (tM-GLV) model to investigate the evolutionary dynamics of prostate cancer cells, specifically distinguishing between sensitive and resistant phenotypes. This model was meticulously calibrated and validated with experimental data. Our findings, illustrated in Figure 1d, show the model's capability to simulate the oscillations in cell populations under different treatment regimens.

Through reinforcement learning, we optimized individualized dosing schedules that significantly extended the time to progression (TTP) and minimized the cumulative drug doses compared to periodic and constant therapies. The impact of these optimized therapies on patient outcomes is captured in Figure 1, where the progression-free survival (PFS) curves demonstrate the superior performance of adaptive therapy over periodic and constant therapies across simulated patient cohorts. In addition, Figure 1c, represented the different drug policies which our algorithm found that for controlling the cancer cell population for a longer period of time.

In this study, we demonstrate the use of advanced reinforcement learning to devise personalized cancer treatment strategies. By integrating mathematical models with experimental data, we optimize drug administration to significantly enhance treatment efficacy while reducing drug dosages. This innovative approach designed to dynamically adjust to patient-specific tumor dynamics for improved outcomes in managing prostate cancer. The effectiveness of this adaptive therapy was compared with traditional constant and periodic therapies, highlighting the potential utility of personalized adaptive therapy in the clinical setting.

Acknowledgements (Optional): This work was supported by the Cancer Prevention and Research Institute of Texas (CPRIT RR210080). JTG is a CPRIT Scholar in Cancer Research.