Poster M10 - FusOn-pLM: a Fusion Oncoprotein-Specific Language Model

Introduction: Fusion oncoproteins, a class of highly disordered proteins arising from chromosomal translocations, play central roles in driving and sustaining various types of cancers, especially those that impact children. These hybrid proteins, such as EWS:FLI1 in Ewing’s Sarcoma and PAX3:FOXO1 in alveolar rhabdomyosarcoma (ARMS), are thus promising drug targets for pediatric cancer treatment. However, due to their intrinsically disordered nature and lack of well-defined, druggable pockets, fusion oncoproteins have historically been challenging to target therapeutically: neither small molecule-based methods, such as molecular glues or PROTACs, nor structure-based approaches for binder design are strong options for this class of molecules. Recently, protein language models (pLMs) have demonstrated success at representing protein sequences with information-rich embeddings, enabling downstream design applications from sequence alone. However, no current pLM has been trained on fusion oncoproteins and may thus not produce optimal representations for these sequences, which possess defined motifs of significance. In this work, we fine-tune ESM-2 on fusion oncoprotein sequences via a masked language modeling (MLM) task, and apply low-rank adaptor matrices (LoRA) to its final three layers to create fusion oncoprotein-aware ESM-2 embeddings. Using both random 15% masking and focused probabilistic masking on binding amino acid positions and breakpoint junctions, we demonstrate that our models improve performance on puncta formation prediction in comparison to the base ESM-2 model, motivating downstream usage of our embeddings for fusion-specific design tasks.

Materials and

Methods: We fine-tune ESM-2-650M on 44,236 fusion oncoprotein sequences collected from two databases, FusionPDB and Fusion Oncoprotein Database (FOdb) (Kumar, et al. 2024, Tripathi, et al. 2023). After data cleaning, we obtained 44,236 sequences for training and 177 held-out FOdb sequences for benchmarking. Breakpoints were determined by running local NCBI BLAST (Basic Local Alignment Search Tool) on each fusion sequence to determine where the head gene sequence ends and the tail gene sequence begins.
FusOn-pLM is a fine-tuned encoder trained via a MLM task to create fusion oncoprotein-aware embeddings (Figure 1A). To force comprehension of the physicochemical features of fusion oncoproteins, we employ various masking strategies of the amino acid sequence: (1) masking amino acids likely to participate in PPIs as predicted by SaLT&PepPr (Brixi, et al. 2023), (2) masking the entire fusion oncoproteins’ breakpoint region, (3) focused masking around the breakpoint region using a Gaussian distribution, and (4) masking a random 15% of amino acids (Figure 1B). To preserve comprehension of wild-type proteins, we train FusOn-pLM with an MLM head on ESM-2-650M (Lin et al. 2023). To avoid overfitting ESM-2 parameters directly on our new sequences, we employ LoRA (Hu, et al. 2021) by freezing pre-trained model weights and injecting trainable low-rank matrices into the final three layers of ESM-2, allowing for efficient adaptation to fusion oncoproteins with a small number of trainable parameters. The model was trained on a NVIDIA H100 GPU for five epochs with batch size of 16 and learning rate of 5e-5.

Results, Conclusions, and Discussions: In recent works, certain fusion oncoproteins have been shown to form puncta, a hallmark pathology preceding cancer phenotypes and tumor proliferation (Jiang, et al. 2020). To determine if FusOn-pLM embeddings produce accurate numerical representations of fusion oncoproteins, we evaluated their performance on predicting the propensity of puncta formation. Here, we utilized 177 sequences from FOdb with experimental data on puncta formation (Tripathi, et al. 2023). We leveraged the XGBoost architecture to train a binary classifier with 50 trees and 5-fold stratified k-fold cross-validation. Label 0 or 1 represented a lack or presence of puncta, respectively. To determine if FusOn-pLM embeddings both capture fusion oncoprotein-specific information and retain important physicochemical properties contained in wild-type pLMs, we compare FusOn-pLM embeddings against two others: 1) wild-type ESM-2-650M embeddings, and 2) dummy one-hot encodings. Our results demonstrate that FusOn-pLM embeddings from the SaLT&PepPr-based, breakpoint-biased, and random 15% masking strategies all strongly outperform ESM-2-650M on key classification metrics across the entire held-out test dataset (Table 1), underscoring the added benefit of incorporating fusion oncoprotein representations into ESM-2-650M embeddings, and encouraging use of FusOn-pLM embeddings for downstream fusion oncoprotein design tasks.

In this work, we introduce FusOn-pLM, the first pLM fine-tuned to specifically represent fusion oncoproteins. To our knowledge, no pLM has successfully captured the unique characteristics of fusion oncoproteins, which differ from most proteins due to their highly disordered nature and altered structural and functional properties driving oncogenic transformation (Knott, et al. 2019). Our benchmarking results demonstrate that FusOn-pLM embeddings outperform ESM-2-650M embeddings on fusion oncoprotein-related tasks.

Recently, our lab has trained ESM-2-based models to generate peptides provided only the sequence of the target protein, facilitating the design of peptide-guided E3 ubiquitin ligases for target-specific proteasomal degradation (Bhat, et al. 2023, Chen, et al. 2023). As such, our next steps will be to replace ESM-2 embeddings in these models with FusOn-pLM embeddings, enabling fusion-specific degrader design. By leveraging recent advances in gene delivery, such as lipid nanoparticles (LNPs) and adeno-associated viral (AAV) vectors (Kulkarni, et al. 2021), we envision that fusion-specific degraders may eventually serve as safe and efficacious therapeutics for fusion-positive cancer patients.

Acknowledgements (Optional):