IDR-pLM: IDR Ensemble Property Prediction via Protein Language Models

Introduction: Intrinsically Disordered Regions (IDRs) make up a large part of eukaryotic proteomes and play several key roles in molecular and cellular functions. IDRs lack a fixed 3D structure and instead exist as an ensemble of conformations, allowing them to interact with multiple binding partners and regulate diverse cellular pathway[1]. The structural ensemble properties of IDRs, such as radius of gyration (Rg), end-to-end distance (Re), asphericity, and polymer scaling exponent, provide valuable insights into their flexibility, compactness, and overall shape. These properties are essential for understanding how IDRs function in cellular processes and how they contribute to disease pathogenesis. Notably, proteins containing IDRs are involved in many diseases, such as neurodegenerative disorders, cancers, and viral infections[2]. As a result, IDRs often serve as molecular targets for drug development, making the understanding and characterization of IDRs crucial for deciphering cellular mechanisms and developing targeted therapeutic strategies.

However, the characterization of IDRs is made difficult by their high degree of conformational heterogeneity. Traditional methods for studying IDRs are technically demanding, time-consuming, and computationally intensive [3]. Previous deep learning-based approaches have relied on complex architectures such as bidirectional long short-term memory (Bi-LSTM) networks while utilizing simple one-hot encodings for protein sequence representation[4]. In this work, we introduce a novel framework to predict the ensemble properties of IDRs in an accurate and computationally efficient manner using a multilayer perceptron network with rich input representations from protein language models (pLMs). We specifically aimed to predict key IDR properties including Rg, Re, asphericity, and polymer scaling exponent.

Materials and

Methods: Data. An extensive library of protein sequences containing intrinsically disordered regions (IDRs) was utilized, which included both synthetic and naturally occurring proteins [4]. The ensemble properties of these proteins were calculated using a state-of-the-art coarse-grained force field called Mpipi-GG [4]. To ensure model robustness, the data was randomly split into training and test sets, while removing any test sequences that were highly homologous to the training sequences.

Model. Protein embeddings were generated from state-of-the-art protein language model ESM-2,[5] which captures the contextual relationships and evolutionary information of protein sequences. A linear probing technique was employed via the use of a Multilayer Perceptron (MLP) network on top of frozen ESM-2 layers. Modules were then developed to predict the aforementioned biophysical property values given protein sequence alone. The MLP network consisted of three fully connected layers with 640, 320, and 1 neuron(s), respectively. ReLU activation functions were applied after the first two layers, followed by layer normalization and dropout with a rate of 0.2. This allowed the model to learn complex non-linear relationships between the ESM-2 embeddings and the target IDR properties. These models were evaluated using a mean squared error (MSE) loss metric, as well as the coefficient of determination (R2), comparing property values predicted by our models to the true values. Finally, Low Rank Adaptor (LoRA) matrices with a rank of 32 were used to fine-tune the last three attention layers of ESM-2. This targeted approach allowed us to generate IDR-aware embeddings without the need to retrain the entire network.

Results, Conclusions, and Discussions: Various models of ESM-2 were benchmarked in order to observe whether or not embeddings generated from a model with more parameters had an intrinsic understanding of IDR properties. To this end, model confidence scores were calculated for the embeddings generated for each protein sequence. From this, it was observed that as the number of parameters of the pLM increased, the greater the correlation coefficient between the confidence and the IDR property of interest. This indicates that higher-parameter protein language models (pLMs) were able to capture contextual and evolutionary information regarding protein sequences, and are therefore valuable tools for downstream tasks.

Four different MLP regression heads were then developed atop frozen ESM-2 layers to predict various IDR properties, namely asphericity, scaling exponent, end-to-end radius, and radius of gyration. The model predictions showed a high degree of similarity to the true property values, as evidenced by the scatter plots comparing the predicted and true values (Figure 1). The coefficient of determination (R^2) values for each property were as follows: 0.90 for asphericity, 0.91 for scaling exponent, 0.92 for end-to-end radius, and 0.88 for radius of gyration. These strong correlations demonstrate the effectiveness of an MLP approach in accurately predicting IDR properties using the embeddings generated by ESM-2.

The last three attention layers of ESM-2 were then fine-tuned via the use of low rank adapter matrices (LoRA) in order to generate IDR-aware embeddings for downstream tasks. It was observed that this finetuning allowed for a higher degree of understanding of protein sequences, resulting in increased R2 scores and therefore improved model performance from the baseline MLP models. In total, our results highlight the importance of leveraging feature-rich, pre-trained sequence embeddings for IDR-based prediction and design tasks.

Acknowledgements (Optional):

References:
[1] Lee, R.V.D. et al. ACS Chemical Reviews 2014, 114, 13, 6589–6631.
[2] Uversky, V.N. et al. ACS Chemical Reviews 2014, 114, 13, 6844–6879.
[3] Necci, M. et al. Nature Methods 2021, 18, 472-481.
[4] Lotthammer, J.M. et al. Nature Methods 2024, 21, 465-476.
[5] Lin, Z et al. Science 2023, 379, 6637, 1123-1130.