Poster L14 - Inappropriate Data Handling Leads to Overoptimistic Machine Learning Results: Small-For-Gestational-Age (SGA) Prediction

Introduction: Small-for-Gestational-Age (SGA) refers to infants with a birth weight below the 10th percentile and are at significantly higher risk of complications, such as prematurity and neonatal asphyxia. Early prediction of SGA allows for appropriate clinical management, which can improve fetal outcomes. Machine learning (ML) has been proposed for early SGA prediction, showing promising results. However, there is a lack of standardization in reporting ML studies. Studies have identified data leakage issues in 294 ML studies, leading to overly optimistic findings. In this study, we (i) investigate the performance of an ML model in SGA prediction, (ii) examine the impact of data leakage on ML performance, and (iii) perform a parametric analysis to understand its effect. Our findings highlight the importance of addressing data leakage to ensure the reliability of ML-based predictions in clinical settings. These results contribute to establishing more robust and standardized reporting practices in ML studies, ensuring the reproducibility of SGA studies and medical research.

Materials and

Methods: The study protocols were approved by the Medical Research Ethical Committee (MREC) at the UMMC (University of Malaya Medical Centre), Malaysia (MREC ID NO: 2021329-9997). Data between January 2019 and September 2022 were extracted. Data collected includes maternal age, parameters measured during the second trimester, spanning from 14 weeks to 27 weeks 6 days, and birth information. SGA is defined as a birth weight less than the 10th centile according to the WHO birth weight chart.

A total of 10,418 data was collected (4,978 consisting of missing values and 5,440 with complete datasets). Using the complete (Original) datasets (Figure 1), all the values were normalized to z-score and trained using eight machine learning models. 5-fold cross-validation is performed, and model performance is reported (Table 1). The results are compared with clinical diagnosis.

To investigate the effect of data leakage, missing data was imputed using the Multiple Imputations by Chained Equation (Overall datasets). The experiment is repeated on the Imputed and Overall datasets (Figure 1). A parametric analysis is performed to investigate the effect of N-imputed values on model performance.

Results, Conclusions, and Discussions: Table 1 shows that logistic regression (LR) performed the best in SGA prediction among all the models, with the highest AUC of 0.7072. Compared to clinical diagnosis, LR shows a superior performance of F1 score and balanced accuracy by 46% and 11%, respectively.

Table 2 tabulates the model performance of eight models using different datasets. Utilizing Imputed datasets in evaluating models reveals a trend of increased performance by 2% to 37% compared to the Original datasets. Conversely, when evaluating the model with the Overall dataset, higher AUC values are observed, which can be up to 6.25%.

Parametric analysis results (Figure 2) showed that AUC increases when the number of imputed parameters (N) increases in the datasets. XGB, KNN, and LightGBM show the highest AUC surge when N increases. These results suggest that when data handling is inappropriate, specifically, when the testing data consists of imputed datasets, the outcomes tend to be overly optimistic.

We showed that when inappropriate data imputation is performed, SGA prediction is overly optimistic by an average of 2.3%. MLP model shows the highest increase of AUC (6.25%), followed by RF (5.72%). This further supports the need for standard reporting practices in ML studies to ensure the reproducibility of SGA prediction.

Acknowledgements (Optional):