Feature-Limited Performance in Machine Learning Prediction of Endometriosis from Clinical Symptoms

Abstract

Background: Endometriosis affects approximately 10% of women of reproductive age worldwide, yet diagnosis remains challenging due to nonspecific symptoms and reliance on invasive laparoscopic confirmation, resulting in diagnostic delays averaging seven to ten years. Machine learning approaches have shown promise for noninvasive screening, but fundamental questions remain regarding whether performance limitations arise from model architecture constraints, insufficient training data, or intrinsic information ceilings imposed by symptom-based clinical features.

Methods: Five machine learning architectures (logistic regression with L1 regularization, support vector machines with radial basis function kernels, gradient-boosted decision trees, random forests, and deep neural networks) were systematically compared for endometriosis prediction using six base clinical variables from 10,000 patient records. The pipeline incorporated stratified data splitting (80%/10%/10% train/validation/test), label noise mitigation through ambiguity-based instance weighting, cost-sensitive learning prioritizing false negative reduction, and cross-validated threshold optimization. Feature engineering expanded the base features to 21 variables through interaction terms, polynomial transformations, and discretized bins. Learning curve analysis assessed whether performance was constrained by training set size or feature informativeness.

Results: All five model architectures converged to similar test performance (AUC range: 0.653–0.674), with the selected logistic regression model achieving test AUC of 0.674, recall of 0.566, precision of 0.562, and specificity of 0.696 at the Youden-optimized threshold. Feature engineering yielded negligible improvements, with mean test AUC changing by only 0.002 between baseline (6 features) and engineered (21 features) configurations. Learning curves plateaued beyond 60% of training data, with training and validation AUC converging to 0.667 and 0.641 respectively, and the gap narrowing from 0.058 to 0.026.

Conclusions: The convergence of multiple model families to similar performance limits, minimal gains from feature engineering, and plateaued learning curves provide empirical evidence that model performance is constrained by the information content of symptom-based clinical features rather than by model architecture, sample size, or feature representation sophistication. The observed AUC ceiling of approximately 0.65–0.67 aligns with published literature on symptom-based endometriosis screening and indicates that clinically actionable discrimination performance requires data enrichment through incorporation of imaging findings, biomarkers, or genomic risk factors rather than algorithmic innovation.