Using Machine Learning to Predict the In-Hospital Mortality in Women with ST-Segment Elevation Myocardial Infarction

Background: Several studies have shown that women have a higher mortality rate than do men from ST-segment elevation myocardial infarction (STEMI). The present study was aimed at developing a new risk-prediction model for all-cause in-hospital mortality in women with STEMI, using predictors that can be obtained at the time of initial evaluation. Methods: We enrolled 8158 patients who were admitted with STEMI to the Tianjin Chest Hospital and divided them into two groups according to hospital outcomes. The patient data were randomly split into a training set (75%) and a testing set (25%), and the training set was preprocessed by adaptive synthetic (ADASYN) sampling. Four commonly used machine-learning (ML) algorithms were selected for the development of models; the models were optimized by 10-fold cross-validation and grid search. The performance of all-population-derived models and female-specific models in predicting in-hospital mortality in women with STEMI was compared by several metrics, including accuracy, specificity, sensitivity, G-mean, and area under the curve (AUC). Finally, the SHapley Additive exPlanations (SHAP) value was applied to explain the models. Results: The performance of models was significantly improved by ADASYN. In the overall population, the support vector machine (SVM) combined with ADASYN achieved the best performance. However, it performed poorly in women with STEMI. Conversely, the proposed female-specific models performed well in women with STEMI, and the best performing model achieved 72.25% accuracy, 82.14% sensitivity, 71.69% specificity, 76.74% G-mean and 79.26% AUC. The accuracy and G-mean of the female-specific model were greater than the all-population-derived model by 34.64% and 9.07%, respectively. Conclusions: A machine-learning-based female-specific model can conveniently and effectively identify high-risk female STEMI patients who often suffer from an incorrect or delayed management.


Introduction
ST-elevation myocardial infarction (STEMI), the most serious type of cardiovascular disease, is one of the leading causes of mortality worldwide [1][2][3].Multiple longitudinal studies have shown that mortality from STEMI is higher in women than in men [4][5][6][7][8].Risk stratification is critical in identifying high-risk patients and assisting physicians in decision making [9,10].The traditional risk assessment tools are the Global Registry of Acute Coronary Events (GRACE) [11] score and the Thrombolysis in Myocardial Infarction (TIMI) score [12,13], but the following three conditions are usually taken as major limitations for these tools: (1) the predictors are not immediately available on admission, and medical history is unreliable; (2) these tools were used without accounting for sex-specific disease characteristics of STEMI, whereas growing evidence has demonstrated sex differences in both symptom presentation and management efficacy STEMI patients [8,14].The symptoms of myocardial infarction (MI) in women patients are atypical, which make women often suffer from an incorrect or delayed management [15][16][17][18][19]; (3) These tools were developed based on a traditional statistical method, which may lead to the loss of important information [20][21][22][23].Recently, the GRACE 3.0 score, based on machine learning (ML), was developed to reduce sex inequalities, but it was specially designed for the risk assessment of non-ST-elevation acute coronary syndromes (NSTE-ACS) [7].Therefore, it is necessary to develop a new risk-prediction model for women with STEMI using predictors that can be obtained at the time of initial evaluation.
ML algorithms can capture nonlinear relationships among clinical variables, and have many successful applications [24][25][26][27][28].However, real-world medical data are often imbalanced.When trained with imbalanced data, the developed ML models can be overwhelmed by the majority class (i.e., survival group) and can ignore the minority class (i.e., death group) [29], which is the focus of clinical attention.To alleviate this problem, an effective strategy is data preprocessing.The data-preprocessing approach is to resample the imbalanced training set prior to model training.In order to create the balanced training set, the original imbalanced data set can be oversampled for the minority class and/or undersampled for the majority class [30].Since the undersampling strategy leads to the loss of information from the majority class, we adopted the adaptive synthetic (ADASYN) oversampling approach [31], which has been proven effective [32].Due to the "black box" nature of ML algorithms, the SHapley Additive exPlanations (SHAP) value was employed to explain the predictors' impact on the outcome [33].
The aims of this study were to: (1) develop prediction models for all-cause in-hospital mortality in women with STEMI using four commonly used ML algorithms combined with the ADASYN sampling approach, and (2) explain the prediction models with SHAP values.

Study Sample
The present study was conducted with information from a hospital-based dataset as described previously [24].In brief, a total of 8158 patients, from January 2015 to December 2021, with STEMI, were retrospectively enrolled.This sample included 6084 (74.58%

Data Collection and Preprocessing
The basic clinical data of the patients were collected, including demographic information (sex, age), physical examination (heart rate, systolic blood pressure, diastolic blood pressure, etc.), laboratory tests (cardiac troponin I), admission pathway, and treatment.All the clinical variables could be obtained at the time of initial evaluation.The primary endpoint was all-cause in-hospital mortality.
Variables with a missing-data percentage of less than 20% were retained.For continuous variables, the mean imputation method was used to supply the missing values, which replaces the missing values of a certain variable with the mean of the available cases.For categorical vari- ables, the mode imputation method was applied to supply the missing values, which replaces the missing values of a certain variable with the mode of the available cases.Respiration, heart rate, systolic blood pressure, diastolic blood pressure, cardiac troponin I, and time from symptom to first medical contact, were missing in 0.06%, 0.05%, 0.07%, 0.07%, 3.36% and 0.33% cases, respectively.Because the range of different variables varied widely, and some of the used algorithms required quantitative data normalization, zscore normalization was used [35].

Statistical Analysis
Categorical variables are reported as counts (%) and continuous variables as mean (SD) or median (IQR).The Kolmogorov-Smirnov test was used to test the normality of distribution.We used Student's t test to assess the differences between parametric continuous variables and the Mann-Whitney-U test for non-parametric variables.We used the Chi-squared test to evaluate the differences between categorical variables.All statistical analysis were performed using Python 3.7.3(Python Software Foundation, Wilmington, Delaware, USA) with the scientific libraries "scipy.stats".A two-tailed p ≤ 0.05 was considered statistically significant.

Model Development and Validation
According to whether the endpoint occurred, the entire set of data was divided into a survival group and a death group.Each of the two groups was randomly split into a sub-training set (75%) and a sub-testing set (25%), and then the two sub-training sets were merged to get the Training Set, and the two sub-testing sets were merged to get the Testing Set as shown in Fig. 2. The Training Set was pretreated using the ADASYN sampling technique to achieve a balance between the minority class (death group) and the majority class (survival group).Four commonly used ML algorithms, including decision trees (DT), random forests (RF), support vector machines (SVM), and extreme gradient boosting (XGBoost), were selected for the development of models to predict the in-hospital mortality in patients with STEMI.A Grid Search method with 10fold cross validation was used to optimize the ML models.The hyperparameter settings of each model were shown in Table 1.Model performance was assessed according to several learning metrics (accuracy, specificity, sensitivity, Gmean, and area under the receiver operating characteristic curve [AUC]).The performance of all-population-derived models and female-specific models in predicting in-hospital mortality in women with STEMI was compared to demonstrate the effectiveness of the female-specific model proposed in this study.In addition, a 5 × 2 cross validation paired t test was used to evaluate the difference between two models [36].The model development and validation were performed using Python (Version 3.7.3)software with the packages "scikit-learn", "xgboost", and "imblearn".

Model Interpretation
Although the ML models can provide more accurate predictions than traditional statistical models, the results cannot be explained.To show the decision-making process in an intuitive way, the SHAP value was included.SHAP is an approach based on game theory, proposed by Lundberg and Lee, to interpret ML models [33].The optimal SHAP value was calculated for each feature of each sample after the model was trained, and the impact of each feature on predictions can be represented by SHAP values [37].Note: the SHAP value has a stronger theoretical basis than other methods [38] and the performance of its explainabil-ity has been validated in previous work [25,39,40].The ML model explanation was performed using Python (Version 3.7.3)software with the package "shap".

Patient Characteristics
In all, 8158 STEMI patients were included in this study, including 6084 male patients (74.58%) with a median age of 61.00 (53.00, 68.00) years, and 2074 female patients (25.42%) with a median age of 70.00 (63.00, 77.00) years.The median age of all patients was 63.00 (55.00, 71.00) years.The overall in-hospital mortality rate was 3.02% (n = 246).Table 2 shows the baseline characteristics and the comparisons between patients who died and those who survived.Compared with surviving patients, dead patients were more likely to have had higher rates of emergency medical services (EMS) admissions, higher Killip classification, lower reperfusion rates, higher age, faster respiration, higher heart rates (HR), lower systolic blood pressure (SBP), lower diastolic blood pressure (DBP) and higher cardiac troponin I (cTnI).Additionally, patients in death group were more likely to have been unconscious.

Development of All-Population-Derived Models and Validation in Women
The performance of different all-population-derived models was shown in Table 3 and the analysis of receiver operating characteristic (ROC) curves was shown in Fig. 3.The performance of models was significantly improved by ADASYN according to G-mean and AUC.The SVM combined with ADASYN achieved the best performance (Gmean: 80.33%; accuracy: 75.98%; sensitivity: 85.29%; specificity: 75.66%; and AUC: 85.36%).As shown in Fig. 4, 1492 of 1972 patients and 58 of 68 patients were correctly classified into the low-risk group and high-risk group, respectively.However, the all-population-derived models performed poorly in women with STEMI as shown in Table 4.The best performing model achieved only 53.66% accuracy, 51.48% specificity and 70.36% G-mean.Fig. 5 shows that 246 of 507 patients were incorrectly classified into the high-risk group.Additionally, Fig. 6 shows that sex (ranked as 4/12) was highly associated with the outcome, and that women have a higher risk of all-cause mortality.

Sex Differences in Patients with STEMI
The comparison between men and women is shown in Table 5.Compared with men, women were more likely to have higher mortality, higher Killip classification, lower reperfusion rates, higher age, lower DBP, and longer time from symptom to first medical contact (S to FMC).The baseline characteristics and the comparisons between the survival group and the death group in female patients were shown in Table 6.Compared with surviving patients, dead patients had been more likely to have higher EMS admission rates, higher Killip classification, lower reperfusion   rates, higher age, lower SBP and higher cTnI.

Development, Validation and Comparison of Female-Specific Models
The performance of different female-specific models is shown in Table 7 and the analysis of ROC curves is shown in Fig. 7. Similarly, the performance of models was significantly improved by ADASYN.The SVM combined with ADASYN achieved the best performance (G-mean: 76.74%; accuracy: 72.25%; sensitivity: 82.14%; speci-   DT, decision tree; RF, random forest; SVM, support vector machine; XGBoost, extreme gradient boosting; ADASYN, adaptive synthetic; AUC, area under the curve.G-mean is the geometric mean of sensitivity and specificity.

Discussion
STEMI is the leading cause of death among women worldwide [1][2][3][4][5][6][7][8], which may be partly attributed to atypical symptoms and insufficient risk assessment.Therefore, in the present study, four commonly used ML algorithms were selected for the development of models to predict the in-hospital mortality in women with STEMI.Additionally, ADASYN was applied in order to improve the performance of the models [31].The best performing female-specific model achieved an accuracy, sensitivity, specificity, Gmean, and AUC of 72.25%, 82.14%, 71.69%, 76.74% and 79.26%, respectively, leading to a more convenient and effective identification of high-risk patients at the first medical contact.
Consistent with previous studies [4,41], our results demonstrated that women were more likely than men to have a delay between symptom and medical contact (121 min vs. 116 min, p = 0.0006), lower rates of reperfusion treatment (75.27% vs. 82.89%,p < 0.0001), and higher mortality (4.92% vs. 2.37%, p < 0.0001).The mechanisms behind these differences may be the following: (1) women with STEMI are more likely to present with multiple nonchest pain symptoms [19,42,43], which often results in an incorrect or delayed management; (2) competing responsibilities, as well as embarrassment or fear of disturbing others, lead women to be more likely to wait until symptoms subside rather than seek care [44]; and (3) because of lower socioeconomic status and lower perception for the risk of heart disease, women are less willing to opt for invasive coronary angiography [45][46][47].As a result, physicians, patients, and relatives all tend to choose conservative treatments due to the lack of sex-specific guidelines [48].Therefore, it is critical to optimize risk assessment and subsequent management, although the traditional risk-assessment tools are far from perfect.As machine learning blossoms, there are many successful applications of machine-learning models in the cardiovascular field.A machine-learning-based model called the PRAISE score was developed for predicting all-cause death, recurrent acute myocardial infarction, and major bleeding after acute coronary syndrome [49], but  it was not designed for women.Recently, the GRACE 3.0 score, based on machine learning, was developed to reduce sex inequalities, but it was specifically developed for risk assessment of NSTE-ACS [7].
Although ML algorithms are accurate in capturing complex nonlinear relationships between clinical variables, when trained with imbalanced real-world medical data, the developed models are vulnerable to incorrectly predicting the minority class as the majority class [50], which leads the models to ignore high-risk patients.Therefore, an oversampling technology called ADASYN was applied to generate more samples from the minority class to alleviate the above problem.Compared with undersampling technology, which balances the Training Set by discarding the majority class samples, ADASYN can fully utilize precious medical data, resulting in a higher level of robustness [31].Due to the "black-box" nature of ML models, the SHAP value was applied for explanation.The SHAP assesses the effect of each feature on results and presents it in an intuitive way [33], which can help doctors better understand how the model works, rather than blindly trusting the predictions.
The present study demonstrated that the femalespecific models significantly outperformed the allpopulation-derived models in predicting in-hospital mortality in women with STEMI, and sex was considered to be an important predictor according to the feature importance scores (Fig. 4).However, women were not well represented in the study sample of the TIMI trial, where they accounted for only 24.7% [12], and in the  study sample of the GRACE trial, where they accounted for 33.5% [11].Additionally, our models can provide predictive results at the initial evaluation, resulting in an improvement in applicability.The 2017 ESC Guidelines recommend an aggressive treatment strategy for high-risk patients [34].However, physicians, and women with STEMI, are more likely to choose conservative treatment (treatment-risk paradox), which can be inappropriate [51].The proposed female-specific models can conveniently and effectively identify high-risk patients at the first medical contact, which can provide a basis for physicians to choose intensive treatment for high-risk patients, thereby improving treatment compliance.
This study has several limitations to be acknowledged.First, this is a single-center study.Therefore, the models should be validated in external centers to confirm their generalizability.Nonetheless, the risk prediction model proposed in this study still provides a convenient and effective method to predict in-hospital mortality in women with STEMI.Second, this is a retrospective study.Bias in patient enrollment and data collection is inevitable.However, the patients' data were collected from a high-quality database, which reflected the real world.Third, the endpoint of this study included only in-hospital mortality, with no information on myocardial infarction, ischemic stroke, or heart failure; information on longitudinal follow-up was not obtained.Thus, further long-term follow-up studies are needed to obtain more detailed and comprehensive information in order to develop more clinically instructive models.Finally, some important predictors were not included in this study, such as creatinine level, myocardial injury biomarkers, and sex-specific risk factors, which attenuated the performance of models and made the comparison to other risk scores impossible.Conversely, our models can be used conveniently and effectively in pre-hospital or emergency departments.In addition, the symptoms of MI are usually atypical in elderly patients and in patients with diabetes, which makes these patients less willing to seek medical ser-vice.Therefore there are many papers and literature works that focus on diabetes and the elderly as distinct groups [52][53][54][55].Accordingly, future studies should focus on applying machine learning to improve the prognosis of these patients.

Conclusions
In this study, four commonly used ML models (DT, RF, and SVM) were developed to predict in-hospital mortality in women with STEMI.The predictors could be obtained at initial evaluation.Additionally, ADASYN was applied to assess and mitigate the effects of class imbalance, thereby improving model performance.By capturing the non-linear association of predictors, the proposed femalespecific model could conveniently and effectively identify high-risk female patients at the first medical contact.Therefore, the integration of our female-specific model into daily clinical practice may improve the prognosis of women with STEMI who often suffer from an incorrect or delayed diagnosis.
) males and 2074 (25.42%) females.The enrollment criteria for patients are as follows: (1) the diagnosis of STEMI complied with the European Society of Cardiology Guidelines for the diagnosis and treatment of acute ST-segment elevation myocardial infarction [34]; (2) persistent ischemic chest pain for less than 12 hours; (3) electrocardiogram (ECG) findings showing the presence of ST segment elevation in two or more consecutive leads, with ≥0.2 mV in the precordial leads and ≥0.1 mV in the limb leads.The exclusion criteria were as follows: (1) age <20 or >100; (2) incomplete laboratory indexes; (3) missing data on sex; or (4) unknown in-hospital outcome.This observational and retrospective study was approved by the Local Ethics Committee.The flowchart of this study is shown in Fig. 1.

Fig. 4 .
Fig. 4. The confusion matrix of the best performing allpopulation-derived model in the overall population.

Fig. 5 .
Fig. 5.The confusion matrix of the best performing allpopulation-derived model in women with STEMI.STEMI, STelevation myocardial infarction.

Fig. 8 .
Fig. 8.The confusion matrix of the best performing femalespecific model in women with STEMI.STEMI, ST-elevation myocardial infarction.

Table 2 . Basic Characteristics of the overall sample by outcome.
EMS, Emergency medical services; PCI, percutaneous coronary intervention; HR, heart rate; SBP, systolic blood pressure; DBP, diastolic blood pressure; cTnI, cardiac troponin I; S to FMC, time from symptom to first medical contact.

Table 4 . The performance of all-population-derived models in women with STEMI.
DT, decision tree; RF, random forest; SVM, support vector machine; XGBoost, extreme gradient boosting; ADASYN, adaptive synthetic; AUC, area under the curve.G-mean is the geometric mean of sensitivity and specificity.

Table 5 . Basic Characteristics of the overall patient population by sex.
EMS, Emergency medical services; PCI, percutaneous coronary intervention; HR, heart rate; SBP, systolic blood pressures; DBP, diastolic blood pressure; cTnI, cardiac troponin I; S to FMC, time from symptom to first medical contact.