Data analytics approach for short- and long-term mortality prediction following acute non-ST-elevation myocardial infarction (NSTEMI) and Unstable Angina (UA) in Asians.

Background: Traditional risk assessment tools often lack accuracy when predicting the short- and long-term mortality following a non-ST-segment elevation myocardial infarction (NSTEMI) or Unstable Angina (UA) in specific population. Objective: To employ machine learning (ML) and stacked ensemble learning (EL) methods in predicting short- and long-term mortality in Asian patients diagnosed with NSTEMI/UA and to identify the associated features, subsequently evaluating these findings against established risk scores. Methods: We analyzed data from the National Cardiovascular Disease Database for Malaysia (2006–2019), representing a diverse NSTEMI/UA Asian cohort. Algorithm development utilized in-hospital records of 9,518 patients, 30-day data from 7,133 patients, and 1-year data from 7,031 patients. This study utilized 39 features, including demographic, cardiovascular risk, medication, and clinical features. In the development of the stacked EL model, four base learner algorithms were employed: eXtreme Gradient Boosting (XGB), Support Vector Machine (SVM), Naive Bayes (NB), and Random Forest (RF), with the Generalized Linear Model (GLM) serving as the meta learner. Significant features were chosen and ranked using ML feature importance with backward elimination. The predictive performance of the algorithms was assessed using the area under the curve (AUC) as a metric. Validation of the algorithms was conducted against the TIMI for NSTEMI/UA using a separate validation dataset, and the net reclassification index (NRI) was subsequently determined. Results: Using both complete and reduced features, the algorithm performance achieved an AUC ranging from 0.73 to 0.89. The top-performing ML algorithm consistently surpassed the TIMI risk score for in-hospital, 30-day, and 1-year predictions (with AUC values of 0.88, 0.88, and 0.81, respectively, all p < 0.001), while the TIMI scores registered significantly lower at 0.55, 0.54, and 0.61. This suggests the TIMI score tends to underestimate patient mortality risk. The net reclassification index (NRI) of the best ML algorithm for NSTEMI/UA patients across these periods yielded an NRI between 40–60% (p < 0.001) relative to the TIMI NSTEMI/UA risk score. Key features identified for both short- and long-term mortality included age, Killip class, heart rate, and Low-Molecular-Weight Heparin (LMWH) administration. Conclusions: In a broad multi-ethnic population, ML approaches outperformed conventional TIMI scoring in classifying patients with NSTEMI and UA. ML allows for the precise identification of unique characteristics within individual Asian populations, improving the accuracy of mortality predictions. Continuous development, testing, and validation of these ML algorithms holds the promise of enhanced risk stratification, thereby revolutionizing future management strategies and patient outcomes.

While modern techniques in percutaneous coronary intervention (PCI) have evolved, incorporating innovations like drug-eluting stents and the increased use of pharmacological agents such as beta-blockers and renin-angiotensin system blockers, acute myocardial infarction (AMI) still ranks as a major cause of mortality in Asia [[1]].

AMI can be classified as either ST-elevation myocardial infarction (STEMI) or non-ST-elevation myocardial infarction (NSTEMI), depending on the results of an electrocardiogram (ECG). It is crucial to have the ability to predict morbidity and mortality in AMI patients in order to identify those individuals who are at an increased risk, who therefore require intense therapy and vigilant monitoring [[3]].

Frequently used risk assessment techniques, such as Thrombolysis in Myocardial Infarction (TIMI) and the Global Registry of Acute Coronary Events (GRACE), play an important role in predicting the likelihood of death and recurrent cardiovascular events in patients with AMI, particularly those with NSTEMI [[4]]. The improved GRACE risk predictor (version 2.0) has been thoroughly evaluated for its accuracy in predicting outcomes in NSTEMI patients, broadening the scope of risk assessment tools beyond standard models [[6]]. Additionally, other models such as Naples Score [[7]] and Intermountain Risk Score [[8]], have also been used to validate mortality risk in patients with STEMI. It is worth noting that these risk scores were originally established to predict mortality for illnesses such as cancer and other health outcomes, demonstrating their adaptability in measuring the probability of adverse occurrences in many clinical circumstances.

The limitation of commonly used risk assessments tools for NSTEMI such TIMI and GRACE were formulated based on data from Western Caucasian AMI patients during the 1990s to early 2000s. This occurred prior to the introduction of drug-eluting stents and the most recent iteration of antiplatelet medications. Significantly, these methods inadequately represent the Asian demography, despite Asia comprising around 60% of the world's population [[9]]. Furthermore, the prediction models' outcomes were limited to short-term mortality, such as in-hospital, 14-day, and 30-day death [[3], [10]].

Recent research indicates that incorporating machine learning (ML) and stacking ensemble learning (EL) techniques can improve the effectiveness of AMI prediction models, overcoming the limits of conventional risk score methodology. The conventional risk score has several shortcomings, including a major dependence on linearity, parametric assumptions, and a restricted capabilities to examine higher-order interactions [[12]].

ML is tailored to manage complex data, facilitating the development of precise risk-prediction models [[13]]. Several ML algorithms, including Support Vector Machine (SVM), Random Forest (RF), Extreme Gradient Boosting (XGBoost), and Naive Bayes (NB), have validated this concept [[14]–[17]]. The ML and EL algorithms have been implemented at a population-specific level in countries like Malaysia, Korea, and China for mortality predictions. These algorithms outperformed conventional risk scores such as TIMI and GRACE in terms of Area Under the Curve (AUC) for mortality prediction post-ACS, STEMI, and NSTEMI [[9], [13], [18]–[21]]. Accurate prediction is useful for determining the appropriate level of care as well as the appropriate pharmacological or invasive treatment; however, there is only a limited risk classification model on NSTEMI/UA [[22]–[25]], and none of them have been built specifically for NSTEMI/UA Asian patients as of yet.

Furthermore, although some individual ML models for predicting post-ACS mortality demonstrate exceptional outcomes, there are still several obstacles that need to be addressed. Firstly, no singular ML technique consistently surpasses others within the same field. Furthermore, combining multiple algorithms may result in enhanced performance, especially in the medical domain. Accuracy is of utmost importance in this field, as it directly impacts the efficiency of diagnoses and the survival outcomes of patients [[26]]. As a result, new prognostic methods for predicting mortality in Asian NSTEMI/UA patients must be considered throughout time.

Patients who are admitted with NSTEMI/UA in lower-income or Asian nations typically have a younger age, are frequently diabetic, and are current smokers. They also tend to have fewer chronic health conditions. Additionally, a significant majority (76.7%) of these patients satisfy the requirements for an early invasive surgery, indicating an exceedingly high risk [[27]]. These NSTEMI/UA patients exhibit a higher likelihood of adverse outcomes such as mortality, recurrent myocardial infarction, symptomatic ischemia, severe arrhythmias, heart failure, and stroke. According to published studies [[28]–[31]], although the in-hospital mortality rate is lower for patients with NSTEMI/UA compared to those with STEMI, the long-term risk of CVD is similar between them, making risk assessment difficult for this NSTEMI/UA population. Therefore, the therapeutic significance of risk stratification in patients with AMI, especially those with NSTEMI/UA, is substantial. This is due to the wide range of clinical manifestations and subsequent prognosis associated with these conditions [[32]].

The primary objective of this study is to discern features associated with mortality and to develop ML and EL models that can accurately predict the short- and long-term mortality risks in Asian NSTEMI/UA patients. To realize this objective, four established ML algorithms—SVM, RF, XGBoost, and NB—will be employed. Additionally, these algorithms will serve as base learners in the construction of the stacked EL model. Our research has made a substantial contribution by identifying the significant prognostic features correlated with in-hospital, 30-day, and 1-year mortality in a multi-ethnic Asian cohort diagnosed with NSTEMI/UA, achieved using the SVM feature selection approach.

According to existing literature [[34]], the implementation of feature selection techniques is critical for improving the predictive accuracy of ML algorithms. Consequently, the identification of mortality-associated factors is crucial for identifying high-risk patient characteristics, thereby facilitating improved patient care in clinical settings. To achieve this, the SVM feature selection method is employed, followed by the construction of the ML, and stacking EL models utilizing the selected features. Then, we compare the performance of individual ML models to that of stacked EL models with complete and selected feature sets. Lastly, we intend to validate the performance measures of the developed prediction models using multi-ethnic registry data on Asian NSTEMI/UA and TIMI NSTEMI/UA risk scores.

Patient data for this study was collected from the Malaysian National Cardiovascular Disease Database (NCVD-ACS) registry, encompassing multiethnic Asian individuals hospitalized with NSTEMI and UA from 2006 to 2019. The Medical Review & Ethics Committee (MREC) of the Ministry of Health, Malaysia, authorized the establishment of the NCVD registry in 2007, with the approval code NMRR-07-20-250. An exemption from informed consent for participants was permitted by the MREC for this NCVD study. The registry includes standardized clinical, demographic, procedural, and patient outcomes from all consecutive patients at associated centers [[36]]. Ethical clearance for data collection was also obtained from the UITM Ethics Committee (Reference number: 600-TNCPI (5/1/6)) and the National Heart Association of Malaysia (NHAM).

The data was gathered using a standardized case report that covered the period from the patients' admission to the hospital until their discharge, as well as the subsequent follow-up. The major outcome of interest was the patients' survival status, which was classified as alive or dead. The dataset included information on patients' demographics, clinical presentation, initial diagnostic tests, electrocardiography results, therapeutic procedures, and pharmaceutical medication. Each individual patient was assigned a unique identification number to avoid any instances of duplication. The data used in this study were anonymized before analysis since our research focuses entirely on the values and features of the data without gaining access to any personal information about the patients.

The study included all patients who were enrolled in the NSTEMI/UA registry, without any events of exclusion. Patients who received reperfusion treatments such as primary percutaneous coronary intervention (PCI), angiography demonstrating spontaneous reperfusion, further fibrinolysis, or urgent coronary artery bypass grafting (CABG) were included in this group. Within this context, NSTEMI was defined as an incident of AMI with evidence of a positive cardiac marker but no ST-segment elevation in electrocardiographic (ECG) measurements. This study used a complete set of datasets, including a total of 39 features, in adherence to established clinical criteria. This study's features included sociodemographic characteristics, CVD diagnosis and severity, risk factors associated with CVD, comorbidities linked to CVD, comorbidities unrelated to CVD, biomarkers, and medication consumption. The yearly verification of mortality was carried out by establishing data links with the Malaysian National Registration Department. The following data points were originally meant to collect these features; however, they were unfortunately omitted from the research due to a significant number of missing values. The primary goal of this research was the development of an algorithm aimed at modifying policy endpoints, with a special emphasis on severe endpoints such as death, in order to augment the study's importance. This methodology is similarly used in other academic publications [[9], [38]].

The development of ML and stacked EL model was based on complete cases data to ensure the robustness and validity of the research findings. Complete cases refer to instances when there is no missing data related to the predictors. A total of 54,926 cases of NSTEMI/UA were obtained from the registry. Among these cases, 23,682 were classified as complete cases. From these, subsets of 9,518 for in-hospital, 7,133 for 30-day, and 7,031 for 1-year outcomes. The study used an extensive set of 39 features, consisting of 8 continuous and 31 categorical features. The frequency of missing data across these features is detailed in S1 Table.

For the secondary analysis, we included an additional 31,244 instances that had missing data. This resulted in a comprehensive dataset that consists of 54,926 patients. This missing cases was imputed through multivariate imputation by chained equations, utilizing predictive mean matching from the MICE package in R [[40]]. This imputation approach estimates missing values by utilizing actual data from similar cases, thus maintaining the integrity of the dataset.

An "incomplete dataset" is defined in the study as a dataset in which the missing data for any particular feature does not exceed 30%. It is crucial to emphasize that the missing data only refers to patient demographics and clinical features and does not include information on treatment outcomes. Given the prospective design of our dataset and the retrospective approach to data management, it is imperative to recognize that the missing values across all features occurred randomly and were not influenced by the researchers. This randomness of missing data, affirmed by its classification as Missing Completely at Random (MCAR), ensures that these missing values are uncorrelated with any of the features used or unused in the analysis, supporting the assumption that the missingness is independent of the observed or unobserved data. This randomness of missing data, as indicated by its classification as Missing Completely at Random (MCAR), ensures that these missing values are uncorrelated with any of the features used or omitted in the study, supporting the assumption that missingness is independent of observed or unobserved data.

In line with Kuhn and Johnson's methodology [[41]], the dataset was subjected to stratified random sampling and divided into two groups: 70% for model training and 30% for validation. This separation applied to both complete and imputed cases to avoid data leakage, as recommended by best practices [[42]]. To avoid potential duplications due to multiple admissions, patient data were aggregated under unique patient identities, ensuring that all records for a particular patient were solely included in either the training or validation set. For this study, a randomly generated patient ID system was used to replace actual patient identifiers, preserving data anonymity, and conforming to confidentiality procedures.

Data pre-processing techniques, including imputation for handling missing instances and balancing for both full and missing cases, were exclusively used for the training data. Normalization techniques were applied to the training data, and the same scaling parameters were then used to normalize the testing dataset, ensuring the integrity of the model evaluation process, and preventing data leakage.

In the study, distinct normalizing techniques were applied to the training and testing datasets in a different manner. The performance of the resulting model and the TIMI risk score for NSTEMI/UA was assessed using a validation dataset, which constituted 30% of the total data and was entirely excluded from the model training phase to preserve the integrity of the model's performance assessment.

Class imbalance between the minority non-survival group and the majority survival group was addressed by pre-processing the training dataset. To rectify the imbalance and provide an equal representation of both classes, a combined method of up sampling the minority class and down sampling the majority class was used, utilizing the ROSE package as described by Lunardon et al. (2014) [[43]]. This balanced the classes to prevent bias in the subsequent ML model training.

Normalization approaches were used in the study to ensure that all continuous features contribute equally to the model's performance, preventing larger scale features from dominating the pattern recognition process [[41]]. In this study, standardization, also known as z-score normalization, was applied to continuous features such as age, heart rate, systolic and diastolic blood pressure, total cholesterol, HDL, LDL, triglycerides, and fasting blood glucose to ensure a consistent scale across these variables.

Several ML techniques were used to develop prediction models for NSTEMI/ UA using the R programming language. These algorithms include Support Vector Machine (SVM) [[44]], Random Forest (RF) [[45]], Extreme Gradient Boosting (XGBoost) [[46]], and Naive Bayes [[47]].

The SVM is a robust ML algorithm known for its ability to classify by finding the optimal hyperplane that maximizes the margin between two classes. For our study, the SVM model utilized two types of kernels: a linear kernel for linearly separable data and a Radial Basis Function (RBF) kernel for non-linear data separation. RF is an EL technique that builds multiple decision trees during training. It utilizes the bagging technique to determine the classification outcome based on the majority vote from individual trees, improving prediction robustness and accuracy. XGBoost is part of the gradient boosting algorithm family. It is well-known for its regularization strategies, which are intended to prevent overfitting, which is a common problem in complex models, resulting in a more generalized model that performs better on unseen data. The Naive Bayes (NB) algorithm is a probabilistic classifier that applies Bayes' Theorem while making the 'naive' assumption of conditional independence between every pair of features given the class label. Despite its simplicity, NB can perform remarkably well, particularly when the independence assumption is obeyed. The selection of these algorithms was based on prior studies pertaining to mortality associated with cardiovascular disease [[13], [17], [48]–[51]]. A combination of random search and manual adjustment was used to fine-tune the hyperparameters of the ML and stacked EL models, ensuring that the models are optimally configured for the task. Details on the hyperparameters can be found in S2 Table.

Stacking is a meta-learning technique that uses the outputs of various base learners as input for a subsequent meta-learner, which then makes the final prediction. This ensemble approach is well-known for its ability to outperform any individual classifier or regressor. We used several kinds of base learners for developing our stacked EL model, including SVM with both linear and RBF kernels, RF, XGBoost, and NB. The predictions from these base learners were combined using a meta-learner, a Generalized Linear Model (GLM), which combines the individual predictions into a final, ensemble prediction. This method relies on the strengths of each base learner and often generates improved predictive accuracy [[52]–[54]].

Previous research [[19], [49], [51]] has shown that features selected by SVM improve model performance when compared to other ML techniques. This led to the prioritization of SVM-ranked features in our feature selection procedure. The Backward Feature Elimination (BFE) method was used to systematically eliminate less significant features, ranking them ascendingly based on their contribution to the predictive abilities of the model [[55]]. This elimination was applied to the SVM-ranked features [[56]]. The models were trained and evaluated for each feature subset iteration using the 30% hold-out validation set, which was independent of the model training stage. This step was crucial in determining and choosing models that provided optimal performance while utilizing the fewest features possible, resulting in a more generalizable reliable model. These optimized feature sets were then used to develop the final ML and stacked EL models. To reduce the risk of overfitting, 10-fold cross-validation was used on the training dataset during the model training process [[57]].

The integration of ML models into clinical practice is impeded given their 'black-box' nature. The behavior of ML models, which are inherently agnostic, can be deduced by modifying inputs and analyzing the resulting output [[58]]. Changes can be made to align these inputs with human comprehensibility. Shapley Additive Explanations (SHAP) is employed in this study for interpreting the most optimal ML model. Shapley values are used to assess the impact of input features on the output of an ML model. The SHAP values indicate the level of significance that a model assigns to specific features, allowing for the establishment of a feature ranking [[59]].

In order to assess the calibration of the models, standardized metrics were utilized [[60]]. The area under the curve (AUC) was used as a predictor of performance due to its independence from class distribution. Additional performance metrics, including accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV), were used to further evaluate the calibration of the models. A paired resampled t-test was carried out as a means to comparison of the predictive performance of ML and stacked EL models [[61]]. In addition, the net reclassification improvement (NRI) was computed to assess the extent to which the enhanced ML model improved the accurate classification of positive and negative cases, as compared to the TIMI risk score for NSTEMI/UA [[62]].

The TIMI scores for NSTEMI/UA obtained from the NCVD registry were used to validate the dataset's performance. In terms of predictive performance, we compared the TIMI score to our developed ML model and the stacked EL model. The area under the curve (AUC) was used as the primary metric to assess the predictive accuracy of the models. This assessment was carried out on a validation set that comprised 30% of the total data, ensuring an accurate assessment. A performance breakdown graph was generated as well to examine the TIMI score for NSTEMI/UA application in clinical practice with regard to established literature thresholds [[62]]. The ML model's high-risk group in this study is defined as those with a mortality probability greater than 50%, which corresponds to a TIMI score greater than 5.

In our study, we presented continuous variables as the mean ± standard deviation, while categorical variables were displayed as frequency and percentage. The Chi-square test was used to analyze categorical features like Killip class and statin usage. The Man-Whitney test was used for non-normally distributed continuous variables, and the two-sided independent t-test was used for normally distributed continuous features. To determine the normality of the data, the Shapiro-Wilk Test was used, with a p-value less than 0.05 indicating a non-normal distribution. Both the ML and stacked EL models were evaluated using pairwise corrected resampled t-tests [[45], [57]], with a p-value of less than 0.001 considered statistically significant. Fig 1 provides a detailed overview of the workflow and methods used in this study.

Graph: Fig 1 info:doi/10.1371/journal.pone.0298036.g001

Table 1 summarizes the baseline demographic and clinical characteristics of the NSTEMI/UA patient cohort, stratified by survival status at the in-hospital, 30-day, and 1-year time points. The data depict the patients' mean age, which is uniformly distributed around the age of 61, with a significantly higher proportion of males (approximately 73%). The data reveals a progressive increase in mortality rates among the patients, with in-hospital mortality at 3.8%, rising to 6.6% at 30 days, and reaching 16.5% by the 1-year follow-up. A statistically significant difference between survivor and non-survivor groups is observed across multiple variables, including age, smoking status, and a history of various comorbidities. Clinical measures such as heart rate, blood pressure, and Killip class, alongside laboratory parameters including cholesterol levels, glucose, and specific ECG changes, also exhibited significant differences between the two groups (p < 0.001 for all). This table highlights the differences in clinical features and outcomes for NSTEMI/UA patients and provides a statistical basis for further model development and analysis.

Graph

Table 1 Comprehensive baseline characteristics and outcome correlates of the NSTEMI/UA patient population.

1 • An asterisk (*) denotes that a p-value of less than 0.001 signifies a statistically significant difference between the features of survival and non-survival.

2 • Features with statistically significant values are highlighted in bold.

Table 2 presents a comprehensive comparison of the AUC scores of the TIMI risk score with different ML models. The ML models, trained using both the entire set of 39 features and SVM selected subset, demonstrated better predictive abilities when compared to the TIMI NSTEMI/UA risk score in the validation dataset. Specifically, for NSTEMI/UA patient predictions, all ML models, particularly when applying feature selection, outperformed the full-feature models and the conventional TIMI risk scores. The XGBoost model exhibited the highest AUC scores for in-hospital and 30-day predictions, whereas the stacked EL model performed exceptionally well in the 1-year prediction (p-value < 0.001 for all models). Table 3 describes the detailed performance of ML models in relation to the TIMI risk score for NSTEMI/UA.

Graph

Table 2 The AUC for full and selected feature ML models relative to TIMI score in NSTEMI/UA based on a 30% validation dataset.

• 3 Abbreviations
• 4 LinSVM = Support Vector Machine with Linear Kernel
• 5 LinSVM-BFE = LinSVM with Backward Feature Elimination
• 6 RadSVM = Support Vector Machine with Radial Kernel
• 7 RadSVM-BFE = RadSVM classifier with Backward Feature Elimination
• 8 RF = Random Forest
• 9 RF-BFE = RF with Backward Feature Elimination
• 10 XGB = XGBoost
• 11 XGB-BFE = XGB with Backward Feature Elimination
• 12 NB = Naive Bayes
• 13 NB-BFE = NB with Backward Feature Elimination
• 14 EnsembleGLM = Stacked Ensemble Learning with GLM meta learner
• 15 EnsembleGLM-BFE = EnsembleGLM with Backward Feature Elimination

Graph

Table 3 Detailed performance metrics for full and selected feature ML models relative to TIMI score in NSTEMI/UA based on a 30% validation dataset.

This study conducted a secondary analysis to evaluate the predictive performance of the best-performing ML models that were trained on imputed datasets. The analysis showed that the XGBoost model achieved an AUC of 0.89 (95% CI: 0.859–0.918) for predicting in-hospital outcomes and 0.86 (95% CI: 0.83–0.897) for predicting outcomes within 30 days. On the other hand, the stacked EL model achieved an AUC of 0.798 (95% CI: 0.774–0.822) for predicting outcomes within 1 year. The models trained on imputed data consistently performed well, with AUC values of 0.88 (95% CI: 0.851–0.915) for in-hospital mortality, 0.87 (95% CI: 0.843–0.905) for 30-day mortality, and 0.81 (95% CI: 0.782–0.83) for 1-year mortality, when compared to the results from the complete cases dataset. The models' robustness is emphasized by the statistical significance of their performance in predicting short-term and long-term outcomes, both in imputed and complete case datasets (p<0.001 for all comparisons). The results confirm the reliability of utilizing imputed data when complete data is not accessible, thereby enabling the wider application of predictive modeling in clinical environments.

As detailed in the methods section, BFE was combined with SVM algorithms to create models with optimized predictive accuracy. Table 4 displays a comparison of the features identified through ML feature selection and those utilized in the TIMI risk score for NSTEMI/UA. ML consistently identified patient age, Killip class, heart rate, and the use of Low-Molecular-Weight Heparin (LMWH) as significant factors for predicting outcomes within the hospital, at 30 days, and at 1 year. Age was the only predictor that was common to both the ML-selected features and the TIMI risk score across all evaluated timeframes.

Graph

Table 4 Feature selection comparison across timeframes: ML algorithms versus TIMI risk score for NSTEMI/UA.

The SHAP summary plots (Figs 2–4) demonstrate the distribution of SHAP values for the significant features identified through SVM-based feature selection. These features are used to predict in-hospital, 30-day, and 1-year outcomes for NSTEMI/UA patients.

Graph: Fig 2 info:doi/10.1371/journal.pone.0298036.g002

Graph: Fig 3 info:doi/10.1371/journal.pone.0298036.g003

Graph: Fig 4 info:doi/10.1371/journal.pone.0298036.g004

The vertical axis of each plot displays the features in a descending order of their influence on the model's predictions. The x-axis corresponds to the SHAP value, which measures the influence of each feature on the model's output. The color gradient indicates the original value of the feature, with red representing higher values and blue representing lower values.

In the in-hospital prediction model (Fig 2), factors such as Killip class, age, and fasting blood glucose have a significant impact on the model's output. Higher values of these features tend to increase the SHAP value, which indicates a more profound risk of negative outcomes. In the 30-day prediction model (Fig 3), systolic blood pressure and heart rate are identified as significant factors, with higher values indicating an increased risk. Furthermore, the 1-year prognostic model (Fig 4) indicates that chronic ailments like renal disease and prior myocardial infarction play a substantial role in influencing the predictions. The presence or elevated levels of these conditions are closely linked to heightened risk.

The plots clearly demonstrate a consistent association between higher Killip classes and elevated heart rates with higher SHAP values across all models. This suggests that these features are strong predictors of mortality at various stages after NSTEMI/UA. On the other hand, the use of medications such as Low-Molecular-Weight Heparin (LMWH) is shown to have lower SHAP values, suggesting a possible preventative effect.

The mortality rate distributions predicted by the TIMI risk score, and the best performing ML models developed for this study are illustrated in Figs 5–7. These figures are crucial for illustrating the models' reliability in forecasting the mortality risk of NSTEMI/UA patients over various time periods. The ML risk score, determined through SVM feature selection, classifies patients into two groups: low-risk (with a mortality probability below 50%) and high-risk (with a mortality probability of 50% or higher). This classification aligns with the established standards, where a TIMI score of 5 or less indicates low risk, and a TIMI score greater than 5 indicates high risk [[63]].

Graph: Fig 5 info:doi/10.1371/journal.pone.0298036.g005

Graph: Fig 6 info:doi/10.1371/journal.pone.0298036.g006

Graph: Fig 7 info:doi/10.1371/journal.pone.0298036.g007

Table 5 displays the mortality percentages categorized by the TIMI risk score and the best performing ML models that include SVM feature selection. The data classifies patients into two categories based on their risk of mortality: low risk being equivalent to a TIMI score ≤5 and ML probabilities < 0.5, and high risk corresponding to a TIMI score > 5 and ML probabilities ≥ 0.5.

Graph

Table 5 Comparative mortality risk stratification by ML models and TIMI score in NSTEMI/UA patients.

The ML models, specifically the XGB and stacked EL models, which employ SVM-selected features, exhibit a more precise classification of patients into risk groups. When compared to the conventional TIMI scoring system, they demonstrate significantly lower rates of low-risk in-hospital mortality predictions and higher rates of high-risk predictions across all timeframes. This demonstrates an improved predictive capability, as the ML models are able to more accurately identify patients who are at a higher risk of mortality. For instance, the XGB model demonstrates a mortality prediction of 1.27% for low-risk in-hospital cases, which is notably lower than the TIMI score's prediction of 3.78%. Additionally, the XGB model predicts a high-risk rate of 19.12%, while the TIMI score predicts a rate of 6%. Similarly, the stacked EL model provides comparable discrimination, with a 1.61% rate for low-risk and an 18.25% rate for high-risk in-hospital cases. The consistency of these findings is evident in both the 30-day and 1-year forecasts, further confirming the reliability of the ML models in predicting outcomes for patients with NSTEMI/UA over different time frames.

The Net Reclassification Index (NRI) Analysis quantifies the improvement in risk stratification established by ML compared to the conventional TIMI risk scores for predicting in-hospital, 30-day, and 1-year mortality in patients with NTEMI/UA. Tables 6–8 demonstrate a breakdown of the NRI for these respective timeframes, demonstrating a significant enhancement in the accuracy of risk classification. The ML models illustrate an evident improvement in reclassification: 51% for in-hospital, 56.6% for 30-day, and 42% for 1-year outcomes, highlighting their superior predictive abilities. The improvements in classification accuracy are statistically significant, as indicated by a p-value of less than 0.00001. This emphasizes the reliability of the ML approach in accurately identifying patients at different levels of risk, surpassing the TIMI score for NSTEMI/UA.

Graph

Table 6 In-hospital mortality reclassification: NRI of ML models versus TIMI score.

Graph

Table 7 30-days mortality reclassification: NRI of ML models versus TIMI score.

Graph

Table 8 1-year mortality reclassification: NRI of ML models versus TIMI score.

This groundbreaking study represents the initial application of employing ML with stacked EL models to enhance the prediction performance of in-hospital, 30-days and 1-year mortality risk in Asian NSTEMI/UA patients, demonstrating the superiority of over the conventional TIMI scoring method. The ML techniques have been improved using a feature set chosen by SVM and further optimized through backward elimination. The outcomes of this study are: i) Using a selected feature set, the XGBoost model achieved an AUC of 0.883 for in-hospital mortality predictions and 0.874 for 30-day outcomes, while the stacked EL model achieved an AUC of 0.806 for one-year mortality, indicating that models developed with a selected feature set outperform those developed with a full feature set ii) Individual ML models and stacked EL models both outperformed the TIMI scoring system for NSTEMI/UA, with the ML approach resulting in a significant increase in classification accuracy.

ML algorithms have been shown to improve prognostic models for ACS patients using specific datasets from various regions, including the United States [[64]–[67]], Sweden [[32]], Sweden [[35]], Poland [[68]], Israel [[38]], China [[20]], Korea [[9]], and Singapore [[69]]. Despite these advances, ML research on NSTEMI/UA patients in the Asian demographic is limited. In this context, Lee et al. (2021) [[24]] discovered that ML outperformed the TIMI risk score in predicting outcomes for Korean NSTEMI/UA patients, demonstrating higher AUC values (In-hospital: 0.89 vs 0.67, 3 months: 0.85 vs 67, 1-year: 0.86 vs 0.68), which is consistent with the findings of this study.

The TIMI risk score for NSTEMI/UA was developed to estimate the 14-day mortality risk and has remained unchanged for over two decades [[5], [24], [62]]. Although it demonstrated an AUC of 0.80 for 30-day mortality prediction in Western populations [[70]], its applicability in Asian cohorts has been less convincing. Kumar et al. (2021) found a TIMI score AUC of 0.78 in Pakistan [[71]], while a study in China found AUCs of 0.52, 0.50, and 0.52 for 6-month, 1-year, and 2-year mortality risks [[72]]. According to Indonesian research, the 14-day mortality prediction AUC for NSTEMI was 0.61 and 0.63 for UA [[73]], while Malaysian data revealed an in-hospital mortality prediction AUC of only 0.55 [[51]]. The disparities in performance can be attributed to the TIMI score's origins in predominantly Caucasian populations, with a lack of representation of Asians, who frequently present with ACS at younger ages and have higher rates of diabetes, hypertension, renal failure, and delays in seeking medical care [[74]–[76]]. Furthermore, the TIMI score has been criticized for underestimating mortality in high-risk groups [[19], [77]], which is consistent with the current study's findings.

ML models have been improved in the NSTEMI/UA patient group with a selected group of features that proved to be most effective for improving performance—13 features for in-hospital, 11 for 30-day, and 12 for 1-year predictions. These models, which were optimized using Backward Feature Elimination (BFE), outperformed other ML models that included the entire set of features, as well as the TIMI risk score. The importance of feature selection algorithms in mortality prediction utilizing ML is emphasized, as combining these algorithms with classification methods has been shown to produce better results than using classifiers alone [[35], [48], [78]]. Feature selection not only improves ML model performance by reducing the number of predictors to a manageable number, but it also reduces the predictor space's complexity [[79]]. Streamlining predictors improved model performance in this study. This is due to the remaining features being clinically significant for mortality prediction, whereas the excluded features were determined to be non-contributory to the predictive accuracy of the models [[80]].

Treatment strategies were integrated into the best performing ML models for mortality prediction, yielding significant results. In particular, invasive procedures such as cardiac catheterization has been associated with improved 30-day survival rates in NSTEMI/UA patients [[81]]. Depending on the clinical stage of the condition, treatment pathways may differ, with catheter-based interventions frequently being the preferred course of action. Traditional ACS management has focused on antiplatelet and anticoagulant therapies, anti-anginal medications, aggressive lipid-lowering treatments, and risk factor modification. The introduction of thrombolytic therapy transformed the ACS therapeutic landscape. The current study's findings, as validated by SHAP analysis, are consistent with these well-established clinical practices and outcomes.

In this study, non-invasive treatment predictors, specifically pharmacotherapy encompassing antihypertensives (ACE inhibitors, beta-blockers, diuretics, ARBs), antidiabetics (oral hypoglycemic agents, insulin), and antiplatelet medications, were used to predict in-hospital, 30-day, and 1-year mortality. This is consistent with the findings of the SHAP analysis. These medications are typically given as an acute treatment to improve neurohumoral control in response to negative left ventricular remodeling. The requirement for such treatments may indicate a more compromised cardiac state, which explains their strong association with an increased risk of death [[83]].

The Malaysian NCVD registry frequently reports a high incidence of CVD risk factors such as hypertension, diabetes, dyslipidemia, smoking, and obesity [[84]]. Over 95% of people hospitalized for ACS have at least one established CVD risk factor. Concurrently, patients with chronic kidney disease have a threefold increased risk of myocardial infarction, as well as higher morbidity and mortality rates [[85]]. Uncontrolled CVD risk factors cause negative systemic changes that can lead to a variety of cardiovascular diseases such as heart failure, cerebrovascular renal failure, and peripheral arterial disease [[86]].

This study shows that stacked EL has the potential to predict one-year mortality in Asian NSTEMI/UA patients. Even minor improvements in predictive model performance can have a significant impact in healthcare [[87]]. However, it is important to note that the stacked EL method does not always outperform individual ML models; its effectiveness is heavily dependent on the diversity of the base learners and models used [[88]]. The current study supports this by demonstrating that the performance gains of the stacked EL model were not significantly higher than those of the single ML models that served as its base, which is consistent with previous research [[13], [58]].

Data imputation was used to ensure the accuracy of our analysis. Cases with more than 30% missing information were initially excluded to avoid potential biases caused by imputation. Due to the large size of the remaining dataset, this exclusion criterion was not deemed restrictive. The dataset's completeness for all follow-up intervals enabled the development of risk calculators for both ML models and the TIMI risk score. Using complete cases to identify features that predict short- and long-term mortality, on the other hand, is likely to yield more reliable results. Following that, analysis using both incomplete and imputed data produced similar results, confirming the findings' validity.

In line with the findings by Hayıroğlu and Altay (2023) [[89]], our research demonstrates AI's rising impact on healthcare, particularly in cardiology. Artificial intelligence algorithms have demonstrated effectiveness in predicting outcomes such as atrial fibrillation recurrence and measuring cardiovascular risk using complete data analysis that includes medical history and genetic information. These capabilities complement the findings our study, in which ML methods, specifically the stacked EL methodology, enhanced the accuracy of mortality prediction for Asian NSTEMI/UA patients. This is especially important given the complexity of cardiovascular conditions and the need for precise risk classification, which potentially dictate better clinical decision-making and eventually lead to better patient care outcomes.

There are several limitations to consider in this investigation. It primarily compares the performance of ML models across in-hospital, 30-day, and 1-year time periods to the TIMI risk score for NSTEMI/UA, which was designed to predict 14-day mortality. Furthermore, because the study focused on a multi-ethnic Malaysian cohort, the findings may not be directly applicable to countries with predominantly Caucasian populations. Another limitation is the lack of comparisons with other prognostic models such as HEART and GRACE, a decision influenced by the dataset's structure and the large amount of missing data in certain variables. Furthermore, the possibility of selection bias in patient registries should not be overlooked. As a result, future empirical studies in real-world settings will be required to validate these findings.

The risk scores generated by this study have the potential to be integrated into hospital electronic health record systems for clinician use. Instead of relying solely on administrative databases, future research could aim to implement and validate these risk scores within clinical registries. Given the demographic diversity of the NCVD registry, which includes Malay, Chinese, and Indian people, the study's findings are particularly relevant to the Asian context. This is especially important for Malaysia, Brunei, and Singapore, as well as other populous countries like China and India [[90]].

Our study demonstrates that the combination of ML and stacked EL, along with feature selection techniques, can improve the accuracy of mortality prediction in Asian NSTEMI/UA patients. This approach complements and enhances the conventional risk scoring methods, such as the TIMI risk score. With this predictive tool, healthcare professionals can identify patients at high risk right away in the emergency department, allowing for targeted and effective management based on their risk profiles. Future research will focus on integrating this algorithm into online platforms to improve accessibility and practical application in clinical settings.

S1 Table

The rates of missing data for each feature used in this study.

(PDF)

S2 Table

The parameters setting values for optimum ML model performance.

(PDF)

We would like to acknowledge the National Heart Association Malaysia for sharing us with the data for this study.