This study used KNHIS data from January 1, 2005 to December 31, 2018. Since 1995, KNHIS, the single national health insurer, has provided health examinations for all Koreans. The KNHIS database contains complete health information about approximately 50 million Koreans [25]. In this study, case subjects were defined as patients with AF who were newly diagnosed with ischemic stroke, and control subjects were those with AF who had not been diagnosed with ischemic stroke. We used the International Classification of Disease, 10th revision (ICD-10) codes to identify patients with AF and those who had experienced ischemic stroke from the health claim records [26]. We obtained patients diagnosed with AF (ICD-10: I48) between 2005 and 2013. Subsequently, we checked if the selected patients were hospitalized for ischemic stroke (ICD-10: I63) within five years after the diagnosis of AF. Next, we collected demographic, health examination, and medical history information of subjects from the KNHIS database. Demographic information contains gender, age, occupational status, and income level. The medical history includes information on the occurrence of 43 diseases (e.g., hypertensive disease, hemolytic anemia, chronic gastritis, hyperlipidemia, and thyroid diseases). The medical history information in the KNHIS database is built using the medical bills that were claimed by medical service providers for the expenses. Health examination includes results of nine general laboratory tests (e.g., blood pressure, and urinary protein) and six questionnaires on lifestyle and behavior (e.g., smoking, exercise, and drinking). A detailed description of the extracted information was provided in Supplementary Table 1.

The study protocol was approved by the Institutional Review Board of the National Health Insurance Service in Korea (NHIS-2020-4-109). The authors confirm that all methods were performed in accordance with relevant guidelines and regulations. The need for informed consent from participants was waived by the ethics committee of the Chonnam National University because this study involved routinely collected medical data that were anonymized at all stages to protect an individual’s privacy.

Logistic regression is a statistical technique that estimates the causal relationship between categorical dependent variables and several independent variables and is divided into two types according to the number of categories of dependent variables [27]. A binary logistic regression is used when the dependent variable has two categories of 0 or 1, and polynomial logistic regression is used when the dependent variable is composed of two or more categories. The binary logistic regression, used as a statistical technique in this study, was expressed by defining logistic functions in reverse using logits as shown below in Eqns. 1,2, to express linear relationships between independent and dependent variables [28].

(1)$\mathrm{Minimize}p(x)=\sigma (t)={\displaystyle \frac{1}{1+e^{-\left({\beta }_0+{\beta }_{1}x\right)}}}$
$\mathrm{Subjectto}(\because t={\beta }_0+{\beta }_1{x}_1+\mathrm{\cdots }+{\beta }_k{x}_k)$

(2)$\mathrm{Minimize}\mathrm{g}(\mathrm{p}(\mathrm{x}))={\sigma }^{-1}(\mathrm{p}(\mathrm{x}))=\mathrm{logit}(\mathrm{p}(\mathrm{x}))=$ $\mathrm{Subjectto}\ln \left({\displaystyle \frac{\mathrm{P}(\mathrm{x})}{1-\mathrm{P}(\mathrm{x})}}\right)={\beta }_0+{\beta }_1{\mathrm{x}}_1+\mathrm{\cdots }+{\beta }_{\mathrm{k}}{\mathrm{x}}_{\mathrm{k}}$

Regression coefficient, standard error, Wald chi-square, and p-value were used for binary logistic regression analysis with maximum likelihood estimation. The regression coefficient implies that the dependent variable increases or decreases in proportion to the estimated value when the independent variable increases by one unit [29]. If the coefficient is a positive value, it has a positive correlation and vice versa. As the coefficient was close to zero, the effect of the independent variable decreased [29, 30]. The standard error is the standard deviation of the sample means used to determine whether the regression coefficient occurs by accident, revealing the closeness of the sample mean values to the population mean [31]. The smaller the standard error, the closer it is to the population mean, and it spreads closer to the regression line, which implies that the probability of a regression coefficient being accidental is less likely to occur. This shows that the causal relationship between the independent variable and dependent variable is significant. The Wald chi-square is an index for evaluating the importance of each independent variable [27].

(3)$$W={\left(\frac{\beta }{SE(\beta )}\right)}^2$$

where $\beta$ is the coefficient, and SE is its standard error. The Wald chi-square refers to the ratio of the square of the regression coefficient to its standard error and is expressed as a chi-square distribution [27]. The higher the value, the lower the significance level, indicating that it is an important variable in explaining the dependent variable. The p-value is the probability that a value equal to or more than that in the sample is observed, assuming that the null hypothesis is correct [32]. Moreover, a p-value less than a certain significance level implies that the observed result is improbable under the null hypothesis and that there is a significant association between the dependent and the corresponding independent variables. However, a p-value greater than a certain significance level indicates that there is no significant association between the dependent and the corresponding independent variables. Therefore, the p-value for each feature tests the null hypothesis that the feature does not correlate with the occurrence of ischemic stroke. In this study, we set the significance of the p-value as 0.001. Multicollinearity was detected by the tolerance and variance inflation factor (VIF). The tolerance is defined as 1-R${}^{\mathrm{2}}$, where R${}^{\mathrm{2}}$ is the coefficient of determination for the regression of a variable on the other independent variables. The VIF is defined as the reciprocal of tolerance. If the VIF value exceeds 10, it is considered to indicate multicollinearity.

In this study, we used a deep neural network to predict the occurrence of ischemic stroke in AF patients based on KNHIS data (Fig. 1). The deep neural network is composed of multiple hidden layers between an input layer and an output layer. The multiple hidden layers enable the modeling of complex nonlinear relationships through the learning function of a high-level layer formed by combining the features of the lower layer, and learning complex functions mapping the input to the output from data [33]. Among the 75 features extracted from KNHIS, we used 4 demographic information, 31 medical histories, and 13 health examination features, which were considered statistically significant through regression analysis. The dataset was divided into 6:2:2 as training, validation, and test set, respectively. Then, the self-attention mechanism was applied to the deep learning model. The self-attention mechanism improves the prediction performance by estimating the importance of the feature [34]. Input features were fed to the fully-connected and the softmax layers to calculate the self-attention scores.

Fig. 1.

A systematic overview of the deep neural network-based model that predicts the occurrence of ischemic stroke in AF patients. Demographic, medical history, and health examination information was used as input, and occurrence of ischemic stroke in AF patients was used as output. We calculated attention scores for the input features and concatenated the attention scores with input features. The occurrence of ischemic stroke was predicted by a three-layer fully-connected neural network with non-linear activation function.

(4)$$a=\mathrm{softmax}(\mathrm{g}(X))$$

where X is the selected input features, and g($\bullet$) is the fully-connected layer without activation. g($\bullet$) can be represented as below.

(5)$$g(x)=Wx+b$$

where W = [w${}_{\mathrm{1}}$, w${}_{\mathrm{2}}$,…, w${}_n$] is the weight matrix, and b is the bias of each unit. In this study, the output of linear operator g($\bullet$) is the same size as the input; therefore, W$\mathrm{\in }{ℝ}^{\mathrm{48}\mathrm{\times }\mathrm{48}}$ and b$\mathrm{\in }{ℝ}^{\mathrm{48}}$. g($\bullet$) is then fed into the softmax function which return a vector of numbers with equal to one.

(6)$$\mathrm{softmax}(z)=\frac{e^{z_i}}{{\sum }_i{e}^{z_i}}$$

Then, the component-wise multiplication between input features and self-attention score vector was performed.

(7)$$o=a\odot X$$

where $\odot$ is the component-wise multiplication operator. Then, we concatenated the output vector o and input features x as [o${}_i$ ; x${}_i$]. The concatenated vector was used to train the three-layer fully-connected neural network for predicting the occurrence of ischemic stroke of AF patients. The hyperparameters, additional techniques, and library information used to build a fully-connected network are described in Supplementary Section 1 and Supplementary Table 2.

From June 2005 to March 2013, a total of 754,949 patients were diagnosed with AF, of which 62,226 (8.24%) were diagnosed with ischemic stroke five years after diagnosis of AF. Table 1 shows the frequency and proportion of stroke and non-stroke groups in each variable. Exceptionally, insurance fee is continuous data, we reported mean $\pm$ SD (range). The CHA${}_2$DS${}_2$-VASc score ranged from 0 to 9, indicating that the risk increases as the score increases. The mean CHA${}_2$DS${}_2$-VASc score of the non-stroke group was 2.15 points, and the stroke group was 3.01 points. As expected, it was confirmed that the stroke group had higher CHA${}_2$DS${}_2$-VASc scores than the non-stroke group. Next, we checked the individual risk factors of CHA${}_2$DS${}_2$-VASc scores, including five medical history factors, age, and sex. It was confirmed that the stroke group had a high proportion compared with the non-stroke group for the medical history of five diseases considered in the CHA${}_2$DS${}_2$-VASc score. The mean ($\pm$ SD) age of the patients was 64.6 $\pm$ 13.3 years in the non-stroke group and 71.5 $\pm$ 9.5 years in the stroke group. We also observed that the stroke group had a higher proportion of patients aged from 65 to 74 years and above 75 years than the non-stroke group. Regarding sex, the non-stroke group had a higher proportion of males (59.21%), whereas the stroke group had a higher proportion of females (50.34%). These results indicate that the CHA${}_2$DS${}_2$-VASc score reflects the characteristics of stroke because it gave a high score for the five medical history features, elderly, and women in the stroke group. However, the difference between the two groups was not significant, and the proportion of patients who scored five or higher in the stroke group did not reach 20%. To estimate the strength of the association between independent variables and occurrence of ischemic stroke, relative risk were provided. When the relative risk value is 1, it means that the independent variable does not affect the result. When the Relative risk value is higher than 1, it means that the risk of the occurrence of ischemic stroke is increased by the independent variable. Conversely, if the relative risk value is less than 1, the risk is decreased by the independent variable.

We used coefficient values and p-values of logistic regression to identify the features related to ischemic stroke occurrence. The VIF values of independent variables are reported in Supplementary Table 3. The result indicated that VIF values of all variables did not exceed 3. The results showed that age, sex, and occupational status were important factors in demographic information. We identified important features from medical history, including thyroid diseases, other cardiac arrhythmias, chronic lower respiratory diseases, hemolytic anemia, cancer, hemorrhoids, diabetes mellitus, hypertensive diseases, chronic kidney diseases, heart failure, hyperlipidemia, peripheral vascular disease, gout, noninflammatory gynecological problems, pulmonary embolism, and chronic gastritis. Significant features found by the p-values through tests were analyzed based on coefficient, Wald chi-square, and odds ratio with 95% confidence interval (CI) (Supplementary Section 2, Supplementary Table 3 and Supplementary Fig. 1).

Our method predicts the occurrence of ischemic stroke risk in AF patients based on KNHIS data. We evaluated the area under the curve scores of the receiver operating characteristic (AUROC) and corresponding SD for the average of 10-fold cross validation to assess the predictive performance. We tested the performance for five different types of input feature sets: (i) using all features without feature selection (FS); (ii) using all features with FS; (iii) using demographic features with FS only; (iv) using medical history features with FS only; and (v) using health examination features with FS only (Fig. 2a). From the results, we found that using all features with FS (AUROC = 0.722 $\pm$ 0.004) exhibited better performance than using all features without FS (AUROC = 0.714 $\pm$ 0.003) and using a single feature set only (AUROC = 0.613~0.679). These results indicate that the proposed model considers the complex associations of large-scale feature sets. Next, we compared our method with other machine learning methods, including logistic regression, XGBoost, and random forest (Fig. 2b). Both XGBoost and random forest optimized hyperparameters with Bayesian optimizers. In both methods, the maximum depth of the tree was tuned in the range of 5 to 10, and the number of classifiers was tuned in the range of 10 to 500. To prevent overfitting, XGBoost tuned minimum child weight (1–10) and learning rate (0.01–0.1), and random forest tuned minimum sample split (2–10) and minimum sample leaf (0.01–0.5). Results indicate that the proposed deep learning model has better AUROC than logistic regression (AUROC = 0.691 $\pm$ 0.005), XGBoost (AUROC = 0.708 $\pm$ 0.004), and random forest (AUROC = 0.694 $\pm$ 0.009). Furthermore, we compared the prediction performance of our method with the CHA${}_2$DS${}_2$-VASc scores (Fig. 2c). Since CHA${}_2$DS${}_2$-VASc scores are predictors of cardioembolic sources, we screened the patients corresponding to the cerebral infraction due to embolism of cerebral arteries (ICD-10: I63.4) in ischemic stroke patients. Previous study indicated that patients with ICD-10 code of I63.4 includes about 73% subtype diagnosis of cerebral embolism [35]. Through this process, we finally selected 954 ischemic stroke patients and conducted the experiment. The results indicated that an AUROC value of the proposed method was 0.727 $\pm$ 0.003 and an AUROC value of CHA${}_2$DS${}_2$-VASc scores was 0.651 $\pm$ 0.007.

Fig. 2.

AUROC value of models generated from different datasets and methods. (a) We compared the AUROC performance of the proposed method for four different input datasets, including all features, demographic feature only, medical history features only, and health examination features only. (b) We compared the performance of our method with other machine learning methods, including logistic regression, XGBoost, and random forest. (c) We compared the performance of our method with the CHA2DS2-VASc scores.

In the prediction of the occurrence of ischemic stroke in AF patients, we firstly calculated precisions for different positive/negative ratios to evaluate the precision performance in the various skewness of datasets (Table 2) [36]. To do this, a negative set was generated by random sampling of the AF patients without ischemic stroke at various rates. The negative sets were generated ten times at each rate, and the performance for each case was evaluated by averaging the results. Results indicated that the precision performance of the proposed deep learning model was decreased when skewness was increased. In realistic scenario, the precision score of the proposed method was 0.132 $\pm$ 0.011, logistic regression was 0.095 $\pm$ 0.013, XGBoost was 0.121 $\pm$ 0.012, and random forest was 0.109 $\pm$ 0.013. Next, we checked the recall performance. Out of 12,445 ischemic stroke patients of the test dataset, the proposed model covered 9508 patients (r = 0.764 $\pm$ 0.005). Furthermore, the proposed method performed better compared to other machine learning methods (r = 0.678~0.717).

The model output can be interpreted as an approximate probability of ischemic stroke occurrence and has a value between 0 and 1. In general, the decision threshold that predicts ischemic stroke occurrence based on the model output value is often 0.5. However, the default threshold may not represent an optimal interpretation of the predicted probabilities [37]. In this study, the class distribution of the dataset is skewed, and predicted probabilities are not calibrated. This is a classification problem with imbalanced classes [38]. To solve this, we identified the optimal threshold value of the model output to judge the occurrence of ischemic stroke. We calculated the F1-scores, which is the harmonic mean of precision and recall, by changing the threshold of the model output. The best performance (F1-score = 0.223) was when the threshold value was 0.519.