Prediction of hearing preservation after acoustic neuroma surgery based on SMOTE-XGBoost

1.   Introduction

Acoustic neuroma, which is also known as vestibular schwannoma, is a benign tumor that arises from the vestibular nerve sheath ^[1]. It is most commonly found in the internal acoustic canal and the cerebellopontine angle, and can result in hearing loss, tinnitus, dizziness, unsteady walking, face numbness and other symptoms ^[2]. Surgical excision is the most common treatment option as this effectively removes the tumor, but problems such as hearing loss or disabilities can result, which can significantly impact the daily lives of patients ^[3,4]. As a result, the removal of the entire tumor while maintaining hearing is the optimum therapeutic goal for acoustic neuromas. Based on their clinical and pathological appearance, how can the postoperative hearing status of individuals with acoustic neuroma be accurately predicted? This question has clinical importance in terms of relevant clinical trials and for guiding the individualized treatment and management of patients.

For postoperative hearing preservation prediction, if only the experience of the doctor is relied upon, it may be time-consuming and laborious, potentially leading to accidental errors. Machine learning is a fast and effective method for the analysis of patient characteristics, obtaining discriminatory results in a timely manner and assessing the validity of the results. It is widely used in the medical field and is widely regarded as being superior to traditional statistical methods.

For example, in diabetes clinical epidemiology, supervised and unsupervised learning are both widely used, and a study demonstrated that machine learning methods can help improve risk stratification and outcome prediction in diabetes epidemiology applications ^[5]. In the field of cardiology, scholars summarized 40 research papers relating to machine learning since 2018, comparing seven machine learning models and arguing that machine learning algorithms can improve the healthcare industry by making improvements to the entire disease prediction and treatment recommendation process in the vast majority of cases ^[6]. A review summarized 14 studies on machine learning for the prediction colorectal cancer (CRC) risk stratification from 2011 to 2022, arguing that as an adjunct to disease prediction, it can help patients and clinicians identify patients who are at high risk for CRC and diagnose and treat them more quickly ^[7]. In the field of heart disease, scholars have used a variety of machine learning methods for effectively predicting the preincidence of heart disease ^[8,9,10,11]. In addition, machine learning has achieved desirable results in other fields, including biomedicine ^{[12,13,14,15,16,17,18]}.

However, to the best of our knowledge, the study of the prognostic factors that affect postoperative hearing preservation among patients with auditory neuroma remains in its infancy. Most previous research is based on traditional statistical methods such as ANOVA, and this does not provide an available predictive model of postoperative hearing preservation for patients ^{[19,20,21,22,23]}. In the scope of this research, it was recognized that Cha et al. took the integration of machine learning methods with hearing preservation studies a step further in 2020, proposing a regression model to facilitate the use of analysis by physicians to provide a referenceable prediction of postoperative hearing preservation for patients prior to surgery ^[24]. Four models were used to fit and regress the data and obtain the most accurate model through experimental comparison: support vector machine (SVM), gradient growth machine (GBM), deep neural network (DNN), and diffusion random forest (DRF). This work inspired us to expand on the idea that hospital data often has the problem of class imbalance, and obtaining the best prediction results by direct model fitting may be difficult. Therefore, perhaps a data imbalance treatment method could be added. Furthermore, while their best model was DNN, which runs slowly and produces unstable results, other models could be considered for fitting attempts.

XGBoost is an ensemble learning classification method that is currently one of the most popular methods used in machine learning areas. Its main idea is improving the stability and accuracy of the model by combining multiple weak classifiers to form a strong classifier ^[25]. It is widely used and has been found to outperform other models in various medical fields. For example, Wimalarathna et al. proved XGBoost is the best model for the classification of auditory brainstem responses recorded from children with auditory processing disorder ^[26]. Kivrak et al. found XGBoost to be the most suitable model for the prediction of death among COVID-19 patients ^[27]. Shorthouse et al. concluded that XGBoost is effective for separating cancer from noncancerous cells ^[28]. Chen et al. proved that XGBoost provides accurate predictions of disease risk progression stratification and long-term prognosis among kidney patients ^[29]. XGBoost has also demonstrated excellent prediction capabilities in many sectors outside of healthcare ^{[30,31,32,33]}.

As was previously mentioned, a detailed study of data from patients with acoustic neuroma is presented in this paper, and three machine learning methods are used: DNN, classification and regression trees (CART), and XGBoost in the study of hearing preservation among patients with acoustic neuroma. A SMOTE-XGBoost model is proposed that incorporates an oversampling imbalance processing method based on SMOTE. The SMOTE-XGBoost model is compared to other machine learning models that use the same data preprocessing, and to the same model both before and after data imbalance processing in this work. The outcome demonstrates the viability of the data processing approach used based on the properties of the data, in addition to the higher performance of the model, which demonstrates its superiority.

The article is organized in the following manner: Section 2 analyzes the methods used in this article are analyzed in Section 2; Section 3 describes the experimental data and required materials in detail; Section 4 shows the data processing flow and provides a discussion of the experimental results; and Section 5 outlines the conclusions that have been reached.

2.   Methods

In this section, the two most important methods in the model this paper proposes are described in detail: SMOTE and XGBoost. The second subsection outlines the theory foreshadowing for the third subsection. It is taken as a comparative model in numerical experiments.

2.1.   SMOTE

There is an unbalanced sample proportion, which is commonly referred to as data imbalance. With classification problems, assuming the quantity of samples of various classes to be roughly equal is common. However, in the field of acoustic neuroma that is described here, the number of patients with better hearing preservation following surgery is far greater than the number of patients with poorer hearing preservation, and this paper is more concerned with the small percentage of patients who are likely to have poorer hearing recovery.

There are several approaches for dealing with data imbalance, including oversampling and undersampling, which are the two most prevalent processing methods ^[34]. As there is such a small sample of actual medical data, utilizing the oversampling method as a means of retaining as much sample information as possible is preferable.

The initial oversampling approach simply copied the few samples, but this would result in data overfitting and would not increase the accuracy of judgment. This difficulty can be solved by SMOTE ^[35].

This is an improved scheme that is based on the random oversampling algorithm. The basic premise is to analyze the minority samples and artificially synthesize new samples based on them to add to the data set, which is represented by the algorithm flow and Figure 2.

The SMOTE algorithm flow is summarized as follows.

$(1)$ For each minority sample $x_i$, the Euclidean distance from $x_i$ to all other minority samples is calculated to obtain its $K$ nearest neighbors;

$(2)$ The oversampling ratio is specified according to the sample category imbalance ratio, $N$ samples are randomly selected from the $K$ nearest neighbor samples, and it is assumed that the selected sample is $x_j$;

$(3)$ For each randomly selected sample $x_j$, a new sample $x_{new}$ with the original sample $x_i$ is generated according to the following formula

where $gap$ is a random number in the interval (0, 1).

2.2.   Decision tree based on the CART algorithm

The CART algorithm is an extensive decision tree learning method that was initially proposed by Breiman ^[36]. It is used for both classification and regression. The following introduces the classification function that is used in this paper.

A classification decision tree is generated through the process of selecting the optimal feature based on the criterion of minimizing the Gini index and the optimal split point in the optimal feature and recursively generating a binary tree. In the classification problem, if it is assumed that there are $K$ classes, the probability of the sample point belongings to the $k$-th class is $p_{k}$, then the Gini index of the probability distribution is

The specific CART algorithm calculation process can be seen in Figure 3. This can be summarized as follows.

$(1)$ For the current node data set $D$, recursion cannot be performed when one of the following three conditions are met: the number of samples in $D$ is less than the specified threshold; there is no feature in $D$; the Gini index is less than the specified threshold;

$(2)$ The Gini index of each value of each existing feature of the current node is calculated, and the feature with the smallest Gini index is then selected as the division point. Taking $A = a$ as an example, it divides the node data set into two sub-nodes $D_1$ and $D_2$ in Figure 3;

$(3)$ The above steps are performed recursively on the generated sub-nodes until the termination condition is met.

2.3.   XGBoost (extreme gradient boosting method)

XGBoost is the integration of trees. The basic classifier is generally selected as the CART tree, which is also the case in this paper. Each tree is added as a means of improving the classification effect. The calculation formula is

where $\hat{y_i}$ is the predicted output result of the $i$-th sample, and $f_t$ is the $t$-th decision tree. For such prediction output, its objective function is

where $\sum_{i = 1}^{n}l(y_i, \hat{y_i})$ is the loss function. The commonly used loss function is the sum of squares of the residuals $L = \sum_{i = 1}^{n}(y_i-\hat{y_i})^2$, which represents the deviation of the predicted value from the true value. $\Omega({f_t(x_i)})$ is a penalty term, which is, specifically expressed as $\Omega(f_t) = \gamma T+\frac{1}{2}\lambda \sum_{j = 1}^{T}\omega^2_j$, where $T$ represents the number of leaves in the tree, $\omega_j$ represents the score on $j$-th leaf in Figure 4, $\gamma$ and $\lambda$ are both custom coefficients. The equation clearly shows that the more trees there are, the larger the term is, so this is used for preventing overfitting due to too many tree splits.

The objective function is expanded with Taylor's formula binomial and the following is obtained,

where $g_{i} = \partial_{\hat{y}^{t-1}} l\left(y_{i}, \hat{y}_{i}^{t-1}\right), \quad h_{i} = \partial_{\hat{y}^{t-1}}^{2} l\left(y_{i}, \hat{y}^{t-1}\right)$. The expression can be simplified thusly

The objective function is then

In the above formula, each $\omega_j$ is independent of each other. For the quadratic equation $G_j\omega_j+\frac{1}{2}(H_j+\lambda)\omega^{ 2}_j$, it is relatively easy to find $\omega_j$ under the minimum value of the objective function when the structure of this additional tree is known

Substituting Eq (2.7) for Eq (2.6), the objective function can be further simplified as

which is the corresponding optimal objective function value. It is assumed that $I_L$ and $I_R$ are the instance sets of left and right nodes following the split. $I = I_L \cup I_R$ as the tree on the left in Figure 4, where $Obj_{1}$ represents the objective function value of $I$, and $Obj_{2}$ represents the objective function value of $I_L$ and $I_R$, and the loss reduction following the split can be obtained by

From the above criteria, in order to sort the values obtained by dividing each feature at different values, the various divisions must be traversed to identify the best one and generate a decision tree.

3.   Materials

A brief description of the data set used in this paper is provided in this section and the criteria for comparing model quality in subsequent experiments are identified.

3.1.   Data description

The data used in this article was gathered by physicians following surgery from the Department of Neurosurgery, Xiangya School of Medicine, Central South University. There is a high level of authenticity and trustworthiness to the data set that includes 79 individuals who were chosen with the help of doctors for 13 characteristics that are potentially related to hearing following acoustic neuroma surgery, including quantitative and categorical parameters. Hearing preservation following surgery is the dependent variable. Negative class 0 refers to the hearing of a patient not changing or improving following the procedure in comparison to preoperative hearing; positive class 1 refers to the hearing of a patient deteriorating following the operation in comparison to preoperative hearing. The distinguishing characteristics are shown in Table 1.

3.2.   Evaluation criteria

Due to the imbalance in the initial data set and the small sample number, model performance evaluation in this paper employs evaluation metrics that are linked to the ROC curve rather than $Accuracy$ ^[37]. The confusion matrix, which is shown in Figure 5, can be used for depicting the classification performance of the two classification problems. The majority class is represented by the negative class in this paper, while the minority class is represented by the positive class. The real class label of the sample appears in the first column of the confusion matrix table, while the anticipated class label is in the first row. The accurately predicted positive label is represented by TP and the number of samples with negative labels is represented by TN, while the misclassified positive labels and negative labels are represented by FN and FP.

Based on the confusion matrix, the indicators for the evaluation of the discriminant model of unbalanced data are

$Accuracy$ represents overall accuracy, and TP $rate$ is the percentage of correct classification of positive samples, which refers to the proportion of patients whose hearing is not successfully preserved following surgery and is correctly predicted. $Sensitivity$ represents the percentage of correct classification of negative samples, which refers to the percentage of patients whose hearing is preserved following surgery and is correctly predicted. FP $rate$ is the proportion of patients whose hearing is not preserved following incorrect prediction. $Precision$ is the percentage of samples predicted to be positive, which is the percentage of patients who are predicted to be hearing unreserved. $F_1$ is the harmonic average of $Precision$ and $Sensitivity$, which simply and clearly reflects the balance between the two.

The ROC curve is a common indicator for the evaluation of imbalanced data prediction models and is a two-dimensional graph with FP $rate$ as the abscissa and TP $rate$ as the ordinate, where (0, 1) is the ideal point. This curve demonstrates the relationship between FP $rate$ and TP $rate$. AUC is the area under the ROC curve, $AUC = 1$ means a perfect classifier, and when AUC is less than 0.5, this means the model is not as effective as random guessing.

At the same time, the PR curve reflects the trade-off between the accuracy of the classifier in recognizing positive examples and its ability to cover positive examples, taking $Precision$ as the vertical coordinate and $Recall$ as the horizontal coordinate. The higher the accuracy of the model, the higher the recall will be, meaning the performance of the model is better. Average precision (AP) is the average accuracy at different recall points and is represented on the PR graph as the area under the PR curve. The larger the AP value, the higher the average accuracy of the model will be.

4.   Results

The imbalance treatment and model fitting process according to the experimental sequence is detailed in this section and the superiority of the model is demonstrated through a comparison between each index.

4.1.   Data imbalance processing and classification

According to the 79 case samples in the data set used in this paper, 68 patients retained or improved their pre-operative hearing level following acoustic neuroma cutting surgery, while 11 patients had poor hearing after surgery. The distribution of the two types of hearing following surgery is obviously quite uneven, which causes the classification results to be biased toward multi-class samples and classification accuracy to be lower. It is apparent that patients are more concerned about hearing deterioration following surgery and doctors are also interested in those who have lost hearing following surgery.

As a result, this paper performed imbalance processing on the data before classifying and regressing to change the sample classification from 68:11 to 68:68, thereby making training a better classifier easier and enabling a more accurate classification for a small amount of data to be obtained.

4.2.   Empirical analysis

Following the imbalance treatment, five-fold cross-validation was used to more reasonably compare the effects of the models, and predictive entropy was used for estimating the uncertainty of the individual models. DNN and CART were utilized as comparisons for XGBoost in the studies, one of them being the model used in the hearing preservation study by Cha et al. and the other the basis model for XGBoost. In addition to cross-sectional comparisons, these three models provide comparisons with findings before and after imbalance treatment. Table 2 shows that all models were significantly improved in all metrics following SMOTE treatment and the floating range of each metric and $entropy (Accuracy, F_1)$ demonstrate more stable performance with the SMOTE-treated model. At the same time, paired t-test was conducted for the $F_1$ of each model both before and after SMOTE treatment. The results presented in Table 3 were obtained, which shows that $p < 0.1$, thereby indicating a significant difference before and after SMOTE treatment. As a result, class imbalanced data requires imbalanced processing.

XGBoost also demonstrated significant advantages over other models both before and after SMOTE processing. It achieved the best results for $Accuracy$, $Sensitivity$, $Precision$ and $F_1$, while also achieving the minimum predicted entropy to prove the stability of the model. SMOTE- XGBoost outperformed even 0.9 in all evaluation metrics, but also had the lowest predictive entropy of less than 0.1, meaning that it can be considered the best model for the prediction of postoperative hearing preservation among patients with auditory neuroma.

Due to the imbalance of the original dataset, a 7:3 training set test set cut of the data set was made, and ROC curves and PR curves were used on the test set for further exploring model performance. The joint curves of the six models were then obtained, as can be seen in Figure 6.

Multiple dashed lines in Figure 6 show that the models without data imbalance treatment and with AUC and AP values around 0.5 and below are close to the random prediction, meaning the prediction results have no reference value. Conversely, the solid line indicates all models with data imbalance treatment performed well. The SMOTE-XGBoost, which is represented by the blue curve, was found to be the best model, achieving an AUC value of 0.96 and an AP value of 0.97.

Based on the selected model, the importance of each feature in the model was determined, as can be seen in Figure 7. In the figure, F-score represents the weighted summed mean value of the information gain from feature splitting. A higher F-score means the feature has greater importance. Therefore, it is recognized that tumor size, age, pre-class, symptoms and tumor status are the five most important characteristics that affect postoperative hearing preservation.

5.   Discussion

All patients underwent microscopic intracranial occupancy resection via the posterior suboccipital sigmoid sinus approach from the same operator (same treatment group), and subsequent analysis excluded the effect of intraoperative variables. For patients with poor predicted postoperative hearing preservation, a more observational, conservative treatment plan based on individualized care must be adopted, so the surgeon is able to assess the risks and benefits in order to make further choices based on the actual situation. For those with better predicted postoperative hearing preservation, their preoperative fears can be largely eliminated and good surgery ensured.

Based on the previous discussion, the main results of this work can be summarized as follows.

$(1)$ XGBoost was introduced into the field of auditory neuroma and found to be superior to DNN. The proposed model in this study performed well in all metrics, also outperforming the neural network in terms of model fitting process time.

$(2)$ The imbalance treatment was applied to patient data in the field of auditory neuroma based on the inherent distribution characteristics of auditory neuroma. The need for its introduction was demonstrated as this would allow for a greater focus on the small number of patients whose hearing is not preserved postoperatively while avoiding serious treatment misjudgments by physicians.

$(3)$ The preoperative indicators with the greatest impact on postoperative hearing preservation in patients were analyzed using the selected rational prediction model. The importance of age, preoperative hearing AAO-HNS classification, tumor size, and tumor status has been repeatedly mentioned in previous relevant medical studies ^[38,39,40], while the importance of symptoms, a characteristic summarized to represent the number of preoperative symptoms in patients, has rarely been mentioned in previous studies, but this can be further confirmed in future medical observations.

Only 79 samples were used in this study, so the potential of the proposed model was not fully demonstrated, potentially leading to some overfitting of the prediction results. This was due to the fact that there are few patients with auditory neuroma and relatively few hospitals where relevant surgery is conducted, which makes the collection of various types of data quite difficult. In future studies, the information collection system should be improved to enable more patient data to be obtained and the model to be updated.

6.   Conclusions

A practical predictive model for hearing preservation in patients with acoustic neuroma following tumor cutting surgery was proposed by this paper. The model was found to have obvious credibility in many important indicators, thereby validating the feasibility of the introduction of machine learning into the field of acoustic neuromas.

The model also incorporated a data imbalance processing method based on data distribution characteristics, compensating for the lack of direct model regression in previous research that did not consider data distribution characteristics. This served to improve the accuracy and specificity of the proposed model in a variety of ways. For example, it was able to reliably forecast a larger sample of patient data sets from different categories without the need to account for the flaws of unbalanced data distribution.

In practical terms, the results of this paper demonstrate the application of ensemble learning algorithms for the prediction of the preservation of patient hearing following surgery, thereby saving considerable time and money for clinicians diagnosing the postoperative hearing of patients. As a result, it is believed that this model could be applied clinically. As more data on patients is gathered, it is anticipated that the model proposed by this paper could be adjusted appropriately as a means of producing better outcomes.

Acknowledgments

The author is grateful to Miss Yue Li from The Xiangya Hospital of Central South University for her kind support with data collection for this project. This work is partly supported by the Fundamental Research Funds for the Central Universities of Central South University (No. 1053320213996).

Conflict of interest

The author declares there to be no conflict of interest.