Classification of dementia from spoken speech using feature selection and the bag of acoustic words model

1.   Introduction

Memory disorders impair memory, information processing, reasoning, and other cognitive functions. Cognitive decline is associated with difficulties in finding and understanding words and interruptions in thoughts ^[1]. Dementia is an advanced form of memory disorder where the deterioration of memory and decline in cognitive abilities impair the patient's ability to perform routine daily activities and hinder social and professional functioning. Consequently, as self-care becomes more difficult, the need for help increases starting from even a mild form of the disorder, and this significantly affects the patient's close ones as well. The most common cause of progressive memory disorders and dementia is Alzheimer's disease ^[2].

According to the World Health Organization (WHO), every three seconds, someone in the world is diagnosed with a memory disorder ^[2]. Memory disorders affect over 50 million people worldwide, and the number is expected to triple by the year 2050. The aging of the population is the main driver of the increased prevalence of memory disorders. There is a notable increase in countries such as Japan and many European nations where life expectancy has risen, and birth rates have declined. However, memory disorders impact nearly every country worldwide. In the year 2018, the global costs of memory disorders reached nearly a billion dollars ^[2].

Currently, there is no curative drug treatment for memory disorders, but early diagnosis, treatment, and rehabilitation can maintain and improve a patient's functional capacity. Studies have shown that individuals with healthier lifestyles among memory disorder patients experience fewer risk factors for worsening memory disorders and can lead a good and fully functional life ^[3]. The FINGER study was the first in the world to demonstrate that adhering to a comprehensive lifestyle program can improve cognitive functions and prevent the decline of memory functions ^[4]. Diagnosing memory disorders, especially at an early phase, has been challenging because diagnostics and treatment evaluation require specialized expertise and monitoring. In recent years, there has been increased investment and focus on developing diagnostics and rehabilitation.

Traditionally, cognitive decline has been assessed using conventional pen-and-paper tests, including the Mini-Mental State Examination (MMSE) ^[5], the Montreal Cognitive Assessment (MoCA) ^[6], and the Consortium to Establish a Registry for Alzheimer's Disease (CERAD) test, particularly in the early stages of decline ^[7]. Alongside the paper tests, cognitive impairment related to memory disorders and potential brain injuries has been studied in clinical settings by analyzing patients' speech production conducted by experts. However, this approach has not demonstrated practicality as it is subjective and often challenging to replicate. Consequently, there has been a shift toward developing automatic natural language processing (NLP) machine learning techniques in recent years. These methods have demonstrated utility by being cost-effective, scalable, and providing highly rapid diagnoses ^[8]. However, the methods are often too complex, demanding significant training data, computationally intensive, or involve too many adjustable training parameters ^[9,10].

In machine learning, there has often been a focus on diagnosing Alzheimer's disease or mild cognitive impairment ^[11]. Speech has been analyzed based on lexical, acoustic, or both lexical and acoustic content. Analyzing the lexical content of speech remains challenging because the content needs to be extracted from the recording either manually, which is error-prone and time-consuming, or by using speech-to-text methods that typically require large speech databases and are not language-independent. Further, lexical features can more easily compromise patient privacy. For instance, a transcript may include information about a patient's inner circle or address ^[11].

The bag of words approach has been used for processing naturally spoken speech. This approach forms an unordered list from a set of objects, containing a limited number of objects that best describe the set and their occurrences. In the early stage, the model has been used in text categorization ^[12], such as spam email filtering. The word histogram model has been expanded to include visual words (bag of visual words, BoW), used in image categorization ^[13], and acoustic words (bag of acoustic words, BoAW) in speech processing ^[14].

Typically, in a BoaW model, a set of feature vectors is represented as a histogram of quantized vectors. Quantization is performed by clustering the data and associating the feature vectors with the nearest cluster prototypes. The method involves uncertainty caused by the randomness of the clustering result, which this study aims to address by using a robust K-spatial-median clustering method, an efficient K-means$++$-type clustering initialization, and repeated clustering. In this study, feature selection based on statistical testing aims to reduce the computational complexity of clustering and classification. In addition, the selection of features helps in understanding the most relevant information for the research. Word histogram representations are created for the voiced segments derived from patients' speech recordings, based on the 25 most important acoustic features extracted using the openSMILE library^*. The study shows that although the partitioning caused by clustering results in some variance, it is possible to distinguish dementia patients from control patients with an average classification accuracy of 75.2% (SD: $\pm$4.0%) on separate test data and with an accuracy of 75% using leave-one-subject-out (LOSO) cross-validation for the full dataset.

^*https://www.audeering.com/research/opensmile/

2.   Related work and reference results

Hernanz-Dominguez et al. ^[15] conducted a diagnosis based on acoustic features and achieved a classification accuracy of 62.0% using low-level mel-frequency cepstral coefficients features (MFCC 1-13). The features were extracted from 25 millisecond audio segments, and the following statistical measures were computed: averages, kurtoses, skewness, and variances. In ^[16], memory-impaired patients were diagnosed with an accuracy of 68.0% based on both the frequencies of speech and speech pauses, including averages, variances, minimum and maximum values, and entropies. In ^[17], classification accuracies ranging from 60.0% to 93.8% were achieved using speech duration, time- and frequency-domain features, and emotional states. However, the dataset was relatively small (40 individuals). Additionally, the control group included relatively young individuals (25% of subjects were 20–60 years old) compared to Alzheimer's patients (all over 60 years old).

In the development of machine learning methods, it is crucial to keep the training and test datasets separate to prevent classification results from being biased. In many studies based on machine learning and natural spoken speech diagnostics, classification methods are not designed to be independent of patients. Results have been published where multiple recordings from the same patients were used to train and test the machine learning method on data obtained from the same patient. In these cases, classification methods have not learned to identify the disease itself but have instead overfit the speech production characteristics of individual patients, rarely generalizing well to new data. Consequently, classification results on the datasets used in experiments may appear better than if applied to a different dataset, as underlined in ^[18].

In ^[19], classification was performed based on a subset extracted from the Pitt Corpus audio database^*, consisting of control subjects and dementia patients. The database included a total of 156 individuals, half of whom had a diagnosis of dementia. The dataset was divided into training and testing sets based on individuals' diagnoses, ages, and gender distributions. Noise reduction was applied to the audio recordings, and voiced segments were identified using a signal energy-based threshold for speech. Acoustic features were computed for these segments using the functional emobase, computational paralinguistics challenge (ComPaRe) ^[20], and Geneva minimalistic acoustic parameter set (eGeMAPS) features from the OpenSMILE library ^[21]. These included low-level descriptors (LLD) and statistical measures for LLD features ^[22]. Additionally, statistical measures were selected for multi-resolution cochleagram features (MRCG), as well as features related to pronunciation and speech pauses used in ^[16]. Correlated features were removed from the feature sets. Results for an individual were based on the modes of classification results for all speaker segments using different feature sets. The best classification accuracy based on acoustic features was achieved with the ComPaRe feature set. For the test dataset, an individual achieved a prediction accuracy of 62.5%, and with the LOSO cross-validation used in the experiments, the overall classification accuracy for the entire dataset was 56.5%.

^*https://dementia.talkbank.org/access/English/Pitt.html

Syed et al. ^[23], achieved a classification result of 76.9% using LOSO cross-validation on the training data presented in the ADReSS 2020 challenge^{† [19]}. The result was based on acoustic IS10-Paraling features (1582 features sampled from the ComPaRe feature set), a bag of acoustic words (BoAW) model ^[24] provided by the openXBOW tool^‡, and support vector machine (linear kernel) and logistic regression classifiers. In ^[18], a similar data division was utilized as in ^[19], but with a larger number of selected individuals (a total of 164 individuals). The work introduced a new active data representation method based on self-organizing map (SOM) clustering of feature vectors, associating vectors with their nearest clusters, computing durations of voiced segments, and histogram feature extraction. The proposed model methodologically resembles the BoAW model, where clustering is also used to associate feature vectors with their nearest clusters. The best classification result (77.4%) was achieved using non-correlated features from the eGeMAPS feature set (a total of 75 features) and a classifier based on linear discriminant analysis (LDA).

^†https://luzs.gitlab.io/adress/

^‡https://github.com/openXBOW/openXBOW

The results from studies ^[18,23] are promising. However, the papers do not address the challenges of data clustering, such as the SOM network and K-means clustering used in the BoAW model. In K-means, there are Stirling numbers of the second kind, given by $S(N, K) = \frac{1}{K!}\sum_{i = 0}^K (-1)^{K-i} \binom{K}i \left(i\right)^N$, to partition $N$ observations into $K$ groups which is evidently infeasible for computation ^[25]. Typically, clustering methods converge to a local minimum of the error function, which is why methods should undergo multiple initializations (often 100 iterations are used ^[26,27]), and then the initialization with the smallest clustering error is selected. Multiple initializations do not guarantee an optimal result, but it increases the probability of obtaining a cluster model that effectively describes the internal structure of the data.

3.   Data and methods

3.1.   Pitt Corpus audio database

The Pitt Corpus audio database is one of the databases provided by DementiaBank^*. The data for the database was collected between 1983 and 1988 as part of Alzheimer's research at the University of Pittsburgh ^[28]. The study included 282 individuals, out of which 101 were healthy control subjects, and 181 were Alzheimer's patients. Participants had to be over 44 years old, have at least seven years of education, no central nervous system abnormalities, and score at least ten out of thirty points on the Mini-Mental State Examination (MMSE) as a preliminary result ^[5].

^*https://dementia.talkbank.org/access/

The selected participants performed oral tasks, and their performance in everyday tasks was also assessed. Among the participants' oral tasks was a kitchen scene description task designed to measure speech disorders (a mother washing dishes, and children standing on stools stealing pastries). A healthcare professional provided instructions to the patients at the beginning of the image description task, and the patients' responses were recorded with a microphone. For the ADReSS 2020, a random sample of participants in the image description task from the Pitt Corpus database was selected, ensuring that the age and gender distributions of control subjects and dementia patients matched. Furthermore, only one recording was selected from each individual ^[19]. A total of 78 control subjects and 78 dementia patients were chosen. Approximately 69% of the individuals were selected for the training dataset, and approximately 31% were selected for the test dataset (see Tables 1 and 2). Recordings were acoustically enhanced by removing stationary noise, and audio volume normalization was applied across segmented speech to control for variations caused by recording conditions. It is important to note that in all databases age and gender distributions are not always available. In such cases, it would be good to be able to estimate the ages and genders of the participants (see, for example, the study ^[29]).

3.2.   Data preprocessing

The audio recordings, recorded in WAVE format with a microphone (duration Mean: 55.3 seconds, SD: 29.3 seconds), contained a significant amount of noise, which was removed using Adobe Audition's (VERSION 23.6) adaptive noise reduction filter. The adaptive noise reduction filter introduced short noisy segments at the beginning of the recordings, which were manually removed from the recordings. Additionally, extraneous noises not related to participants, such as caregiver speech, overlapping speech, background noise, buzzer sounds, typing sounds, and emergency vehicle sounds, were removed from the recordings. Toward the end, the recordings only contained either participants' speech or silent periods as participants thought about the task. The recordings were resampled^* from a 44 kHz sampling rate to a 16 kHz sampling rate which accommodates most of the speech frequencies. The resampled audio recordings were normalized to standard audio volumes according to the EBU R128 Standard ^[30] to minimize the effects of different recording conditions, such as the impact of microphone placement.

^*https://sourceforge.net/projects/sox/

3.3.   Speech activity detection

Segments of voiced speech were identified from the recordings using the open-source Auditok library^†. The energy threshold for speech detection (65 dB) and the maximum duration of speech segments (10 seconds) were the same as in ^[19]. Default values of the Auditok library were used as limits for the minimum duration of continuous speech (0.2 second) and the maximum duration of speech pauses (0.3 second). As a result, 2107 audio segments were obtained for the training dataset (duration Mean: 1.27 seconds, SD: 0.93 second), and 959 audio segments were obtained for the test dataset (duration Mean: 1.28 seconds, SD: 0.94 second). Finally, the audio segments were normalized to standard audio volumes according to the EBU R128 Standard.

^†https://github.com/amsehili/auditok

3.4.   Extraction of audio segment features

The eGeMAPS is a minimal set of features developed for automatic paralinguistic or clinical speech analysis ^[22]. These features are effective in identifying psychological changes in speech production ^[21]. They are statistical measures for low-level descriptors (LLD) of audio. The LLD features include, for example, F0 fundamental frequency (pitch), speech intensity (loudness), spectral flux rate, MFCC features, pitch vibration frequency and amplitude (jitter and shimmer), F1, F2, and F3 formant regions, energy ratio (difference between upper and lower spectrum frequencies, that is, alpha ratio), harmonic differences, harmonic-to-noise ratio (HNR), Hammarberg index, and predicted spectral slope. For the audio segments, the OpenSMILE library (VERSION 3.0.1) was used for forming the LLD and statistical eGeMAPS features (a total of 88 features). The eGeMAPS features are computed from LLDs using, for example, arithmetic mean, standard deviation, coefficient of variation (standard deviation divided by the arithmetic mean), percentiles (25, 50, and 80), and interquartile range 25 to 80. Additionally, the study measured the relative speech pauses before initiating speech production. The speech pause was normalized to the total recording duration and the cumulative speech pause in the recording (a total of two features).

3.5.   Feature normalization

Normalization is essential, especially in distance-based applications, because otherwise the features with large scales dominate the distance computations. Unlike the z-score normalization method, min-max normalization is not based on the normal distribution assumption. Min-max normalization to the range $[0, 1]$ corresponds to the following linear transformation:

where $x$ is the original variable and $x'$ is the normalized variable. The issue with feature normalization lies in the presence of outliers, which distort normalization parameters. In this study, a total of 159 outliers were replaced with non-outlying data, which is a very small number (less than 0.1% of all of the data), ensuring that the test data were scaled accordingly to the range of $[0, 1]$.

3.6.   Random forest model with statistical feature selection

Random forest (RF) is a non-linear classification and regression model ^[31]. Random forest builds a meta-estimator that aims to improve the prediction accuracy of a simple decision tree model and prevent overfitting by dividing the data into multiple decision tree estimators and utilizing the average for regression tasks or the mode for classification tasks. The leaf nodes of the random forest trees determine the values of the response variables given the inputs to the tree. In this study, the Gini index of the random forest classifier was used to describe the average impurity decrease, and this can be computed using the probabilities of two or more classes at each tree node. The more the impurity decreases, the more important the feature is in a classification task. The sum of the importance values of the dataset features, computed with the Gini index, is 100%.

The index is computed as follows:

where $p_{i}$ represents the probability that samples in node $n$ belong to class $i$ from $k$ classes. In a binary classification task, the formula reads as: $Gini_n = (p_1 + p_2) - p_1^2 - p_2^2 = p_1(1-p_1)+p_2(1-p_2)$.

Search for the best classification accuracy or the simplest model may include the search and selection of only the essential features ^[32]. In this study, the feature selection process is conducted using RF, as detailed below:

(1) The training dataset is normalized to the $[0, 1]$ scale.

(2) The random forest classification model is trained 100 times on the training dataset. For each iteration, feature importance values are computed based on the Gini index of the RF.

(3) The RF is trained 100 times on the training dataset using permuted response variables. For each iteration, permuted importance values are computed for the features.

(4) The distributions of 100 importance values for each feature and the importance values obtained through permutations are compared using the statistical Wilcoxon signed-rank test. The null hypothesis assumes that the values between the groups come from the same median distribution at a 5% statistical significance level ($p = 0.05$).

(5) The final features are selected as the $k$ highest-ranked features in terms of average importance values that rejected the null hypothesis of the Wilcoxon test.

3.7.   Bag of acoustic words model

The word histogram model represents an object as an unordered collection of words. Typically, the content of speech is not directly used in the model. Instead, compact feature vector representations are derived from speech segments, for instance, by analyzing short-term frequency spectra. Following this, a histogram distribution of feature vectors is constructed, which is then utilized in analyzing the speech. The used word histogram method is illustrated in Figure 1. Initially, in the word histogram model, preprocessing, feature extraction, and feature selection are applied to the identified training and test segments as depicted in Sections 3.2–3.6. Subsequently, a limited number of cluster centroids, or cluster prototypes, are formed from the feature vectors of the training data by clustering the data classes separately into a specified number of clusters. K-means, one of the most well-known clustering methods ^[33], is not robust to outliers because the cluster prototypes are represented by the means of the data sets. The K-spatial-median clustering used in the experiments, combined with an appropriate initialization method, is robust because the prototypes are based on the multidimensional medians of the sets ^[34,35]. The initialization of clustering performed with the K-means++ method randomly selects starting points and favors points that are far apart from each other ^[36]. The clustering result chosen is the one that best minimizes the clustering error based on Euclidean distance:

where $\mathbf{X} = \{{\mathbf{x}}_i\}_{i = 1}^N$ denotes the data and $\{{\mathbf{c}}_k\}_{k = 1}^K$ is the set of cluster prototypes that locally minimizes the error function (see Eq (1)). This minimization problem is non-smooth and it is realized using the sequential over-relaxation (SOR) algorithm with $w = 1.5$ as the step size factor ^[37,38].

Cluster centers form a collection of prototype vectors (codebook), and the feature vectors of segmented speech segments are quantized by associating the vectors with the nearest prototype vectors (words) in the codebook based on Euclidean distances. The entire content of the speech is described by a histogram of size $2\times Nc$, where $Nc$ represents the number of clusters for the dementia and control classes. Each position in the histogram represents the occurrence of one codebook word in the speech. After forming the histogram, it is normalized by the total number of words in the histogram. The normalized histograms are used as feature vectors in classification.

3.8.   Comparison of classification methods

Evaluation of the generalization capability of a dementia classifier is based on its accuracy on a test set. As depicted in Section 3.1, a balanced division into training and test data is provided a priori for the ADReSS 2020 dataset. However, more generally for a set of speech audio samples with labeling into AD and non-AD, our primary interest lies in whether a classifier can reliably identify the class of an unseen, new subject. This capability is measured using LOSO cross-validation. There, in each iteration for $N$ subjects, $N-1$ cases are used for training the model, and the remaining case is used for testing the model. The validation process is repeated $N$ times, ensuring that each subject is tested exactly once.

Six different methods were compared and used for the binary classification problem of dementia: five-nearest neighbors (5-NN) ^[39], linear discriminant analysis (LDA) ^[40], random forest (RF, 50 decision trees and 5 leaf nodes) ^[31], extreme minimal learning machine (EMLM, with the whole training set as reference points) ^[41], linear support vector machine (L-SVM, with overfitting parameter $C = 1.0$) ^[42], and support vector machine with chi-squared kernel (Chi2-SVM, overfitting parameter $C = 0.25$) ^[42]. Classifier parameters were determined through manual testing. Among the classifiers, 5-NN and L-SVM were based on MATLAB (VERSION R2022B, 64-BIT)^*, LDA and RF were used from the scikit-learn library in Python (VERSION 1.2.2)^†, Chi2-SVM was based on the libsvm library (VERSION 3.32)^‡, and EMLM was based on the source code from GitLab^§.

^*https://www.mathworks.com/help/pdf_doc/stats/index.html

^†https://scikit-learn.org/stable/user_guide.html#user-guide

^‡https://www.csie.ntu.edu.tw/cjlin/libsvm/

^§https://gitlab.jyu.fi/hnpai-public/extreme-minimal-learning-machine/

Because we consider a binary classification problem, the support vector-based technique that maximizes the margin between the two sets is, by construction, a potential approach. Especially, SVM with a chi-squared kernel has demonstrated to be effective in histogram classification ^[43]. This kernel is based on chi-squared distances, which are incorporated into the kernel function using an extended Gaussian kernel:

where $D(S_i, S_j)$ represents the chi-squared distance between word histograms $S_i$ and $S_j$, and $A$ denotes a scaling parameter, which is the average of all training histogram chi-squared distances.

The performance of the classification models was evaluated using the ADReSS 2020 test dataset (48 individuals) and by conducting LOSO cross-validation on the entire dataset (156 individuals). The same classifier parameters were utilized in both experiments. Note that only SVMs and RF classifiers included adjustable hyperparameters.

4.   Results

4.1.   Feature selection

In feature selection (depicted in Section 3.6), a total of 25 features were identified. Among the most important features were the functional features related to MFCC 1-4 coefficients, speech F0 fundamental frequency, spectral slopes predicted by linear regression (from frequency bands 0Hz–500Hz and 500Hz–1.5kHz), and spectral harmonics (two different ratios). Additionally, important features included relative speech pauses (pauseTotalDurationRatio and pauseTotalPausesRatio). The combined sum of the average importance scores for the selected features was 36.7%. Figure 2 illustrates the average importance scores of selected and non-selected features. All features and scores are listed in Appendix A.

4.2.   Classification of test data

The performance of classifiers was measured with the selected 25 features and different-sized codebooks. The cluster prototypes of control subjects and dementia patients were both increased in increments of five from five prototypes to fifty prototypes. Table 3 illustrates the average accuracies and standard deviations over 100 repeated clustering runs (each with 100 clustering reinitializations). From the table, it can be observed that the Chi2-SVM performed the classification task best, with an average classification accuracy of 75.2% (SD: $\pm$ 4.0%) on the test dataset. The RF model used for feature selection achieved a good classification accuracy of 71.9% (SD: $\pm$ 4.8%). The best results were obtained with a codebook of 30 prototypes. The variability in classification accuracies can be explained in part by the randomness associated with the classification models and partly by the randomness in the selection of clustering starting points. The variation in 5-NN classification results (2.9%–5.3%) is entirely explained by the randomness of the starting points since the classification model itself does not involve randomness. Table 3 indicates that the performance of the classifiers begins to decline as the codebook size increases beyond 30 prototypes.

4.3.   Performance in LOSO cross-validation

The generalization performance of the classification models was measured by LOSO cross-validation on the dataset of 156 individuals from the ADReSS 2020. The size of the codebook was increased from four cluster prototypes to forty prototypes. The uncertainty associated with cluster error was addressed by repeating LOSO cross-validations a total of 25 times, with each clustering iteration involving 100 reinitializations. The results were obtained by consistently using either the modes or the smallest clustering errors from the 25 binary classification outcomes. Figure 3 shows the classification results for the entire dataset using different classifiers, always choosing the better result between modes and the smallest clustering errors. Note that the results are given separately in Appendix B. Figure 3 illustrates that the best results for most classifiers were achieved with a codebook size of 22 prototypes. The Chi2-SVM classifier performed the classification task best, with a classification accuracy of 72.4%. Additionally, RF and LDA produced classification results of over 70%. Table 4 provides LOSO cross-validation accuracies and F1 scores with a codebook size of 22 prototypes and 75 clustering iterations (each with 100 replicates). The classification results are based on the smallest clustering error outcome. Clearly, the Chi2-SVM classifier emerged as the top performer in LOSO cross-validation with an accuracy of 75.0%. Among the classifiers, RF and EMLM achieved classification accuracies of 71.8% and 71.2%, respectively. Figure 4 shows receiver operating characteristic (ROC) curves for Alzheimer's dementia and non-Alzheimer's dementia groups. The probability values required for forming the ROC curves of the groups were computed from binary classification results over 75 repetitions. The final classes were based on the classification results obtained with the smallest clustering errors. The performance of individual classifiers is represented by the area under the curve (AUC) values, which are provided in the legends of Figure 4. Clearly, control subjects are easier to distinguish from Alzheimer's patients.

5.   Discussion

In this study, diagnostic analysis related to dementia was conducted on a dataset extracted from the Pitt Corpus audio database, consisting of both control subjects and dementia patients. Statistical features for acoustic speech analysis were extracted from audio segments using the OpenSMILE library, and histograms were constructed based on the BoAW approach. Similar analyses, based on the same dataset and a comparable histogram representation, have been conducted in previous studies, with reported LOSO cross-validation accuracies of 76.9% and 77.4% ^[18,23]. However, these studies do not provide details on how they accounted for randomness in the clustering process employed for histogram formation.

The study experiments with the ADReSS 2020 test dataset reveal that, through 100 iterations of clustering, the variability in classification outcomes using a deterministic 5-NN classifier typically falls within the range of 4.0% to 5.0%. Therefore, in the experiments, 25 repetitions of LOSO cross-validation were employed when searching for the number of clusters, and 75 repetitions were used in the final results computation. It is important to note that the LOSO cross-validation method itself involves repetitions of grouping equal to the number of subjects to be validated, leading to a multiplication of the final repetitions by the number of subjects (see the beginning of Section 3.8).

In ^[23], a large set of acoustic features (a total of 1582 features) has been used, leading to computational complexity for the methods. In this study, the computation of average feature importance scores was performed for relative speech pauses and eGeMAPS features (a total of 90 features) using a random forest classifier based on the Gini index. Feature selection was conducted based on the null hypothesis of the Wilcoxon statistical test. Feature selection relying on statistical testing is a straightforward and relatively efficient method for selecting final features ^[44].

In previous literature, feature selection has been performed for the eGeMAPS feature set in emotion recognition based on speech recordings ^[45]. The study utilized four different feature selection mechanisms. In the infinite latent feature selection (ILFS) method, feature importance values are based on all possible subsets of features, representing different paths in the feature graph. The ReliefF method uses weight vectors to represent the connections of features to actual class labels. The generalized Fisher score method (Fisher) seeks a set of features that maximizes the lower bound of a credibility function based on the Fisher metric. The active feature selection method (AFS) clusters individual features of the dataset and selects final features based on the distinctiveness of the clusters. In ^[45], three datasets related to emotion recognition were combined into one set, and the feature selection methods proposed 44–79 eGeMAPS features as the final quantities of features.

Although K-spatial-median clustering is robust, it is important to note that it is computationally more complex than the K-means algorithm, which can become a bottleneck with larger datasets. One may use random swap (RS) clustering, which is computationally more efficient than K-means ^[46]. In RS clustering, it is possible to estimate the expected number of iterations needed to find the correct clustering. The expected processing time is known to be linearly dependent on the number of data vectors, quadratically dependent on the number of clusters, inversely dependent on the neighborhood size, and logarithmically dependent on the number of successful swaps needed.

In this study, among the key features, statistical features of the MFCC, features related to the fundamental frequency (F0), spectral slopes predicted by linear regression, and harmonic features of spectra were particularly prominent. Especially, spectral slope and harmonic features are known to be associated with cognitive load, that is, the capacity of our working memory to handle information at any given moment ^[47,48]. Excessive stress and high cognitive load are known to be harmful to brain structure and function, which is known to increase the risk of cognitive decline and dementia ^[49].

The fundamental frequency of the speech is known to be associated with Parkinson's disease ^[50]. Therefore, the fundamental frequency of vowel sounds, along with frequency oscillation, has been investigated in the diagnosis of Parkinson's disease ^[51]. Schmitt et al. ^[14] used low-level MFCC features together with signal energy for emotion recognition ^[14]. However, instead of speech activity detection, features were extracted from unsegmented audio recordings in 25-millisecond durations using a 10-millisecond audio sampling. For low-level features, statistical features (means and standard deviations) were also computed. The final classification results were based on the creation of histograms using the BoAW model. The presented previous research results support the broader scalability of the features and methods used in this study to other speech-identifiable common diseases or different emotions, which could serve as indicators for illnesses or depression ^[52,53].

6.   Conclusions

In this research, individuals with dementia were identified from healthy controls based on naturally spoken speech. Features related to speech production are known to correlate with the speaker's cognitive ability and changes. The study did not aim to use lexical features derived from challenging speech recognition tasks, such as converting speech to text or analyzing the content of the transcribed text. Instead, spoken speech was categorized based on the acoustic characteristics of the speech. Classifiers and histogram feature extraction based on the BoAW model have demonstrated promise in distinguishing dementia patients from control subjects. The implementation of the BoAW model accounted for the variability in classification results caused by the multiple ways to partition the data by using repeated clusterings, each with multiple reinitializations, as part of the model. The final results were based on either the modes of the classification results or the results with the smallest clustering errors.

The generalization capabilities of the classifiers were tested using the ADReSS dataset, where subjects from the Pitt Corpus database were evenly selected based on age and gender distribution, ensuring that only one audio recording was selected from each individual to avoid overfitting by the classifiers. Furthermore, the classification results were measured with separate test data and by validating the entire dataset using LOSO cross-validation. In the experiments, the classifiers were trained for both tasks using the same classifier hyperparameters. The best accuracies with the test data and the entire dataset were consistent, indicating that overfitting did not occur. Feature selection in this work improved the computational efficiency of the methods. The results were based on only a small number of acoustic features, emphasizing spectral features (slope and harmonics), MFCCs, and statistical characteristics of the fundamental frequency. This minimal feature set provides researchers with opportunities to further develop feature extraction in a more reliable manner. Further, acoustic features are independent of the content of speech. Therefore, the developed diagnostic methods have the potential to be expanded to multiple languages. Overall, the obtained results are promising when fast, cost-effective, and scalable solutions are needed for the rapid diagnosis of cognitive decline or dementia.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

The work of the first Author (MN) was supported by the Finnish Cultural Foundation (Grant Number 30231766). The work of the second author (MvB) was supported by the Samfundet Folkhälsan, and the Research Council of Finland (Grant Number 349336).

Conflict of interest

The authors declare that there is no conflict of interest.

Appendix A.   Average importance scores for eGeMAPS features and relative speech pauses

Appendix B.   LOSO cross-validation results for the ADReSS 2020 audio dataset