Semi-supervised regression with adaptive graph learning for EEG-based emotion recognition

1.   Introduction

Emotion is a multifaceted psychological state, which includes not only an individual's psychological reaction to external environment or self-stimulation, but also the physiological reactions that accompany this response ^[1]. In the past period, researchers have studied emotion recognition mainly rely on data derived from an individual's physical behaviours such as sounds from microphones, gestures and expressions from cameras, and text from websites. However, the performance of recognition methods based on the information from these non-physiological cues may be affected by the users' intentionality. When users deliberately disguise their true feelings, the accuracy of emotion recognition methods can be significantly affected ^[2]. To overcome this limitation, researchers have focused on using EEG signals, which are resistant to camouflage and commonly utilized in the field of emotion recognition in recent years. EEG signals are the reflection of the central nervous system activities and have become a reliable data source for objective emotion recognition ^[3,4].

Recent advances in computer technology, biological science, electronic information, modern informatics, and other related fields have led to significant progresses in brain-computer interface (BCI) technology. In the meantime, affective brain-computer interface (aBCI) systems, which are applicable to human emotion state recognition, have received increasing attention in both academia and industry ^[5]. A typical aBCI system consists of the following several steps including signal acquisition, preprocessing, feature extraction or learning, and emotional state identification by recognition models ^[6]. In this paper, the study corresponds mainly to the latter two stages. In recent years, more and more machine learning models have been proposed for EEG emotion recognition ^[7]. For example, Li et al. explored robust EEG features in cross-subject sentiment recognition using support vector machines combined with the "leave-one-subject-out" strategy ^[8]. Dan et al. proposed a probabilistic cluster-promoting semi-supervised learning method which introduces a regularization term on fuzzy entropy to obtain a more generalized label affiliation function to reduce the effect of EEG noise ^[9]. Moreover, the experimental results also confirm its robustness.

However, there are two fundamental problems in the existing studies within the machine learning-based EEG emotion recognition research. On one hand, it is how to leverage the properties of EEG data to fit well into the emotion recognition process and improve its effectiveness. Specifically speaking, the trained model should consider the samples' local manifold structure, i.e., the high-similar samples can be grouped into the same cluster and the low-similar samples can be separated. On the other hand, in addition to improving recognition accuracy, we hope to gain some additional insights related to emotional effects from the model. In specific, the learned graph similarity matrix can well depict the information of emotional states of EEG data. However, in some existing graph-based models, the learned graphs may not have been further investigated. In this paper, we use semi-supervised regression combined with adaptive graph similarity learning strategy to deal with these two problems.

Least square regression (LSR) has been widely used in pattern classification due to its efficacy and simplicity ^[10]. However, LSR is easily influenced by specific labeled samples, which may result in the selected features not necessarily being optimal ^[11]. Therefore, to improve the performance of LSR-based learning models in EEG emotion recognition, we construct a similarity matrix to explore the manifold structure of EEG data. In graph-based semi-supervised learning models, a two-stage strategy is commonly used. That is, a similarity graph is firstly constructed based on a certain distance metric and then, label propagation or other learning tasks is performed on the constructed graph ^[12,13]. However, these models have some shortcomings that cannot be ignored. Firstly, it is not appropriate to construct a graph in the original space where there might be some noises. In addition, the two-stage strategy breaks the internal connection so that a flawed similarity matrix is learned, which may produce suboptimal results and accordingly reduce the model performance.

Therefore, how to effectively construct a high quality graph becomes a common challenge in many pattern classification tasks. Though the traditional KNN-based methods are simple to implement, they are not always effective because of the noise in the raw data. To alleviate the existing problems, Kang et al. proposed a scalable graph learning framework which used the relationship between samples and anchor points depicted by the bipartite graph to address the challenges of data noise ^[14]. However, the number of anchor points is a key factor in determining the quality of the learned graph. Generally, the best number of anchor points is selected by expert experience or trial-and-error, which is risk and inefficient. In order to reduce this factor's effect, it is necessary to construct graphs that are adaptive to the number of neighboring points.

Adaptive graph learning is a popular approach that combines graph construction and learning tasks into a unified framework. In recent years, many adaptive graph learning methods have been proposed. For example, a structured optimal graph based sparse feature extraction model was proposed in ^[15], which integrated sparse representation, local structure learning and label propagation into a unified learning framework. Besides, Lin et al. adopted an automatic learning scheme by taking advantage of the data properties and structure information to learn a reliable and precise graph ^[16]. The methods mentioned above can adaptively learn the similarity matrix, but they may not fully utilize label information in constructing the similarity matrix, which still has the room to be improved.

Inspired by ^[17,18], to enhance the reliability of manifold structures and adaptive graph learning, we utilize label information in both preserving manifold structures and adapting the graph learning process. In specific, we use semi-supervised regression to obtain the emotional states of the unlabeled EEG samples, and then the obtained label information is used to construct the sample similarity graph. These two processes iterate back and forth towards the optimum. In summary, we propose a semi-supervised regression with adaptive graph learning (SRAGL) model for EEG-based emotion recognition. In comparison to previous research, the following aspects are the primary contributions of this paper.

● A novel semi-supervised regression with adaptive graph learning (SRAGL) model is proposed for cross-session EEG emotion recognition. It achieves joint optimization of semi-supervised regression, structured graph learning and label propagation in a unified framework. In addition, the objective function of SRAGL is optimized using an efficient iterative algorithm.

● SRAGL efficiently captures the underlying data connections of both unlabeled and labeled EEG samples, and dynamically updates the graph similarity matrix through LSR and label propagation to make the learned graph more reliable. Intuitively, the graph block diagonals are gradually apparent, respectively corresponding to the emotional states.

● The performance of SRAGL is verified on the public SEED-IV data set, and the results demonstrate that it outperforms several other compared methods in terms of emotion recognition accuracy. In addition, according to the learned projection matrix in SRAGL, we provide an automatic and quantitative approach to identify key EEG bands and brain regions in cross-session emotion recognition.

Notations. The matrices and vectors in this paper are denoted by uppercase letters in bold and lowercase letters in bold. ${{\mathbf{1}}_{n}} = {{(1, 1, \ldots, 1)}^{T}}$ is a column vector whose elements are all ones and the subscript $n$ denotes its length. For matrix $\mathbf{A}$, ${{\mathbf{a}}^{i}}$ and ${{\mathbf{a}}_{j}}$ stand for the $i$-th row vector and the $j$-th column vector, respectively. The $\ell_{2, 1}$-norm of matrix $\mathbf{A}\in\mathbb{R}^{m\times n}$ is described as $\|\mathbf{A}\|_{2, 1} = \sum_{i = 1}^{m}\sqrt{\sum_{j = 1}^{n}A_{ij}^2} = \sum_{i = 1}^{m}\|\mathbf{a}^i\|_2$.

2.   The Proposed Method

2.1.   Preliminary

Suppose that we have $n$ EEG samples $\mathbf{X}\text{ = }\left[{{\mathbf{x}}_{1}}, {{\mathbf{x}}_{2}}, \ldots, {{\mathbf{x}}_{l}}, {{\mathbf{x}}_{l+1}}, {{\mathbf{x}}_{l+2}}, \ldots, {{\mathbf{x}}_{l+u}} \right]\text{ }\in \text{ }{{\mathbb{R}}^{d\times n}}$, the first $l$ samples are labeled and the remaining samples are unlabeled. $d$ is the dimensionality of the EEG feature. We define $\mathbf{Y} = [{{\mathbf{Y}}_{l}}; {{\mathbf{Y}}_{u}}]\in {{\mathbb{R}}^{n\times c}}$ as the label matrix for all samples, where $c$ represents the number of emotional states and ${\mathbf{Y}}_{l}$ stands for the known label matrix. In specific, if the sample ${{\mathbf{x}}_{i}}$ belongs to the $j$-th class and the label vector is ${{\mathbf{y}}^{i}}$, ${{Y}_{ij}} = 1$ and the rest elements are zeros.

Figure 1 illustrates the general framework of the proposed SRAGL model. It is composed of three main parts, which are the semi-supervised regression, graph construction, and semi-supervised label propagation, respectively. Based on SRAGL, our task are two folds. On one hand, we aim to accurately predict the emotional states of the target session's EEG data ${\mathbf{Y}}_{u}$ based on the inputs $\mathbf{X}$ and ${\mathbf{Y}}_{l}$. On the other hand, we need to determine the relative importance of all EEG features, and further the importance of all frequency bands and channels in a quantitative manner. This will allow us to identify the most critical features that contribute significantly to the accuracy of our predictions.

2.2.   Model Formulation

In the traditional classification task, the sample's feature can be projected into the label space and the optional transformation matrix is learned by minimizing the fitting error. Accordingly, the model can be established as

where $\mathbf{P}\in {{\mathbb{R}}^{d\times c}}$ is the transformation matrix. The first term is the loss function when projecting samples' features into label space and the second term represents the regularization term which controls the row sparsity of the transformation matrix $\mathbf{P}$.

According to the graph theory, we use EEG samples as vertices and the connection weights between samples as edges. We define the sample similarity matrix $\mathbf{S}\in {{\mathbb{R}}^{n\times n}}$ to describe the essential relationship between samples. Different from obtaining the similarity between two samples by calculating the Euclidean distance of their features, we expect to learn $\mathbf{S}$ from data adaptively. Therefore, we can obtain the similarity matrix $\mathbf{S}$ using the following formula

where the constraint $\mathbf{S}\ge \mathbf{0}$ represents all elements of the similarity matrix are non-negative and $\mathbf{S1}_n = \mathbf{1}_n$ denotes that the sum of degree of similarity between the sample ${{\mathbf{x}}_{i}}$ and the others is 1. From equation (2.2), it is realized that if the closer the Euclidean distance of two samples, the more similar two samples are, and the corresponding weight should be larger.

The local structure preservation method is introduced in order to thoroughly examine the local structure features of samples so that the proposed model can explore more discriminative information in EEG data. To be specific, if two samples in the original space are similar, they should maintain a similar connection when they are projected into the subspace. Therefore, we define $\mathbf{W}\in {{\mathbb{R}}^{d\times m}}$ as the projection matrix and $m$ is dimensionality of the subspace. Moreover, there exists a trivial solution for problem (2.2); that is, for a certain sample $\mathbf{x}_i$, the weight ${{S}_{ij}}$ would be 1 if $\mathbf{x}_j$ is its nearest neighbor. Above all, in order to avoid such non-ideal case and make data similarities more accurate, we introduce a term in equation (2.2) to shrink the elements in ${{\mathbf{s}}^{i}}$. Besides, we use ${{\mathbf{W}}^{T}}{{\mathbf{x}}_{i}}$ to replace ${{\mathbf{x}}_{i}}$ for better exploring the local structure information. Therefore, we have

where $\alpha$ is a regularization parameter to balance the impacts of the two terms. The constraint on the projection matrix $\mathbf{W}$ defines an orthogonal subspace.

In problem (2.3), it just evaluates data similarity in feature space and ignores label space consistency. In other words, if two EEG samples are similar, then their labels should also belong to the same emotional state as much as possible. Mathematically, the model is represented as

where ${{\mathbf{L}}_{S}}\in {{\mathbb{R}}^{n\times n}}$ is the Laplacian matrix and can be calculated by ${{\mathbf{L}}_{S}} = \mathbf{D}-\mathbf{S}$. $\mathbf{D}$ is a diagonal matrix and the value of the $i$-th element ${{D}_{ii}}$ is calculated by $\sum\nolimits_{j = 1}^{n}{{{S}_{ij}}}$. $\mathbf{y}^i$ and $\mathbf{y}^j$ respectively correspond to the label indicator vectors of samples $\mathbf{x}_i$ and $\mathbf{x}_j$. In equation (2.4), the two constraints require that for each row in $\mathbf{Y}_u$, all its elements are non-negative and their summation is 1. Equivalently, each element of ${{\mathbf{y}}^{i}}$ means the probability of the $i$-th unlabeled sample belongs to a certain emotion. For example, if the predict label of an EEG sample is [0.12, 0.75, 0.08, 0.05], then this sample has a high probability to the second emotional state.

Finally, by combining equations (2.1), (2.3), and (2.4) together, the objective function of SRAGL is obtained as

where $\lambda$, $\beta$, $\alpha$, $\gamma$ are the regularization parameters.

2.3.   Optimization

Below the optimization method to objective function (2.5) is introduced; that is, we propose to update the four variables $\mathbf{W}$, $\mathbf{S}$, $\mathbf{P}$ and ${\mathbf{Y}}_{u}$ by solving one variable and fixing others.

● $\mathbf{Update}\text{ }\mathbf{P}$. The objective function with respect to $\mathbf{P}$ is

Because the regulation term $||\mathbf{P}|{{|}_{2, 1}}$ is equal to $\text{Tr}({{\mathbf{P}}^{T}}\mathbf{QP})$, we can change equation (2.6) as

where $\mathbf{Q}$ is a $d\times d$ diagonal matrix and the value of its $i$-th diagonal element is

Here $\varepsilon$ is a small value which is used to prevent the denominator from being $0$ due to the row sparsity of $\mathbf{P}$. By making that function's partial derivative corresponding to problem (2.7) with respect to $\mathbf{P}$ to $0$, we can obtain

● $\mathbf{Update}\text{ }\mathbf{W}$. The objective function with respect to $\mathbf{W}$ is

It is equal to

$\mathbf{W}$ is solved by computing the eigenvectors that correspond to the $m$ smallest eigenvalues of $\mathbf{X}{{\mathbf{L}}_{S}}{{\mathbf{X}}^{T}}$.

● $\mathbf{Update}\text{ }\mathbf{S}$. The objective function with respect to $\mathbf{S}$ is

Because each $i$ of problem (2.12) is independent, we can solve $\mathbf{S}$ in row-wise manner. For the $i$-th row, i.e., $\mathbf{s}^i$, we have the following objective function

where ${{\mathbf{a}}^{i}}\in {{\mathbb{R}}^{1\times n}}$ is a row vector and the $j$-th value is ${{a}_{ij}} = a_{ij}^{x}+a_{ij}^{y}$. Among, we define $a_{ij}^{x} = ||{{\mathbf{W}}^{T}}{{\mathbf{x}}_{i}}-{{\mathbf{W}}^{T}}{{\mathbf{x}}_{j}}||_{2}^{2}$ and $a_{ij}^{y} = \frac{\gamma }{\beta }||{{\mathbf{y}}^{i}}-{{\mathbf{y}}^{j}}||_{2}^{2}$. The Lagrangian function of the problem (2.13) is

where $\mu$ and $\mathit{\boldsymbol{\theta}}_i$ are Lagrangian multipliers. On the basis of the KKT condition ^[19], we can obtain the solution of ${{\mathbf{s}}^{i}}$

where ${{(g(x))}_{+}} = \max (g(x), 0)$.

● $\mathbf{Update}\text{ }{\mathbf{Y}}_{u}$. The objective function with respect to ${\mathbf{Y}}_{u}$ is

By completing the squared form for each $i|_{l+1}^n$, the problem (2.16) is changed into

where $\mathbf{m}\triangleq \sum\nolimits_{j = l+1, j\ne i}^{n}{{{\mathbf{y}}^{j}}{{S}_{ij}}}\in {{\mathbb{R}}^{1\times c}}$. Obviously, for each $i$, the above objective function can be expressed as

Obviously, the above equation implies the following standard quadratic programming problem

where $\mathbf{v}\triangleq {{\mathbf{y}}_{i}}\in {{\mathbb{R}}^{c\times 1}}$, $\mathbf{A}\triangleq (1+\gamma)\mathbf{I}_c\in {{\mathbb{R}}^{c\times c}}$, $\mathbf{b}\triangleq 2({{\mathbf{P}}^{T}}{{\mathbf{x}}_{i}}+\gamma \mathbf{m})\in {{\mathbb{R}}^{c\times 1}}$. It can be efficiently solved by the method proposed in ^[20].

To sum up, the optimization procedure is presented in Algorithm 1. Below, we analyze the time complexity of our proposed SRAGL by using the big $\mathcal{O}$ notation. For each iteration, the time is mainly consumed in updating the graph adjacency matrix $\mathbf{S}$, the label matrix of the unlabeled samples ${{\mathbf{Y}}_{u}}$ and the projection matrix $\mathbf{W}$. To be specific, the complexity of updating ${{\mathbf{s}}^{i}}$ is $\mathcal{O}(dm)$ and therefore we need $\mathcal{O}(dmn)$ complexity to update $\mathbf{S}$. Then, we need $\mathcal{O}({{n}^{2}}d)$ to update $\mathbf{W}$ and $\mathcal{O}(uc)$ to obtain ${{\mathbf{Y}}_{u}}$. Suppose that the number of iterations is $t$, and the total computational complexity of the SRAGL model learning is $\mathcal{O}(t({{n}^{2}}d+uc+dmn))$.

 

3.   Experiments

3.1.   Data Set

SEED-IV ^[21] is an emotional EEG data set that was collected in response to 72 movie clips, each of which intended to induce one of four emotional states (i.e., happiness, sadness, fear, and neutrality). This data set was developed by Shanghai Jiao Tong University, and each subject participated in three sessions. In total, there were 15 subjects who contributed EEG data to this data set. During each session, the participants watched a total of 24 movie clips. Within each session, 6 of the 24 clips were intended to induce a particular emotional state. In Figure 2, it describes the procedure of each trial. EEG signals were collected using a 62-channel ESI NeuroScan System while the subjects were watching the movie clips. Then, the raw EEG data were downsampled to 200 Hz and processed by an 1-50 Hz bandpass filter. In their experiments, differential entropy features, extracted from five frequency bands, Delta (1-3 Hz), Theta (4-7 Hz), Alpha (8-13 Hz), Beta (14-30 Hz) and Gamma (31-50 Hz), were used for emotion recognition. Previous works have shown that differential entropy features perform well for emotion recognition task ^[22,23]. Therefore, the dimensionality of the extracted EEG signal features is 310 (i.e., 62 channels multiplied by 5 frequency bands). Due to the time durations of the movie clips in each session are slightly different, the numbers of EEG samples in the three sessions are accordingly different. To be specific, there are 851,832 and 822 EEG samples in these three sessions.

3.2.   Experimental Setup

In this section, we compare the performance of SRAGL with some related learning models on the SEED-IV data set, which are listed below.

$1)$ Feature Selection with Orthogonal Regression (FSOR) ^[24], which can be used for high-dimensional data feature selection in supervised learning tasks.

$2)$ Discriminative LSR (DLSR) ^[25] which introduces the $\epsilon$-dragging technique into LSR and obtains stronger discriminative power and generalization ability.

$3)$ Efficient Anchor Graph Regularization (EAGR) ^[26], which optimizes the performance of anchor graph by introducing anchor points, and leads to more accurate classification results.

$4)$ Rescaled Linear Square Regression (RLSR) ^[27], which proposes an adaptive weight assignment strategy to quantitatively learn the contributions of different features and explicitly illustrates the underlying rationale of applying $\ell_{2, 1}$-norm for feature selection.

$5)$ RLSR with Orthogonal constraint (ORLSR), which adds orthogonal constraints onto the projection matrix in RLSR in order to better maintain the structure information of data.

$6)$ Sparse Discriminative Semi-supervised Feature Selection (SDSSFS) ^[28], which can be considered as the extension of both RLSR and DLSR. That is, the $\epsilon$-dragging technique is extended into semi-supervised paradigm on the basis of the joint emotional state estimation of unlabeled EEG samples.

$7)$ Semi-supervised Feature Selection via Adaptive Structure Learning and Constrained Graph Learning (ASLCGLFS) ^[29], which is a semi-supervised feature selection method based on adaptive structure learning and constraint graph learning. Its central idea is to explore the discriminative information of labeled data and the structure information of unlabeled data to improve the learning performance.

FSOR has no parameters to be adjusted. For DLSR, RLSR, SDSSFS and ORLSR, the involved regularization parameter is adjusted in $\{2^{-20}, 2^{-19}, \cdots, 2^{20}\}$. For EAGR, the parameters $(\lambda, \gamma)$ are tuned from $\{10^{-3}, 10^{-2}, \cdots, 10^{3}\}$ and anchor number is adjusted in $\{10, 20, \ldots, 100\}$. For ASLCGLFS, the parameters $(\alpha, \lambda, \beta)$ are tuned from $\{10^{-3}, 10^{-2}, \ldots, 10^{3}\}$. In SRAGL, we have four parameters, i.e., $\lambda, \alpha, \beta, \gamma$, to be tuned from $\{10^{-3}, 10^{-2}, \cdots, 10^{3}\}$ and the subspace dimension $m$ is adjusted in $\{30, 50, 70,100\}$.

In our experiment, we perform the cross-session EEG emotion recognition. More specifically, for 'session2$\to$session3' classification task, the EEG samples from the third session are unlabeled while those from the second session are labeled. Then, we aim to estimate the emotional states of these unlabeled EEG samples as accurately as possible, within the transductive semi-supervised learning framework. For a certain sample, if it is from the unlabeled session, the elements in its label indicator vector (i.e., the corresponding row in $\mathbf{Y}_u$) are all initialized to $1/c$, meaning that we have no prior knowledge to its emotional state. When the criterion $\frac{\|\text{obj}^{(i+1)}-\text{obj}^{(i)}\|_2}{\|\text{obj}^{(i)}\|_2}\leq 1e-4$ is satisfied or the maximum number of iterations (i.e., 30) is reached, we terminate the iteration process. Here, $\text{obj}^{(i)}$ indicates the objective function value in the $i$-th iteration. It should be noted that the value of 30 for maximum iterations is typically much larger than the actual number of iterations required to achieve convergence with the algorithm in this experiment.

3.3.   Recognition Results and Analysis

In Tables 1, 2, and 3, the recognition results of all comparison models and our SRAGL model for cross-session EEG emotion classification tasks are shown, where the bolded result(s) in each row indicate the best accuracy among all the compared models. By comparing and analyzing the recognition accuracy of these eight models, the following points can be realized.

$1)$ The proposed SRAGL model achieves the highest recognition accuracy on most subjects in the cross-session emotion recognition tasks, and the average recognition accuracies of all the three cross-session tasks are higher than the remaining models, i.e., 78.18%, 80.55%, and 81.99%, respectively. Compared to the second best model ASLCGLFS, SRAGL improves by 3.47%, 6.63%, 3.85% in the three cross-session tasks. Overall, the proposed SRAGL model is verified to be effective and the adaptive graph learning-based similarity matrix facilitates the improvement of emotion recognition performance.

$2)$ Though EAGR, ASLCGLFS, and SRAGL are all graph-based models, the recognition results of EAGR are generally worse than those obtained by ASLCGLFS and SRAGL. To be specific, EAGR only obtained 69.18%, 69.47%, and 72.58% average recognition accuracies in three cross-session tasks. The reason accounting for it is that the similarity matrix constructed by EAGR is not powerful enough to depict the desirable data connections. EAGR selects a few representative samples, and then uses them to represent all sample points. Though the computational complexity of this anchor point based model is greatly reduced, recognition results and the quality of the constructed graph depend severely on the selected anchor points. In contrast, the graph learning in ASLCGLFS and SRAGL is by projecting the original data into the subspace to suppress the noisy and unwanted data components, and then constructing the graph similarity matrix based on the distance of the projected data.

$3)$ The remaining comparison models can be considered as variations of LSR. Among them, RLSR outperforms DLSR and FSOR due to its ability of adaptively learning the significance of different features through a feature self-weighting variable. By comparing the recognition results obtained by RLSR, ORSLR, and SDSSFS, the latter two models are superior to RLSR. ORLSR additionally includes the orthogonal constraint to the transformation matrix in RLSR, which on one hand can better remove the data redundancy information and on the other hand can retain the EEG data metric. SDSSFS employs the label dragging strategy to correct the labels of the originally misclassified samples, thus improving the recognition accuracy.

In Table 4, it shows the analysis of the variance (ANOVA) between SRAGL and each of the other models ^[30]. The ANOVA is a popular statistical method used to determine whether there are significant differences between two or more groups. In our experiment, we adopt a standard one-way ANOVA. Specifically, the performance accuracy is the dependent variable and the algorithm is the independent variable. The null hypothesis of ANOVA is that there is no difference between the group means. When performing ANOVA, it is usually necessary to set a significance level $\alpha$, which means the probability of incorrectly rejecting the null hypothesis. If $\alpha$ is set at 0.05 and p-value is less than 0.05, it means that there are differences between the models. If $\alpha$ is set to 0.01 and p-value is less than 0.01, the models are considered to be significantly different from each other. From Table 4, F-value and p-value are got by comparing the recognition results obtained by SRAGL (i.e., a8) and the other seven methods (i.e., a1-a7) respectively. These results demonstrate that SRAGL statistically outperforms the other seven models in terms of recognition accuracy.

In Figure 3, the confusion matrix is used to represent the accuracy of each model for four emotion states. From the figure, we can obtain more information from the recognition results, which includes 1) the eight models' average recognition accuracy in each emotion state. For example, the average recognition accuracy for the neutral emotional state using SRAGL was 84.58%, which was the highest recognition rate among the four emotional states. Conversely, the fear state had the lowest average recognition accuracy at 76.67%. 2) the accuracy improvement of SRAGL compared to the other models in each state. For example, SRAGL improved ASLCGLFS by 6.91%, 4.81%, 6.40%, and 1.01% for identifying the four states. 3) the proportion of each state that is misclassified into the other states. For example, SRAGL misclassified the EEG samples belonging to the fear state as sad, happy and neutral with 11.25%, 4.40% and 7.68%, respectively.

In SRAGL, we learn an optimal graph according to iteratively updating the graph adjacency matrix which describes the relationship between the subspace represented EEG samples. Based on the structured graph learning theory ^[31,32], the number of connected sub-components (i.e., diagonal blocks in visualized graph similarity matrix) in an ideal graph should be equal to the number of classes in the data set. In the SEED-IV data set, there are four emotional states (i.e., $c = 4$); therefore, the ideal graph adjacency matrix should have four diagonal blocks, respectively corresponding to the sad, fear, happy, and neutral states. In Figure 4, we visualize the learned graphs in terms of different iterations, i.e., when the numbers of iterations are 1, 2, 5 and 12, respectively. From figures, we can see that as SRAGL iterates, the learned graph for the SEED-IV data set has four diagonal blocks, each of which corresponds to one of the four emotional states. This result demonstrates that the adjacency matrix of the learned graph can represent the relationship between EEG samples effectively. Take Figure 4(a) as an example, we can clearly observe that the four diagonal blocks gradually become more visible as the number of iterations increases, which means the connection values within the block become larger while the connection values between the blocks become smaller. In other words, EEG samples belonging to the same emotional state should be connected to each other while EEG samples belonging to different states should be unconnected. Based on the learned graph adjacency matrix, the recognition accuracy of this case is 97.36%. Similarly, other cases also illustrate the above facts. However, since the EEG samples have inter-session variablities to some extent, it is not guaranteed that the learned graph adjacency matrix has the exact four diagonal blocks in all cases. In addition, this figure shows us that the objective function of SRAGL exhibits good convergence, with the objective function value decreasing quickly as the number of iterations increases.

Below, we show how the emotion recognition performance of SRAGL is influenced by the regularization parameters, i.e., $\lambda, \beta, \alpha, \gamma$, by some example cases. In SRAGL, the parameters $\alpha, \beta$ correspond to the graph adjacency matrix learning, $\gamma$ corresponds to graph label propagation and $\lambda$ controls the sparsity of the transformation matrix $\mathbf{P}$. It is found experimentally that the performance of SRAGL is insensitive to parameter $\beta$. In Figure 5, we fix $\beta$ as 1 and take the case of 'sub1: session1$\to$session3' as an example to investigate how the recognition results of SRAGL varies in response to different settings of $\alpha$, $\lambda$, and $\gamma$. From this figure, we observe that the parameter $\lambda$ is the most sensitive one to SRAGL. In model training, we can set $\lambda$ within $\{0.1, 1, 10\}$ to make SRAGL achieve good recognition results. In contrast, SRAGL is not very sensitive to the parameters $\alpha$ and $\gamma$ due to $\alpha$ mainly affects the learning of the graph adjacency matrix and thus implicitly influences the label matrix of the unlabeled EEG samples.

In Figure 6, three cases were used to show the iterative optimization process of SRAGL according to the learned label indicator matrix of unlabeled EEG samples, which was proposed in ^[33]. More specifically, with the number of iterations increasing, we reconstruct the graph adjacency matrix at different iterations by ${{\mathbf{Y}}_{u}}\mathbf{Y}_{u}^{T}$. From this figure, we observe that the correlations for the intra-class EEG samples are increasing over time, whereas the relationships for inter-class EEG samples are gradually decreasing. This is consistent with our expectation that SRAGL gradually depicts the underlying semantic information of EEG samples whilst suppressing the interference of different sessions and trials. In addition, the more accurate the label matrix of the unlabeled EEG samples was predicted, the clearer the four diagonal blocks in the reconstructed graph become. Taking Figure 6(a) 'sub2: session1$\to$session2' as an example, four diagonal blocks correspond to four emotion states in the SEED-IV data set become more and more visible as the accuracy increases which illustrates the effectiveness of the SRAGL model optimization. Meanwhile, we show the ground truth graph as a comparison in the last column.

3.4.   EEG Emotional Activation Pattern Mining

EEG data is typically characterized as multi-frequency and multi-channel signals, and DE features are extracted from different frequency bands and channels. In this section, we quantify the contributions of EEG features to cross-session emotion recognition tasks to investigate key frequency bands and channels. As shown in Figure 7, the coupling relationship between each EEG feature dimension and each frequency band (channel) is illustrated.

Inspired by the $\ell_{2, 1}$-norm based feature ranking theory ^[34,27], we calculate the normalized $\ell_{2}$-norm of each row of the projection matrix $\mathbf{W}$ to determine the contribution of each EEG feature (i.e., ${{\theta }_{i}}$). In mathematics, it is expressed by

It is obvious that the larger ${{\theta }_{i}}$ is, the greater contribution of the $i$-th EEG feature to the emotion classification tasks. For each subject, we calculate the contribution of each feature dimension. Due to the inter-subject variability, there may be variations for the learned feature importance values of different subjects. To mitigate the impact of individual variability and improve the reliability of the identified important features by SRAGL, we average the feature contributions across all the subjects. As a result, we obtain the contributions of the 310 feature dimensions, shown in Figure 8. Suppose that we have $a$ EEG bands and $b$ channels in total, based on the established relationship between EEG features and their correlated frequency band (channel) ^[35], we can calculate the contribution of the $i$-th frequency band by

Similarly, the contribution of the $j$-th EEG channel is

In SEED-IV, the feature dimension is 310, the number of frequency bands is 5 (i.e., $a$ = 5) and the number of EEG channel is 62 (i.e., $b$ = 62). According to equation (3.2), Figure 9 displays the relative importance of different frequency bands, represented as bar charts. From this figure, we observe that Gamma is the most significant frequency band for cross-session EEG emotion recognition, while the second one is Delta. Similarly, according to equation (3.3), we obtained the contribution values of the 62 EEG channels and represented their relative importance using brain topography maps in Figure 10. As a result, we conclude that prefrontal, left/right temporal, especially the (central) parietal lobes contributed more to emotion recognition, which is consistent with the existing studies ^[21]. In Figure 11, we show the 10 most important channels in cross-session emotion recognition, which are CZ, CPZ, CP2, P2, C2, FP1, CP1, POZ, FP2, and FC6. The emotional activation pattern analysis method utilized above can be extended to any EEG data set, regardless of the number of bands or channels.

3.5.   Discussion

To highlight the significance of this work, this section provides thorough discussions by putting the emphasis on the connections and differences between SRAGL and some of our previous studies including the Self-weighted Semi-supervised Classification (SWSC) ^[36], the retargeted semi-supervised regression with robust weights (RSRRW) ^[37], the semi-supervised Joint Sample and Feature importance Evaluation (sJSFE) ^[38], the Joint label-Common and label-Specific Features Exploration (JCSFE) ^[39], and the Optimal Graph coupled Semi-Supervised Learning (OGSSL) ^[40]. The connections are first analyzed, followed by the differences.

Generally, all of these models are proposed for EEG-based emotion recognition under the semi-supervised classification paradigm. On the experimental setup, EEG data from different sessions was utilized in the comparative studies for their performance evaluation, i.e., cross-session emotion recognition. Besides, the important EEG frequency bands and channels in emotion recognition were identified according to the explicit/implicit feature importance exploration and the correspondence between EEG features and the associated frequency bands (channels).

On the differences between SRAGL and the previously proposed ones, we have the following two understandings.

$1)$ From the perspective of model formulation, SWSC is motivated by explicitly introducing a feature weighting variable to characterize the different contributions of different features in emotion recognition while JCSFE is a more problem-driven model and puts more focus on the common activation patterns across different emotional states and the specific patterns associated with a certain state. Both RSRRW and sJSFE mainly pay attention to the model robustness by respectively introducing discrete and continuous weights to characterize the different contributions of different samples in emotion recognition, which do not involve the graph data structure in their model formulations. Though graph regularization terms were used in both SWSC and JCSFE, the graph similarity matrices were directly calculated according to fixed rules such as the '0-1' and 'heatkernel' weighting schemes. In SRAGL, the graph similarity matrix is dynamically learned within the iterative model optimization process, which aims to enforce the label consistency of unlabeled EEG samples. In the experiments, the effectiveness of the graph learning technique in SRAGL is intuitively analyzed from two aspects. On one aspect, we showed how the learned graph similarity matrix depicts the semantic information of EEG samples; that is, each of four diagonal blocks in Figure 4 corresponds to one of the four emotional states. On the other aspect, we showed how the learned graph facilitates the estimation of the label indication matrix of the unlabeled samples in Figure 6.

$2)$ Both OGSSL and SRAGL involve the dynamic graph learning strategy in their model objective functions. The main difference between them is how the label indicator matrix of unlabeled samples is estimated. In OGSSL, the estimation of the label indicator matrix corresponds to a label-propagation process over the learned graph. However, in SRAGL, the label matrix is obtained by learning a projection matrix to bridge the data space and the label space. Obviously, different optimization strategies were employed in respective models to handle the updating of the label indicator matrix. In terms of the recognition accuracy, the proposed SRAGL model in the present work improves OGSSL by 1.67%, 3.47%, and 0.70% corresponding to three cross-session tasks respectively.

4.   Conclusions

In this paper, a semi-supervised regression with adaptive graph learning model is proposed for EEG emotion recognition, which is termed SRAGL. In SRAGL, semi-supervised regression, adaptive graph structure learning and label propagation are seamlessly integrated together. SRAGL updates the graph similarity matrix gradually by leveraging label information from both labeled and unlabeled samples, which is jointly estimated by LSR, ultimately resulting in the optimal structured graph. Extensive results on SEED-IV indicated that the learned graph adjacency matrix effectively improves the performance of emotion recognition. Furthermore, by the established relationship between EEG features and their correlated frequency band (channel), and utilizing the learned projection matrix, we calculated the contribution of each EEG feature and analyzed the emotional activation patterns. Our findings using SRAGL suggest that the Gamma band is the most significant one, and that the central parietal lobe is more closely related to emotional expression. Based on these findings, it is expected to customize the hardware and computing models for emotion recognition applications in the future. For example, we can minimize the number of electrodes required in wearable devices such as caps for EEG data collection. In addition, we explore the possibility of utilizing only Gamma band to reduce computational requirements. These conclusions shed new light on emotional processing for human-computer interaction and provide valuable insights for future research in this field.

Acknowledgments

This work was partially supported by the National Natural Science Foundation of China (No. 61971173) and the Natural Science Foundation of Zhejiang Province (No. LY21F030005).

Conflict of interest

The authors declare there is no conflict of interest.