Functional division of the dorsal striatum based on a graph neural network

1.   Introduction

A BCI provides a pathway for people suffering from neuro-muscular dysfunctions to communicate with the world ^[1], by decoding electroencephalography (EEG) signals that reflect synchronous activities of neurons in the cerebral cortex beneath the skull ^[2]. An ERP-based BCI speller, also referred to as the P300 speller in literatures ^[3,4], relies on characteristics of the ERP components elicited by the attended stimuli in a spelling task ^[5].

Multi-sensor recordings are normally required to capture the widely distributed ERP features over the cortical surface, as well as to compensate for the poor signal quality from a single sensor. More sensors typically yield better classification accuracies, while it is at the cost of increased complexity and reduced usability of the speller, limiting the popularization of mobile/wearable BCI devices ^[6,7,8] for the end use. Reduction and centralization of sensors may bring much more comfort to the user, decreases installation time duration, reduce the cost and improve portability and convenience of a BCI ^[9]. To reduce the number of sensors, recently many computational sensor-selection methods, from the perspectives of the dimensionality reduction and the spatial filtering, have been developed, such as those based on the swarm algorithm ^[10,11], independent component analysis ^[12], automatic relevance determination ^[13], and other strategies ^[14,15], etc. However, aside from an additional computational burden, these sensor-selection methods are of no help to centralization of sensors, because they only select but do not change the underlying sources. On the contrary, the localization of brain activities could be a solution to both reduction and centralization of sensors.

The basic concept of localization is to reinforce local rather than global brain activities, and make useful information concentrated on centralized sensors. Some BCI modalities, such as motor imagery BCI ^[12] and steady-state visual evoked potential (SSVEP)-based BCI ^[16], are born with alocalization feature. Motor imagery BCIs are based on contralateral ERD/ERS rhythms that are mainly concentrated around the central area over the sensorimotor cortex, and SSVEP BCIs are based on homogeneous SSVEP rhythms that are originated from the visual cortex and spread over the whole cortical surface. Different from these BCI modalities, ERP-based BCIs may suffer from a dispersed distribution of the underlying ERP activities, mainly involving those associated with visual N1, P2, N2, and P3 components. Visual P2 and N2 preponderating over the frontal-central area, P3 preponderating over the parietal area, and N1 preponderating over the occipito-temporal area, have been thought to make major contributions to the working of an ERP speller ^[5,17,18]. Such a dispersed distribution poses a huge challenge for the localization of an ERP-based BCI, because discarding any of these ERP components may remarkably deteriorate the classification performance, which now is still too low even if considering all of these components. Because characteristics of ERPs are closely related to the perceptual and cognitive meanings of the visual stimulus processed by the brain, one possible way to handle the challenge is to optimize the visual stimulus to obtain more localized activities.

The traditional ERP speller uses a type of character-flashing stimulus, where luminous intensification of characters is used for accentuation ^[4]. Recently, new stimulus types, such as the face-flashing stimulus ^[19,20,21], the object rotation stimulus (the FLIP paradigm) ^[22], and the object motion stimulus (motion-onset paradigm) ^[23], have been shown to have advantages over the traditional character-flashing type. The face-flashing paradigm, which superimposes face images onto characters for accentuation, was shown to benefit from additional ERPs related to face recognition, such as N170 and N400, and thus significantly improved the classification performance. The FLIP paradigm was revealed to be able to suppress the refractory effect of the P3 component in an ERP speller, and the motion-onset paradigm was revealed to elicit the motion-onset N200 component, and have a merit of low contrast and luminance tolerance. In our previous work, we had investigated a novel visual graphic stimulus, and demonstrated its effectiveness for improving overall classification performance of an ERP speller ^[18]. Different from the stimulus types examined in literatures ^{[19,20,21,22,23]}, the visual graphic stimulus has significantly increased complexity and unpredictability, which profoundly affects the perceptual processing and leads to significant enhancement of ERP responses ^[24,25]. Although many stimulus types have been proposed, hardly any of them has been evaluated referring to the localization performance.

The novelty and main contributions of the present study is to reveal the specific localization effect of brain activities elicited by the visual graphic stimulus, and verify its effectiveness and significance for the centralization and reduction of sensors, for an ERP speller. For this purpose, three sensor settings, i.e., the FS, NS, and LS settings, were tested, and the classification performance of an ERP speller with the visual graphic stimulus, were reevaluated, and compared to those with the traditional character-flashing stimulus. The FS condition uses a full set of all sensors and provides the baseline performance, whereas for the NS condition, ten sensors, Fz, Cz, C3, C4, Pz, Oz, O1, O2, T5, T6, were selected in accordance with relevant state-of-the-art studies ^[13,15,18], and for the LS condition, only five posterior sensors, Pz, O1, O2, T5, T6, were selected, according to our findings about ERP characteristics under both graphic and character-flashing paradigms. The sensor placement adopted in the present study is illustrated in Figure 1.

Figure 1.  Sensor placement adopted in the present study. Sensors with red dotted circles refer to the NS setting, whereas those with blue background refer to the LS setting. All sensors were used under the FS setting.

DownLoad: Full-Size Img PowerPoint

2.   Materials and method

2.1.   Experimental paradigm

A description of the traditional character-flashing paradigm could be found in ^[4]. The visual graphic stimulus-based paradigm modifies the traditional paradigm by changing the stimulus type from the character-flashing type to a varied graphic pattern flashing type, as shown in Figure 2. For each row/column flashing, the characters involved are accentuated by being superimposed with 6 highlighted geometric patterns, which are selected randomly and uniquely from a set including eleven patterns shown in Figure 2c. We adopt a dynamic presentation scheme to ensure the unpredictability of the stimulus: 1) All patterns in a row/column flashing should be selected randomly and uniquely from the set; 2) patterns for two successive flashings of a character should be distinct. In this way, the user would be uncertain about the morphology of the target stimulus in the next flashing, as well as the morphologies of its non-target surrounding stimuli. More details about the varied graphic pattern flashing paradigm could be found in ^[18].

Figure 2.  Experimental paradigm. (a) Blank speller; (b) one row is transiently accentuated by superimposed visual graphic stimulus; (c) visual graphic patterns.

DownLoad: Full-Size Img PowerPoint

A subtrial is defined as the twelve row and column flashings of the matrix, and a trial is defined as several consecutive subtrials required for spelling a character. In our experiment, a trial contained 5 subtrials. Stimulus onset asynchrony (SOA), defined as the time interval from the onset of one flashing to the onset of the next flashing, was set as 160 ms with an accentuation period of 80 ms.

2.2.   Experiment

Sixteen healthy participants (12 males and 4 females), all right-handed, with normal or corrected to normal vision, took part in the experiment. All participants were naïve to BCIs. Ethical approval and informed consents were obtained in compliance with the Declaration of Helsinki.

Participants were instructed to copy-spell 72 characters without feedback under both the graphic stimulus-based paradigm (GP) and traditional character-flashing paradigm (CP). Target characters were selected randomly from 36 characters in the character matrix, with each character selected twice. These target characters were divided into 6 groups, and each group contained 12 characters. In each experimental run, a group of characters was selected as targets. Therefore, there were totally 12 runs, with 6 runs under the GP, and the other 6 runs under CP. For counterbalancing, the runs were carried out alternately, and half of the participants began with GP, while the others began with CP.

A run contained 12 trials. For each trial, participants were instructed to attend to the flashings of one target character. When a trial began, they had 3 seconds to focus on the target item in the matrix before the flashings occurred. Then, they should mentally count the number of flashings of the target character. At the end of the trial, there was a 2-second blank, during which they should report the number they counted. Then, they would continue with the next trial until the end of the run. When a run finished, there was a 2-minute break before the next run began. To reduce the ocular artefacts, participants were told to try not to blink during flashings.

Finally, we got 144 trials for each participant, with 72 trials for each paradigm.

2.3.   Data acquisition and processing

Brain signals were acquired with a Nuamps 40 amplifier (Neuroscan Inc.) at a sampling rate of 250 Hz, with linked-mastoid reference and a forehead ground. Thirty-two Ag/AgCl sensors, including 30 recording sensors and 2 reference sensors, were placed according to the international 10–20 EEG systems, as shown in Figure 1. The sensor impedance was kept below 5 kilohm. Data from all sensors were recorded in the experiment. Data storage and speller implementation are achieved by BCI2000 ^[26].

For classification, the recorded brain signals were filtered successively using a causal 3-order Butterworth high-pass filter with a cutoff of 0.5 Hz, and a causal 6-order Butterworth low-pass filter with a cutoff of 5.5 Hz. Whereas for ERP analysis, non-causal zero-phased filters were used instead, with the cutoff of the low-pass filter broadened to 40 Hz. To overcome possible ERP distortion arising from epoch overlapping due to a short SOA, only epochs with target-to-target intervals greater than 5*SOA were used for examining ERP responses.

The epoch length was 800 ms, starting from 200 ms preceding stimulus onset. For classification, data epochs were down sampled to a rate of 25 Hz, and then the resultant epochs from the selected sensors were concatenated to form feature vectors. A stepwise linear discriminant analysis (SWLDA) classifier ^[27] was adopted.

2.4.   Data analysis

Grand-average difference ERP responses between both spellers were compared, obtained by subtracting non-target responses from target responses averaged across all participants.

Based on the comparison on ERP responses, a localized sensor set was determined. Then, classification accuracies and information transfer rates (ITR) ^[28] between GP and CP, were compared respectively, through three-way 2 × 3 × 5 repeated measures ANOVA (PARADIGM [GP, CP] × SENSORSETTING [FS, NS, LS] × TRIAL LENGTH ^[1,2,3,4,5]). Trial length was defined as the number of subtrials used in a trial. For a trial length level less than 5, a corresponding number of subtrials in the front of each trial were always used, without screening, for the classification, e.g., for trial length 1, only the first subtrial in each trial was used for the classification. For the FS condition, all the 30 recording sensors were selected. ANOVA were carried out with the Statistic Package for Social Science (SPSS).

A 4-fold cross validation was adopted to obtain the estimation of classification accuracies. For each participant, the obtained 72-character dataset were divided sequentially into four groups, with 18 characters in each group. Every time, one of the four groups was selected for training the classifier, while the remaining were used for evaluating the character-wise accuracy. Then, accuracies from the 4 folds were averaged to obtain a final result. ITRs were calculated using ^[28]

where N indicates the number of items in the speller matrix, P indicates the character-wise classification accuracy, T indicates the time span of a trial (unit: Min (minute)), and ITR means the number of bits being sent in a trial (unit: Bits/min).

3.   Results

3.1.   ERP responses

Grand-average difference ERP responses for both paradigms are shown in Figure 3. Several differences could be found between GP and CP. First, at occipito-temporal sensor sites, e.g., O1 and O2 sites, an enhanced negative N1 component with a latency of about 200 ms could be observed for GP. Second, at T5, T6, O1, and O2 sites, a pronounced positive peak with a latency around 320 ms is observed for GP, which seems absent for CP. Third, at frontal-central sites, typically Fz and Cz, a more negative N2 component with a latency around 300 ms, is elicited for GP than that for CP. While for the P2 component at frontal-central sites, with a latency around 230 ms, and for the P3 component at parietal and occipito-temporal sites, with a latency around 400 ms, there seem no much differences between both spellers.

Figure 3.  Grand-average difference ERP responses for GP and CP.

DownLoad: Full-Size Img PowerPoint

From the ERP responses, it can be found that GP elicited enhanced ERP features on occipito-temporal sensor sites. These differences are mainly concentrated on several minority posterior sites, indicating a localization effect. The localization effect brought out by GP could be seen in more detail in individual scalp distributions of amplitudes of the posterior negative (named N1) and positive (named P2b) peaks in difference responses between GP and CP, as shown in Figures 4 and 5, respectively. It should be noted that, in Figure 4, amplitudes of inverse N1 are shown such that a greater value in the map indicates more pronounced N1 amplitudes. For evaluating the amplitudes of these components, these negative and positive peaks were first found in a range of 100 to 400 ms referring to Oz, and then mean amplitudes within 40 ms around the peaks were evaluated. It can be seen obviously that, for most participants, enhancement of ERP activities by GP is primarily localized over the posterior region. Therefore, four occipito-temporal sites, T5, T6, O1, and O2, were selected for the localized sensor setting. On the other hand, a parietal site Pz was also included in the localized sensor setting, in consideration of the contribution of P3 component to classification, which preponderates at the parietal area. Finally, five posterior sensors, at T5, T6, O1, O2, and Pz sites, respectively, were selected for the localized sensor setting.

Figure 4.  Individual distributions of amplitudes of inverse N1 in difference responses between GP and CP (GP-CP).

DownLoad: Full-Size Img PowerPoint

Figure 5.  Individual distributions of amplitudes of P2b in difference responses between GP and CP (GP-CP).

DownLoad: Full-Size Img PowerPoint

3.2.   Classification accuracies and ITRs

Mean accuracies and mean information transfer rates of both spellers, obtained from the 4-fold cross validation, are shown in Figure 6a and b, respectively.

Figure 6.  Accuracies and information transfer rates (ITR) under different trial lengths and sensor sets. (a) Mean accuracies at various trial lengths; (b) mean ITRs at various trial lengths. The error bar shows the standard deviation.

DownLoad: Full-Size Img PowerPoint

Results from ANOVA on classification accuracies revealed a significant PARADIGM × SENSORSETTING × TRIALLENGTH interaction (F(8,120) = 2.68, p = 0.010). GP significantly outperforms CP irrespective of sensor setting and trial length (5 trial length levels (1~5) within 3 sensor settings (FS, NS, LS): F(1, 15) = 34.85, 47.23, 18.50, 13.71, 9.23; 41.68, 32.99, 23.07, 15.75, 9.95; 25.20, 39.56, 31.51, 32.33, 19.59 respectively. All p-values < 0.01). PARADIGM × TRIALLENGTH interactions are also revealed to be significant for each sensor settings (F(4, 60) = 12.60, 10.30, 2.62; p = 0.000, 0.000, 0.043 respectively). At short trial lengths (1~2), the differences between both paradigms become more obvious than those at long trial lengths (3~5).

Results from ANOVA on information transfer rates revealed a significant PARADIGM × TRIAL LENGTH interaction (F(4, 60) = 21.61, p = 0.000). Similar to the results of ANOVA on accuracies, at each trial length level, GP outperformed CP (F(1, 15) = 34.77, 53.25, 33.75, 29.15, 18.13 respectively; all p-values < 0.01), and the differences become more obvious for short trial lengths than for long trial lengths. The PARADIGM × SENSOR × TRIALLENGTH interaction does not reach the significant level (F(8,120) = 1.70, p = 0. 104).

From Figure 6, it can be seen that even if with the localized sensor setting, GP still achieves better results than CP with any of the three sensor settings. To verify this argument, we further performed two-tailed paired-samples T-tests between the LS-GP condition and each of the FS-CP, NS-CP, and LS-CP conditions. Results from T-tests on accuracies show that, at short trial lengths (1~2), the LS-GP outperforms either of the FS-CP and NS-CP (p < 0.05); while when trial length increases (3~4), differences between the LS-GP and the NS-CP become less significant (p < 0.1), and differences between the LS-GP and the FS-CP do not reach the significant level (p > 0.1); at trial length 5, there are no significant differences between the LS-GP and either of the NS-CP and the FS-CP (p = 0.271 and 0.332 respectively); however, at all trial lengths (1~5), the LS-GP is significantly better than the LS-CP (all p-values = 0.000). T-tests on information transfer rates give the similar results to T-tests on accuracies. Therefore, it can be concluded that, at short trial lengths (1~2), GP with the localized sensor set outperforms CP with any of FS, NS, and LS, while the differences become less significant as the trial length increases.

4.   Discussion

Sensor reduction is an important issue for lowering complexity of an ERP speller. However, a tradeoff between the accuracy and the sensor set size should always be considered when pursuing the minimization of sensors. Because the amount of information available for classification may decrease with reduced sensors, classification accuracy would typically degenerate as well. One way for sensor reduction is to exploit signal processing methods to find the optimal sensor set. These methods, also commonly called the sensor/channel selection methods ^{[9,10,11,12,13,14,15]}, depend on the established characteristics of the ERP signals within an experimental paradigm. Another way for sensor reduction is through localization of brain activities, which is the main purpose of the present study. Different from the signal processing methods, which do not change the physical properties of the underlying ERP components, localization methods could directly improve the quality of the ERP components, and make the useful information more concentrated on minority sensors, so it would be more powerful for sensor reduction.

We evaluated the localization performance under a novel visual graphic stimulus paradigm. Performance of GP was compared to that of CP under three sensor set conditions, i.e., the full 30-sensor set, the normal 10-sensor set, and the localized 5-sensor set. Results showed that, GP achieves significantly greater classification accuracies and information transfer rates than CP, irrespective of sensor settings. Furthermore, even with the localized sensor set, GP still shows its advantage over CP using any of the three sensor settings, especially at short trial lengths. For instance, at trial length 2, the average accuracy of GP under LS reaches up to 80.24%, significantly greater than those of CP under FS, NS, and LS, i.e., 75.38, 75.00 and 67.64% respectively (all p-values < 0.05). To increase the output rate of an ERP speller, researchers have always been finding ways, from either signal processing or experimental paradigm directions, to increase the classification accuracy at short trial lengths. In this sense, GP is also especially valuable for BCI studies with short trial lengths. GP also obtained a maximum average ITR of 69.76 bit/min under LS, significantly higher than those of CP under FS, NS, and LS, i.e., 58.36, 56.47 and 49.38 bit/min respectively (all p-values < 0.01). In conclusion, for GP, the sensor-set size could be reduced half, from the normal frontal-central and posterior sites to only the localized posterior sites, whereas the performance remains better than CP. Therefore, we think GP is remarkably effective for sensor reduction and localization of brain activities of an ERP BCI. Besides the localization performance, the trial length effect needs also be discussed. As shown in Figure 6, a clear trial length effect can be observed for both accuracies and ITRs, under each of the six conditions, where mean accuracies show a positive correlation trend, and mean ITRs show a negative correlation trend, with respect to the trial length. That is because, a greater trial length means more stimulus repetitions, and thus brings about increased signal-to-noise ratio for the obtained ERP samples, which naturally leads to increased accuracies. This may explain the trial length effect on accuracies shown in Figure 6a. However, an increase of the trial length may also prolong the time span of a trial, and thus, according to the definition of the ITR, may cause a reduction of ITRs, as observed from our results given in Figure 6b.

According to our results, five posterior sensors, including one parietal and four occipito-temporal sensors, were enough for GP to obtain a comparable or even higher performance than CP. Because these sensors are concentrated over the local posterior area, the proposed method may facilitate the design and setup of an ERP speller device, and so it would be of great value for the practical use. These results could be supported by the underlying ERP responses at these sensors. As shown in Figure 3, for GP, at Pz site, differences between target and non-target responses mainly come from P2, N2, and P3 components, whose latencies are 230,300 and 400 ms respectively, whereas for O1, O2, T5, and T6 sites, N1, P3, and a pronounced positive component with a latency of about 320ms, provide discriminative information for classification. Performance enhancement of GP should probably come from the enhancement of N1, and the elicitation of a pronounced positive component (denoted as P2b here) earlier than P3 at occipito-temporal sensor sites.

5.   Conclusion

In this paper, we examined the localization issue for an ERP speller, which may be especially useful for reduction and centralization of sensors, and thus for popularization of wearable/mobile BCIs for the end use. A novel visual graphic stimulus was used to yield localized posterior activities, which, from our experimental results, was demonstrated to achieve comparable or even better classification performance than those obtained by the traditional character-flashing stimulus using global activities, and was also revealed to be able to reduce the number of sensors required by half. Participants also reported that they felt more concentrated and less fatigue with the graphic stimulus than the character-flashing stimulus. Future work may involve incorporating dimensionality reduction approaches, such as the Grassmannian subspace method ^[29,30], to further reduce the computational load and boost the classification performance, as well as revealing the cognitive origins of the localized posterior activities under visual graphic stimulus, by solving an EEG inverse problem ^[2,31].

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (61772508, U1713213, 61906183, 61671105), Shenzhen Technology Project (JCYJ20170413152535587, JCYJ20180507182610734), Key Research and Development Program of Guangdong Province (2019B090915001), CAS Key Technology Talent Program, Shenzhen Engineering Laboratory for 3D Content Generating Technologies (NO. [2017]476).

Conflict of Interest

All authors declare that they have no conflict of interest in relation to this scientific work.