Gesture recognition of continuous wavelet transform and deep convolution attention network

1.   Introduction

With the frequent occurrence of diseases, traffic accidents, and natural disasters, the number of patients with physical disabilities is gradually increasing. The most significant proportion of patients have some upper limb disabilities ^[1]. The rehabilitation robot for assisting movement using surface electromyography (sEMG) is becoming a hot issue of research. By exploring different recognition methods, sEMG is used to control intelligent prosthetics more efficiently and accurately. Intelligent prosthetics can help people complete simple tasks in many daily activities. For example, intelligent prosthetics can help disabled patients finish hand rehabilitation training. SEMG reflects neuromuscular activity and contains some vital information with muscle activity ^[1,2,3]. SEMG is one of the most widely used signals for identifying gestures due to being easily acquired ^[3].

Although the original sEMG contains a large amount of information about the intention of hand movements, it cannot be directly applied to recognition due to the non-smooth and non-periodic nature of the signal ^[2]. Features of sEMG are very important, and they can determine the performance of the recognition method. It is not easy to extract significant features from sEMG. The traditional manual extraction method can only extract some low-level features of sEMG. So, extracting meaningful features of sEMG has been a huge challenge for gesture recognition. Many researchers are trying to explore new methods for extracting sEMG features ^[3]. sEMG contains a large amount of noise, due to the differences between individuals and the influence of the collection environment. Therefore, the raw sEMG needs to be preprocessed. In recent years, researchers have explored new networks to improve gesture recognition accuracy. As the depth of some networks increases, some factors affecting network performance appear ^[2,3,4,5]. This paper explores a new gesture recognition algorithm by improving CNN to reduce the adverse effects of network deepening.

CNN-based methods can capture local features in 2D or 3D spaces ^[3]. Na Duan from Fudan University preprocessed the sEMG using short-time Fourier transform, and extracted the spectrum diagram of sEMG, to extract the depth features ^[4]. However, the receptive field of convolutions limits CNNs ^[3]. The depth of a CNN can affect its ability to extract features. Especially, the gradient's disappearance and the increasing depth of the network can cause a lack of data feature information, and result in lower identification accuracy ^[5].

In order to solve the above problems, many researchers focused on various preprocessing methods for improved CNN models to enhance the recognition accuracy ^[6,7,8,9]. Liukai Xu et al. changed the energy kernel phase map of sEMG into grayscale images. The grayscale images were used as the input of a CNN to recognize different gestures ^[8]. In the latest study, Mehmet Akif Ozdemir et al. obtained time-frequency images of sEMG by applying short-time Fourier transform, continuous wavelet transform, and Hilbert-Yellow transforms. The pre-trained ResNet-50 network was used for gesture classification, and the best recognition accuracy was 93.75% ^[9]. Although the accuracy of gesture recognition has improved, it is still far from being usable for practical applications.

This paper proposes continuous wavelet transform combined with a deep convolutional neural network-spatial attention module (DCNN-SAM) model to realize gesture recognition. The main content of the paper is as follows.

● sEMG is preprocessed by filtering, and then the time-frequency graph of sEMG is extracted by using continuous wavelet transform.

● The DCNN-SAM model is designed, and the model can enhance the feature expression of crucial regions and reduce features loss of sEMG. The time-frequency graph of sEMG is used as the input of the model.

● An experiment is designed to verify the effectiveness of the proposed method using the sEMG collected from 10 subjects.

The structure of the paper is as follows: The second part mainly presents the construction of DCNN-SAM, including the spatial attention mechanism and residual module. In section 3, the experiment is designed, including the preprocessing and feature extraction, as well as results and analysis. Finally, we conclude the full text and discuss future work.

2.   Materials and methods

In order to get the frequency components, sEMG is processed by continuous wavelet transform. DCNN-SAM is improved by embedding the residual module. Then, the time-frequency spectrogram of the sEMG is selected as input of the DCNN-SAM.

2.1.   Spatial attention module

The spatial attention mechanism is used to enhance the feature representation of some critical regions ^[10]. It is essentially transforming the spatial information in the original image to another space through the module of spatial transformation and retains the essential information in it ^[11]. It generates a weight mask for each location and weights the output, and in turn it enhances the feature representation of some critical specific target regions. At the same time, features of irrelevant regions are also weakened ^[12]. Figure 1 shows the structural diagram of the spatial attention mechanism.

Figure 1.  Spatial attention mechanism model.

DownLoad: Full-Size Img PowerPoint

The process is as follows: The input is an H × W × C sEMG time-frequency spectrogram. First, a maximization and averaging operation is performed along the dimensional direction of the channel to produce two feature maps representing different information. A two-dimensional convolution with a convolution kernel size of 1 × 1 is used to perform the feature fusion operation ^[12,13]. Then, the two feature maps are combined to obtain a feature map of H × W × 1 size. Finally, the two-dimensional spatial attention feature map is output after activation by the sigmoid activation function. The original input time-frequency spectrogram is superimposed to obtain the spatial attention output feature map ^[13], and this in turn enables the features of the target region to be enhanced and retains the critical feature information that helps in gesture recognition classification.

2.2.   Residual module

The idea of the residual module is that it changes the mapping from the initial feature X to the learned feature Y. Finally, the learned residual feature is added to the original feature ^[14]. The residual module can contain multiple convolutional layers. The original input feature is directly connected with the result of the output feature by skipping some convolutional network layers. Finally, all of them are taken as input of the activation function to get the learning output feature ^[15]. Figure 2 shows the structure diagram of the residual module. The residual module can optimize the deep learning network, alleviate the problem of network degradation, and ensure the integrity of signal features.

Figure 2.  Residual module structure.

DownLoad: Full-Size Img PowerPoint

2.3.   Network model

The CNN is one of the classic algorithms of deep learning ^[14,15,16]. In this section, four deep convolutional modules are designed to build the deep convolutional neural network (DCNN). The spatial attention mechanism module is added to construct the DCNN-SAM model. Figure 3 shows the model structure diagram.

Figure 3.  Network structure diagram.

DownLoad: Full-Size Img PowerPoint

The DCNN contains two parts, DCNN-1 and DCNN-2. DCNN-1 carries out the convolution operation on the time-frequency spectrogram of the sEMG. Then, the output of DCNN-1 is used as input for DCNN-2 to obtain more detailed sEMG features. DCNN-1 is composed of three convolutional layers. DCNN-2 is composed of one convolutional layer. A batch-normalized BN layer exists between each convolutional block and activation layer. A residual module is embedded between two substructures, and a spatial attention mechanism is introduced to ensure the integrity of the features in the convolution operation. Finally, the full connection and softmax layer are used to get the result.

The process is divided into three stages, the input stage of data, the deep convolution stage, and the feature output stage. The input of the model is a time-frequency spectrogram of size 40 × 40 × 1. The features are extracted and learned at the deep convolution stage. Finally, the trained model is used to recognize the corresponding gesture.

The convolutional module consists of two-dimensional convolutional kernels, a batch normalization (BN) layer, and an activation layer. The size of the convolutional kernel is 3 × 3, the moving step is 1, and the numbers of convolutional kernels in the four layers are 16, 32, 64 and 128, respectively. The BN layer improves the speed of network training by inhibiting network gradient disappearance or gradient explosion during network training. The activation function can better extract sparse features and improve the learning speed and learning accuracy by using a rectified linear unit (Relu). The spatial attention mechanism is added between the third and fourth convolutional modules to strengthen the features in specific regions. A residual module is embedded for optimizing DCNN-SAM. The numbers of convolutional kernels of two residual structures are 64 and 128, respectively. The residual structure can effectively alleviate the network degradation problem by connecting the initial input with the output feature. Then, a global average pooling layer is used to calculate the average of all pixels of each channel to obtain a new 1 × 1 channel map. It can suppresses overfitting and reduces the redundant network parameters. Finally, a fully connected layer of size 128 is used to connect the data and collect the information related to the sample feature space. A softmax function is used as the output layer to get the final classification results.

3.   Experiments and results analysis

A dataset containing ten different gestures was acquired by using the Myo EMG armband with ten subjects. The experimental included preprocess sliding window segmentation ^[15,16], continuous wavelet transform, and the DCNN-SAM model training and testing ^[17]. Figure 4 shows the process diagram of the experiment.

Figure 4.  Experimental flow chart.

DownLoad: Full-Size Img PowerPoint

3.1.   Acquisition of sEMG

The Myo EMG armband was used to acquire sEMG. It has eight sEMG channels with a sampling frequency of 200 Hz ^[18,19]. The Myo EMG armband is worn on the subject's right forearm, as shown in Figure 5. Before the signal acquisition, the subject is told not to do strenuous exercise to avoid the effects of muscle fatigue during the experiment. The skin surface is wiped with alcohol pads to remove the influential oil and dead skin to ensure good contact between the subject's skin and the electrodes ^[20].

Figure 5.  Wearing position of the Myo EMG armband.

DownLoad: Full-Size Img PowerPoint

The data was collected from 10 healthy subjects, including 5 males and 5 females, all aged between 20 and 30 years. Table 1 shows the details of the subjects. All subjects were informed of the details of the experiment and voluntarily completed the relevant written informed consent form.

Table 1.  The information of the subjects.

Number of subjects	Male to female ratio	Average height (cm)	Average weight (kg)	Average BMI (kg/m2)
10	1:1	170.5 ± 9.2	60.17 ± 3.12	21.35 ± 3.18

---

 | Show Table

DownLoad: CSV

The designed ten gestures included fist, one, two, OK, open hand, praise, six, up, down, and eight. Figure 6 shows the gestures. Subjects wore the Myo EMG armband on the right forearm and extended the arm straight and perpendicular to the ground. Subjects were trained on the corresponding gestures before formal experiments. All gestures were repeated 15 times, and each action was recorded about 16 s. Each gesture yields 3200 sampling points under the sampling frequency of 200 Hz.

Figure 6.  Classified gestures: (a) fist; (b) one; (c) two; (d) OK; (e) open hand; (f) praise; (g) six; (h) up; (i) down; (j) eight.

DownLoad: Full-Size Img PowerPoint

3.2.   sEMG data filtering and denoising

Effective information of sEMG is mainly distributed between 20 and 200 Hz ^[21]. SEMG will be interfered with by the industrial frequency signal at 50 Hz and the low-frequency signal below 20 Hz ^[20,21,22]. So, the Butterworth high-pass and band-pass filters were selected to process sEMG. The low-frequency noise below 20 Hz was filtered using the high-pass filter, and the industrial frequency noise at 50 Hz was filtered using the band-pass filter ^[23].

The Butterworth filter has been widely used due to being simple and easy to design ^[24]. The Butterworth high-pass filter is shown in Eq (1), where $w$ represents the frequency, ${w}_{c}$ represents the cut-off frequency, $\left|H\left(w\right)\right|$ represents the amplitude, and $n$represents the order of the filter. We selected a third-order high-pass filter to filter out low-frequency noise below 20 Hz.

A band-pass filter can block the passage of specific frequency. Equation (2) expresses an ideal filter, where ${w}_{0}$ = 50 Hz. It can filter out 50 Hz power-frequency noise.

Figure 7 shows the spectrum of sEMG after filtering.

Figure 7.  sEMG before and after filtering.

DownLoad: Full-Size Img PowerPoint

3.3.   Sliding window data segmentation

SEMG must be segmented before continuous wavelet transform. The sliding window segmentation method is used to cut the sEMG to ensure signal continuity ^[25]. The number of overlapping windows is shown in Eq (3) ^[26].

W is the number of windows, n is the number of sampling points, k is the window size, and s is the sliding step length. The sliding window and sliding step size are 200 and 100, respectively. Figure 8 shows the schematic diagram of the sliding window segmentation.

Figure 8.  Schematic diagram of sliding window segmentation.

DownLoad: Full-Size Img PowerPoint

3.4.   Continuous wavelet transform (CWT)

The wavelet transform is used for time-frequency analysis ^[27,28]. It can analyze information on time-frequency and adapt to the requirements of time-frequency analysis automatically ^[29]. Equation (4) shows the wavelet transform.

In Eq (4), $a$ represents the scale parameter, $\tau$represents the time parameter, $f\left(t\right)$ represents the original signal, $WT(a, \tau )$ represents the transform value, and $\varphi \left(t\right)$represents the wavelet transform function. Equation (4) shows that the frequency w and time T correspond to the scale ^[30]. The wavelet transform can obtain the frequency components of the signal and the moments when each frequency component appears ^[31]. Through continuous wavelet transform, the time-frequency spectrogram can be extracted to obtain information on the signal frequency and the time. Figure 9 shows a channel's sEMG map and the time-frequency spectrum map.

Figure 9.  sEMG map with the time-frequency spectrum map.

DownLoad: Full-Size Img PowerPoint

3.5.   Analysis of experimental results

The sEMG for the experiment was divided into three parts: 70% of the data are used for training, 20% of the data are used for testing, and 10% of the data are used for validation. DCNN-SAM uses an adaptive optimizer (Adam), the loss function uses an inter-class cross-entropy function, the random deactivation is 0.5, the initial learning rate is 0.001, and the number of iterations for network training is 100.

One subject is evaluated individually to verify the feasibility of the experiment. In order to demonstrate the superiority of this method, the result is compared with the classification models commonly used.

Figure 10 shows the accuracy and loss curves for the classification recognition with 100 iterations of training. From the graph, we can see that the accuracy value of the set increases rapidly, and the loss value decreases rapidly in the first 10 iterations. The accuracy value is greater than 90%, and the loss value is less than 0.2 after 10 iterations. The model can converge rapidly. The test set's accuracy and loss value remain stable after the number of iterations reaches 18.

Figure 10.  Curves of accuracy and loss functions for training and testing sets.

DownLoad: Full-Size Img PowerPoint

The result is compared with other models, including DCNN-SAM without continuous wavelet transform, DCNN and CNN. We chose four measures to evaluate the recognition results: Accuracy, Recall, Precision, and F1-score.

To illustrate the concept of four measures, we introduce several definitions: TP, TN, FP, and FN. TP is the number of correctly classified samples, TN is the number of incorrectly classified samples, FP is the number of incorrectly classified samples as correct samples, and FN is the number of correctly classified samples as incorrect samples. The accuracy is the ratio of the number of correctly predicted samples to the total number of samples, the precision is the ratio of the number of correctly predicted samples to all of the number of correctly predicted samples, the recall is the ratio of the number of correctly predicted samples to all of the number of correctly predicted samples, and the F1-score can be regarded as a weighted average of the accuracy and recall. A higher F1 score means a better performance of the network model. Equations (5)–(8) shows the formulas for the four measures.

Table 2 shows the results of the experiment of ten subjects.

Table 2.  Results of ten gestures categories.

Classifier	Accuracy	Recall	Precision	F1-score
CWT+DCNN-SAM	0.961	0.963	0.973	0.958
DCNN-SAM	0.929	0.931	0.937	0.928
DCNN	0.900	0.904	0.911	0.898
CNN	0.820	0.822	0.828	0.817

---

 | Show Table

DownLoad: CSV

From the experimental results, we can see that the method proposed significantly improved accuracy compared to other model methods, with an accuracy of 96.1%. Using continuous wavelet transform to analyze the sEMG has a more significant impact on the classification. We can see that continuous wavelet transform extracted the useful frequency information, helping the model improve the accuracy of gesture recognition by about 4 percentage points. This indicates that the improvement method of the Spatial Attention Module and the residual module achieve a significant effect, and it not only improves the recognition accuracy of the model but also avoids the problems that affect the classification accuracy, such as gradient disappearance due to the increased of network depth. Thus the model can achieve better performance.

Recently, some SOTA algorithms have been developed in gesture recognition, including gesture recognition based on the postural graph convolutional network (GCN) method. The network takes related gestures of hand and body as the input, and then the GCN, the residual connection, and the residual module structure are used for classification. The pyramid structure of multi-scale features is extracted by using the multi-scale multi-attention time-frequency converter network (MSMHA-VTN), the multi-scale head attention model of the transformer. The model adopts different attention dimensions for each head of the transformer, and it can provide an attention mechanism at the multi-scale level. In some recent studies, researchers proposed a new gesture recognition method based on AlexNet transfer learning and Adam optimizer. The classification of different configuration transfer learning is tested. Others proposed some new methods based on CNN, building a 12-layer CNN as the backbone. The improved 20-channel data enhancement method was used to avoid overfitting. CNN was used to recognize different gestures. We propose a new gesture recognition method of this present paper is compared with the SOTA algorithms mentioned above. Table 3 shows the comparison results. According to the data, the accuracy of our method is improved compared with the SOTA algorithms, indicating the feasibility of the algorithm proposed.

Table 3.  The recognition accuracy of different SOTA classification algorithms.

Classifier	Accuracy	Recall	Precision	F1-score
CWT+DCNN-SAM	0.961	0.963	0.973	0.958
GCN	0.926	0.929	0.931	0.926
MSMHA-VTN	0.912	0.912	0.918	0.910
AlexNet+Adam	0.915	0.914	0.921	0.916
CNNSP+CNN	0.908	0.908	0.913	0.906

---

 | Show Table

DownLoad: CSV

Figure 11 shows the confusion matrix of classification results, and we can see the recognition accuracy of each gesture. Most of the gestures have a relatively high recognition accuracy, with an overall concentration on the diagonal. The recognition accuracy of individual gestures is relatively low, such as the two gestures of one and down. The reason may be that there are some differences in the strength of signals in the process of acquisition, and it affects the subsequent data processing and classification. In addition, the low degree of differentiation between gestures is also a reason for the different recognition accuracy values of different gestures, as it is easy for them to be mistaken for other similar gestures. The magnitude of the action varies from gesture to gesture, so the part of the arm muscles involved also varies, causing different recognition accuracies simultaneously.

Figure 11.  Confusion matrix of classification and recognition of ten gestures.

DownLoad: Full-Size Img PowerPoint

We selected five healthy subjects without muscle diseases for a separate experimental evaluation of gesture recognition. The five subjects included three males and two females. Table 4 shows information of the five subjects. Figure 12 shows the evaluation results.

Table 4.  Information of five healthy subjects.

Subject	Gender	Age	Height [cm]	Weight [kg]
Subject 1	Male	24	174	118
Subject 2	Male	25	176	138
Subject 3	Male	25	183	130
Subject 4	Female	26	168	110
Subject 5	Female	25	165	106

---

 | Show Table

DownLoad: CSV

Figure 12.  Measurement indexes of five subjects under different classification models.

DownLoad: Full-Size Img PowerPoint

It can be seen that the accuracy of the proposed method is significantly improved compared to other classification algorithms. The recognition accuracy can reach 98.5% in the experiments with a single subject. This shows that the algorithm also applies to the gesture recognition of a single subject.

4.   Conclusions

We propose a DCNN-SAM model for sEMG gesture recognition. First, time-frequency analysis of sEMG is done using continuous wavelet transform. Then, the spatial attention mechanism is applied to construct the DCNN-SAM model. Finally, DCNN-SAM is optimized by embedding the residual module to avoid the gradient disappearance and local information loss. The experiment demonstrated that the method has a good recognition and classification. Although the proposed DCNN-SAM model improves the recognition accuracy, the training time of the model is slightly increased due to the deepening and increasing complexity of the network. In future research, the model can be further improved to reduce the training time.

Acknowledgments

We would like to thank all the colleagues that have supported this work. This work is jointly supported by the Natural Science Foundation of Hebei Province (No. F2021201002); National Natural Science Foundation of China (No. 62276087); Science and Technology Project of Hebei Education Department (No. ZD2020146).

Conflict of interest

The author(s) declare(s) that there is no conflict of interest regarding the publication of this paper. We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and/or company that could be construed as influencing the position presented in, or the review of, the manuscript entitled "Gesture recognition based on continuous wavelet transform and deep convolution attention network".

Data availability

The programs' data used to support the findings of this study are available from the corresponding author upon request.