Spatiotemporal and kinematic characteristics augmentation using Dual-GAN for ankle instability detection

1.   Background

Chronic ankle instability (CAI) is common for patients with lateral collateral ligament injuries (LCL injuries) of the ankle ^{[1, 2]}. About up to 40% of patients who have previously encountered lateral ankle sprain had a CAI problem ^[3]. These patients often require radical treatment for stabilizing their ankle, such as using orthoses, taping, or accepting surgical repairs on the lateral ligaments ^[4].

In current practice, physicians make diagnoses of CAI based on physical and medical examinations. So, medical imaging and gait kinetics on CAI is rapidly developing into a world research focus ^{[5, 6, 7, 8]}. Deep learning algorithms are developed to improve the detection of radiographs accordingly ^[9]. To precise manage CAI, physicians will need to perform gait analysis on patients. Lacking gait analysis impedes our understanding of patients' injuries and impacts on locomotion ^[10]. Many scientists argued that gait analysis is necessary for patients with foot and ankle ligament injuries, especially in those serious and frequent cases ^[11]. Gait analysis includes spatiotemporal and kinematic characteristics that can describe pathological changes in foot and ankle motion during walking ^{[12, 13, 14]}. This quantitative analysis will provide a solid foundation for deep learning to carry out an automatic, accurate, and immediate detection method for patients with ankle instability ^[15]. However, there are few studies on intelligent detection on gait analysis for patients with CAI now.

For this issue, our study aimed to augment gait features in CAI patients with highly representative, yet fully private and synthetic data for intelligent detection. To achieve this goal, we first employed the Dual Generative Adversarial Networks (Dual-GAN), a dataset augmentation method, to synthesize the significant spatiotemporal and kinematic characteristics measured in our previous study ^{[16, 17]}. The Dual-GAN was trained by the injury and control group independently to learn the probability distribution of spatiotemporal and kinematic characteristics in walking. Then we used the t-distribution Stochastic Neighbor Embedding (t-SNE) algorithm to evaluate the relevance between the real and synthesized features visually. These real and synthesized data were fed to the Long Short-Term Memory (LSTM)-, LSTM- Fully Convolutional Networks (FCN)-, and Convolutional LSTM-based CAI detection models under different strategies for training. Finally, the confusion matrix was used to profile the diagnosis results using these three intelligent CAI detection models.

2.   Related works

The Generative Adversarial Networks (GAN) proposed by Goodfellow et al. is an unsupervised deep learning model playing a Zero-Sum game between two competition networks (generator and discriminator). The generator attempts to synthesize samples to confuse the discriminator, while the discriminator is a binary classifier to detect the synthesized samples from real-synthesis mixed samples as much as possible. In current studies, several improved GANs were proposed aiming for the state-of-the-art application. The study of improved GANs mainly focused on optimizing the network structure and the objective function.

Several altered network structures have been presented in GAN to improve performance. Radford et al. ^[18] presented the Deep Convolutional GAN (DCGAN) using the transposed/general deep convolutional layers to replace the fully connected layers in the generator/discriminator to increase the performance. The DCGAN uses the fractional-stride convolution for pooling to improve their stability. Zhang et al. ^[19] synthesized photo-realistic images from text with Stack GAN (SGAN). The SGAN includes two conditional GANs decomposing a hard problem into sub-problems. The Stage-I GAN is used to synthesize low-resolution images. The Stage-II GAN takes Stage-I results as inputs and generated high-resolution data with realistic details. Yi et al. ^[20] proposed the Dual-GAN for mutual translation between two domains. Dual learning trains two opposite translators (source-to-target domain and target-to-source domain) simultaneously by minimizing the reconstruction loss via these coupled GANs. The coupled GANs form a nested feedback loop to reinforce the learning. Compared to the classic GAN model, the Dual-GAN model is easier to converge, as well as it has better stability and performance in data synthesizing. Moreover, several studies, such as Triple-GAN for hyperspectral classification on small training dataset ^[21], Star GAN for multi-domain image-to-image translation ^[22], Big-BiGAN progressing in image generation quality to improve representation learning performance ^[23], and Cycle-Dublur GAN to improve the image quality of chest cone-beam computed tomography images ^[24], etc., revealed that the altered networks make the GANs available for a wide range of image applications.

Furthermore, the improved objective function and penalty function are used to solve vanishing gradient and mode collapse in GAN. F-divergence is a general divergence measure between two given probability distributions including Kullback-Leibler divergence, Jensen-Shannon divergence, etc. Nowozin et al. showed a unified framework f-GAN and discussed the benefits for GAN using various f-divergence functions on model training ^[25]. F-GAN offers a powerful approach to measure complex distributions without factorizing assumptions and inspires the following researchers. The researchers try to design modified GANs to work well, fast, and concisely. Compared to the cross-entropy loss function, GAN with the least-squares loss function makes the synthesized samples closer to the real data. The least-squares loss function provides a smooth and non-saturating gradient in the discriminator forcing the synthesized samples toward the decision boundary ^[26]. Wasserstein GAN replaces the Jensen-Shannon divergence with Wasserstein distance to deal with the mode collapse and provide a meaningful loss function ^[27]. However, the training process and the convergence of Wasserstein GAN are more time-consuming than the original GAN. Given these issues, Ishaan Gulrajani et al. proposed gradient penalty instead of weight clipping in Wasserstein GAN and demonstrated that the Wasserstein GAN with gradient penalty (WGAN-GP) performs a stable training ^[28].

Various GANs were widely applied in image synthesizing for machine learning of small sample sets in recent studies, particularly in brain images, while rarely in time-series synthesizing. Bidirectional GAN, a novel end-to-end network, uses image contexts and latent vectors to optimize brain MR-to-PET synthesis for Alzheimer's Disease (AD) ^{[29, 30]}. Multi-directional perception GAN (MP-GAN) can visualize the morphological features indicating the severity of AD by the capture capability of salient global features ^[31]. Tensor-train, High-order pooling, and Semi-supervised learning-based GAN (THS-GAN) is proposed to assess mild cognitive impairment and AD ^[32]. Compared to image synthesizing, time-series synthesizing will learn their distribution and dynamics synchronously in time-series embedding space ^{[33, 34]}.

3.   Participants and data preparation

3.1.   Participants

This controlled laboratory study was conducted at the Peking University Third Hospital. The methods used in this study were reviewed and approved by the Institutional Research Board at the Peking University Third Hospital. Each participant provided informed, written consent before entering the study.

Three patients (all male, mean age of 34 years, range: 32–37 years) diagnosed with CAI resulting from the Grade III LCL injuries of the ankle (severe, complete rupture of the ligament fibers) and ten paired control subjects (all male, mean age of 28 years, range: 24–32 years) were recruited. The control subjects from the student and staff of the hospital were no known lower extremity abnormalities, previous injuries, or surgeries. There was a total of 211 normal gait cycles and 30 injured gait cycles from the participants. The training dataset for Dual-GAN consisted of 132 normal gait cycles for six control subjects and 20 injured gait cycles from two patients with CAI. The rest of the data from another four control subjects and a patient was used to test the proposed model.

3.2.   Data acquisition and pre-processing

As shown in Figure 1, the reflective markers were stuck on the shank and foot following the Heidelberg Foot Measurement Model (HFMM) ^[35]. A visual anthropometric model was built in the software Vicon Nexus 1.8.5 (Vicon Motion Systems Ltd., Oxford, UK). The Vicon MX Motion Capture System (Vicon Motion Systems Ltd., Oxford, UK) with eight cameras was used to track the markers within the capture volume and record their three-dimensional coordinate data at 100 Hz. The raw streaming data during walking were exported as.csv files for future analysis.

The raw data was filtered by a low-pass zero phase shift first-order Butterworth filter and segmented into gait cycles. The gait cycles were normalized to 100 sample points by linear interpolation for comparison among subjects.

3.3.   Spatiotemporal and kinematic characteristics

In this study, we selected basic gait variables, velocity and micro-adjustment variables, and range of motion (ROM) variables as spatiotemporal and kinematic characteristics for CAI detection.

3.3.1.   Basic gait variables

The basic gait variables reported in our previous study, including the percentage of first/second rocker phase in the gait cycle (%), the stride length (mm), and the duration (s/stride), were significantly different between the patients with LCL injuries of the ankle and control subjects ^[16].

3.3.2.   Velocity and micro-adjustment variables

Five markers covering knee, ankle, and foot in HFMM were chosen for measuring velocity and micro-adjustment variables, including TTU, LML, CCL, DMT2, and HLX.

The results ^[16] revealed that in the gait cycle significant differences in velocity between patients with CAI of the ankle and control subjects involved the maximum velocity (V_max, mm/10^-2s) of the knee (TTU), ankle (LCL), and foot (CCL, DMT2, HLX); the percentage of time to maximum velocity (TV_max, %) of the ankle (LML) and foot (CCL, DMT2, HLX); the minimum velocity (V_min, mm/10^-2s) of the TTU, LML, and DMT2; the percentage of time to minimum velocity (TV_min, %) of the TTU.

In the 2^nd rocker phase, the significant differences in micro-adjustment and velocity between the two groups included the number of valleys of the TTU, LML, and DMT2; the number of peaks of the TTU, LML, CCL, and DMT2; the mean velocity of the TTU, LML, DMT2, and HLX.

3.3.3.   Range of motion variables

According to our research, eight ROM variables calculated in ^[17], including tibiotalar flexion, forefoot/ankle abduction, medial arch angle, lateral arch angle, subtalar rotation, forefoot/ankle supination, MT I-V angle, and lateral malleolus scale, all made a certain contribution to automatic detection for the ankle instability.

4.   Data augmentation and CAI detection model

4.1.   Dual Generative Adversarial Networks

4.1.1.   Network architecture

The Dual-GAN is a closed-loop feedback system consisting of a pair of tasks. We can obtain detection results from unlabeled data, and then use the results to improve the machine learning models in dual tasks. The dual network structure translates in the cross-domain, which means that the network learns the bidirectional mapping between the features in domain $F$ and the results in domain $R$. The generator ${G}_{A}$ maps feature $f\left(f\in F\right)$ to result $r\left(r\in R\right)$ and generator ${G}_{B}$ maps result $r\left(r\in R\right)$ to features $f\left(f\in F\right)$. The discriminator ${D}_{A}$ can recognize the difference between the generated result of ${G}_{A}$ and the real result in domain $R$. The discriminator ${D}_{B}$ can recognize the difference between the generated features of ${G}_{B}$ and the real features in domain $F$.

As shown in Figure 2, the real feature $f$ translates to the result ${G}_{A}\left(f, z\right)$ via ${G}_{A}$, and the discriminator ${D}_{A}$ evaluates how appropriate the translation ${G}_{A}\left(f, z\right)$ is in domain $R$. Here, $z$ and ${z}^{{'}}$ appeared below are both random noises. ${G}_{A}\left(f, z\right)$ then translates back to domain $F$ using ${G}_{B}$, which outputs ${G}_{B}\left({G}_{A}\left(f, z\right), {z}^{{'}}\right)$ as the reconstructed $f$. Similarly, the real result $r$ is translated as ${G}_{B}\left(r, {z}^{{'}}\right)$ and then reconstructed as ${G}_{A}\left({G}_{B}\left(r, {z}^{{'}}\right), z\right)$.The discriminator ${D}_{A}$ is trained with positive samples $r$ and negative examples ${G}_{A}\left(f, z\right)$, whereas ${D}_{B}$ takes $f$ as positive samples and ${G}_{B}\left(r, {z}^{{'}}\right)$ as negative samples. The generator ${G}_{A}$ and ${G}_{B}$ emulate "fake" outputs to blind the correlation between discriminator ${D}_{A}$ and ${D}_{B}$ and minimize the reconstruction loss $\parallel {G}_{A}\left({G}_{B}\left(r, {z}^{{'}}\right), z\right)-r\parallel$ and $\parallel {G}_{B}\left({G}_{A}\left(f, z\right), {z}^{{'}}\right)-f\parallel$ for optimizing the Dual-GAN.

We employ the gradient penalty in WGAN-GP replacing the weight clipping to insure gradient stability in this study. The loss functions of ${D}_{A}$ and ${D}_{B}$ are defined as:

Here, loss function ${L}_{A}^{d}$ is the objective of ${D}_{A}$, and loss function ${L}_{B}^{d}$ is the objective of ${D}_{B}$. $\lambda$ is a hyperparameter, and it is set to 10 in this study. We also define ${\mathbb{P}}_{{\overline x }}$ as sampling uniformly along straight lines between pairs of points sampled from the data distribution (${\mathbb{P}}_{f}$ or ${\mathbb{P}}_{r}$) and the corresponding generator distribution ${\mathbb{P}}_{g}$.

Generator ${G}_{A}$ and ${G}_{B}$ use the same loss function:

where $f\in F$; $r\in R$; ${\lambda }_{F}$, ${\lambda }_{R}$ are two constant parameters within [100.0, 1000.0]. Here, the Manhattan distance is used to measure the recovery error in order to force the reconstruction samples to obey the domain distribution.

4.1.2.   Network configuration

Dual-GAN consists of two generators and two discriminators. The discriminators have the same structure while the generators are different. The architecture of generators ${G}_{A}$ and ${G}_{B}$ is shown in Figure 3.

Generator ${G}_{A}$ adopts the FCN. The input of ${G}_{A}$ is a feature vector, and the output is diagnosis result vector for CAI. Specifically, ${G}_{A}$ is built with a convolutional layer with $5\times 5$ kernel-size and 2 stride-size, 4 convolutional layers with $3\times 3$ kernel-size and 2 stride-size, as well as 3 convolutional layers with $3\times 3$ kernel-size and 3 stride-size. Each layer includes the leaky rectified linear unit (LReLU) activation function and batch normalization (BN) except for the first layer.

Generator ${G}_{B}$ also uses the FCN. But the input of ${G}_{B}$ is diagnosis result vector for CAI, and the output is a feature vector. When the input is a real result vector, a $1\times 1$ convolutional layer is set for matching the dimension between the real result vector and ${G}_{B}$. Generator ${G}_{B}$ is built with a convolutional layer with $1\times 1$ kernel-size, 3 deconvolutional layers with $3\times 3$ kernel-size and 3 stride-size, 4 deconvolutional layers with $3\times 3$ kernel-size and 3 stride-size, as well as a convolutional layer with $5\times 5$ kernel-size and 2 stride-size. Each layer included the rectified linear unit (ReLU) activation function and BN except for the last layer. The last layer configures a TanHyperbolic (tanh) activation function to output the generated feature.

The generators ${G}_{A}$ and ${G}_{B}$ make up a U-shaped generator net with skip connections for learning both high- and low-level features. So, we can establish contact between the gait characteristics and CAI detection results through these skip connections.

The architecture of discriminators ${D}_{A}$ and ${D}_{B}$ is shown in Figure 4. When the input is a real result vector, a $1\times 1$ convolutional layer is set for matching the dimension between the real and generated result vectors. Discriminators ${D}_{A}$ and ${D}_{B}$ consist of a convolutional layer with $5\times 5$ kernel-size and 2 stride-size, 4 convolutional layers with $3\times 3$ kernel-size and 2 stride-size, as well as a fully connected layer. The LReLU activation function follows each convolutional layer, and BN is configured at layers 2–5.

4.1.3.   Model training

We used mini-batch Stochastic Gradient Descent and applied the Root Mean Square Propagation (RMSProp) solver to optimize this Dual-GAN model. The batch size was set to 8 and the dropout rate was 0.3. The learning rate $\alpha$ and the attenuation rate $\beta$ were set to 0.001 and 0.9, respectively. We adopted Manhattan distance to measure the recovery error between synthesized samples and real data. The gradient penalty replaced weight clipping in the loss function for high-quality data generation. The Dual-GAN training procedure is shown in Figure 5.

We randomly selected a group of real characteristics and synthesized samples using Dual-GAN for visualization (Figure 6). The parameters of Dual-GAN are initialized at epoch 0. After 50 epochs, the network begins to learn the general trend of spatiotemporal and kinematic characteristics in the gait cycle. After 100 epochs, the generator can synthesize an unsmooth curve closing to real characteristics. While training after 150–200 epochs, the synthesized sample resembles the real data, but it seems relatively oscillatory. After 250 epochs, the Dual-GAN is convergence, and the synthesized sample coordinate with the real gait characteristics. In the following section, we will feed the synthesized samples to the detection model and verify the synthesized samples using the t-SNE algorithm.

4.2.   CAI detection

The network architecture based on LSTM learns the long-term sequence characteristics of spatiotemporal and kinematic variables well. We adopted three LSTM-based deep learning models, including LSTM, LSTM-FCN, and Convolutional LSTM to detection CAI (Figure 7) ^{[36, 37, 38, 39]}. The synthesized and real data were mixed for training the detection model.

4.2.1.   LSTM-based detection model

LSTM can solve the problem of gradient disappearance, especially in the long-distance dependence task. We used LSTM to CAI detection in this study. As shown in Figure 7, the LSTM cell is composed of an input gate, an output gate, and a forget gate. The cell traces the dependency between features in the input sequence. The input gate selectively retains the information from the previous time step. The forget gate gives up some input information from the previous node. The output gate controls the current cell state output to the next cell. The training procedure of the LSTM-based detection model is shown in Figure 8. The activation function of LSTM is the logistic sigmoid function. The input gate $i$, forget gate $f$, and output gate $O$ are:

where ${X}_{t}$ is the input of current time; ${h}_{t-1}$ is the output in the previous time; ${C}_{t-1}$ is the cell status in the previous time; $W$ is the connection weight; $b$ is bias item; $\sigma$ is a sigmoid activation function; $\mathrm{tanh}$ is a hyperbolic tangent function.

4.2.2.   LSTM-FCN-based detection model

As shown in Figure 7, the LSTM-FCN model in this study includes a fully convolutional block and a LSTM block. The fully convolutional block is comprised of three stacked temporal convolutional blocks and a global average pooling layer. The LSTM block consists of a dimension shuffle layer, two general LSTM sub-modules, and a dropout layer. The dimension shuffle layer can reduce the training time improving the model's efficiency. The training procedure of the LSTM-FCN-based detection model is shown in Figure 9.

4.2.3.   Convolutional LSTM-based detection model

The Convolutional LSTM (Figure 7) not only possesses the capacity of temporal modeling from LSTM but also can describe spatial local features. This is because the Convolutional LSTM model applies convolution structure in both input-to-state and state-to-state transitions to extract spatial features. The "peephole connection" in each gate is employed to supervise the cell state. The training procedure of the Convolutional LSTM-based detection model is shown in Figure 10.

5.   Experiments and results

5.1.   Data feeding strategies

We designed three data feeding strategies (DFSs) for training the CAI detection model and comparing their outcomes. The first data feeding strategy (DFS1) only included significant spatiotemporal characteristics (basic gait variables, velocity variables, and micro-adjustment variables). The second data feeding strategy (DFS2) only included kinematic characteristics (ROM variables). The third data feeding strategy (DFS3) included both spatiotemporal and kinematic characteristics.

Moreover, we created three sub-DFSs for comparing the effects of the synthesized data on the detection model. The first sub-DFS (DFS*-1), as same as the training set for the Dual-GAN, only included real characteristics. The second sub-DFS (DFS*-2) included the real characteristics plus 200 synthesized data for each group. The third sub-DFS (DFS*-3) included the real characteristics plus 1000 synthesized data for each group.

5.2.   t-SNE algorithm-based synthesized data verification

The t-SNE algorithm is a nonlinear descending dimension method for visualizing the high-dimensional spatiotemporal and kinematic characteristics ^[40]. What t-SNE does is present a way to project high-dimensional data into a low-dimensional space ^[41]. The data point on the 2-dimensional plane is attracted to data it is semblable in high-dimensional space and repelled by data it is unlike ^[42]. Figure 11 shows the visualization of 132 real control data (), 20 real injury data (), 200 randomly selected synthesized control data (), and 200 randomly selected synthesized injury data () reported by t-SNE. Here, the perplexity was 30 to keep a balance between local and global features.

The synthesized data overlapped with the same classification of real data by learning the distribution from the control group and injury group, respectively. The distribution between real and synthesized data for tibiotalar flexion (B), lateral arch angle (E), forefoot/ankle supination (G), MT I-V angle (H), and lateral malleolus scale (I) were more similar than the others. Compared to patients, the synthesized data of control subjects () were more consistent with the real () due to their routine gait rhythm and pattern. The synthesized data is generally in good shape for training the CAI detection model.

5.3.   Ankle instability detection

The CAI detection is a binary classification including CAI and non-CAI. Here classification metrics such as accuracy, precision, recall (sensitivity), and f1-score (balanced score) are selected to evaluate the performance of the CAI detection modes. the relevant components including $TP$ (the number of correct predicting non-CAI samples), $TN$ (the number of correct predicting CAI), $FP$ (the number of false predicting non-CAI), and $FN$ (the number of false predicting CAI). The definitions of the classification metrics are explained below.

Accuracy is the proportion of all correctly predicted samples to total samples for measuring the performance of a detection model.

Precision is the proportion of all correctly predicted non-CAI samples to all predicted non-CAI samples.

Recall is the proportion of all correctly predicted non-CAI samples to all real non-CAI samples.

The f1-score is the harmonic mean of precision and recall measuring their difference.

5.3.1.   LSTM-based detection

The LSTM-based detection model in this study consisted of an input layer, two hidden layers, and an output layer. For each input/hidden layer, there were 32 neurons. The output layer was a Softmax classifier with two nodes to output the result of detection (CAI/non-CAI). We adopted a binary cross-entropy loss function after the output layer to estimate the consistency between the outcome and true value. Here the RMSProp solver was used to restrain the swing amplitude and speed up the convergence during the gradient descent. The iteration was set to 300 and the batch size was 32 in model training. The accuracy, precision, recall, and f1-score for CAI detection using this LSTM model are presented in Table 1.

5.3.2.   LSTM-FCN-based detection

We tried to replace the LSTM module with LSTM-FCN in the CAI detection model for improving the classification performance. In this model, the LSTM and FCN sub-module perceived the same input in different views. The LSTM sub-model included a dimension shuffle layer and two LSTM cell layers followed by a dropout layer. The shuffle layer for receiving the input was a multivariate time series with a single time step. For each LSTM cell layer, there were 32 neurons. The FCN sub-module consisted of three stacked convolutional blocks with filter sizes of 128, 256, and 128 respectively and a global pooling layer. Each convolutional block comprised a temporal convolutional layer accompanied by BN (momentum was 0.99, epsilon was 0.001) and a ReLU activation function. The output of the dropout and global pooling layer was concatenated and input into the Softmax classifier. Here the iteration was set to 50 and the batch size was 32 in model training. The accuracy, precision, recall, and f1-score for CAI detection using this LSTM model are presented in Table 2.

5.3.3.   Convolutional LSTM-based detection

Considering the temporal-spatial characteristics of gait variables, we further adopted the Convolutional LSTM-based model detecting the CAI. The convolutional layer with filter size 64 and kernel size $1\times 3$, and was accompanied by a ReLU activation function. The convolutional layer was followed by a dropout layer (dropout rate was 0.5) and a flattened layer for inputting a dimensional array into the Softmax classifier. Here the iteration was set to 25 and the batch size was 64 in model training. The accuracy, precision, recall, and f1-score for CAI detection using this LSTM model are presented in Table 3.

6.   Discussion

Based on the results of spatiotemporal and kinematic characteristics augmentation using Dual-GAN, we employed three LSTM-based models (the LSTM-based model, LSTM-FCN-based model, and Convolutional LSTM-based model) in the study to compare their performance and reveal the effect of synthesizing data on CAI detection. The detection outputs by LSTM, LSTM-FCN, and Convolutional LSTM models under three DFSs are shown in Tables 1–3 and the Receiver Operating Characteristic (ROC) curves for the CAI and non-CAI classification are shown in Figure 12.

We took a step to optimize the feeding by pre-selecting crucial features for intelligent detection in our previous study ^{[16, 17]}. The spatiotemporal and kinematic characteristics in gait were both effective to predict CAI. However, the detection performance using the model trained by only real spatiotemporal characteristics (DFS1) was unsatisfactory (Area Under Curve (AUC): 0.122–0.718). When we used real kinematic characteristics (DFS2) for training, the outcome was improved (AUC for DFS2: 0.923–0.995). When we added real kinematic characteristics to real spatiotemporal characteristics (DFS3) for training, the outcome was also good (AUC for DFS3: 0.738–0.987). In general, the training set of only the real data did not yield adequate classification outcomes. The accuracy of DFS2 and DFS3 were all 88.76% for LSTM, LSTM-FCN, and Convolutional LSTM-based detection models. The accuracy, precision, recall, and f1-score were moderate.

So, we further included 200 and 1000 synthesized characteristics in the training set, and the detection accuracy, precision, recall, and f1-score improved dramatically. Adding 200 synthesized data for training, the outcome of the detection model was improved. When we included 1000 synthesized data, the detection accuracy, precision, recall, and f1-score improved dramatically for either or both spatiotemporal and kinematic characteristics. The volume of training samples is crucial for the models' detection performance. Spatiotemporal and kinematic characteristics augmentation using Dual-GANs can improve the diagnostic accuracy of the CAI detection models presented in this study.

When we added 200 synthesized data to the real data set training the LSTM-based CAI detection model (Table 1), the accuracy was improved (from 74.16% to 86.62% for DFS1; from 88.76% to 94.38% for DFS2 and DFS3). After adding 1000 synthesized data, the accuracy for the three DFSs was more than 95% (DFS1: 98.88%, DFS2: 100%, DFS3: 95.51%), and the precision, recall, and f1-score also look good.

For LSTM-FCN-based CAI detection model (Table 2), the accuracy was improved (from 83.15% to 87.64% for DFS1; from 88.76% to 96.63% for DFS2; from 88.76% to 94.38% for DFS3) because of adding 200 synthesized data (DFS*-2). Adding 1000 synthesized data (DFS*-3) enhanced the detection accuracy of DFS1 (from 87.64% to 94.38%) and DFS3 (from 94.38% to 95.51%), but there was no positive effect on DFS2.

The results in Table 3 reveal that the performance of the Convolutional LSTM-based classifier is better than LSTM and LSTM-FCN-based CAI detection model. The Convolutional LSTM-based detection model showed better performance for spatiotemporal and kinematic characteristics. Adding more synthesized data helped the Convolutional LSTM-based classifier enhance the detection performance. when we added 1000 synthesized data to the training set, the accuracy of DFS1, DFS2, and DFS3 was up to 94.38, 100, and 100% respectively. Their precision, recall, and score were all superior. The detection results were good, but several limitations of the current study need to be noted. First, the high-quality gait information from the optical motion capture system helped us find several significant spatiotemporal and kinematic characteristics of CAI enhancing the detection efficiency, but the data collection is time-consuming. At present, wearable devices, such as smartphones, wearable biofeedback sensors, and sensing fabrics, are widely applied to gait monitoring ^{[43, 44]}. Unlike laboratory-based motion tracking, wearable devices are convenient to record daily gait data. Based on our current results, we will explore effective gait features from the data recording by wearable devices. It is possible to get more practical applications for our studies, such as multi-sensor-based human activity recognition, bipedal walking robot, exoskeleton, et al. ^{[45, 46, 47, 48]}. Second, this paper focused on the performance of LSTM-based models for ankle instability detection. We will explore more CAI detection models based on characteristics augmenting and combine them to improve the detection effectiveness in our further research.

7.   Conclusions

In this paper, we first present an approach for augmenting spatiotemporal and kinematic characteristics using the Dual-GAN to train a series of modified LSTM detection models to make the training process more data-efficient. The Dual-GAN enables the synthesized data to approximate the real data distribution visualized by the t-SNE algorithm.

Then we use LSTM-, LSTM-FCN-, and Convolutional LSTM-based detection models training by the real data collected from the controlled laboratory study and mixed data from real and synthesized gait features respectively to identify the patients with CAI. The detection models were tested in real data to validate the positive role in data augmentation as well as to demonstrate the capability and effectiveness of the modified LSTM algorithm for CAI detection using spatiotemporal and kinematic characteristics in walking.

The experimental results show that the proposed data augmentation method promoted the detection performance in cases of the limited real dataset and the modified LSTM algorithm yielded an enhanced classification outcome to identify those CAI patients from a group of control subjects based on gait analysis data than any previous reports. This is because the Dual-GAN addresses the insufficient limitation of the training set, and the Convolutional LSTM-based detection model works well both in spatial and temporal features during walking.

The techniques proposed in this study develop a new way to extract significant features from a small clinical gait analysis dataset to improve computer-assisted diagnosis on CAI patients. This is a concrete step toward the long-term goal to develop artificial intelligence-based instruments for clinical diagnosis and rehabilitation. In subsequent studies, we will expand our research to posture control in more sports situations, such as running, jumping, and cutting, et al., and improve the performance of deep learning models to enhance the medical treatment in sports medicine.

Acknowledgments

The publication of this paper is funded by the Peking University Third Hospital—Haidian Innovation and Transformation Project under Grant Y74482-09, the National Natural Science Foundation of China under Grant 61801019, the China Scholarship Council under Grant 201906465021, and the Fundamental Research Funds for the University of Science and Technology Beijing under Grant FRF-BD-19-012A.

Conflict of interest

The authors declare there is no conflict of interest.