Hybrid self-supervised monocular visual odometry system based on spatio-temporal features

1.   Introduction: Background and motivation

1.1.   What FMR is

Flight Maneuver Recognition (FMR), the standard terminology given by the Federal Aviation Administration ^[1], refers to a series of flight states of an aircraft under the control of the pilot. The raw data of the maneuvers obtained from onboard recorders allows for the identification of trends and patterns in operations as well as further identification of basic movements, complex maneuvers, and even training subjects.

There are many types of aircraft, including airplanes, spacecraft, rockets, and missiles, which can be divided into fighter airplanes and civilian airplanes. According to the principle of power, aircraft can be divided into lighter-than-air aircraft (balloons, airships) and heavier-than-air aircraft, including fixed-wing aircraft (aircraft, gliders), rotary-wing aircraft (helicopters, rotorcraft), flutter aircraft, and tiltrotor Rotorcraft. These aircraft are called Unmanned Aerial Vehicles (UAVs) or Unmanned Combat Air Vehicles if they do not carry people and are operated by radio-controlled equipment and self-contained program control devices.

A complete flight mission is defined as starting from the aircraft's start engine via taxiing, taking off, and performing a series of maneuvers, and then landing and shutting down to finish, which is a complex process, and this process contains a series of complex maneuvers, and these complex maneuvers are made up of a series of fundamental maneuvers.

An airplane rotates in bank, pitch, and Yaw angle (YAW) while also moving horizontally, vertically, and laterally. The four fundamentals (straight-and-level flight, turns, climbs, and descents) are the principal maneuvers that control the airplane through the six motions of flight. As they have the same handling principles, different types of aircraft have similar fundamental maneuvers, also called basic maneuvers.

The combination of several basic maneuvers makes up complex maneuvers, flight subjects, and procedures. A takeoff is a combination of a straight-and-level flight and a climb. Turning on course to the first navigation fix after departure is a climb and a turn, and the landing at the destination is a combination of airplane ground handling, acceleration, pitch, and a climb. Owing to their considerably different structural limitations and missions, different types of aircraft have different complex maneuvers.

FMR, sometimes referred to as flight action or activity identification, provides a systematic tool for the automatic recognition of flight maneuvers from raw onboard data. The raw data, which is automatically recorded at a certain frequency in devices such as the Quick Access Recorder (QAR) ^[6], Flight Data Recorder (FDR), Cockpit Voice Recorder (CVR) (Figure 1), Health Usage Monitoring System (HUMS), and Integrated avionics systems. These are generated from a variety of onboard sensors in a complex human–machine environment, which is a nonlinear chaotic dynamic system with continuous noise and complex deformation. This is why fast and reliable automatic recognition of flight maneuvers is difficult.

1.2.   Why we need FMR

FMR, a kind of pattern recognition, is a precise understanding of the current aircraft status used to predict the next situation. It is considered to be an active research area owing to applications such as improving pilot training, flight safety, and autonomous air combat.

1) Pilot Training

The pilot training includes simulator training and airplane training. Manual evaluation by coaches based on experience is the main evaluation method, which is subjective, qualitative, expensive, and inefficient. Automated evaluation using artificial intelligence (AI) represents the future, which is objective, quantitative, convenient, and efficient ^[7]. In order to achieve the goal, it is necessary that A. the data source is sufficient and accurate to reflect the flight process correctly, and B. the maneuvers made by the pilot are automatically recognized. FMR is necessary for automatic assessment in pilot training.

FMR is also necessary for air combat flight training. In the 1970s, for one-on-one air-to-air combat training, NASA developed an adaptive maneuvering logic computer program (AML) ^[8,9], which provides a virtual competitor for human pilots at NASA Langley Research Center's Differential Maneuvering Simulator. As AI, AML recognizes the maneuvers and intentions of the opponent and makes the right decisions to drive the next maneuvers.

2) Flight Safety

As the primary means of flight safety, an aircraft's load and load-related requirements determine the safety factor ^[10,11,12]. Measuring maneuvering loads is necessary for aircraft design and certification. Brandt et al. ^[13] examined how the Navy could process raw parameter data generated by HUMS to identify the maneuvers flown so as to support the structural monitoring function.

3) Autonomous Air Combat

Since AML was introduced, and with the development of the UAV, autonomous air combat is developing rapidly. Similar to ^[8], real-time accurate identification of enemy aircraft maneuvers is the key. The difference is that vision-based FMR or a kind of laser-based detection FMR is needed by UAVs.

In summary, FMR is a key technology in many fields and is an important research topic in pattern recognition and pervasive computing. It is not the end goal but is an important and necessary step.

Theoretically, the same maneuver should have a similar pattern, but owing to the complex subjective and objective conditions of the aircraft, a considerable gap exists between the raw data generated by the same flight maneuver in terms of data values and duration. It is impossible to directly identify the specific flight maneuver from the original raw data. The question of how to efficiently and automatically identify an accurate flight maneuver is a common problem in many research fields.

Aiming to discuss state-of-the-art FMR techniques, we considered classification approaches, their advantages, challenges, datasets, and applications of FMR. This paper discusses machine learning and deep learning techniques for FMR and gives a comprehensive measurement based on a unified data set. It also includes challenges and potential future work based on FMR in terms of machine learning, deep learning, and hybrid techniques.

Essentially, the original flight data are typical of temporal series data, and FMR is a pattern recognition problem. The most similar to FMR is sensor-based Human Action Recognition (HAR). Owing to limited domain knowledge and data sources, compared to HAR's booming research ^[2,3,4], research on FMR is lacking and limited to a few institutions.

Although some related surveys have been conducted in flight data mining ^[5], there has been no survey focusing on FMR areas. To the best of our knowledge, this is the first paper to present the latest research on FMR. We hope this paper can provide a helpful summary of existing work along with the challenges and potential future research directions.

This paper is organized as follows. Section 2 introduces the comparison to HAR. Section 3 lists and summarizes the various methods and models for FMR and gives a comprehensive measurement based on a unified data set. Section 4 covers the challenges and future directions, and concluding remarks and a summary are given in Section 5.

2.   Comparison with HAR

Similar to HAR, FMR is essentially a pattern recognition problem. HAR and FMR are not the end goal, but an important and necessary step in various applications. Research on FMR has been lacking owing to limited domain knowledge and data sources compared to HAR's booming research. Moreover, HAR involves human activity monitoring tasks in different areas, such as medicine, education, entertainment, visual surveillance, video retrieval, and abnormal activity identification. Because of an increase in the usage of cameras and sensors, there are two kinds of HAR: video-based and sensor-based. Both in terms of research subjects and methods, FMR is similar to HAR. Therefore, this is a brief description of the state of HAR and a comparison with FMR.

According to Poppe ^[14], Pareek and Thakkar ^[15], vision-based HAR is the process of labeling image sequences with action labels. Robust solutions to this problem have applications in domains such as visual surveillance, video retrieval, and human-computer interaction. Various machine learning and deep learning techniques for HAR have been introduced. According to Wang et al. ^[16], compared to conventional pattern recognition approaches, deep learning-based methods as unsupervised and incremental learning have been widely adopted for the sensor-based HAR.

This section focuses on the following aspects to compare FMR and HAR: 1) research goals to achieve, as well as the data source and datasets, and 2) a process roadmap.

For convenience, we discuss four items here: sensor-based HAR and FMR and vision-based HAR and FMR.

2.1.   Goals and data

Essentially, the research goals of HAR and FMR are different. HAR's research target is humans, while FMR's is the airplane; HAR's goal is automatic recognition of various human actions from data sources and FMR's goal is flight maneuvers. Whether sensor-based or vision-based, FMR and HAR both have stable sources of data generation; the differences are data format, capacity, and frequency. Specifically, the current state of existing public datasets is different ^[17]. The specific comparison is shown in Table 1. Note that there is no review of FMR, so the review is chosen for HAR, and the specific reference is chosen for FMR.

The comparison shows that HAR has rich datasets, while FMR has none, which means that research has been limited, and FMR lacks a standard baseline. Therefore, we experimented with, analyzed, and compared the above methods with our unified dataset. As far as we know, this is the first time such an effort has been made.

Our dataset was generated by fixed-wing general aviation aircraft during flight training, and includes sensor-based onboard raw data and vision-based onboard videos. Other methods, with the exception of rotorcraft and UAV studies, could not be replicated. See Section 3 for details.

2.2.   Process roadmap

Figures 2 and 3 present a typical flowchart of sensor-based HAR ^[4] and vision-based HAR ^[15]. Figures 4 and 5 present a typical flowchart of sensor-based FMR and vision-based FMR.

Figure 2 is from ^[4]; the left picture presents a typical flowchart of HAR using conventional PR approaches, and the right picture shows how deep learning works for HAR with different types of networks.

As shown in Figure 3, Pareek and Thakkar ^[15] provide an overview of the general steps, including data collection, pre-processing, feature extraction and encoding, dimensionality reduction, action classification, and the final analysis.

Figure 4 presents an illustration of vision-based FMR for onboard equipment, including flight video and image collection, feature extraction to detect the horizon, model training, and inferences of the maneuvers. In particular, it is important to note that the vision-based FMR this paper refers to is based on airborne equipment, so it does not include surveillance equipment, such as airfield surveillance equipment and satellites.

Figure 5 presents an illustration of sensor-based FMR (also limited to airborne equipment), including flight raw data collection, preprocessing, feature extraction, model training, and inferences of the maneuvers. As an expertise field, the use of expert knowledge is woven throughout the process. As a typical multivariate time-series analysis problem, traditional pattern matching methods, expert knowledge inference machines, and machine learning and deep learning methods can be used.

3.   FMR method and model

It can be seen that vision-based FMR is based on horizon detection to estimate flight attitude and identify related objectives. Sensor-based FMR is based on huge volume time-series raw data processing in order to extract spa-tio-temporal features and represent and distinguish flight maneuvers. This section introduces the state-of-the-art methods and measures those models with our unified dataset. As our dataset was generated by fixed-wing general aviation aircraft during flight training, we included sensor-based onboard raw data and vision-based onboard videos. Other methods, with the exception of rotorcraft and UAV studies, could not be replicated.

3.1.   Vision-based FMR

3.1.1.   Sky/ground segmentation method

In terms of the shape of the Earth, when the flight altitude reaches a certain height, the general airborne camera cannot capture the sky/ground horizontal line, so vision-based FMR is limited to visual flight or the low-altitude phase. The advantages are that theycan be used for real-time attitude analysis during critical visual flight phases (e.g., takeoff, landing) and for UAV control stabilization or intelligent mission profiles.

Todorovic and Nechyba ^[31] introduced a unified computer vision framework in 2004. In the method, first, onboard camera shots are used as input, and real-time image feature extraction is done to discriminate the horizon. Second, the sky and ground are split using tree-structured belief networks. Lastly, real-time recognition of artificial targets is done for tracking-like tasks. Horizon detection and tracking are key for this method, and the multiscale linear discriminant model is used for real-time feature extraction. The result of horizon detection is a range of images differing in various features. In order to deal with the distortion caused by the wide angle lens widely used in UAVs, Lukács ^[33] presented an adaptive contrast detection method based on horizon recognition for attitude determination. Through these studies, we found that the horizon detection-based approach could identify the aircraft attitude in real time, not only for UAVs but also for low-altitude aircraft. Therefore, we performed recognition experiments on general aviation training cockpit videos, and the details and results are as follows.

3.1.2.   Experimentation and analysis

The experimental material was taken from the cockpit video surveillance of an actual training flight, and a frame was taken during the low-altitude flight phase, as shown in Figure 6(a), The restored flight state is shown in Figure 6(b) using our visualization tools.

The whole approach is as follows: 1) Determine the threshold using Gaussian Blur, image binarization, and Otsu's method; 2) identify the horizon line; 3) calculate the slope k; 4) do correlation analysis of label values (Roll) and eigenvalues (k and Pitch) to output correlation coefficient matrix; and 5) model the relationship between label values (Roll) and eigenvalues (k and Pitch). After processing, we captured the actual horizon outside the cockpit, as well as the indicated horizon on the attitude director indicator inside the cockpit (Figure 7).

3.2.   Sensor-based FMR

3.2.1.   Background

The vast majority of FMR research has focused on sensor-based FMR, which originated in the field of aircraft design and manufacturing and is widely used in flight accident investigation, safety monitoring, and pilot training evaluation.

Although there are some differences in data formats for different types of aircraft with different purposes, all have the characteristics of accessibility, source stability, high frequency, rich information, and continuous time series. As an example, the unified dataset used in this paper, named Civil Aviation Flight University of China (CAFUC), is presented here.

The CAFUC dataset was generated from realistic fixed-wing aircraft flight training at CAFUC and contains 64 parameters, four artificial labels, 150,000 total frames, 41.6 hours of total flight duration in CSV format, 14356 basic maneuvers, and 168 flight training subjects. A sample is given in Table 2.

Other methods, with the exception of rotorcraft and UAV studies, can be replicated. In order to unify standards and establish a baseline, this dataset was used for all model measurements described in this section.

Essentially, FMR is a multiple nonlinear time-series pattern recognition problem. Pattern recognition problems mainly include classification and clustering.

In the time-series classification problem, feature volume construction and classifier design are the core problems. Time-series classification aims to take the whole time series as input, and its purpose is to assign a discrete label to this series. It is more difficult than the general classification problem owing to the unequal length of the classified time-series data, which makes it impossible to apply the general classification algorithm directly.

In order to solve these difficulties, there are usually two approaches. First, define the appropriate distance degree, such as Dynamic Time Warping ^[82], such that sequences that are similar in the sense of this metric have the same classification labels, which are domain-independent methods, such as the 1) distance-based method.

Second, using knowledge rules or context-dependent modeling, each sequence is represented by an equal-length and same-dimension feature vector of model parameters and then trained and classified by a conventional classification algorithm, which is a domain-related approach called the model-based method. In general, model-based FMR methods can be divided into four categories: 2) feature extraction-based, 3) expert knowledge rule-based, 4) probabilistic graphical model-based, and 5) neural network-based.

Naturally, hybrid methods combining multiple methods have also been proposed, and they are called 6) hybrid methods.

In addition, some scholars have also conducted FMR from the perspective of 7) clustering.

3.2.2.   Related definitions

The flight raw dataset without labels is defined as

The flight dataset with labels is defined as

These datasets can be divided into several time series according to the division granularity, with each series representing a flight maneuver (i.e., corresponding to a maneuver label):

The goal of sensor-based FMR is to divide these datasets into sequences and identify the labels corresponding to these series by classification, clustering, or pattern recognition:

By definition, the length of ${\boldsymbol{D}}_{\boldsymbol{u}}^{\left(\boldsymbol{k}\right)}, {\boldsymbol{D}}_{\boldsymbol{l}}^{\left(\boldsymbol{k}\right)}$ and ${\boldsymbol{S}}_{p}^{\left(\boldsymbol{k}\right)}$ is variable, and MC as the flight maneuvers class collection is artificially customized.

According to the requirements of the International Civil Aviation Pilot Practical Examination and MIT Challenge ^[83,84], combined with the dataset content restrictions, we set two MC collections, ${MC}_{\beta }$ and ${MC}_{\gamma }$, which are listed as follows for uniformity as Table 3. ${MC}_{\beta }$ contains basic maneuvers, such as Level. ${MC}_{\gamma }$ contains complex maneuvers, such as Rectangular Course. As complex maneuvers are composed of basic ones, the two collections are mutually exclusive and should be validated separately.

3.2.3.   MPSR

Zhang and Zhang ^[84] concluded that flight data have typical chaotic characteristics. Chaos is the pseudo-randomness of deterministic systems, and therefore FMR can be considered from the perspective of a deterministic nonlinear dynamical system. If a time series is generated by a deterministic nonlinear dynamical system, the recovery and inscription of the original dynamical system by the time series are called phase space reconstruction (PSR), and the multivariate recovery is called multiparameter phase space reconstruction (MPSR). The most commonly used method is the Takens Delayed Embedding Theorem, also known as Takens Theorem (Takens Theorem). According to Qu et al. ^[50], MPSR based FMR flowchart is as Figure 8.

Experimental results show that the dataset as a multivariate time series does fit the chaotic nonlinear dynamical system characteristics. Similar maneuvers show similar characteristics on the recurrence graph, with close values of approximate entropy (ApEn), while different maneuvers vary widely. Thus, the phase space reconstruction recognition method based on approximate entropy can distinguish the recognition of flight maneuvers, especially complex maneuvers. The ApEn results are given in Table 4, and a samples of the trace recovery visualization and recurrence map experiment are given in Figure 9.

Based on the combined analysis of the results in Table 5, it can be inferred that the number of parameters involved in the calculation is proportional to the difference between the approximate entropies of the different types of maneuvers, and different maneuvers have a unique ApEn and a large gap between each other. In practice, after optimizing the parameters to further increase the difference, this is enough to distinguish maneuvers.

3.2.4.   RF-SVM

Yang et al. ^[42] proposed an FMR method based on fuzzy least-squares support vector machines (FLS-SVM). It defines five kinds of flight maneuvers by 10 parameters, which are relative barometric altitude (H), one-second change in relative barometric altitude ($\Delta H$), aero pitch ($\theta$), one-second change in track pitch ($\Delta \theta$), slope angle ($\gamma$), one-second change in slope angle ($\Delta \gamma$), heading angle ($\phi$), one-second change in heading angle ($\Delta \phi$), indicated airspeed ($v$), and indicated airspeed change per second ($\Delta v$). We added the parameter NormAC to the 11 parameters, and defined 12 kinds of flight maneuvers tagged by the y label.

Jia et al. ^[68] proposed a two-stage recognition method for online FMR. The first stage involves the extraction of the trajectory parameters to identify basic maneuver elements using RF, called meta tags. The second stage identifies tactical maneuvers using support vector machine (SVM) with the sequence of generated maneuver elements and motion parameters from the previous level. According to Jia, online identification means reading the radar real-time data frame by frame. Based on the read and calculated feature values, it is determined whether a change in the maneuver basic element has occurred. If the maneuver basic element changes, the recognition of the maneuver basic element is performed using random forest, and then the recognition of the maneuver is performed using SVM, the total flow chart as Figure 10.

We compared the accuracy of the adopted random forest classifier with the non-integrated decision tree classifier and the Adaboost integrated decision tree classifier, where CART is used for the decision tree classifier and CART is also used for the basic classifier of Adaboost. Because of the iterative update mechanism used by Adaboost, the random forest classifier was faster to train than Adaboost, and because the random forest includes multiple decision trees, the training time for a single decision tree was much less than that of the other two algorithms. The results are shown in Table 5. For specific experimental results, see: Supplementary file - Experimental results.

3.2.5.   Expert-System

Y. Wang et al. ^[57] established a characteristic data library for all types of maneuvers based on the idealization standard time history of each maneuver type so as to conduct automatic maneuver identification. The overall approach was: 1) Data splitting, 2) Standard maneuver definition, 3) Rules serialization, 4) Comparison with the Expert Knowledge Database, 5) Regular expression matching, and 6) Result output.

We selected five parameters, which are Longitude, Latitude, Altitude of Mean Sea Level (AltMSL), Vertical Speed (VSpd), and Heading (HDG), used the Savitzky-Golay filter for preprocessing, 55 window lengths for the lift rate, 11 window length for HDG, and third-order polynomial fit.

We divided the flight maneuvers into ascent (replaced by the letter A), descent (replaced by the letter B), and turns (replaced by the letter C), where the permutations of flight maneuvers are represented by strings, and turns are divided into C1 (30-degree turn), C2 (60-degree turn), C3 (90-degree turn), C4 (180-degree turn), and C5 (360-degree turn). The main research method was to discriminate the flight maneuvers and form the maneuver sequence characteristics of the flight subjects, which were matched with the maneuver sequences in the expert database. For example, as shown in Figure 11, the green segment represents climbing, the blue segment represents level flight, the yellow segment represents turning, and the red segment represents descending.

The results show that the maneuver sequences cannot be matched because flightdata1 splits the descent phase into two segments, and the maneuver sequences of flightdatas 2 and 3 match the action sequences of rectangular routes, so they can be identified as Rectangular Course. For specific experimental results, see: Supplementary file - Experimental results.

3.2.6.   DBM

Meng et al. ^[70] explored the relationship between maneuvers and feature parameters and built a dynamic bayesian network model (DBM) for maneuver recognition based on such a relationship. Meng et al. ^[70] used five parameters as features: Altitude (ALT), Altitude Change Rate (ALR), YAW, heading angle change rate (YAR), and speed (VK). According to experience, the specific relationship between various maneuvers and these five parameters was summarized. The selected five parameters were used as the observation nodes of the dynamic Bayes network model, and the root node was the recognition result of maneuvers. The intermediate nodes of the network were established according to the feature classification of the flight parameters and their dependence on the maneuvers, and then the dynamic DBM shown in Figure 12 was established for FMR. Considering the differences in aircraft types and missions, the probability matrix used in our experiments was set for the CAFUC dataset corresponding to the model and task, and the setting criteria were also expert experience. In addition, compared to Meng et al. ^[70] experiments, the maneuver types and data set timing lengths were different.

Experiments showed that the method could quickly identify flight maneuvers without prior subsequence segmentation.

3.2.7.   CNN-LSTM

Fang et al. ^[67] proposed a method based on a symbolic neural network to realize efficient FMR for both basic flight maneuvers and complex flight maneuvers. The overall approach was: 1) Input training data; 2) extract the flight feature matrix as the training set; 3) slice the flight parameters by differential segmentation; 4) build and train the convolutional neural networks (CNN) model to output the basic maneuvers weight, ${\rho }_{i}^{basic}$; 5) decompose complex maneuvers to basic maneuvers by redoing step 1 and step 2 with basic maneuver weight; 6) do symbolic basic flight maneuvers; 7) build and train the long short-term memory (LSTM) model to output complex maneuvers weight (${\rho }_{i}^{Complex}$); 8) input test data; 9) extract the flight feature matrix as the test set; 10) slice the flight parameters by differential segmentation; 11) decompose the test set to meta flight maneuvers with step 4's basic weight (${\rho }_{i}^{basic}$); 12) determine if there are complex maneuvers by manually setting thresholds; 13) output the identity-specific categories directly if they are basic; and 14) output the identify specific categories with ${\rho }_{i}^{Complex}$ if it is complex. The flow chart as Figure 13.

Our experiment used 23,254 rows of data from CAFUC, 68 columns of data per row, and a total of 15,812,272 numbers. Using the random forest algorithm to do correlation analysis on y labels.

According to the importance, we selected the top 10 correlation coefficient features, which are Oil Temperature (OilT), Oil Pressure (OilP), Waypoint Distance (WptDst), Wind Speed (WndSpd), Wind Direction (WndDr), Latitude, Waypoint Bearing (WptBrg), HDG, Ground Speed (GndSpd), and Track Angle(TRK), for the next step of data preprocessing. The CNN model uses three convolutional layers to depict the flight action features. Each convolutional layer contains convolution, pooling, and normalization for parameter training. After that, a spreading layer is added to flatten the two-dimensional data into one-dimensional data, and a dropout layer is added to discard some repetitive parameters to prevent overfitting. The output layer consists of 13 fully-connected neurons, and calculates the probability of the data being labeled with 13 types of labels, and the activation function is SOFTMAX.

The model was trained using the data after preprocessing. It was trained for 500 iterations, with 25 rows of data fed into the model each time, and training was stopped when the loss value did not decrease after 50 iterations. Adam was used as the optimizer. The accuracy and loss value changed during the training process and are shown in Figure 14.

3.2.8.   ISO-DATA

Zhang et al. ^[49] proposed a method based ISO-DATA clustering. This method extracts maneuver segments from the flight data based on the trend of the normal overload data and uses clustering to group the maneuver segments into several classifications. The overall approach is: 1) Select a parameter as the key parameter; 2) identify trends of the key parameter sequence by Double Window Recognition Algorithm; 3) segment trends into a single trend subsequence; 4) merge adjacent segments as maneuver segments according to certain rules; 5) construct feature vectors to represent each maneuver segments; 6) cluster feature vectors to obtain the set of fragments by ISO-DATA; and 7) determine the correspondence between these sets and maneuvers.

As in the original paper, our experiments used six parameters to construct the feature vectors: Indicated Air Speed (IAS), TRK, AltMSL, Pitch angle (Pitch), Roll angle (Roll), and Normal overload Acceleration Change (NormAC), and NormAC was the key parameter. We used the slope to divide the key into three primitives and output the index of all trend segments with a double sliding window model. The change of flying parameters within a maneuver will not be only in a single trend, but often a combination of multiple adjacent trend segments, which is a maneuver segment. Usually, the aircraft tends to return to a level flight state after only maneuvering maneuvers, so the segment between two level flight segments is identified as a maneuvering segment, and there will be similar maneuvers to level flight in the maneuvering maneuver, so the shortest level flight segment threshold, ${\theta }_{h},$ needs to be introduced when the length exceeds the threshold horizontal maneuvering segment that can be identified as the level flight state. Considering the model differences, ${\theta }_{h}$ was 53. The maneuver segments were time-series data and could not be clustered directly using the ISO-DATA algorithm. The statistical features of the maneuver segments could be extracted, and a feature vector could be used to characterize the maneuver segments before bringing them into the clustering algorithm. The mean and variance of each flying parameter in the maneuver fragment were used to build the feature vector.

Among the six clustering parameters of ISO-DATA, K, L, and I were relatively easy to choose, K = 7, L = 1, and I = 100 were selected in the experiment, and the genetic algorithm was used to find ${\theta }_{n}$, ${\theta }_{s}$, and ${\theta }_{c}$. The final optimal parameter setting values obtained were ${\theta }_{n}$ = 1, ${\theta }_{s}$ = 0.0373 and ${\theta }_{c}$ = 0.0043, and the evaluation result was 6.3823.

The categories and corresponding segments generated by the clustering results are shown in Table 6. The specific maneuvers corresponding to each category are shown in Figure 15.

3.3.   Summary

This section introduces the state-of-the-art methods, and describes how we measured those models with our unified dataset, the CAFUC dataset. The CAFUC dataset was generated by realistic fixed-wing aircraft flight training at CAFUC. Similar to HAR, FMR is essentially a pattern recognition problem, and there are two kinds: video-based and sensor-based. We reproduced and tested six methods: Qu et al.^[50], Jia et al.^[68], Wang et al. ^[57], Meng et al. ^[70], Fang et al. ^[67] and Zhang et al. ^[49]. We summarize the advantages and disadvantages of these methods based on the results in Table 7.

In general, various methods solved parts of the problem of FMR from various perspectives, although none of the methods performed as well as claimed in the original paper with the CAFUC real data set. In addition, most of this literature has no dataset presentation, lacks partial model parameter details, expert design thresholds, and rules, not to mention the different types of aircraft and the different nature of the missions performed. However, in this review, we comprehensively reproduced the typical model analytically, validated its feasibility, and tested the performance with the same dataset. As we know, this is the first time such an effort has been made.

There are several common shortcomings in this field of research, such as poor interpretability of the research process, low visualization of the research results, the lack of a solid theoretical foundation for the whole study, and the lack of systematicity to explain its completeness and correctness.

4.   Challenges and future directions

In this section, we discuss the challenges and future directions of FMR. Unlike ordinary pattern recognition problems, FMR has many similarities to HAR, but the overall state of research cannot be compared to it. In general, the main gap lie in the quantity and quality of datasets and the maturity of the unified common architecture.

4.1.   Datasets

The current state of the HAR dataset is mentioned in Section 2.3, and is characterized by a large number, high quality, and a wide variety. On the contrary, FMR research objects are aircraft, the subjects are various maneuvers and phases of the aircraft, and the data collection method is mainly based on onboard sensors. Therefore, the number of subjects is scarce, the subject behavior is difficult to generate (e.g., maneuvers cannot be changed at will, pilots need high training costs, and each flight maneuver is expensive), data collection is extremely difficult (it needs to consider the aviation industry data usage norms), and each line of data is expensive and requires professional knowledge. Therefore, there are no mature public datasets in the industry, which severely limits the development of FMR research.

4.2.   General architecture

Traditionally, methods from the signal processing domain are used to analyze and extract features from the collected sensor data. Such methods are used for feature engineering, creating domain-specific, sensor-specific, or signal processing-specific features as well as views of the raw data. machine learning models are then trained and evaluated on the processed data. The limitation of this approach is that analyzing the raw data and designing features suitable for the model requires expertise in signal processing and related fields. This expertise is required for each new dataset or sensor data. In most daily HAR tasks, these methods may rely heavily on heuristic manual feature extraction, which is often limited by human domain knowledge. Moreover, these methods can only learn shallow features, which leads to degraded performance for unsupervised and incremental tasks; that is, they require human feature engineering and poor generalization of the trained models. Owing to these drawbacks, the performance of traditional pattern recognition methods is limited in terms of classification accuracy and generalization ability of the models in human activity recognition tasks.

The same is true of FMR. Traditional pattern recognition methods are not directly applicable to FMR, and the performance is limited in terms of classification accuracy and model generalization ability in FMR tasks because of the above drawbacks.

Ideally, learning methods can be used to automatically learn the features needed to make accurate predictions directly from the raw data. Deep neural network models are now beginning to demonstrate their feature extraction capabilities and are gaining performance for human activity recognition tasks. They can perform automatic feature learning from raw sensor data and outperform models trained using feature engineering. Feature extraction and model building processes are typically performed simultaneously in deep learning models. These features can be learned automatically by the network without the need for manual design. In addition, deep neural networks can extract deep high-level representations, which makes them more suitable for complex activity recognition tasks. Therefore, the next step for FMR should be more research on DNN architectures that automatically learn features from raw flight parameters and train models without the need for complex manual knowledge, thus building generic architectures that can be widely used for various flight tasks of various aircraft types.

The urgent need for online real-time identification in the field of aviation security needs to be highlighted in particular. Aviation safety is an important guarantee for humans to carry out safe and reliable flight activities. If we can realize online real-time recognition of flight maneuvers, then combined with aviation safety model and flight expert experience, we can achieve a real-time understanding of the current aircraft state, identify the next state, and judge whether the next state will pose a danger. This way, we can guide the pilot in a timely manner or aid in aircraft automatic recovery, avoiding aircraft destruction and loss of life. This requires predictive models with anomaly detection capabilities, of course. This also creates another challenge: There is an imbalance in the number of anomaly samples, which can be more severe in the aviation industry than in other industries.

In summary, the next step for FMR is to investigate a generic architecture with real-time recognition capabilities. We have already tentatively proposed an architecture and done some experiments. ^[87].

4.3.   Multisource data fusion

The problem of multi-source data fusion arises from the complexity of data association. Data correlation refers to how many flight maneuvers or even other flight activities are associated with certain data, what the data association rules are for different data sources, and what the meaning of the associated data is. Multi-source multimodal data can provide more information than single-source data, and by supporting, complementing, and correcting each other, such data can provide more accurate information. Therefore, the fusion and integration of multi-source data is an important basis for improving the quality of data analysis when high information quality is required, such as comprehensive real-time understanding of flight dynamics.

For example, suppose we have the raw flight parameter data of a certain aircraft at a certain time, along with radar data, weather data, and flight crew information for the corresponding period. We can identify the landing maneuvers among them and evaluate them as a result of the evaluation of that pilot's landing technique ^[85]. We can fuse the wind shear from the meteorological data and analyze how that landing maneuver differs from other maneuvers under such a wind shear edge. We can also fuse radar data to analyze the aircraft trajectory and determine the flight landing forward and approach accuracy ^[86]. We can even fuse aircraft structural data to determine the aircraft landing gear load at the moment of landing to determine whether the landing gear needs to be repaired, and so on.

5.   Conclusions

Automated FMR is considered a domain for understanding flight maneuvers. It involves flight maneuver detection and recognition tasks in different areas, including pilot training, aviation safety, and autonomous air warfare. This review provided a survey of existing techniques used for FMR and the existing dataset for FMR. We discussed what is FMR, why we need FMR, and compared FMR with HAR. We introduced a unified dataset (the CAFUC dataset), presented an evaluation of vision-based FMR and sensor-based FMR using this dataset, and visualized the results. Based on the results of evaluation, we discussed the advantages, disadvantages, and highlights of those methods. We have discussed the challenges and future directions of FMR. We hope that this survey can contribute to the development of FMR and provide the reader with a more comprehensive understanding of it.

Acknowledgments

This research was funded by the Sichuan Key R & D Projects of China, NO: 21ZDYF; The Civil Aviation Flight Technology and Flight Safety Key Laboratory of China research projects, NO: FZ2020ZZ02; The CAFUC Research Project, NO: CJ2021-01; Sichuan Science and Technology Program, NO:2022YFG0027.

Conflict of interest

The authors declare that there is no conflict of interest.