A survey of generative adversarial networks and their application in text-to-image synthesis

Wu Zeng; Heng-liang Zhu; Chuan Lin; Zheng-ying Xiao; Wu Zeng; Heng-liang Zhu; Chuan Lin; Zheng-ying Xiao

doi:10.3934/era.2023362

Electronic Research Archive

2023, Volume 31, Issue 12: 7142-7181. doi: 10.3934/era.2023362

Previous Article Next Article

Review Special Issues

A survey of generative adversarial networks and their application in text-to-image synthesis

1.
Engineering Training Center, Putian University, Putian 351100, China
2.
College of Computer Science and Mathematics, Fujian University of Technology, Fuzhou 350118, China
3.
Fujian Provincial Universities Key Laboratory of Industrial Control and Data Analysis, Fuzhou 350118, China
4.
School of Mechanical, Electrical & Information Engineering, Putian University, Putian 351100, China

Received: 13 September 2023 Revised: 29 October 2023 Accepted: 30 October 2023 Published: 09 November 2023

With the continuous development of science and technology (especially computational devices with powerful computing capabilities), the image generation technology based on deep learning has also made significant achievements. Most cross-modal technologies based on deep learning can generate information from text into images, which has become a hot topic of current research. Text-to-image (T2I) synthesis technology has applications in multiple fields of computer vision, such as image enhancement, artificial intelligence painting, games and virtual reality. The T2I generation technology using generative adversarial networks can generate more realistic and diverse images, but there are also some shortcomings and challenges, such as difficulty in generating complex backgrounds. This review will be introduced in the following order. First, we introduce the basic principles and architecture of basic and classic generative adversarial networks (GANs). Second, this review categorizes T2I synthesis methods into four main categories. There are methods based on semantic enhancement, methods based on progressive structure, methods based on attention and methods based on introducing additional signals. We have chosen some of the classic and latest T2I methods for introduction and explain their main advantages and shortcomings. Third, we explain the basic dataset and evaluation indicators in the T2I field. Finally, prospects for future research directions are discussed. This review provides a systematic introduction to the basic GAN method and the T2I method based on it, which can serve as a reference for researchers.

Keywords:

Citation: Wu Zeng, Heng-liang Zhu, Chuan Lin, Zheng-ying Xiao. A survey of generative adversarial networks and their application in text-to-image synthesis[J]. Electronic Research Archive, 2023, 31(12): 7142-7181. doi: 10.3934/era.2023362

Related Papers:

[1]	Haoqing Wang, Wen Yi, Yannick Liu . An innovative approach of determining the sample data size for machine learning models: a case study on health and safety management for infrastructure workers. Electronic Research Archive, 2022, 30(9): 3452-3462. doi: 10.3934/era.2022176
[2]	Tej Bahadur Shahi, Cheng-Yuan Xu, Arjun Neupane, William Guo . Machine learning methods for precision agriculture with UAV imagery: a review. Electronic Research Archive, 2022, 30(12): 4277-4317. doi: 10.3934/era.2022218
[3]	Xiaoyan Wu, Guowen Ye, Yongming Liu, Zhuanzhe Zhao, Zhibo Liu, Yu Chen . Application of Improved Jellyfish Search algorithm in Rotate Vector reducer fault diagnosis. Electronic Research Archive, 2023, 31(8): 4882-4906. doi: 10.3934/era.2023250
[4]	Xi Luo, Ran Yan, Shuaian Wang, Lu Zhen . A fair evaluation of the potential of machine learning in maritime transportation. Electronic Research Archive, 2023, 31(8): 4753-4772. doi: 10.3934/era.2023243
[5]	Jialiang Xiao, Yongtao Zheng, Wei Wang, Xuedong Hua . Robustness analysis of a multimodal comprehensive transportation network from the perspectives of infrastructure and operation: A case study. Electronic Research Archive, 2025, 33(4): 1902-1945. doi: 10.3934/era.2025086
[6]	Yufei Duan, Xian-Ming Gu, Tingyu Lei . Application of machine learning in quantitative timing model based on factor stock selection. Electronic Research Archive, 2024, 32(1): 174-192. doi: 10.3934/era.2024009
[7]	Bangna Li, Qingqing Zhang, Xingshi He . Multi-label feature selection via constraint mapping space regularization. Electronic Research Archive, 2024, 32(4): 2598-2620. doi: 10.3934/era.2024118
[8]	Nihar Patel, Nakul Vasani, Nilesh Kumar Jadav, Rajesh Gupta, Sudeep Tanwar, Zdzislaw Polkowski, Fayez Alqahtani, Amr Gafar . F-LSTM: Federated learning-based LSTM framework for cryptocurrency price prediction. Electronic Research Archive, 2023, 31(10): 6525-6551. doi: 10.3934/era.2023330
[9]	Qing Tian, Yanan Zhu, Yao Cheng, Chuang Ma, Meng Cao . Unsupervised domain adaptation through transferring both the source-knowledge and target-relatedness simultaneously. Electronic Research Archive, 2023, 31(2): 1170-1194. doi: 10.3934/era.2023060
[10]	Pinchao Meng, Xinyu Wang, Weishi Yin . ODE-RU: a dynamical system view on recurrent neural networks. Electronic Research Archive, 2022, 30(1): 257-271. doi: 10.3934/era.2022014

Abstract

1. Introduction

Rapid urbanization generates growing amounts of travel flows, urging the requirement for efficient transport planning policies. During 1980's, in Shanghai, China, the urbanization processes were in their nascent stages, with an approximate daily total of 20 million trips. By 2019, this daily total had surged to approximately 57.31 million trips. Similarly, in Beijing, since the inception of the first travel survey in 1986, the daily volume of trips in the central urban area has exhibited an average annual increase of 4%. By 2019, the daily volume of trips had reached a significant 39.57 million. The substantial growth has resulted in severe traffic congestion, with an average congestion duration of three hours in 2019 ^[1,2]. These trends underscore the urgent need for the development and implementation of efficient transport planning policies. Accurate comprehension of an individual's transport mode choice enables the development of customized guidance policies for different demographic groups. These policies can significantly contribute to the optimization of transportation system infrastructure, the promotion of increased public transportation utilization and the relief of traffic congestion. Consequently, the development of precise models for transport mode detection assumes paramount importance ^[3].

1.1. Background

In fact, the choice of transport modes is influenced by many types of factors, including individual, household and built environment factors. Individual factors include age, gender, income, occupation, attitudes and trip purpose ^{[4,5,6,7,8,9]}. Household factors include household structure, household characteristics, number of household members and car ownership, etc. ^[10,11,12]. Furthermore, several attributes of the built environment have been recognized to influence the choice of transport modes, such as building density, land use mix and distance to the transport facilities ^{[13,14,15,16]}. The primary method of collecting the data is via travel survey. Collecting individual mobility patterns and rich sociodemographic information are critical advantages of travel surveys ^{[17,18,19,20]}. However, respondents may provide incorrect information out of a desire to protect their privacy, such as by providing lower-income information, etc. This reason results in insufficient data collection accuracy and impacts the validity of the subsequent study ^[21,22]. Additionally, ensuring the security of respondents' privacy can be challenging. Balancing the detection performance of transport mode with the imperative of ensuring respondents' privacy remains a significant challenge in this field of research.

Most detection models for transport mode are predicted by econometric discrete choice models (DCMs) based on the random utility framework, such as multinomial logit model, nested logit model and mixed logit model ^[23,24,25]. The main advantage of these models is that they have good interpretability because of explicit mathematical formulas. The utility functions in the model are defined manually before fitting the model. This step allows for the integration of established behavioral theory into the model ^[26,27]. However, manual definitions may be violated under real complex problems, leading to biased predictions ^[28,29]. In contrast to traditional DCMs, machine learning models applied to model the choice of transport modes are promising methods to address current limitations ^{[30,31,32,33,34]}. Instead of making the necessary prior definitions, machine learning models represent complex relationships between variables through data-driven learning. Although machine learning models have better predictive accuracy, they focus on prediction and lack model interpretability ^[35]. Novel techniques and experiments such as t-SNE, SHapley Additive exPlanations (SHAP) and ablation experiments are proposed to overcome the non-interpretability issue of the machine learning models ^[36,37,38]. At the same time, most of the current detection models detect transport modes, with their categories including car, walk, bike, and public transport. Some studies refine "bike" into bike and e-bike. For public transport, studies consider them as a broad category. For example, they do not distinguish between the bus and the subway ^[30,39,40]. The coarse-grained classification of public transport results in the inability to target groups with specific transport modes for interventions in transportation planning. Therefore, achieving accurate individual transport mode detection for improved travel demand management still presents many areas for research.

1.2. Literature review

1.2.1. Features for transport mode detection

The detection of transport mode plays a crucial role in managing travel demand. The choice of transport mode is influenced by a variety of factors including age ^[4], gender ^[5], income ^[6], occupation ^[7], attitudes ^[8], trip purpose ^[9], household structure ^[10], household characteristics ^[11] number of household members ^[12], car ownership ^[13] and built environment ^[14]. For example, Kim et al. found that the built environment has a greater impact on young and elderly individuals compared to middle-aged individuals in the choice of public transportation ^[4]. Subjective determinants like travel satisfaction can also influence transport mode detection. A study found that pro-environmental attitudes have a positive impact on the utility of subway and walking as modes of travel, and this preference increases with the duration of walking trips ^[41]. Another study found that cars promote subjective well-being by facilitating leisure activities, but they also diminish a sense of belonging and achievement through shopping activities ^[42]. The acquisition of this information can be time-consuming and challenging ^[26,43]. Respondents may provide inaccurate information due to privacy concerns, which affect the efficacy of subsequent research. Therefore, there is potential to propose a method to both protect individual privacy and accurately detect the transport mode.

Research on individual spatial mobility features has provided the potential for detecting modes. Gonzalez et al. first introduced physical concepts such as the COM and radius of gyration to study individuals' mobility features ^[44]. They found that individuals' trajectories exhibit high spatiotemporal regularities, with each individual displaying time-invariant features such as travel distance and a significant probability of returning to specific frequent locations. Another study revealed that if a person's future mobility behaviour is influenced by their historical movements, it can be considered a memory-dependent human mobility model ^[45]. Later, Hong et al. observed that individuals from both datasets maintain a consistent number of combinations of travel modes and activity locations over time ^[46]. However, the currently proposed mobility features alone may not be sufficient for detecting travel modes. Velocity and acceleration play crucial roles in transport mode detection. To improve the accuracy of transport mode detection using trip information, we have introduced velocity and acceleration of the COM in the mobility features.

1.2.2. Models for transport mode detection

Traditionally, DCMs have been employed to detect transport modes of individuals. DCMs based on the random utility maximization theory commonly include forms such as the multinomial logit model, nested logit model and mixed logit model ^[34]. These models have a mathematical structure, making them highly interpretable. Nevertheless, DCMs require many specific features for assessing their impact on transport mode detection, demanding the collection of extensive and detailed data ^[26]. The acquisition of this data is time-consuming and involves significant implementation costs. Additionally, DCMs are based on model assumptions, such as the independence of irrelevant alternatives (IIA) for the multinomial logit model. If these assumptions are violated, it will lead to biases in parameter estimation and model detection.

Machine learning models which are driven by data are emerging as feasible alternatives for detecting individual transport modes. Various machine learning methods like support vector machines (SVM) ^[47], k-nearest neighbors (KNN) ^[27], decision trees (DT) ^[48] and artificial neural networks (ANN) ^[49] have been employed in transport mode detection without stringent assumptions. As machine learning models develop, individual models have exhibited limitations in predicting various aspects. To enhance detection accuracy and robustness, ensemble models such as random forests (RF) ^[30], XGBoost ^[50] and CatBoost ^[51] have been developed. For example, a study compared the performance of seven different machine learning models in detecting transport modes and successfully detected four modes: walk, bike, public transport and car ^[27]. Omrani used four machine learning models to detect individual transport modes in Luxembourg, aggregating walking and biking into one category and recognizing three transport modes: car, public transport and soft mode ^[52]. In another study, Zhao et al. compared four different types of machine learning models, namely Naive Bayes, tree-based models, SVM and neural networks (NN) for predicting individual choices among four transport modes: car, walk, bike and public transport ^[26].

Recently, a novel machine learning model called Deep Forest (DF) has been developed and has achieved success in various domains, including image recognition, natural language processing and anomaly detection ^[53]. The DF model demonstrates the ability to handle high-dimensional data, capture intricate patterns, and make accurate predictions with limited data. The advantages of the DF model make it a potential method to enhance the accuracy and reliability of transport mode detection models. However, most of the methods proposed in literature lack interpretability, which limits their applicability in policy-making. Additionally, these models can detect only a limited number of transport modes, typically three or four. They can detect public transport modes but are unable to distinguish between buses and subways within the category of public transport.

1.3. Objectives of this research

The review of these studies reveals that some aspects need to be improved in the current research: (ⅰ) The traditional DCMs employ individual and household attribute data to detect transport modes. These attribute data involve individual privacy and affect the applicability of the model. (ⅱ) The DCMs produce biased detections, though they reveal the contribution of the variables. The machine learning models are more effective for mode detection, but most models need more interpretability. (ⅲ) Current studies mainly treat public transport as a broad category, making it ill-suited for implementing fine transportation planning.

Therefore, an interpretable machine learning model is proposed for the detection of transport modes. We ignore individual and household attribute data and consider individual mobility features to train the detection models. A wide range of multi-source data that can be utilized to detect transport modes is easily accessible with the rapid development of big data, such as land use mix, GDP, population and online map service data. We combine individual mobility features and multi-source data as detection features for the models. The different machine learning models using detection features were applied to detect fine-grained transport modes, which were divided into seven categories. Specifically, the categories of public transport are subdivided into bus, metro and public transport (including transfers). We calculate four performance metrics to verify the performance of the proposed models for the transport modes. Furthermore, t-SNE and ablation experiments are applied to improve the interpretability of the machine learning models.

The main contributions of this study are as follows. First, the individual and household factors are ignored in the transport mode detection models in order to protect individual privacy and improve the applicability of the model. Thus, the study innovatively considers individual mobility features and combines them with multi-source data, including land use mix, GDP, population and online map service data as detection features. Third, the performance of the deep forest machine learning models has been investigated in a comparative study with traditional classifiers for transport mode detection. We detect the transport modes at the fine-grained level, which are divided into seven categories: car, walk, bike, e-bike, bus, metro and public transport (including transfers). Finally, t-SNE and ablation experiments are employed to enhance the interpretability of the machine learning models.

The remainder of the paper is organized as follows: Section 2 describes the proposed method for transport mode detection, including the extraction of individual mobility features, processing of multi-source data and interpretable machine learning detection models. Sections 3 and 4 discuss the experimental results of different machine learning models using Nanjing as an example. Finally, in Section 5, we summarize the study and propose future research directions.

2. Methodology

The proposed model for detecting transport modes compromises three steps. First, individual mobility features are extracted, making use of the spatial movement. Next, land use mix and online map service data are extracted as multi-source data to enhance the performance of transport mode detection. Finally, these extracted features and multi-source data serve as inputs for the proposed detection model. The performance of models is assessed using performance metrics such as accuracy, recall, precision and F1-measure to verify its effectiveness. The following sections discuss in detail the steps and substeps of the proposed method.

2.1. Overview

Historical movements influence mobility of individuals. Information about the choice of transport modes is implied in individual mobility features ^[45,54]. The COM of trips is the center of individual spatial mobility, which is a vital feature in describing individual mobility. Radius of gyration is used to measure the extent of individual travel activity. These are the essential features of individual mobility ^[44,55]. Velocity and acceleration are critical features in transport mode detection. These two features are introduced to detect the transport mode more accurately from the trip information. We innovatively introduce the velocity and acceleration of the COM in the mobility features.

Many multi-source datasets used to detect transport modes are readily available with the rapid development of big data ^[56,57]. First, the share of different transport modes varies in areas with different land uses. Land use mix is an important indicator of the choice of transport mode ^[58,59]. Second, individual income levels and regional development levels also influence the choice of transport modes ^[60]. Owing to the anonymity of the data, it is impossible to identify the socioeconomic and demographic backgrounds of individuals. The proposed model considers the analysis in different administrative regions, using regional GDP and population as detection features. Finally, online map service data can be used to obtain the travel path, distance and time of different transport modes, which can further improve the performance of the detection models ^[61]. Hence, this study uses the land use mix, GDP, population, and online map service data as the features to detect the transport modes of individuals.

We combine individual mobility features and multi-source data, including land use mix, GDP, population and online map service data as detection features. The machine learning models are applied to detect transport modes, which are divided into seven categories: car, walk, bike, e-bike, bus, metro and public transport (including transfers). To enhance the interpretability of the machine learning models, t-SNE and ablation experiments are conducted in the analysis.

2.2. Individual mobility features extraction

Center of mass. COM ${\overrightarrow c _m}$ for individual trips is determined by considering an individual's stay locations and the frequency of their stays at various spatial positions. This metric can be viewed as the central location within an individual's spatial mobility range. To calculate the COM for individual trips, we map the stay location of an individual onto a two-dimensional (2D) coordinate system and use the following formula:

${\overrightarrow c _m} = \frac{{\sum\nolimits_i {{n_i}\overrightarrow {{r_i}} } }}{{\sum\limits_{i \in L} {{n_i}} }}$

(1)

where i is the stay location of the individual, n_i is the number of stays at location I, L is the set of stay locations and $\overrightarrow {{r_i}}$ is the 2D vector coordinate of stay location i.

Velocity and Acceleration of the COM. The velocity and acceleration of the COM are applied to calculate the velocity and acceleration of the COM for the trips of an individual. The velocity of the COM indicates the speed at which individuals move through space, while the acceleration of the COM reflects changes in an individual's mobility, such as instances of acceleration or deceleration during a journey. Stay location i to stay location i + 1 constitutes one trip i, where each trip includes a transport mode. The distance from stay location i to i + 1 is l_i, and the trip duration is t_i. The velocity ${\overrightarrow v _i}$ and acceleration ${\overrightarrow a _i}$ of trip i are calculated for each individual trip.

${l_i} = \sqrt {{{\left( {{{\overrightarrow r }_{i + 1}} - \overrightarrow {{r_i}} } \right)}^2}}$

(2)

${\overrightarrow v _i} = \frac{{{l_i}}}{{{t_i}}} , {\overrightarrow a _i} = \frac{{{l_i}}}{{t_i^2}}$

(3)

Except for the velocity of the COM mentioned below, the calculation procedure for the acceleration of the COM is the same. The calculation procedure is illustrated in . Specifically, the velocity of trip i is decomposed into velocity in the horizontal ${\overrightarrow v _{ix}}$ and velocity in the vertical direction ${\overrightarrow v _{iy}}$ .

${\overrightarrow v _{iy}} = \frac{{\left| {{y_{{r_{i + 1}}}} - {y_{{r_i}}}} \right|}}{{{l_i}}}{\overrightarrow v _i} = \frac{{\left| {{y_{{r_{i + 1}}}} - {y_{{r_i}}}} \right|}}{{{t_i}}} , {\overrightarrow v _{ix}} = \frac{{\left| {{x_{{r_{i + 1}}}} - {x_{{r_i}}}} \right|}}{{{l_i}}}{\overrightarrow v _i} = \frac{{\left| {{x_{{r_{i + 1}}}} - {x_{{r_i}}}} \right|}}{{{t_i}}}$

(4)

Figure 1. Procedure for calculating the velocity for the center of mass (COM).

DownLoad: Full-Size Img PowerPoint

where ${x_{{r_i}}}$ is the latitude of stay location I and ${y_{{r_i}}}$ is the longitude of stay location i.

Then the velocity of COM ${v_c}$ is calculated using the frequency of each stay location.

${\overrightarrow v _{cx}} = \frac{{\sum\limits_{i = 1}^{N - 1} {{n_i}{{\overrightarrow v }_{ix}}} }}{{\sum\limits_{i \in L} {{n_i}} }} , {\overrightarrow v _{cy}} = \frac{{\sum\limits_{i = 1}^{N - 1} {{n_i}{{\overrightarrow v }_{iy}}} }}{{\sum\limits_{i \in L} {{n_i}} }}$

(5)

${v_c} = \sqrt {{{\overrightarrow v }_{cx}}^2 + {{\overrightarrow v }_{cy}}^2}$

(6)

where ${\overrightarrow v _{cx}}$ is the horizontal velocity of the COM and ${\overrightarrow v _{cy}}$ is the vertical velocity of the COM.

Radius of gyration. Radius of gyration r_g represents the weighted average of distances from an individual's stay locations to the COM. This metric is employed to measure the spatial extent covered by individual mobility. COM is used as the axis of rotation, and the frequency of the individual at the stay location is used as the mass. r_g for the individual trip is calculated as follows:

${r_g} = \sqrt {\frac{1}{N}\sum\limits_{i \in L} {{n_i}{{\left( {\overrightarrow {{r_i}} - {{\overrightarrow c }_m}} \right)}^2}} }$

(7)

2.3. Multi-source data process

Land Use Mix. Land use mix is an important indicator that reflects the choice of the transport mode. In this study, the land use mix were used as detection features. Land use data were obtained from the Chinese essential urban land use categories (EULUC) dataset ^[62]. We draw on the code for planning standards of development land, and classify land use into six categories: residential, administration and public services, commercial and business facilities, industrial, street and transportation and green space. The processing method involves the following steps, as illustrated in Figure 2.

Figure 2. Calculation process of land use mix.

DownLoad: Full-Size Img PowerPoint

First, the size of the land use map was chosen as the basis for creating the fishnet. The width and height of the cell size were set to 1 km in the fishnet. We clip the fishnet and overlay it with land use data. The entropy index (ENT) is usually applied to measure land use heterogeneity. The calculation procedure is as follows:

$ENT = \frac{{ - \sum {\left( {{A_{ij}}\ln {A_{ij}}} \right)} }}{{\ln {N_j}}}$

(8)

where A_ij is the proportion of land use type i in cell j and N_j is the number of land use types in grid j. ENT ranges from zero to one.

Finally, the coordinate data of the trip origin and destination were imported to calculate ENT at each destination and origin.

Online Map Service Data. We introduce online map service data as multi-source data to improve detection performance ^[61]. Specifically, the Baidu application programming interface (API) is utilized to obtain the shortest travel path of individuals in seven transport modes: car, walk, bike, e-bike, bus, metro and public transport-including transfers.

The Baidu API is an open map solution interface for developers that provides route planning, real-time navigation, global positioning and other services. We mainly used the route planning function, which can query travel routes. The parameters include the latitude and longitude coordinates of the origin and destination of the trip, geographic coordinate system type and developer key. We consider that the four transport modes of route change less in a short period (including driving, riding a bike or e-bike and walking), while public transport needs to consider departure time. Therefore, the parameters of public transport include departure time, date and tactics. It is worth mentioning that the parameters named "Departure_date" and "Departure_time" respectively represent the date and time of the trip's departure. Therefore, it is possible to return the route parameters based on the specified date and time. Table 1 lists the public transport API web addresses and parameters.

Table 1. Public transport API web address and parameters.

Parameters	Value	Description
Origin	32.048245,118.796519	Trip origin (Latitude, longitude)
Destination	32.052371,118.890583	Trip destination (Latitude, longitude)
Coord_type	WGS84	Type of geographic coordinate system
Departure_date	20150618	Date of trip origin
Departure_time	08:00	Time of trip origin
Tactics_incity	0	0: Recommended mode; 1: Minimal interchanges mode; 2: Minimal walking mode; 3: No metro mode; 4: Short time mode; 5: Metro priority mode.
URL	https://api.map.baidu.com/direction/v2/transit?

| Show Table

DownLoad: CSV

These parameter values are input to a uniform resource locator (URL) and requested using Python to obtain the shortest travel path for each transport mode. The returned results include the travel time, distance and path for the entire trip. This method determines the transport mode of individual trips by comparing the travel time from the survey data with the time taken by the Baidu API.

2.4. Machine learning models for transport mode detection

Many machine learning models are used for classification. The task of the models is to detect the transport modes using key feature sets, with the entire process being supervised. Python and open-source libraries were used for all these models, and the parameters were transparent during the detection process.

Nonetheless, it is unclear which model is the best for our detection. Therefore, we tested several machine learning models. Specifically, we used logistic regression (LogisticReg), SVM, KNN, DT and RF. All the models were applied using the scikit-learn open-source library. In addition, we used artificial neural networks (ANN) ^[63], XGBoost ^[50], CatBoost ^[51] and DF ^[53]. Moreover, one or more hyperparameters must be optimized to achieve the best performance for each model. We tuned each model to the optimal parameters using a grid search to avoid the problem of unfairness due to improperly set hyperparameters. Unless otherwise stated, the default hyperparameter configuration of the respective models was maintained.

To verify the performance of the fine-grained detection models for the transport modes, we used the confusion matrix to calculate the number of true positives (TP), true negatives (TN), false positives (FP) and false negatives (FN). Then, we calculated four performance metrics through the classification results, including accuracy, recall, precision and F1-measure ^[64]. The performance metrics are calculated using the following equation:

$Accuracy = \frac{{TP + TN}}{{TP + FP + TN + FN}}$

(9)

$Precision = \frac{{TP}}{{TP + FP}}$

(10)

$Recall = \frac{{TP}}{{TP + FN}}$

(11)

$F1 - measure = \frac{{2TP}}{{2TP + FP + FN}}$

(12)

3. Dataset

The dataset used in this study is collected in the Nanjing Transport Development White Paper project. The collection date was June 18, 2015. The survey randomly selected 22,191 participants from 8547 households in 11 administrative districts of Nanjing.

In this study, the dataset was cleaned because the scope of our research included intra-city transport modes. The cleaned dataset comprises a total of 43,039 trips, including 13,301 walking trips, 4692 bike trips, 10,300 e-bike trips, 4996 bus trips, 8135 car trips, 335 public transport trips (including transfers) and 1280 metro trips. Public transport encompasses a broad category, including both buses and metro transfers. By adopting this classification approach, we can better capture the diversity and nuances of travel behaviours, particularly concerning the usage of public transport involving bus and metro transfers. We only use trip information in the survey data, including coordinates of the trip origin, coordinates of the trip destination, time of the trip origin, time of the trip destination and the label of the transport mode. The extracted features of the proposed method are presented in . These features are used as inputs to the model. It is worth mentioning that the indicators of the calculation process are used as features, including ${\overrightarrow v _{ix}}$ , ${\overrightarrow v _{iy}}$ , ${\overrightarrow v _{cx}}$ , ${\overrightarrow v _{cy}}$ , ${\overrightarrow a _{ix}}$ , ${\overrightarrow a _{iy}}$ , ${\overrightarrow a _{cx}}$ and ${\overrightarrow a _{cy}}$ .We used 80% of the dataset for training and 20% for testing.

Table 2. Descriptive statistics of extracted features.

Variables	Source of variables	Description
Continuous Variables		Mean	Std.
Longitude of trip origin	Collected	118.801758	0.077547
Latitude of trip origin	Collected	32.025352	0.156764
Longitude of trip destination	Collected	118.801786	0.077544
Latitude of trip destination	Collected	32.025306	0.156759
Time of trip origin (hours, minutes)	Collected	11.63, 16.04	4.68, 16.70
Time of trip destination (hours, minutes)	Calculated	11.92, 24.71	4.69, 16.82
Trip duration (h)	Calculated	0.44	0.30
Distance ${l_i}$ (km)	Calculated	4.30	6.60
GDP of administrative districts	Collected	781.53	336.50
Population of the administrative district	Collected	90.16	30.12
Center of mass ${\overrightarrow c _m}$	Calculated	13224.95, 3766.65	8281.63, 20296.36
Velocity of COM ${v_c}$ (km/h)	Calculated	8.30	8.56
Acceleration of COM ${a_c}$ (km/h²)	Calculated	24.15	28.40
Radius of gyration ${r_g}$ (km)	Calculated	2.28	3.33
Entropy index ENT	Calculated	0.46	0.35
Categorical variables
Transport modes determined by Baidu API	From Baidu API	Car; walk; bike; e-bike; bus; metro; public transport (including transfers)

| Show Table

DownLoad: CSV

Many studies have determined the radius of gyration distribution P(r_g) using a truncated power-law statistic for all individuals to explore the statistical properties of individual trip features ^[65].We also used statistics to distribute the mobility of individuals in Nanjing:

$P\left( {{r_g}} \right) = {\left( {{r_g} + r_g^0} \right)^{ - {\beta _r}}}\exp \left( {{{ - {r_g}} \mathord{\left/ {\vphantom {{ - {r_g}} {{k_r}}}} \right. } {{k_r}}}} \right)$

(13)

where $r_g^0$ = 0.91, ${\beta _r}$ = 0.81, ${k_r}$ = 8. The r-squared statistic is used to show how well the data fit the regression model. The r-squared statistic is calculated as follows:

${r^2} = 1 - \frac{{\sum\nolimits_i {{{\left( {{y_i} - {{\hat y}_i}} \right)}^2}} }}{{\sum\nolimits_i {{{\left( {{y_i} - {{\bar y}_i}} \right)}^2}} }}$

(14)

where y_i is the observed values of the data. ${\hat y_i}$ represents the predicted values by the model. ${\bar y_i}$ represents the mean of the observed values.

The r-squared statistic for the truncated power-law function mentioned above is 0.999. It is evident that the radius of gyration decreases rapidly from 1 to 10 km, and the long tail part shows a discrete state, as shown in Figure 3. This phenomenon indicates that the daily activities of most individuals were limited to from 1 to 10 km. The share of long-distance travelers was relatively low.

Figure 3. Distribution P(r_g) of the radius of gyration measured for the individuals.

DownLoad: Full-Size Img PowerPoint

Meanwhile, we counted the velocity and acceleration of the COM introduced in this study to discover the patterns. Surprisingly, the distribution is also very close to the truncated power-law:

$P\left( {{v_c}} \right) = {\left( {{v_c} + v_c^0} \right)^{ - {\beta _v}}}\exp \left( {{{ - {v_c}} \mathord{\left/ {\vphantom {{ - {v_c}} {{k_v}}}} \right. } {{k_v}}}} \right)$

(15)

$P\left( {{a_c}} \right) = {\left( {{a_c} + a_c^0} \right)^{ - {\beta _a}}}\exp \left( {{{ - {a_c}} \mathord{\left/ {\vphantom {{ - {a_c}} {{k_a}}}} \right. } {{k_a}}}} \right)$

(16)

where $v_c^0$ = 1954.9, ${\beta _v}$ = -0.015, ${k_v}$ = 7.03, r-squared = 0.998, $a_c^0$ = 0.9, ${\beta _a}$ = -0.11, ${k_a}$ = 16.4 and r-squared = 0.997. As previously depicted in Figures 4 and 5, slow-speed transport modes account for a relatively large proportion of all transport modes, whereas the acceleration of all travel modes is low. The main reason is that the study only considers intra-city transport modes excluding high-speed rail and airplanes, resulting in a lower velocity of the COM.

Figure 4. Distribution P(v_c) of the velocity for the COM.

DownLoad: Full-Size Img PowerPoint

Figure 5. Distribution P(a_c) of the acceleration for the COM.

DownLoad: Full-Size Img PowerPoint

4. Results

In order to validate the detection performance of various machine learning models, experiments were conducted using travel survey data from Nanjing, China in 2015. The primary focus of these experiments was to assess the accuracy and effectiveness of these models in transport mode detection. For the most promising model in terms of performance, further analysis was performed to provide detailed detection results for each transport mode, aiming to explore the reasons for any shortcomings in the detection performance of model. Furthermore, we explored the interpretability of the model, examining its role in the process of transport mode detection.

4.1. Performance results of machine learning models

The four standard metrics used to assess the performance of the detection process are accuracy, recall, precision and the F1-measure. The F1-measure is the harmonic average of precision and recall. Hence, the accuracy and F1-measure were selected as performance metrics in this study. Meanwhile, we selected several machine learning models for comparison. These models include logisticReg, SVM, KNN, DT, RF, ANN, CatBoost, XGBoost and DF. We also compared the results obtained directly from the Baidu API for detecting transport modes. The SVM used a radial basis function kernel to obtain the best results, and the cascade layer of the deep forest was automatically fitted to four layers. We used five layers (including three hidden layers) in the ANN to detect transport modes. The ANN includes 28 input neurons and 7 output neurons. Figure 6 shows the performance of all models.

Figure 6. Performance results of each single-mode detection.

DownLoad: Full-Size Img PowerPoint

Table 3 demonstrates that the deep forest model achieves the best detection results among all the models, with an accuracy and F1-measure of approximately 0.82 and 0.74, respectively. While the RF model also performs well, it does not match the performance of the deep forest model. Our results suggest that the deep forest may have better applicability to our proposed detection model and further supports our decision to use the deep forest model for the remainder of the study.

Table 3. Performance results of machine learning models.

Models	Accuracy	Precision	Recall	F1-measure
Baidu API	0.45	0.37	0.39	0.35
logisticReg	0.58	0.35	0.34	0.31
SVM	0.64	0.43	0.73	0.41
KNN	0.68	0.6	0.56	0.58
DT	0.71	0.71	0.71	0.71
RF	0.8	0.67	0.83	0.73
ANN	0.63	0.52	0.64	0.56
CatBoost	0.74	0.62	0.72	0.65
XGBoost	0.75	0.64	0.76	0.68
DF	0.82	0.82	0.7	0.74

| Show Table

DownLoad: CSV

4.2. Performance results of the DF model

In the previous experiments, we evaluated the performance of several machine learning models. These results suggest that the deep forest model is an effective and robust machine learning model for detecting fine-grained transport modes. The specific detection results for each transport mode were output further to explore the reasons for the imperfect detection performance of the DF model. The results are presented in Figure 6.

As seen in Figure 6, the performance of detecting walking is the best for either performance metric. The walking speed is lower than that of other transport modes, resulting in better detection through the velocity of the COM. Meanwhile, public transport has the worst detection performance, with an accuracy of only 0.31. Public transport may involve a transfer process that includes waiting and transfer times. Moreover, interchange transport modes are unknown, and detecting public transport (including transfers) is the most challenging.

4.3. Interpretability of the DF model

In fact, it is unclear which individual transport mode is actually misidentified in the data or which features are more important. The reason is due to the need for more interpretability of machine learning models. We can find the reason for the detection error and improve detection accuracy if a model can capture the misidentification among different modes or obtain the importance ranking of features. Therefore, we performed t-SNE and ablation experiments to overcome the non-interpretability issue of the machine learning models. We also analysed the confusion matrix of the model.

First, we extracted augmented features of the deep forest detection process; there were four layers of features in the detection process of this study in total. They are challenging to understand and visualize because their features are highly dimensional. t-SNE, which is used for high-dimensional feature visualization, is used to reduce the dimensionality of the features ^[36,66] achieved by minimizing the Kullback-Leibler divergence between a joint probability distribution in high and low dimensional spaces. High-dimensional data are eventually mapped into 2D, which is more suitable for human observation. A visualization of t-SNE is shown in Figure 7. Subsequently, we extracted the confusion matrix of the model, as shown in Table 4.

Figure 7. t-SNE visualization of four layers.

DownLoad: Full-Size Img PowerPoint

Table 4. Confusion matrix of deep forest model.

		Actual result
		walk	bike	e-bike	bus	car	public transport	metro
Predicted Result	walk	2578	26	47	18	10	0	0
	bike	133	637	90	33	30	0	1
	e-bike	154	43	1642	75	129	0	9
	bus	38	11	98	750	127	1	8
	car	35	12	158	77	1303	5	7
	public transport	0	0	2	13	21	24	2
	metro	1	1	26	28	60	2	143

| Show Table

DownLoad: CSV

Figure 7 and Table 4 illustrate the reasons for the imperfect detection of transport modes by the deep forest model. As depicted in the figure, some of the green, orange and blue-green dots are heavily intermingled. This is consistent with the results of the confusion matrix. This phenomenon means that the detection of cars, e-bikes and buses is biased. One potential explanation is that their travel speeds are not significantly different when urban traffic congestion occurs, resulting in inaccurate detection. Additionally, some of the red and yellow dots are also mixed, indicating a biased detection of walking and bikes due to similar travel speeds. The biased detection of transport modes is a result of the slight speed differences during actual travel process. Therefore, the results indicate that, although the velocity of the COM introduced by the paper can substantially enhance the detection performance, further improvements are necessary in the detection of details.

To further investigate the effectiveness of each feature and obtain the importance ranking of a feature, we designed an ablation experiment (including four features and two multi-source data) ^[67,68]. The results of the experiments are listed in Table 5. ENT and OMS respectively represent the entropy index of land use and the travel modes recognized by the Baidu API. The checked ones in the table are the reduced features. With each feature reduction, we kept the rest of the features and compared the performance metrics between the current model and our model. This step allows for an explicit analysis of the features that more significantly impact mode detection.

Table 5. Results of the ablation experiment.

Features						Performance metrics
${\overrightarrow c _m}$	${r_g}$	${v_c}$	${a_c}$	ENT	OMS	Accuracy	Precision	Recall	F1-Measure
Ours						0.82	0.82	0.70	0.74
					√	-0.02	-0.02	-	-0.01
				√		-0.01	-0.02	-0.01	-0.01
			√			-0.04	-0.03	-0.05	-0.04
		√				-0.07	-0.04	-0.06	-0.06
	√					-0.01	-0.01	-0.02	-0.02
√						-0.02	-0.02	-0.02	-0.02

| Show Table

DownLoad: CSV

The accuracy decreases by 0.07 when the velocity of the COM is ignored. The second important feature is the acceleration of the COM, which decreases the accuracy by 0.04. The results directly indicate that the two mobility features introduced in this study play crucial roles in mode detection. Additionally, there is a relationship between transport mode and gyration radius, indicating that travel distance influences the choice of transport mode. Thus, individual mobility features are the most critical features that improve the detection performance, enhancing accuracy by 0.14. Furthermore, the land use mix improves the accuracy of detection by 0.01, implying little correlation between the urban land use mix and the choice of transport modes. The potential reason may be that the dataset of our study includes weekdays and the effect of the land use mix is restricted ^[69].

In summary, our proposed model exhibits several advantages. Firstly, the DF model has demonstrated exceptional performance in accurately detecting between seven different transport modes, exhibiting a superior performance compared to alternative models. The high detection performance of the DF model emphasizes the model's robustness in effectively detecting various transport modes. Additionally, the DF model performs well in terms of interpretability, providing valuable insights into the factors that influence individuals' mode choices. However, the model also has some disadvantages. Specifically, the model faces challenges when it comes to detecting public transport modes, particularly those involving transfers. Furthermore, the influence of land use mix on the detection of the model is relatively modest. In conclusion, there is the potential for further research in the future.

4.4. Managerial implications

Precisely detecting various transport modes of individuals provides valuable insights for transportation management authorities. Tailored policies can be formulated to encourage the adoption of sustainable and low-carbon transportation modes for groups with specific transport modes, such as public transit, cycling and walking. This proactive approach helps address growing urban transportation challenges. For example, managers can adjust transportation policies, including fare structures, service frequencies and accessibility to offer more appealing public transportation options. This approach can help reduce urban traffic emissions, relieve congestion issues and promote the overall sustainability of cities.

Meanwhile, recent research has witnessed a surge in the development of innovative approaches for data collection, especially Global Positioning System (GPS) loggers or smartphone travel surveys. Once we detect individual transport modes through our proposed method, these approaches eliminate the necessity for extensive follow-up surveys or direct participant interactions, making data collection more efficient. Moreover, these data collection approaches emphasize the potential of smartphone and GPS travel surveys in providing highly precise transport mode data while preserving participant privacy. Managers can actively embrace these innovative data collection methods to better understand individuals' travel behavior, guide policy formulation and inform transportation planning.

5. Conclusions

Analysis of transport mode choice is a crucial element in developing and evaluating planning policies. Though many kinds of factors influence the transport mode choice of the individual, many of these factors involve the privacy of individuals. Meanwhile, detection models of transport modes widely use discrete choice models. In order to improve the limitations of existing discrete choice models, we propose an interpretable machine learning model for detecting transport modes by multi-source data. The mobility features of individuals, which innovatively introduce the velocity and acceleration of the center of mass (COM), are considered in the detection model. We combine these features with multi-source data as detection features and apply machine learning models to detect transport modes at the fine-grained level, which were divided into seven categories.

Four performance metrics are calculated through the classification results, including accuracy, recall, precision and F1-measure. The results show that the deep forest algorithm exhibits the best detection performance, achieving a detection accuracy of 0.82 in the test dataset. Meanwhile, recall, precision and F1-measure also achieved optimal performance. The evaluation shows that the detection performance for other transport modes is excellent apart from public transport detection. Compared to previous work, this study is a significant step forward in terms of granularity and detection performance. To address the issue of non-interpretability of machine learning models, we also perform t-distributed stochastic neighbor embedding (t-SNE) and ablation experiments and rank the importance of features. These experiments demonstrate a weak correlation between the land use mix and the choice of transportation modes. Moreover, the most critical features are the individual mobility features in the study, and the accuracy improves by 0.13.

Meanwhile, recent research has witnessed a surge in the development of innovative approaches for data collection, especially Global Positioning System (GPS) loggers or smartphone travel surveys. They leverage these travel surveys to collect accurate travel behavior details without involving the collection of individual and household attribute data. Our proposed approach enhances the potential of these data collection methods in maintaining privacy protection for survey participants without the need for extensive follow-up surveys. On the other hand, if individual travel modes can be accurately detected, transportation government agencies can intervene by targeting specific groups with specific transport modes and effectively promote sustainable and low-carbon transportation options. The results of the study can guide government transportation agencies in formulating effective policies to achieve long-term sustainable and efficient urban transportation goals.

In future research, the utilization of multi-source data will be further expanded to enhance the performance and applicability of the proposed model. The expansion may involve incorporating additional data, such as real-time traffic information, weather conditions or other factors that can influence transport mode choice. The objective is to continuously improve the accuracy and reliability of the model in detecting transport modes, and its effectiveness in guiding transportation policies and addressing urban transportation challenges.

Use of AI tools declaration

The authors declare that they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

This work was supported by the Postgraduate Research & Practice Innovation Program of Jiangsu Province (Grant No. SJCX22_0061); the General Project of National Natural Science Foundation of China (Grant No. 52172317); and the Key Program of National Natural Science Foundation of China (Grant No. 52131203).

Conflict of interest

The authors declare there is no conflict of interest.

References

[1]	W. Yu, M. Luo, P. Zhou, C. Si, Y. Zhou, X. Wang, et al., MetaFormer is actually what you need for vision, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2022), 10809–10819. https://doi.org/10.1109/CVPR52688.2022.01055
[2]	Y. Chen, X. Dai, D. Chen, M. Liu, X. Dong, L. Yuan, et al., Mobile-former: Bridging mobilenet and transforme, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2022), 5270–5279. https://doi.org/10.1109/CVPR52688.2022.00520
[3]	A. Priya, K. M. Narendra, F. Binish, S. Pushpendra, A. Gupta, S. D. Joshi, COVID-19 image classification using deep learning: Advances, challenges and opportunities, Comput. Biol. Med., 144 (2022), 105350. https://doi.org/10.1016/j.compbiomed.2022.105350 doi: 10.1016/j.compbiomed.2022.105350
[4]	Y. L. Li, Research and application of deep learning in image recognition, in 2022 IEEE 2nd International Conference on Power, Electronics and Computer Applications (ICPECA), IEEE, (2022), 994–999. https://doi.org/10.1109/ICPECA53709.2022.9718847
[5]	H. E. Kim, A. Cosa-Linan, N. Santhanam, M. Jannesari, M. E. Maros, T. Ganslandt, Transfer learning for medical image classification: A literature review, BMC Med. Imaging, 22 (2022), 69. https://doi.org/10.1186/s12880-022-00793-7 doi: 10.1186/s12880-022-00793-7
[6]	Z. Zou, K. Chen, Z. Shi, Y. Guo, J. Ye, Object detection in 20 years: A survey, Proc. IEEE, 111 (2023), 257–276. https://doi.org/10.1109/JPROC.2023.3238524 doi: 10.1109/JPROC.2023.3238524
[7]	S. B. Xu, M. H. Zhang, W. Song, H. B. Mei, Q. He, A. Liotta, A systematic review and analysis of deep learning-based underwater object detection, Neurocomputing, 527 (2023), 204–232. https://doi.org/10.1016/j.neucom.2023.01.056 doi: 10.1016/j.neucom.2023.01.056
[8]	T. Diwan, G. Anirudh, J. V. Tembhurne, Object detection using YOLO: Challenges, architectural successors, datasets and applications, Multimedia Tools Appl., 82 (2023), 9243–9275. https://doi.org/10.1007/s11042-022-13644-y doi: 10.1007/s11042-022-13644-y
[9]	S. Frolov, A. Sharma, J. Hees, T. Karayil, F. Raue, A. Dengel, AttrLostGAN: Attribute controlled image synthesis from reconfigurable layout and style, in DAGM German Conference on Pattern Recognition, Springer International Publishing, (2021), 361–375. https://doi.org/10.1007/978-3-030-92659-5_23
[10]	D. Pavllo, A. Lucchi, T. Hofmann, Controlling style and semantics in weakly-supervised image generation, in Computer Vision–ECCV 2020: 16th European Conference, Springer International Publishing, (2020), 482–499. https://doi.org/10.1007/978-3-030-58539-6_29
[11]	R. Wadhawan, T. Drall, S. Singh, S. Chakraverty, Multi-attributed and structured text-to-face synthesis, in 2020 IEEE International Conference on Technology, Engineering, Management for Societal impact using Marketing, Entrepreneurship and Talent (TEMSMET), IEEE, (2020), 1–7. https://doi.org/10.1109/TEMSMET51618.2020.9557583
[12]	Y. Mei, Y. Fan, Y. Zhang, J. Yu, Y. Zhou, D. Liu, et al., Pyramid attention network for image restoration, Int. J. Comput. Vis., 131 (2023), 1–19. https://doi.org/10.1007/s11263-023-01843-5 doi: 10.1007/s11263-023-01843-5
[13]	N. Liu, W. Li, Y. Wang, Q. Du, J. Chanussot, A survey on hyperspectral image restoration: From the view of low-rank tensor approximation, Sci. China Inf. Sci., 66 (2023), 140302. https://doi.org/10.1007/s11432-022-3609-4 doi: 10.1007/s11432-022-3609-4
[14]	A. Dabouei, S. Soleymani, F. Taherkhani, N. M. Nasrabadi, SuperMix: Supervising the mixing data augmentation, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2021), 13789–13798. https://doi.org/10.1109/CVPR46437.2021.01358
[15]	S. C. Huang, W. N. Fu, Z. Y. Zhang, S. Liu, Global-local fusion based on adversarial sample generation for image-text matching, Inf. Fusion, 103 (2023), 102084. https://doi.org/10.1016/j.inffus.2023.102084 doi: 10.1016/j.inffus.2023.102084
[16]	M. Hong, J. Choi, G. Kim, StyleMix: Separating content and style for enhanced data augmentation, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2021), 14857–14865. https://doi.org/10.1109/CVPR46437.2021.01462
[17]	L. Liu, Z. X. Xi, R. R. Ji, W. G. Ma, Advanced deep learning techniques for image style transfer: A survey, Signal Process. Image Commun., 78 (2019), 465–470. https://doi.org/10.1016/j.image.2019.08.006 doi: 10.1016/j.image.2019.08.006
[18]	I. Goodfellow, P. A. Jean, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, et al., Generative adversarial nets, in Advances in Neural Information Processing Systems 27, 27 (2014), 1–9.
[19]	S. Singh, H. Singh, N. Mittal, H. Singh, A. G. Hussien, F. Sroubek, A feature level image fusion for Night-Vision context enhancement using Arithmetic optimization algorithm based image segmentation, Expert Syst. Appl., 209 (2022), 118272. https://doi.org/10.1016/j.eswa.2022.118272 doi: 10.1016/j.eswa.2022.118272
[20]	N. E. Khalifa, M. Loey, S. Mirjalili, A comprehensive survey of recent trends in deep learning for digital images augmentation, Artif. Intell. Rev., 55 (2022), 2351–2377. https://doi.org/10.1007/s10462-021-10066-4 doi: 10.1007/s10462-021-10066-4
[21]	Z. L. Chen, K. Pawar, M. Ekanayake, C. Pain, S. J. Zhong, G. F. Egan, Deep learning for image enhancement and correction in magnetic resonance imaging—state-of-the-art and challenges, J. Digital Imaging, 36 (2023), 204–230. https://doi.org/10.1007/s10278-022-00721-9 doi: 10.1007/s10278-022-00721-9
[22]	Q. Jiang, Y. F. Zhang, F. X. Bao, X. Y. Zhao, C. M. Zhang, P. Liu, Two-step domain adaptation for underwater image enhancement, Pattern Recognit., 122 (2022), 108324. https://doi.org/10.1016/j.patcog.2021.108324 doi: 10.1016/j.patcog.2021.108324
[23]	T. Rahman, A. Khandakar, Y. Qiblawey, A. Tahir, S. Kiranyaz, S. B. A. Kashem, et al., Exploring the effect of image enhancement techniques on COVID-19 detection using chest X-ray images, Comput. Biol. Med., 132 (2021), 104319. https://doi.org/10.1016/j.compbiomed.2021.104319 doi: 10.1016/j.compbiomed.2021.104319
[24]	G. F. Li, Y. F. Yang, X. D. Qu, D. P. Cao, K. Q. Li, A deep learning based image enhancement approach for autonomous driving at night, Knowledge-Based Syst., 213 (2021), 106617. https://doi.org/10.1016/j.knosys.2020.106617 doi: 10.1016/j.knosys.2020.106617
[25]	M. Mirza, S. Osindero, Conditional generative adversarial nets, preprint, arXiv: 1411.1784v1.
[26]	A. Radford, L. Metz, S. Chintala, Unsupervised representation learning with deep convolutional generative adversarial networks, preprint, arXiv: 1511.06434.
[27]	K. R. Chowdhary, Natural language processing, in Fundamentals of Artificial Intelligence, Springer, (2020), 603–649. https://doi.org/10.1007/978-81-322-3972-7_19
[28]	W. Xu, H. Peng, X. Zeng, F. Zhou, X. Tian, X. Peng, A hybrid modelling method for time series forecasting based on a linear regression model and deep learning, Appl. Intell., 49 (2019), 3002–3015. https://doi.org/10.1007/s10489-019-01426-3 doi: 10.1007/s10489-019-01426-3
[29]	M. Atlam, H. Torkey, N. El-Fishawy, H. Salem, Coronavirus disease 2019 (COVID-19): Survival analysis using deep learning and Cox regression model, Pattern Anal. Appl., 24 (2021), 993–1005. https://doi.org/10.1007/s10044-021-00958-0 doi: 10.1007/s10044-021-00958-0
[30]	X. A. Yan, D. M. She, Y. D. Xu, Deep order-wavelet convolutional variational autoencoder for fault identification of rolling bearing under fluctuating speed conditions, Expert Syst. Appl., 216 (2023), 119479. https://doi.org/10.1016/j.eswa.2022.119479 doi: 10.1016/j.eswa.2022.119479
[31]	W. Khan, M. Haroon, A. N. Khan, M. K. Hasan, A. Khan, U. A. Mokhtar, et al., DVAEGMM: Dual variational autoencoder with gaussian mixture model for anomaly detection on attributed networks, IEEE Access, 10 (2022), 91160–91176. https://doi.org/10.1109/ACCESS.2022.3201332 doi: 10.1109/ACCESS.2022.3201332
[32]	H. Li, Deep learning for natural language processing: Advantages and challenges, Natl. Sci. Rev., 5 (2018), 24–26. https://doi.org/10.1093/nsr/nwx110 doi: 10.1093/nsr/nwx110
[33]	B. Pandey, D. K. Pandey, B. P. Mishra, W. Rhmann, A comprehensive survey of deep learning in the field of medical imaging and medical natural language processing: Challenges and research directions, J. King Saud Univ. Comput. Inf. Sci., 34 (2022), 5083–5099. https://doi.org/10.1016/j.jksuci.2021.01.007 doi: 10.1016/j.jksuci.2021.01.007
[34]	A. G. Russo, A. Ciarlo, S. Ponticorvo, F. D. Salle, G. Tedeschi, F. Esposito, Explaining neural activity in human listeners with deep learning via natural language processing of narrative text, Sci. Rep., 12 (2022), 17838. https://doi.org/10.1038/s41598-022-21782-4 doi: 10.1038/s41598-022-21782-4
[35]	Y. T. Vuong, Q. M. Bui, H. Nguyen, T. Nguyen, V. Tran, X. Phan, et al., SM-BERT-CR: A deep learning approach for case law retrieval with supporting model, Artif. Intell. Law, 31 (2023), 601–628. https://doi.org/10.1007/s10506-022-09319-6 doi: 10.1007/s10506-022-09319-6
[36]	R. K. Kaliyar, A. Goswami, P. Narang, FakeBERT: Fake news detection in social media with a BERT-based deep learning approach, Multimedia Tools Appl., 80 (2021), 11765–11788. https://doi.org/10.1007/s11042-020-10183-2 doi: 10.1007/s11042-020-10183-2
[37]	B. Palani, S. Elango, K. V. Viswanathan, CB-Fake: A multimodal deep learning framework for automatic fake news detection using capsule neural network and BERT, Multimedia Tools Appl., 81 (2022), 5587–5620. https://doi.org/10.1007/s11042-021-11782-3 doi: 10.1007/s11042-021-11782-3
[38]	S. Reed, Z. Akata, X. C. Yan, L. Logeswaran, B. Schiele, H. Lee, Generative adversarial text to image synthesis, in Proceedings of the 33rd International Conference on International Conference on Machine Learning, PMLR, (2016), 1060–1069.
[39]	J. Y. Zhu, T. Park, P. Isola, A. A. Efros, Unpaired image-to-image translation using cycle-consistent adversarial networks, in 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, (2017), 2242–2251. https://doi.org/10.1109/ICCV.2017.244
[40]	T. Karras, T. Aila, S. Laine, J. Lehtinen, Progressive growing of GANs for improved quality, stability, and variation, preprint, arXiv: 1710.10196.
[41]	H. Zhang, I. Goodfellow, D. Metaxas, A. Odena, Self-attention generative adversarial networks, in Proceedings of the 36th International Conference on Machine Learning, PMLR, (2019), 7354–7363.
[42]	A. Odena, C. Olah, J. Shlens, Conditional image synthesis with auxiliary classifier GANs, in Proceedings of the 34rd International Conference on International Conference on Machine Learning, PMLR, (2017), 2642–2651.
[43]	H. Park, Y. Yoo, N. Kwak, MC-GAN: Multi-conditional generative adversarial network for image synthesis, preprint, arXiv: 1805.01123.
[44]	T. T. Qiao, J. Zhang, D. Q. Xu, D. C. Tao, MirrorGAN: Learning text-to-image generation by redescription, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019), 1505–1514. https://doi.org/10.1109/CVPR.2019.00160
[45]	H. C. Tan, X. P. Liu, X. Li, Y. Zhang, B. C. Yin, Semantics-enhanced adversarial nets for text-to-image synthesis, in 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2019), 10500–10509. https://doi.org/10.1109/ICCV.2019.01060
[46]	M. Tao, H. Tang, F. Wu, X. Jing, B. Bao, C. Xu, DF-GAN: A simple and effective baseline for text-to-image synthesis, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2022), 16494–16504. https://doi.org/10.1109/CVPR52688.2022.01602
[47]	W. T. Liao, K. Hu, M. Y. Yang, B. Rosenhahn, Text to image generation with semantic-spatial aware GAN, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2022), 18166–18175. https://doi.org/10.1109/CVPR52688.2022.01765
[48]	X. T. Wu, H. B. Zhao, L. L. Zheng, S. H. Ding, X. Li, Adma-GAN: Attribute-driven memory augmented GANs for text-to-image generation, in Proceedings of the 30th ACM International Conference on Multimedia, ACM, (2022), 1593–1602. https://doi.org/10.1145/3503161.3547821
[49]	Z. X. Zhang, L. Schomaker, DiverGAN: An efficient and effective single-stage framework for diverse text-to-image generation, Neurocomputing, 473 (2022), 182–198. https://doi.org/10.1016/j.neucom.2021.12.005 doi: 10.1016/j.neucom.2021.12.005
[50]	H. Zhang, T. Xu, H. S. Li, S. T. Zhang, X. G. Wang, X. L. Huang, et al., StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks, in 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, (2017), 5908–5916. https://doi.org/10.1109/ICCV.2017.629
[51]	H. Zhang, T. Xu, H. S. Li, S. T. Zhang, X. G. Wang, X. L. Huang, et al., StackGAN++: Realistic image synthesis with stacked generative adversarial networks, IEEE Trans. Pattern Anal. Mach. Intell., 41 (2019), 1947–1962. https://doi.org/10.1109/TPAMI.2018.2856256 doi: 10.1109/TPAMI.2018.2856256
[52]	A. Dash, J. C. B. Gamboa, S. Ahmed, M. Liwicki, M. Z. Afzal, TAC-GAN - text conditioned auxiliary classifier generative adversarial network, preprint, arXiv: 1703.06412.
[53]	Z. Z. Zhang, Y. P. Xie, L. Yang, Photographic text-to-image synthesis with a hierarchically-nested adversarial network, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, (2018), 6199–6208. https://doi.org/10.1109/CVPR.2018.00649
[54]	L. L. Gao, D. Y. Chen, J. G. Song, X. Xu, D. X. Zhang, H. T. Shen, Perceptual pyramid adversarial networks for text-to-image synthesis, in Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, (2019), 8312–8319. https://doi.org/10.1609/aaai.v33i01.33018312
[55]	T. Xu, P. C. Zhang, Q. Y. Huang, H. Zhang, Z. Gan, X. L. Huang, et al., AttnGAN: Fine-grained text to image generation with attentional generative adversarial networks, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, (2018), 1316–1324. https://doi.org/10.1109/CVPR.2018.00143
[56]	T. Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, et al., Microsoft COCO: Common objects in context, in Computer Vision–ECCV 2014: 13th European Conference, Springer, (2014), 740–755. https://doi.org/10.1007/978-3-319-10602-1_48
[57]	T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, Improved techniques for training GANs, in Advances in Neural Information Processing Systems 29, 29 (2016), 1–9.
[58]	M. F. Zhu, P. B. Pan, W. Chen, Y. Yang, DM-GAN: Dynamic memory generative adversarial networks for text-to-image synthesis, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019), 5795–5803. https://doi.org/10.1109/CVPR.2019.00595
[59]	H. Zhang, J. Y. Koh, J. Baldridge, H. Lee, Y. Yang, Cross-modal contrastive learning for text-to-image generation, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2021), 833–842. https://doi.org/10.1109/CVPR46437.2021.00089
[60]	Z. B. Shi, Z. Chen, Z. B. Xu, W. Yang, L. Huang, AtHom: Two divergent attentions stimulated by homomorphic training in text-to-image synthesis, in Proceedings of the 30th ACM International Conference on Multimedia, ACM, (2022), 2211–2219. https://doi.org/10.1145/3503161.3548159
[61]	Z. W. Chen, Z. D. Mao, S. C. Fang, B. Hu, Background layout generation and object knowledge transfer for text-to-image generation, in Proceedings of the 30th ACM International Conference on Multimedia, ACM, (2022), 4327–4335. https://doi.org/10.1145/3503161.3548154
[62]	Y. Ma, L. Liu, H. X. Zhang, C. J. Wang, Z. K. Wang, Generative adversarial network based on semantic consistency for text-to-image generation, Appl. Intell., 53 (2023), 4703–4716. https://doi.org/10.1007/s10489-022-03660-8 doi: 10.1007/s10489-022-03660-8
[63]	Y. K. Li, T. Ma, Y. Q. Bai, N. Duan, S. Wei, X. Wang, Pastegan: A semi-parametric method to generate image from scene graph, in Advances in Neural Information Processing Systems 32, 32 (2019), 1–11.
[64]	B. Zhu, C. W. Ngo, CookGAN: Causality based text-to-image stynthesis, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2020), 5518–5526. https://doi.org/10.1109/CVPR42600.2020.00556
[65]	J. Cheng, F. X. Wu, Y. L. Tian, L. Wang, D. Tao, RiFeGAN: Rich feature generation for text-to-image synthesis from prior knowledge, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2020), 10908–10917. https://doi.org/10.1109/CVPR42600.2020.01092
[66]	J. Cheng, F. X. Wu, Y. L. Tian, L. Wang, D. Tao, RiFeGAN2: Rich feature generation for text-to-image synthesis from constrained prior knowledge, IEEE Trans. Circuits Syst. Video Technol., 32 (2022), 5187–5200. https://doi.org/10.1109/TCSVT.2021.3136857 doi: 10.1109/TCSVT.2021.3136857
[67]	Y. H. Yang, L. Wang, D. Xie, C. Deng, D. Tao, Multi-sentence auxiliary adversarial networks for fine-grained text-to-image synthesis, IEEE Trans. Image Process., 30 (2021), 2798–2809. https://doi.org/10.1109/TIP.2021.3055062 doi: 10.1109/TIP.2021.3055062
[68]	S. L. Ruan, Y. Zhang, K. Zhang, Y. B. Fan, F. Tang, Q. Liu, et al., DAE-GAN: Dynamic aspect-aware GAN for text-to-image synthesis, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2021), 13940–13949. https://doi.org/10.1109/ICCV48922.2021.01370
[69]	M. Lee, J. Seok, Controllable generative adversarial network, IEEE Access, 7 (2019), 28158–28169. https://doi.org/10.1109/ACCESS.2019.2899108 doi: 10.1109/ACCESS.2019.2899108
[70]	T. T. Qiao, J. Zhang, D. Q. Xu, D. C. Tao, Learn, imagine and create: Text-to-image generation from prior knowledge, in Advances in Neural Information Processing Systems 32, 32 (2019), 1–11.
[71]	S. Nam, Y. Kim, S. J. Kim, Text-adaptive generative adversarial networks: Manipulating images with natural language, in 2018 Advances in Neural Information Processing Systems 31, 31 (2018), 42–51.
[72]	B. W. Li, X. J. Qi, T. Lukasiewicz, P. Torr, ManiGAN: Text-guided image manipulation, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2020), 7877–7886. https://doi.org/10.1109/CVPR42600.2020.00790
[73]	J. Peng, Y. Y. Zhou, X. S. Sun, L. J. Cao, Y. J. Wu, F. Y. Huang, et al., Knowledge-driven generative adversarial network for text-to-image synthesis, IEEE Trans. Multimedia, 24 (2020), 4356–4366. https://doi.org/10.1109/TMM.2021.3116416 doi: 10.1109/TMM.2021.3116416
[74]	T. Hinz, S. Heinrich, S. Wermter, Semantic object accuracy for generative text-to-image synthesis, IEEE Trans. Pattern Anal. Mach. Intell., 44 (2022), 1552–1565. https://doi.org/10.1109/TPAMI.2020.3021209 doi: 10.1109/TPAMI.2020.3021209
[75]	H. Wang, G. H. Lin, S. C. H. Hoi, C. Y. Miao, Cycle-consistent inverse GAN for text-to-image synthesis, in Proceedings of the 29th ACM International Conference on Multimedia, ACM, (2021), 630–638. https://doi.org/10.1145/3474085.3475226
[76]	H. C. Tan, X. P. Liu, B. C. Yin, X. Li, Cross-modal semantic matching generative adversarial networks for text-to-image synthesis, IEEE Trans. Multimedia, 24 (2022), 832–845. https://doi.org/10.1109/TMM.2021.3060291 doi: 10.1109/TMM.2021.3060291
[77]	M. Cha, Y. L. Gwon, H. T. Kung, Adversarial learning of semantic relevance in text to image synthesis, in 2019 Proceedings of the AAAI conference on artificial intelligence, AAAI, (2019), 3272–3279. https://doi.org/10.1609/aaai.v33i01.33013272
[78]	K. E. Ak, J. H. Lim, J. Y. Tham, A. A. Kassim, Semantically consistent text to fashion image synthesis with an enhanced attentional generative adversarial network, Pattern Recognit. Lett., 135 (2020), 22–29. https://doi.org/10.1016/j.patrec.2020.02.030 doi: 10.1016/j.patrec.2020.02.030
[79]	G. J. Yin, B. Liu, L. Sheng, N. H. Yu, X. G. Wang, J. Shao, Semantics disentangling for text-to-image generation, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019), 2322–2331. https://doi.org/10.1109/CVPR.2019.00243
[80]	W. H. Xia, Y. J. Yang, J. H. Xue, B. Y. Wu, TediGAN: Text-guided diverse face image generation and manipulation, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2021), 2256–2265. https://doi.org/10.1109/CVPR46437.2021.00229
[81]	J. C. Sun, Y. M. Zhou, B. Zhang, ResFPA-GAN: Text-to-image synthesis with generative adversarial network based on residual block feature pyramid attention, in 2019 IEEE International Conference on Advanced Robotics and its Social Impacts (ARSO), IEEE, (2019), 317–322. https://doi.org/10.1109/ARSO46408.2019.8948717
[82]	M. E. Nilsback, A. Zisserman, Automated flower classification over a large number of classes, in 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, IEEE, (2008), 722–729. https://doi.org/10.1109/ICVGIP.2008.47
[83]	C. Wah, S. Branson, P. Welinder, P. Perona, S. Belongie, The caltech-ucsd birds-200-2011 dataset: Technical report CNS-TR-2011-001, (2011), 1–8.
[84]	Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, Reading digits in natural images with unsupervised feature learning, in NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, NeurIPS, (2011), 1–9.
[85]	A. Krizhevsky, Learning Multiple Layers of Features from Tiny Images, Master's thesis, University of Toronto, 2009.
[86]	T. Karras, T. Aila, S. Laine, J. Lehtinen, Progressive growing of GANs for improved quality, stability, and variation, preprint, arXiv: 1710.10196.
[87]	Y. Lecun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition, Proc. IEEE, 86 (1998), 2278–2324. https://doi.org/10.1109/5.726791 doi: 10.1109/5.726791
[88]	M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, S. Hochreiter, GANs trained by a two time-scale update rule converge to a local nash equilibrium, in Advances in Neural Information Processing Systems 30, 30 (2017), 1–12.
[89]	T. Sylvain, P. C. Zhang, Y. Bengio, R. D. Hjelm, S. Sharma, Object-centric image generation from layouts, preprint, arXiv: 2003.07449.
[90]	Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process., 13 (2004), 600–612. https://doi.org/10.1109/TIP.2003.819861 doi: 10.1109/TIP.2003.819861
[91]	R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang, The unreasonable effectiveness of deep features as a perceptual metric, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2018), 586–595. https://doi.org/10.1109/CVPR.2018.00068
[92]	S. R. Zhou, M. L. Gordon, R. Krishna, A. Narcomey, L. F. Fei-Fei, M. Bernstein, HYPE: A benchmark for human eye perceptual evaluation of generative models, in Advances in Neural Information Processing Systems 32, 32 (2019), 3449–3461.
[93]	M. Wang, C. Y. Lang, L. Q. Liang, S. H. Feng, T. Wang, Y. T. Gao, End-to-end text-to-image synthesis with spatial constrains, ACM Trans. Intell. Syst. Technol., 11 (2020), 2157–6904. https://doi.org/10.1145/3391709 doi: 10.1145/3391709
[94]	F. W. Tan, S. Feng, V. Ordonez, Text2Scene: Generating compositional scenes from textual descriptions, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019), 6703–6712. https://doi.org/10.1109/CVPR.2019.00687
[95]	B. Zhao, W. D. Yin, L. L. Meng, L. Sigal, Layout2image: Image generation from layout, Int. J. Comput. Vision, 128 (2020), 2418–2435. https://doi.org/10.1007/s11263-020-01300-7 doi: 10.1007/s11263-020-01300-7
[96]	S. Hong, D. D. Yang, J. Choi, H. Lee, Inferring semantic layout for hierarchical text-to-image synthesis, in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2018), 7986–7994. https://doi.org/10.1109/CVPR.2018.00833
[97]	M. Q. Huang, Z. D. Mao, P. H. Wang, Q. Wang, Y. D. Zhang, DSE-GAN: Dynamic semantic evolution generative adversarial network for text-to-image generation, in Proceedings of the 30th ACM International Conference on Multimedia, ACM, (2022), 4345–4354. https://doi.org/10.1145/3503161.3547881
[98]	F. Fang, Z. Q. Li, F. Luo, C. J. Long, S. Hu, C. Xiao, PhraseGAN: Phrase-boost generative adversarial network for text-to-image generation, in 2022 IEEE International Conference on Multimedia and Expo (ICME), IEEE, (2022), 1–6. https://doi.org/10.1109/ICME52920.2022.9859623
[99]	Z. H. Li, M. R. Min, K. Li, C. L. Xu, StyleT2I: Toward compositional and high-fidelity text-to-image synthesis, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2022), 18176–18186. https://doi.org/10.1109/CVPR52688.2022.01766
[100]	H. Luo, Y. R. Wang, J. Cui, A SVDD approach of fuzzy classification for analog circuit fault diagnosis with FWT as preprocessor, Expert Syst. Appl., 38 (2011), 10554–10561. https://doi.org/10.1016/j.eswa.2011.02.087 doi: 10.1016/j.eswa.2011.02.087
[101]	M. Versaci, F. C. Morabito, G. Angiulli, Adaptive image contrast enhancement by computing distances into a 4-dimensional fuzzy unit hypercube, IEEE Access, 5 (2017), 26922–26931. https://doi.org/10.1109/ACCESS.2017.2776349 doi: 10.1109/ACCESS.2017.2776349

Reader Comments

Your name:*

Email:*
© 2023 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)