Tuning extreme learning machine by an improved electromagnetism-like mechanism algorithm for classification problem

1.   Introduction

Classification is a very important issue in many fields such as face detection, big data, and disease diagnosis. Especially in recent years, with the development of internet and smart devices, various types of data are exploding. In order to obtain accurate results more efficiently, the traditional image analysis method and signal detection are being replaced by machine learning methods gradually. In all these methods, artificial neural networks (ANNs) ^[1] and support vector machine (SVM) ^[2] are the two most popular methods. For ANNs, many neural network models have been developed, such as back propagation algorithm (BP) ^[3] and convolutional neural networks (CNN) ^[4]. However, these techniques are time-consuming, easy to be trapped in local optima and require the setting of many parameters. To overcome these disadvantages, the extreme learning machine (ELM) method has been proposed for single-hidden layer feed-forward neural network ^[5].

ELM has advantages of high learning speed and excellent classification performance owing to its inherent characteristics of simple structure. Due to the above advantages, ELM has been widely used in various fields, such as localization ^[6], industrial production ^[7], solar radiation prediction ^[8], finite-time optimal control of nonlinear systems ^[9], etc. In addition, ELM has several variants to solve complex problems. Zhang et al. proposed a multilayer probability extreme learning machine for device-free localization ^[10]. Youngmin Park combined convolutional neural network and ELM for image classification ^[11]. In ELM, the input weights and hidden bias are randomly generated without iterative learning ^[12]. Although these settings bring certain advantages, they also increase the risk of overfitting. Besides, the hidden neurons are sensitive to unknown testing data. Traditionally, choice of these parameters mainly depends on prior knowledge and expertise. To solve these problems, it is important to optimize the input weights, hidden bias and the structure of the network.

Intelligent algorithms are naturally considered as the solution to the above problems, such as particle swarm optimization (PSO) ^[13], ant colony optimization (ACO) ^[14], and artificial bee colony algorithm (ABC) ^[15]. Electromagnetism-like mechanism (EM) algorithm, which was developed by Birbil and Fang in 2003 ^[16], is a population-based random search algorithm similar to genetic algorithm (GA). Because of its strong search capability and easy implementation, EM has been successfully applied to optimization problems ^{[16,17,18,19,20,21,22,23]}, such as function optimization ^[17] and flow shop scheduling ^{[18,19,20,21]}. All these previous studies have demonstrated the excellent optimization performance of EM. Therefore, the integration of EM and ELM should be a promising approach in training feedforward neural network.

In this study, an improved EM algorithm called DAEM is proposed by incorporating some theories of dragonfly algorithm (DA) ^[24] into EM approach. By using the new algorithm, we optimized the input weights and hidden biases, and minimum norm least-square scheme was used to analytically determine the output weights in ELM. In the selection of input weights and hidden biases, the improved EM considers not only the classification error rate but also the norm of the output weights as well as constrains the input weights and hidden biases within a reasonable range. In addition, the k-fold cross-validation method is adopted to avoid the problem of overfitting.

The rest of the paper is organized as follows. The theories related to ELM and EM are briefly introduced in Section 2. Section 3 describes the establishment of the DAEM-ELM algorithm. Section 4 presents the results and discussion on eight classification problems to demonstrate the effectiveness of the proposed algorithm. Finally, the conclusions are summarized in Section 5.

2.   Theories and methods

2.1.   Extreme learning machine

The core idea of ELM is to transform the training process of traditional SLFN model into solving the least square solution problem. The main process of ELM consists of random generation of the parameters of hidden neurons, followed by fixing of the hidden layer parameters and then algebraically solving the output weights. The specific theoretical basis of ELM is as follows.

For N arbitrary distinct samples (x_i, t_i), where x_i = [x_i1, x_i2, …, x_in]^T∈Rⁿ, t_i = [t_i1, t_i2, …, t_im]^T∈R^m. The i-th sample x_i is an n × 1 feature vector, and t_i is an m × 1 target vector. The standard mathematical model of SLFNs with L hidden neurons and activation function g (x) is as follows:

where O_j denotes the corresponding actual output vector of x_j, ω_i = [ω_i1, ω_i2, …, ω_in]^T indicates the weight vector connecting the i-th hidden neuron and input neurons, β_i = [β_i1, β_i2, …, β_im]^T represents the weight vector connecting the i-th hidden neuron and output neurons, b_i is the bias of i-th hidden neuron, also known as the threshold, and ω_i·x_j is the inner product of ω_i and x_j. The purpose of training SLFNs is to minimize the error of output value, which means:

Then, the N equations represented by equation (1) can be expressed in matrix form as follows:

where $H = H\left({{\omega _1}, {\omega _2}, ..., {\omega _L}, {b_1}, {b_2}, ..., {b_L}, {x_1}, {x_2}, ..., {x_N}} \right) = {\left[{\begin{array}{*{20}{c}} {g\left({{\omega _1} \cdot {x_1} + {b_1}} \right)} & {...} & {g\left({{\omega _1} \cdot {x_1} + {b_L}} \right)} \\ {...} & {...} & {...} \\ {g\left({{\omega _1} \cdot {x_N} + {b_1}} \right)} & {...} & {g\left({{\omega _L} \cdot {x_N} + {b_L}} \right)} \end{array}} \right]_{N \times L}}$, $\beta {\rm{ = }}{\left[{\begin{array}{*{20}{c}} {\beta _1^T} \\ \vdots \\ {\beta _L^T} \end{array}} \right]_{L \times m}}$, $T = {\left[{\begin{array}{*{20}{c}} {t_1^T} \\ \vdots \\ {t_N^T} \end{array}} \right]_{N \times m}}$, H is the hidden layer output matrix, β is the output weight matrix, and T is the output matrix.

In the algorithm of ELM, when the input weights and hidden layer biases are randomly generated, the determination of the output weights is to find the least-square (LS) solution to the linear system:

where H⁺ is obtained by singular value decomposition of Moore-Penrose (MP) generalized inverse matrix.

The pseudo-code of ELM is as follows:

2.2.   Electromagnetism-like mechanism algorithm

The basic principle of EM is that every feasible solution is compared to a charged particle and the charge of each particle is calculated by the value of the preparative optimization objective function ^[16]. The charge determines not only the type of the force between two particles (either attraction or repulsion), but also the strength of the force. Under the action of attraction and repulsion forces, the population moves to a new generation. The whole process of particle movement under the force in a population is shown in Figure 1. As can be clearly seen from Figure 1, the blue particle is subject to the forces of other particles in the population, both attraction and repulsion. Besides, the optimal particle in the population will always attract other particles to move towards it. Specifically, the EM algorithm mainly includes the following four steps.

(1) Initialization: m particles are randomly selected from the feasible region as initial population. Each coordinate of the particle is uniformly distributed between corresponding upper and lower bounds. Then, the objective function value is calculated for each particle and the particle with the best objective function value is stored in X_best.

(2) Local search: The procedure of the local search conducted on a single particle is to improve the solution obtained. Each dimension of the current optimal particle X_best is searched according to a certain step size. Once a better solution is found, the optimal particle is updated. The effective local information obtained from this procedure can contribute to the abilities of EM of both exploration and exploitation.

(3) Calculation of the resultant force: The magnitude of the force of particle i is strongly related to its charge q_i, which can be calculated by formula (5):

The resultant force F_i exerted on particle i can be calculated as follows:

According to the above formula, the particle with better and poorer objective function values attracts and repulses other particles, respectively. The better the objective function value is, the stronger the attraction will be, and vice versa.

(4) Movement of the population: After calculating the resultant force, the particles are moved in the direction of the force, thus forming a new generation of population. The direction and step of the movement are determined by the following formula:

λ is a random number of 0 to 1, which guarantees that the particles with a nonzero probability move to unvisited regions. RNG is a movable range between the upper and lower sectors.

The pseudo-code of EM algorithm is as follows:

3.   The improved extreme learning machine (DAEM-ELM)

From the introduction of ELM, it can be seen that the input weights and hidden layer biases are randomly generated at the initialization stage. The network constructed in this way may give rise to a problem of overfitting. More specifically, ELM usually requires more hidden neurons than conventional neural networks to achieve the expected performance. Larger network size results in longer running time of the testing phase of ELM, which may hinder its efficient development in some test time sensitive scenarios ^[25]. To solve this problem, an improved approach designated as DAEM-ELM, which combines EM with ELM, is proposed in this paper. This new ELM adopts a novel EM called DAEM to optimize the input weights and biases of ELM to improve the generalization performance and the conditioning of the SLFN. In this section, we will first provide a detailed description of the DAEM algorithm, and then present the DAEM-ELM algorithm.

3.1.   DAEM

The convergence speed of EM slows down gradually during the iterations and the algorithm easily falls into the local optimal solution, that is, prematurity. In addition, the position and number of adjacent particles influence the step length and position update of the population according to formula (1). But it remains unclear under what conditions individual particles can be defined as adjacent to each other. In order to solve these problems, we propose an improved EM algorithm called DAEM, by incorporating some theories of dragonfly algorithm (DA) into EM.

First, the adjacency of particles is defined. A neighborhood (circle in a 2D, sphere in a 3D space, or hypersphere in an nD space) with a certain radius r is assumed around each particle ^[24]. If the Euclidean distance between particle i and particle j is less than r, particle i and particle j are considered as adjacent. In order to accelerate the convergence speed, the radius r increases with increasing number of iterations. The specific calculation formula of r is as follows:

where iter is the number of current iterations, Max_iteration is the maximum number of iterations, ub is the upper limit of variables, and lb is the lower limit of variables. The neighborhood can be represented as (r₁, r₂, ..., r_n) and n represents the number of dimensions.

Taking particle i as an example, it can be expressed as X_i = (x_i1, x_i2, ..., x_in), and then the neighborhood range of particle i in the j-th dimension (j = 1, 2, …, n) is [x_ij-r_j, x_ij + r_j]. When another particle X_k is within the neighborhood range of particle i of each dimension, it is considered that X_k is adjacent to X_i.

For different problems, the fixed solution updating strategy of EM may not be always reasonable, and cannot guarantee the discovery of global optimal solution or approximate global optimal solution. Therefore, DAEM provides three different solution updating strategies motivated by DA. In this way, the suitable updating strategy can be selected according to the prior information of different problems. Besides, variable searching step is adopted to solve the conflicts between solution accuracy and computation time in the optimization process. The three updating strategies are described as follows.

Strategy 1: when the distance between the current particle and the optimal particle in a certain dimension is smaller than the neighborhood radius, and there are adjacent particles in the neighborhood of the current particle, the updated formula of the particle is as follows:

where c = 0.9-iter × (0.5/Max_iteration), η is a random number between [0, 1]. F_i is still calculated by formula (6), but only the particles in the neighborhood instead of the entire population are considered.

Strategy 2: when the distance between the current particle and the optimal particle in a certain dimension is smaller than the neighborhood radius, and there is no particle adjacent to the current particle in its neighborhood, the updated formula of the particle is as follows:

where r₁ and r₂ are two random numbers in [0, 1], and α is a constant (equal to 1.5 in DAEM).

Strategy 3: when the distance between the current particle and the optimal particle in any dimension is greater than the neighborhood radius, the updated formula of the particle is as follows:

where ${\varepsilon _W}$ and ${\varepsilon _B}$ are random numbers in [0, 1], $Worst = {X_{worst}} + {X_i}$, $Best = {X_{best}} - {X_i}$, X_worst and X_best represent the worst and best particle of current population respectively, l is a random numbers in [0, 1], L is the difference between the position of X_best and position of X_i when X_best is in the neighborhood of X_i, otherwise L is a m-dimensional vector of 0's, e = 0.1 – iter × (0.2/Max_iteration) (when e < 0, let e = 0), C is an n-dimensional vector generated randomly.

3.2.   DAEM-ELM

The basic process of DAEM-ELM is described below. Data samples are divided into training sample set and testing sample set. The training set is trained by DAEM to build the classification model. Then, the testing set is tested by the obtained classification model. The detailed steps of the proposed method are as follows.

Firstly, the population is randomly generated. Each particle in the population is composed of a set of input weights and hidden biases. The specific coding form is as follows:

where N is the number of input neurons, and L is the number of hidden neurons, that is, the dimension of each particle is (N + 1) × L. All components in the particle are randomly initialized within the range of [-1, 1].

Secondly, for each particle, the corresponding output weights are computed according to formula (4). Then, the fitness of each particle is evaluated. The fitness function formula of DAEM-ELM is:

where N is the number of training samples, and MCR_i is the misclassification of the algorithm.

Neural network training should not solely rely on the misclassification of training set as the fitness function, because a higher training accuracy does not guarantee a higher test accuracy. It has been reported that neural networks tend to have better generalization performance with weights of smaller norms ^[26,27]. Hence, the output weights are also considered in selection strategy design in order to further enhance the performance of our algorithm. It is stipulated that when the fitness values of different particles are similar, the particle with smaller norm of output weights is chosen as a better solution. The algorithm introduces the tolerance rate λ to meet this requirement and the effect is better when λ is set to 0.04 as validated by experiment. Besides, to reduce the computational complexity of the algorithm, the following regulation formula of current optimal solution is used only after each iteration (the fitness function is still the misclassification, and only the tolerance rate is added for adjustment).

where f (X_i), f (X_ibest) and f (X_gbest) are the corresponding fitness values for the i-th particle, the best position of the i-th particle and global best position of all particles, respectively. $\beta \left( {{X_i}} \right),{\rm{ }}\beta \left( {{X_{ibest}}} \right){\rm{ and }}~\beta \left( {{X_{gbest}}} \right)$ are the corresponding output weights obtained by MP generalized inverse when the input weights are set as the i-th particle, the best position of the i-th particle and global best position of all particles, respectively.

Thirdly, the k-fold cross-validation (k-fold CV) method is applied in order to get an unbiased estimate of the generalization accuracy and make full use of samples in case of insufficient sample. In k-fold CV, the data sample set is divided into k mutually disjoint subsets (approximately equal in size), such as S₁, S₂, …, S_k, and then DAEM-ELM is performed for k iterations. S_k is selected as the testing set and the rest of subsets are used as training set in the i-th iteration. Here, the parameter value of k is 5 and the final classification results are the average value of five iterations.

The pseudo-code of DAEM-ELM is as follows:

4.   Experiment and discussion

4.1.   Datasets and experimental setup

In this section, the performance of the proposed algorithm is evaluated on eight real-world classification problems (Thyroid, WDBC, Wine, Bupa Liver, Australian, Breast Cancer, Parkinson and Iris), and all these data-sets are taken from the University of California Irvine (UCI) repository ^[28]. The specification of these datasets is listed in Table 1.

Iris, Australian and Breast cancer datasets have missing values (A data-set contains a certain number of instances, and an instance includes several attributes. If some instance in a data-set lack some attributes, it is said that the data-set has missing values.). The missing categorical attributes are replaced by the mode of the attributes, and the missing continuous attributes are replaced by the mean of the attributes in order to ensure the integrity of the sample data. Besides, normalization is employed to avoid the influence of the feature values in larger numerical ranges on those in smaller numerical ranges, which can also reduce the computational complexity. Every feature value can be normalized by scaling them into the interval of [-1, 1] according to:

where x′ is the normalization value, x is the original value, min_i is minimum value of feature i and max_i is maximum value of feature i.

The results obtained with the proposed algorithm are then presented and analyzed, and compared with those obtained using other related algorithms. The parameters in all algorithms of experiments are determined by trial and error. For DAEM-ELM, the maximum optimization epochs are 70, and the population size is 50. The sigmoid function g(x) = 1/(1 + exp(-x)) is adopt as the ELM activation function to compute the hidden layer output matrix. All the programs are run in MATLAB 7.0 environment.

4.2.   Benchmark classification

The performance of DAEM-ELM algorithm is tested with the number of hidden neurons increasing from 5 to 40 at a step size of 5. The reason for choosing this range is that an excessive number of hidden neurons will lead to an overfitting problem for ELM. In addition, better parameters more suitable for associated networks can be found with the DAEM algorithm. Hence, the ELM only needs a small number of hidden neurons to obtain better results. The experimental results are shown in Table 2.

As shown in Table 2, the accuracy rate does not simply increase with the number of hidden neurons. When the hidden neurons increase to a certain number, further increase will lead to a decline in accuracy. Secondly, the optimal number of hidden neurons varies for different problems. Specifically, the optimal number is 20, 25 and 35 for the Thyroid dataset, and we chose the smallest number (20) to reduce the computational time of the network. For the Parkinson, Iris, Bupa Liver, Australian, Breast Cancer, Wine and WDBC datasets, the optimal number is 40, 5, 15, 35, 20, 10 and 10, respectively.

Table 3 and Table 4 show the results achieved with all seven investigated methods (ABC-ELM ^[29], ELM, PSO-ELM ^[29], IPSO-ELM ^[29], E-ELM ^[29], LM and DAEM-ELM) for the eight benchmark classification datasets based on five trials. The last column of Table 3 shows the smallest number of hidden neurons to be used in order to achieve the best results. It is evident that a higher accuracy rate and a smaller number of hidden neurons represent a better mode. In addition, to test the accuracy, the nearly optimal number of hidden neurons for these algorithms and the standard deviations are shown in the table in the form of mean ± standard deviation. From Table 3 and Table 4, we can draw the following conclusions.

For the Thyroid dataset, although the number of hidden neurons of DAEM-ELM is not the smallest (only slightly bigger than that of ABC-ELM), its classification accuracy is the highest among seven methods and the standard deviation of the acquired performance is also the smallest (equal to 0), indicating the consistency and stability of the proposed method. The results of the Thyroid dataset are shown in Figure 2(a).

For the Parkinson dataset, ABC-ELM has the fewest hidden neurons. But the proposed method outperforms other six methods in terms of classification accuracy by around 3% (more than 9% compared with that of LM). Besides, DAEM-ELM also has the smallest standard deviation among these methods. Figure 2(b) shows the accuracy obtained on the Parkinson dataset.

For the Iris dataset, DAEM-ELM only needs five hidden neurons to achieve the highest classification accuracy. In this way, the proposed method achieves both the highest classification accuracy and the most compact network structure. The results of the Iris dataset are shown in Figure 2(c).

For the Bupa Liver dataset and WDBC dataset, the proposed method outperforms others in all cases (classification accuracy, number of hidden neurons, standard deviation), as shown in Figure 2(d) and Figure 2(h).

For the Australian dataset and Breast Cancer dataset, the DAEM-ELM still maintains the highest classification accuracy and the smallest standard deviation with a medium number of hidden neurons. These results are also confirmed in Figure 2(e) and Figure 2(f).

For the Wine dataset, ABC-ELM is the algorithm with the best performance. The proposed algorithm has the same number of hidden neurons as ABC-ELM, and only slightly poorer performance in accuracy and standard deviation. Figure 2(g) illustrates the results on the Wine dataset.

In summary, the DAEM-ELM algorithm can achieve better performance by using DAEM to select the input weights and hidden biases of the SLFN than the ELM, PSO-ELM, IPSO-ELM, E-ELM and LM algorithms, indicating that the optimal network structure tuned by the DAEM algorithm contributes greatly to the reduction of hidden neurons in the models and a reasonable generalization performance for these datasets.

5.   Conclusion

In this paper, a novel extreme learning machine based on electromagnetism-like mechanism (DAEM-ELM) is proposed. In the new algorithm, an improved EM is used to optimize the input weights and hidden biases, and minimum norm least-square scheme is employed to analytically determine the output weights. In the optimization process, the improved EM considers not only the misclassification but also the norm of the output weights as well as constrains the input weights and hidden biases within a reasonable range. In addition, the 5-fold CV method is adopted to prevent the overfitting problem. Experimental results show that DAEM-ELM outperforms other methods (original ELM, PSO-ELM, IPSO-ELM, E-ELM and LM) and has a more compact network structure. It is also confirmed that due to the selection of optimal parameters, the results are more stable with fewer hidden neurons. Hence, it can be concluded the developed DAEM-ELM algorithm can be a feasible and effective algorithm for classification problems. Future research work will be focused on the identification of the optimal hidden neuron number, input weights and hidden biases at the same time.

Acknowledgments

This research work is supported in part by National Natural Science Foundation of China (NSFC) under Grant No. 61603145.

Conflict of interest

All authors declare no conflict of interest in this paper.