Feature fusion based artificial neural network model for disease detection of bean leaves

1.   Introduction

In order to feed the world population, productivity in agricultural production should be increased by 60% annually until 2050 ^[1]. Plant diseases are one of the most important agricultural problems affecting crop yield and quality. Studies have shown that plant diseases reduce crop yields by 20–40% ^[2]. Most of the strategies used to prevent plant diseases today are based on the use of chemical pesticides ^[2]. It has been determined that 78–79% overdose pesticides are used in pesticide applications without considering the plant need and the prevalence of the disease ^[3]. Excessive use of chemicals causes the emergence of disease and pest species that are more resistant to pesticides ^[4,5]. In addition, due to climate change, there are changes in the emergence time and periods of diseases ^[6,7,8]. For these reasons, there is a need for systems that will detect the location of the disease and apply chemicals only to these areas, thus preventing excessive pesticide use.

According to past studies, 42% of agricultural production is in loss due to plant leaf diseases ^[9]. Detection of leaf diseases is commonly made with the diagnosis of an experienced plant pathologist. However, this method can be used in a limited way due to the small number of specialists and the slow diagnosis made in this way. Complex systems are frequently modelled using machine learning techniques ^[10,11]. The complexity of agriculture makes machine learning techniques well-suited for detecting leaf diseases with expert-level precision and significantly faster speed. ^[12]. Since there are many different diseases, variations within the disease, and plant varieties, more studies are needed to obtain a generalizable leaf disease classification solution in this regard ^[13]. The generalization of a machine learning model is that the classification success achieved during training and validation is also sustainable on new samples that the model has never seen ^[14].

Artificial neural network models for computer vision need very large image datasets for training ^[15]. However, collecting and tagging large image datasets is expensive in terms of both time and money ^[16]. Methods such as transfer learning, data regularization and data augmentation are used to obtain a model that can generalize the maximum from the available data ^[17,18,19]. In addition, it has been seen in various studies that fusion of descriptive features obtained by different methods can increase classification accuracy and generalization success, and that the machine learning model will be converged to the solution faster ^[20,21,22].

1.1.   Literature review

Chen et al. ^[23] used AlexNet modificated architecture-based CNN on the Android platform to predict tomato leaf diseases using training and testing data containing 18,345 and 4585 images respectively. The best model accuracy is 98%.

Fan et al. ^[24] presented a general framework for identifying plant diseases by fusing deep feature descriptors and traditional handcrafted features. Extensive experiments have been conducted to validate the efficacy of the proposed method, which achieves classification accuracies of 99.79, 92.59 and 97.12% on three datasets (Two apple leaf datasets and one coffee leaf dataset). They used InceptionV3 architecture as deep feature descriptors.

Elfatimi et al. ^[25] suggested a method for classifying bean leaf diseases based on a publicly available bean leaf image dataset using MobileNetV2 model architecture as a deep feature extractor. The trained model achieved 97% accuracy on training data and 92% accuracy on test data.

Harakannanavar et al. ^[26] conducted a research by using machine learning and image processing to detect leaf diseases in tomato plants. The extracted features are classified using machine learning approaches such as Support Vector Machine (SVM), Convolutional Neural Network (CNN) and K-Nearest Neighbor (K-NN). The accuracy of the proposed model is tested using SVM (88%), K-NN (97%) and CNN (99.6%). The proposed model extracts informative features using computer vision techniques such as RGB conversion, Histogram Equalization, K-means clustering, contour tracing, Discrete Wavelet Transform, Principal Component Analysis, and GLCM (gray level co-occurrences matrix). They suggest that they can improve accuracy by using fusion techniques in their future works.

Annrose et al. ^[27] proposed a hybrid deep learning model with an Archimedes optimization algorithm (HDL-AOA) for the classification of soybean diseases. The hybrid deep learning strategy is built of Wavelet packet decomposition (WPD) and long short-term memory (LSTM). The WPD disassembles the input pictures into four subseries, and four LSTM networks were created. The HDL-AOA model is applied in a cloud-based framework for collaboration and achieves a lower MAPE (mean absolute percentage error) than other existing techniques such as RNN, DCNN, LSTM and CNN. The suggested HDL-AOA model has an accuracy of 98.23%.

Singh et al. ^[28] employed an artificial intelligence and computer vision approaches to construct and develop an intelligent leaf disease classification system. The PlantVillage data set (for apple, maize, potato, tomato, and rice plants) photos are augmented, and deep features are retrieved using a convolutional neural network (CNN). The picture was then subjected to preprocessing and feature extraction via color moments, HOG, and GLCM. For the selection of these hybrid features, binary particle swarm optimization is utilized, followed by random forest classification for classification results. Five convolutional neural network architectures were trained and evaluated, namely LeNet, ShuffleNet, AlexNet, EffNet, and MobileNet MobileNet achieved the highest accuracy (96.1%).

1.2.   Proposed method

In this study, we aimed to detect the disease in bean leaves, which is one of the commonly produced cultivars, via digital images. For this purpose, we used an image dataset consisting of healthy, bean rust and angular leaf spot classes for the training of the machine learning model. In order to increase the classification accuracy of the developed model and the generalization success on the test dataset, an artificial neural network model was created and trained by fusion of descriptive vectors obtained by transfer learning feature extraction and HOG feature extraction methods. CNN-based machine learning models trained in the detection of leaf diseases will converge to the solution in less time and with more accuracy by employing fewer parameters according to the proposed method.

2.   Materials and methods

2.1.   Dataset

The bean leaf dataset consists of images taken with smartphone cameras. The captured images consist of bean rust, angular leaf spot and healthy classes (Figure 1). The dataset was collected by the Makerere AI research lab and tagged by experts at the National Crops Resources Research Institute (NaCRRI) ^[29]. Images are 500 × 500 pixels in RGB (red-green-blue) digital format.

We divided the image dataset into training, test and validation datasets to be used in the training, validation and testing of the machine learning model. There are 1034 images in the training dataset, 128 images in the test dataset, and 134 images in the validation dataset. The total number of images in the healthy, bean rust and angular leaf spot classes are given in Table 2. We used training and validation datasets in the training of the machine learning model. We tested the developed model using a test dataset that has not been used in training before.

2.2.   Data augmentation

Data augmentation is a technique to improve the diversity of datasets. It is used in machine learning and deep learning to increase the size of training data, which can be very useful in cases where the training dataset is too small. Data augmentation involves adding more samples to the original dataset so that each sample has a different representation from the others. This can be done by translating/rotating/cropping/scaling/randomly changing pixels of an image ^[30]. In the data augmentation phase, each image in the training and validation dataset was subjected to randomly selected augmentation process (es) before entering to the artificial neural network (Figure 2). Test dataset representing real-world, unseen data. Consequently, we did not apply data augmentation to the test dataset for this reason.

2.3.   HOG feature extraction

The histogram of oriented gradients (HOG) is an image feature descriptor and widely used in computer vision applications ^[31]. It simplifies images by discarding extraneous information and retaining only local texture and appearance information. Since diseased areas on the leaf have different local texture characteristics from healthy areas, HOG is effective for use in disease detection. The HOG function in the scikit-image 0.19.3 library was used with its default settings for HOG feature extraction (Figure 3) ^[32].

The calculation in HOG feature extraction was performed as follows.

1) RGB image ($R\left(r, c\right)$, $G\left(r, c\right)$, $B\left(r, c\right)$) (r: row, c: column) firstly converted to grayscale image ($I\left(r, c\right)$)

2) The grayscale image was normalized with gamma correction to reduce the effects of change in brightness.

3) Gradients were calculated separately for each pixel in the horizontal (${G}_{x}(r, c)$) and vertical (${G}_{y}(r, c)$) directions.

4) Then, the magnitude ($\mu$) and direction angle ($\theta$) values that make up the gradient matrix were calculated for the gradient values in the horizontal and vertical directions.

5) The gradient matrix was divided into blocks of 8x8 cells and 9 bins histograms (corresponding to 20° per bin) were generated for each cell in the block. The bin that the cell belongs to was calculated according to the direction angle. The value in each bin was calculated according to magnitude. By summing these values, the histogram of that block was found and histogram matrix was created.

6) After the histogram matrix calculation was completed, the blocks in the histogram matrix were scanned with a 3 × 3 kernel so that the kernel shifted one block after each calculation. The histograms of the 9 blocks in the 3x3 kernel were combined and a feature vector (${f}_{{b}_{i}}$) of 81 × 1 size was created for each step.

7) All feature vectors are normalized with the L2 norm

The HOG feature extraction of the images of Figure 1 are given in Figure 4. We performed HOG feature extraction of all images in the dataset Later, we used these images in the training of the artificial neural network.

2.4.   Artificial neural networks

Artificial neural networks are a type of machine learning algorithm inspired by the way the human brain works. Artificial neural networks are composed of layers of neurons, with each neuron connecting to the neurons in the previous layer. This allows for the information processed in one layer to be passed on to subsequent layers, helping the network learn how to solve certain problems by recognizing patterns in the data ^[33].

We used TensorFlow 2.10.0 library to create artificial neural networks and NVIDIA A100 SXM4 40 GB GPU to accelerate training. We coded with Python 3.8.8 in Google Colab environment. We used the following layers in the artificial neural network models: Convolutional layer: A 2-D convolutional layer applies sliding convolutional filters to two-dimensional input. The convolution operation is performed by sliding the filters along the input horizontally and vertically, and calculating the dot product of the weights and input. A bias term is added afterwards. Dense layer: A dense layer, also known as a fully connected layer, connects every neuron in the preceding layer to every neuron in the present one. Flatten layer: These layers flatten the multi-dimensional input tensors into a single dimensional array for inputting it to the next layer. Batch normalization layer: Each input is normalized and zero-centered across all mini-batches by batch normalization layer. This layer is helping to reduce the risk of vanishing/exploding gradients problems at the artificial neural network training. Dropout layer: The Dropout layer randomly sets the input units to 0 with a frequency of rate at each step during training time, which prevents overfitting. Global Average Pooling 2D layer: A 2-D global average pooling layer computes a mean of the height and width dimensions of the input to downsample it. Concatenation layer: The concatenation layer concatenates a list of inputs ^[34].

A loss function is a function that compares the target (${y}_{i}$) and predicted output (${\widehat{y}}_{i}$) values; measures how well the artificial neural network models the training data. Because we have sparse labels we choose sparse categorical cross-entropy as a loss function.

Optimizers are algorithms used to update the parameters of an artificial neural network such as weights and biases to reduce the losses. Adam optimizer ^[35] used in all trials.

An activation function in an artificial neural network defines how the weighted sum of the input ($x$) is transformed into an output. ReLU (rectified linear unit) activation function is used in conv and dense layers in all models. In the last layer, the softmax activation function ($\sigma$) is used to calculate which class the sample will belong to ($k$) based on feature vector ($\overrightarrow{z}$).

Accuracy is a metric that generally describes how the model performs across all classes. It is calculated as the ratio between the number of correct predictions to the total number of predictions.

2.5.   Artificial neural network with HOG feature extraction

We created the artificial neural network model using HOG feature extraction as in Figure 5. Images already have compressed information as they are converted with HOG transform. For this reason, we created a simpler artificial neural network structure. After the RGB images were converted to HOG images, the data was subjected to augmentation and the artificial neural network was trained with these images. We selected batch size as 32 and the training of the model continued for 100 epochs. We used sparse categorical cross-entropy as the loss function and Adam as the optimizer by setting the learning rate to 0.001. We used the softmax activation function in the output layer, and the ReLU activation function in the dense and convolution layers.

2.6.   Artificial neural network with transfer learning feature extraction

Transfer learning allows us to build better performing artificial neural network models faster and with a better accuracy by using previously pre-trained models. In transfer learning, we used a pre-trained model which is trained on a large and general enough dataset to serve as a generic model for our needs. We can use these pre-trained models without having to train a model from scratch on a large dataset.

MobileNet V2 model was developed at Google ^[36], pre-trained on the ImageNet dataset with 1.4M images and 1000 classes of web images. Since ImageNet is a large dataset in terms of number of images and classes, models trained with this dataset and giving good results can be easily adapted for other datasets.

We created an artificial neural network using transfer learning feature extraction as in Figure 6. We chose 32 as the batch size in the artificial neural network. In this artificial neural network, we first subjected RGB images to data augmentation, then we extracted features with MobileNet V2 feature extractor, transformed the extracted features into vector by the global average pooling layer and passing them through various dense, batch normalization, dropout layers. All MobileNet V2 feature extractor weights are frozen for 5 epochs in order to heat the weights of other layers. Learning rate was 0.001 during initial 5 epochs. Then we applied the first fine-tuning process for 45 epochs by unfreezing the last 4 layers of MobileNet V2, reducing the learning rate to 0.0001. Later we applied the second fine-tuning process for 50 epochs by unfreezing the last 6 layers of MobileNet V2 and reducing the learning rate to 0.0001. So, we used 100 epochs in total for training.

2.7.   Artificial neural network with feature fusion

Feature fusion is the process of combining features extracted each of based on a different feature extraction method. Combining features can improve the accuracy of a classification model. We performed feature fusion by concatenating the descriptive feature vectors obtained from the artificial neural network which is using transfer learning feature extraction and the artificial neural network which is using HOG feature extraction.

The artificial neural network using feature fusion is shown in Figure 7. This network consists of two separate branches through which HOG and RGB images pass. After the descriptive feature vectors which contains 128 neurons obtained from each of these branches are combined with a concatenation layer, they reach the output layer by passing through the dense, batch normalization layer, and dropout layers. By using the softmax activation function in the output layer, an estimate is made about which class the input image belongs to. The images were given to the network in batches of 32 images during the training. After training with a learning rate of 0.001 for 5 epochs, we unfreeze the last 4 layers of the MobileNet V2 feature extractor on the transfer learning feature extraction branch to apply the first fine tuning. We changed the learning rate to 0.0001 and trained the model for 45 epochs. Then, for the second fine tuning, we unfreeze the last 6 layers of the MobileNet V2 feature extractor, changed the learning rate to 0.00001 and trained the model for 50 more epochs.

3.   Results

Figures 8 and 9 show the results of accuracy and loss results obtained from training and validation datasets of the artificial neural network using only HOG feature extraction. The accuracy and loss results in the training and validation datasets are not far from each other. Accuracy results remained approximately the same from the start of training and showed minor oscillations. Although the loss values followed a similar structure, especially after the 85th epoch, the loss values of the training dataset remained constant, while the loss value of the validation dataset made some peaks.

Figures 10 and 11 show the graphs of the accuracy and loss results obtained from the training and validation datasets of the artificial neural network using only transfer learning feature extraction. The training and validation accuracy values, which converged during the first 35 epochs, then began to diverge from each other. In addition, after the 80th epoch, both the validation and training set accuracy values remained approximately constant, sitting on a plateau around 92 and 96%, respectively.

Figures 12 and 13 show the graphs of the accuracy and loss results obtained from the training and validation datasets of the artificial neural network using feature fusion, in which HOG and transfer learning feature extraction methods are used simultaneously. When we examine the accuracy and loss results of the feature fusion model, it is seen that the training and validation accuracy and loss values converge after the 20th epoch, and that the oscillation of the accuracy and loss values decreases after the 50th epoch.

We gave the accuracy values of all three models on the training, validation and test datasets in Table 3. When the model using feature fusion is compared to the model using transfer learning feature extraction method, it has been seen that the accuracy value is 1.65, 5.53 and 1.53% higher in the training, validation and test datasets, respectively. These results show us that the feature fusion model increases classification accuracy compared to the other two models. When we compared the accuracy values in the training set of the feature fusion model with the accuracy values in the validation and test datasets, we found a difference of -0.07 and -0.9%, respectively.

Confusion matrix of the HOG feature extraction model on the test dataset is shown in Table 4. It is seen that this model predicts the majority of the samples given in the test dataset as healthy class. Confusion matrix of the transfer learning feature extraction model on the test dataset is given in Table 5. In this model, the most error was estimating the images of the healthy class as belonging to the bean rust class. Confusion matrix of the feature fusion model on the test dataset is given in Table 6.

In Table 4, we examined the confusion matrix obtained using the test dataset with the HOG feature extraction model. 77.27% of the leaves belonging to the healthy class were classified as healthy, 2.27% as bean rust, and 20.45% as angular leaf spot. 80% of the leaves belonging to the bean rust class were classified as healthy, 4.44% as bean rust, and 15.56% as angular leaf spot. 47.72% of the leaves belonging to the angular leaf spot class were classified as healthy and 52.28% classified correctly.

In Table 5, we examined the confusion matrix obtained using the test dataset with the Transfer learning feature extraction model. 93.18% of the leaves belonging to the healthy class were classified as healthy and 6.82% as bean rust. All of the leaves belonging to the bean rust class and angular leaf spot classes were classified correctly.

In Table 6, we examined the confusion matrix obtained by using the test dataset with the feature fusion model. All of the leaves belonging to the healthy and angular leaf spot classes were classified correctly. On the other hand, 97.78% of the leaves belonging to the bean rust class were classified as bean rust and 2.2% as angular leaf spot.

In Table 7, we examined the classification report of three different models on the test dataset Feature fusion model have best values on precision, recall and F1-scores.

4.   Discussion

In the HOG feature extraction model, HOG images have compressed information. Because of that the HOG feature extraction model was kept shallow to prevent overfitting. Also HOG feature extraction model has been trained from scratch with a shallower structure, so the loss and accuracy values have remained low compared to other methods (Figures 8 and 9). Fluctuation in validation losses can be about the small validation dataset However, the accuracy and loss values obtained in the training and validation sets are not far from each other. In this sense, we can say that HOG images are successful in preventing overfitting.

In the transfer learning feature extraction model, the training and validation accuracy values, which converged during the first 35 epochs, then started to diverge from each other. his indicates overfitting. In addition, after the 80th epoch, both validation and training dataset accuracy values remained approximately constant, reaching a plateau in the 92 and 96% bands, respectively (Figure 10).

When we examine the accuracy and loss results of the feature fusion model, in which HOG and transfer learning feature extraction methods are used simultaneously, it is seen that they training and validation results converge after the 20th epoch, and that the oscillation of accuracy and loss values decreases after the 50th epoch (Figures 12 and 13). The results indicate that this artificial neural network with feature fusion shortens the converge time, increases the accuracy, and a more generalizable model can be obtained ^[24,25].

The results of the accuracy performances of all three models on training, validation and test datasets are given in Table 3. When the feature fusion model is compared with the transfer learning feature extraction model, it is seen that it provides 1.65, 5.53 and 1.53% performance increase in accuracy on training, validation and test datasets, respectively. These results show us that the feature fusion model increases classification accuracy compared to the other two models.

In Table 4, we examined the confusion matrix obtained using the test dataset with the HOG feature extraction model. 77.27% of the leaves belonging to the healthy class were classified as healthy, 2.27% as bean rust, and 20.45% as angular leaf spot. 80% of the leaves belonging to the bean rust class were classified as healthy, 4.44% as bean rust, and 15.56% as angular leaf spot. 47.72% of the leaves belonging to the angular leaf spot class were classified as healthy and 52.28% classified correctly. These results show that the HOG feature extraction model is not very successful in classification and especially fails to distinguish diseased leaves from healthy ones. The shallow nature of the HOG feature extraction model and the use of only HOG images that already have compressed information are the most important reasons for this underfitting.

In Table 5, we examined the confusion matrix obtained using the test dataset with the Transfer learning feature extraction model. 93.18% of the leaves belonging to the Healthy class were classified as healthy and 6.82% as bean rust. All of the leaves belonging to the Bean rust class and Angular leaf spot classes were classified correctly. These results show that while the diseased leaves are correctly detected with the transfer learning feature extraction model, some healthy leaves are misclassified as bean rust.

In Table 6, we examined the confusion matrix obtained by using the test dataset with the feature fusion model. All of the leaves belonging to the healthy and angular leaf spot classes were classified correctly. On the other hand, 97.78% of the leaves belonging to the bean rust class were classified as bean rust and 2.2% as angular leaf spot. These results show that while healthy and angular leaf spot leaves were detected correctly with the feature fusion model, some healthy leaves were misclassified as angular leaf spot.

There are two main reasons for the misclassifications in Tables 5 and 6. The first and most important is that the dataset we use is not large enough and the second there are background scenes in most of the images. The fact that the background scenes have similar color and texture to the diseases may be the reason for the error ^[26,28].

Figure 14 shows some images and visual detection results in the test dataset by using feature fusion model. Lime v0.2.0.1 library was used to visualize the results ^[37]. As seen in visual detection column, the most important parts of the image used by the model to make a decision for classification, are outlined in green, and the unimportant parts for the model are outlined in red.

It is seen that the proposed feature fusion model has shown state of the art accuracy when compared with other studies using ibean dataset and transfer learning method ^[25,38].

In recent years, image classification by transformer-based methods has been reported to give more successful results in terms of accuracy than CNN-based methods ^[39,40]. However, training transformer models from scratch requires much more data than CNN models, but agricultural production has a periodic structure, and it is often not possible to collect more images of the problem in question ^[41]. Also, transformer methods usually result in larger model size. Therefore, the operation of models using the transformer method may be slower than CNN models in computer vision ^[42]. If the models developed to detect leaf diseases are used in smart spraying machines (only spraying the areas where the disease is present) machines, they should be able to work in mobile and edge devices with low computational resources ^[43]. In this respect, it is important that the models developed are not only accurate but also suitable for working on mobile and edge devices. Yu et al. ^[44] developed the Inception Convolutional Vision Transformer (ICVT) model by mixing CNN and transformer architecture and tested it on the ibean dataset. The number of parameters, which was 49M in the ViT/16 model ^[45], has decreased to 25M. The model we developed that is based on CNN feature fusion achieved slightly better accuracy compared to the ICVT model with 34% less (16M) parameters.

Furthermore, inductive bias plays an important role in the ability of machine learning models to generalize to unseen data, and CNN models have a higher inductive bias than transformer models ^[43].

5.   Conclusions

In this study, we made feature fusion of descriptive vectors obtained from two different machine learning models, which are using HOG feature extraction and transfer learning feature extraction, to classify healthy, bean rust and angular leaf spot classes on bean leaf images. The model developed in the study is unique in terms of its network structure, pre-processing processes used, input types and the use of this method for disease detection in bean leaves. This model is composed of two different branches that pass-through HOG and RGB images. After combining the descriptive characteristic vectors that contain 128 neurons obtained from each of these branches with concatenation layer, they reach the output layer through the dense, batch normalization and dropout layers. The softmax activation function in the output layer is used to estimate the class of input image. The model using feature fusion has higher accuracy on training, validation and test datasets than models using only HOG feature extraction and only Transfer Learning feature extraction. In addition, this new method provides a faster converge to the solution during the training of the artificial neural network. The present study has only examined bean leaf dataset Therefore, we will apply this model on various datasets to obtain better generalizability. Also, we will focus on model size optimization for future mobile applications.

Conflicts of interest

The authors declare that there are no conflicts of interest.