Image classification of Chinese medicinal flowers based on convolutional neural network

1.   Introduction

Chinese medicinal flowers are very important in traditional Chinese medicine. The pharmacology of Chinese medicinal flowers is widely used in the prevention and treatment of various diseases. It has been an indispensable medicinal tool for thousands of years ^[1,2]. Tree flowers are one of the important categories of traditional Chinese medicine and serve not only as food but also as herbal medicine to treat diseases. The variety and complexity of traditional Chinese medicine flowers are many and many medicinal plants look alike and confusing. Only experienced experts can identify and classify the tree flowers of Chinese medicine accurately. However, relying on visual assessment from experts for herb identification is time consuming and subjective ^[3]. It is suggested to develop an efficient and fast identification model to identify flowers correctly and prevent herb poisonings ^[4].

Common identification methods on traditional Chinese medicinal materials use DNA barcodes, FT-IR, SD-IR and 2D-IR biochemical identification ^[5,6]. Nowadays, deep learning recognizes images automatically and effectively and convolutional neural networks have achieved favorable results in image classification. Jahanbakhshi et al. used Multilayer perceptron (MLP), Fuzzy, k-nearest neighbor (KNN), support vector machine (SVM), gradient boosting tree (GBT) and evolution decision tree (EDT) to classify the extracted features ^[7]. The feature extraction of traditional machine learning mainly relies on manual labor while deep learning automatically extracts features from images. Convolutional neural networks have achieved good results in the classification research of flower recognition ^[8] and traditional Chinese medicine recognition ^[2]. The related literatures using convolutional neural networks on the classification of multiple traditional Chinese medicines are few. Chai et al. ^[9] and Jahanbakhshi et al. ^[7] classify a single type and its counterfeits. There are also some literatures that classify multiple types of Chinese medicines. For example, Xu et al. ^[10] created a NH-98 Chinese medicine dataset by themselves and used multiple Chinese medicines for classification prediction. However, an unbalanced dataset may affect the research results.

In addition, all of the related Chinese medicine datasets are self-photographed or collected datasets from the internet. Currently, there is no open public dataset of Chinese medicine flowers available. We selected twelve most commonly used and economically valuable Chinese medical flowers and created a dataset of Chinese medicinal flowers in previous study ^[11]. The dataset provides a collection of blossom images both close-up photos and remote photos.

ResNet-101 is a deep residual neural network ^[12] developed by researchers at Microsoft. It is a residual network. The architecture of ResNet-101 is a variant of the ResNet series of models. Compared to the shallower architecture (for example: ResNet-50), it has more layers. The residual learning proposed by ResNet makes the network deeper. It is easier to train and can solve the problem of gradient disappearance (Vanishing Gradient). Since ResNet-101 has deeper layers, it has stronger modeling ability and can capture more complex features and patterns. This makes it perform better in many computer vision tasks such as image classification, object detection ^[13] and semantic segmentation ^[13]. Due to better initialization and better gradient flow, ResNet-101 can often converge faster relative to shallower models. This enables the model to achieve better performance in less time.

SENet is an attention mechanism module to improve network performance. Many literatures add attention modules to the convolutional neural network model ^[14,15]. Neural architecture search (NAS) designs the architectures automatically and has been applied in related works ^{[16,17,18,19]}. A comprehensive review of the NAS algorithm was done to summarize the design principles and enlighten the future direction ^[20]. At present, SENet has been flexibly applied to existing network architectures such as MobileNet and ResNet50 ^[21]. At the same time, SENet is also applicable to other networks. He et al. ^[22] proposed a YOLOv4-based model Mina-Net in 2022 to detect insulator explosion images by using channel attention mechanism to improve the recognition accuracy of the network. The results show that the accuracy of Mina-Net is 88.07% which can effectively improve the detection accuracy of self-explosion of different sizes which is 4.78% higher than that of YOLOv4. Yang et al. ^[23] proposed a DAN-EffcinetNet-B2 model for fish feeding behavior recognition in 2021 with a dual attention network of EffcinetNet-B2. The output feature vectors of the last layer are sent to channel attention and spatial attention respectively. Finally, the outputs of the two attention modules are fused which can focus on the interdependence of space and channels and solve the feature extraction problem of fish gathering areas during feeding. The experimental results show that the test accuracy is 89.56% which is higher than results from AlexNet, VGG, InceptionV3 and ResNet. Zhang et al. ^[24] proposed an aggressive posterior retinopathy of prematurity (AP-ROP) in 2022, adding a channel attention module and a bilinear pooling module to obtain complementary information between layers. The results show that the accuracy of the combined network is 95.81% which is better than ResNet50 and ResNet101.

Zhou et al. ^[14] proposed a model for automatic ore classification in 2022. After data augmentation and transfer learning, MobileNet model has an accuracy of 94% while the accuracy reached 96.89% when adding SENet to MobileNet. Zhao et al. ^[15] developed a vegetable disease recognition model DTL-SE-ResNet50 in 2022, the results show that the recognition accuracy is 97.24% which is better than models such as EfficientNet, AlexNet, VGG19 and InceptionV3.

This study proposes SE-ResNet101 model. To our understanding, this is the first study combining SENet and ResNet101 on Chinese medicinal flower classification. We compare the performances of the proposed SE-ResNet101 models with four models including AlexNet, InceptionV3, ResNet50 and ResNet101. In addition, the CBAM-ResNet101 and SE-CBAM-ResNet101 models are conducted in this study to observe if there is synergy between the different changes. The main contributions of the study are as follows:

1) Develop classification models for the 12 most commonly used Chinese medicinal flowers.

2) The first study combines SENet with ResNet101 models on classification of Chinese medicinal flowers.

3) Compare the performances on connection of attention mechanism and ResNet101 models on classification of Chinese medicinal flowers.

The remainder of this paper is structured as follows. Section 2 introduces the dataset used in this research and the proposed models in this study. Section 3 presents and compares the results of the proposed models with other models. Section 4 presents the ablation experiments. The conclusion of this study is made in Section 5.

2.   Materials and methods

2.1.   Chinese medicinal blossom-dataset

The dataset of blossom images for traditional Chinese medicinal flowers were searched through the web. There are twelve categories: 1) Syringa, 2) Bombax malabarica, 3) Michelia alba, 4) Armeniaca mume, 5) Albizia julibrissin, 6) Pinus massoniana, 7) Eriobotrya japonica, 8) Styphnolobium japonicum, 9) Prunus persica, 10) Firmiana simplex, 11) Ficus religiosa and 12) Areca catechu. The total number of searched images is 1716. The sub-folders are named by blossom categories in Mendeley. Figures 1 and 2 display examples of the close-up photos and remote photos for twelve categories.

2.1.1.   Image preprocessing

Some original images quality are poor. We crop letters and frames, remove handwriting and blurry parts, center flowers, and adjust length and width, etc. Before data augmentation, the numbers of images for each category are 1) Syringa, 191; 2) Bombax malabarica, 172; 3) Michelia alba, 122; 4) Armeniaca mume, 236; 5) Albizia julibrissin, 222; 6) Pinus massoniana, 87; 7) Eriobotrya japonica, 115; 8) Styphnolobium japonicum, 213; 9) Prunus persica, 89; 10) Firmiana simplex, 75; 11) Ficus religiosa, 126; 12) Areca catechu, 68.

2.1.2.   Image augmentation

Data augmentation enhances classification performance and stability of models through creating image diversity. Data augmentation methods are many and we selected Gaussian filtering, image brightness augmentation, image brightness reduction, mirror rotation, noise increase, 90° and 180° rotation methods, seven methods in total in the training and validation datasets. After that, all the images are augmented through mirror flip. Finally, dataset was increased to eight times. Figure 3 shows an example of the original image and the images after data augmentation. Table 1 presents the number of training, validation and test images before and after data augmentation.

2.2.   Research methodologies

Figure 4 displays the flowchart of this study.

Step 1: The blossom images of traditional Chinese medicinal flowers were searched and created through the web. There are both close-up photos and remote photos for each category.

Step 2: Divide the original images into training set, validation set and test set at 80:10:10 ratio.

Step 3: Increase the number of images to eight times through data augmentation.

Step 4: Apply several classification models including AlexNet, InceptionV3, ResNet50 and ResNet101.

Step 5: Compare the performance of the above classification models.

Step 6: Evaluate the ablation experiments of the proposed ResNet101 models on Chinese medicinal flower dataset.

Step 7: Compare the performances of the proposed ResNet101 models with VGG and DenseNet models on Chinese medicinal flower dataset.

2.3.   The proposed SE-ResNet101 model

The more layers of neural network, the more complex the architecture. residual etwork (ResNet) proposes a residual block with shortcut connections which solves the degradation problem of deep networks, accelerates network training and conquers the gradient disappearance and gradient explosion as the number of network layers increases in the model. ResNet101 is a commonly used convolutional neural network. As the name implies, it is a deep convolutional neural network with 101 layers. The network can be divided into five parts: conv1, conv2, conv3, conv4, and conv5. Figure 5 details the architecture of ResNet101 model.

Squeeze-and-excitation network (SENet) is a representative channel attention module with simple structure, easy deployment, no new functions or layers ^[25], and easily combined with the existing network. SENet reduces the error rate of the model, and has low complexity, low calculation amount. Figure 6 presents the architecture of SENet where C, H and W represent the number of image channels, image height and image width, respectively.

Transformation: ${\mathrm{F}}_{\mathrm{t}\mathrm{r}}$ represent the transformation relationship from X to U.

Squeeze: squeeze H and W to one dimension and transform the input of $\mathrm{H}\times \mathrm{W}\times \mathrm{C}$ to 1$\times$1$\times$C.

Excitation: The dimension of s is 1$\times$1$\times$C; C represents the number of channels.

Scaling: The multiplication between s and U to scale the weights. ${\mathrm{u}}_{\mathrm{c}}$ is a two dimension matrix; ${\mathrm{s}}_{\mathrm{c}}$ represents the weight.

The proposed SE-ResNet101 model adds SENet to pre -trained ResNet101 on ImageNet after the last global average pooling layer (GlobalAveragePooling) to achieve better performance. Figure 7 presents the present the layer-by-layer architecture of the proposed SE-ResNet101 model used in this study and the proposed model is detailed as follows:

1) Adding SENet after the last global average pooling layer (GlobalAveragePooling)

2) Adding one more global average pooling layer and dense layer after SENet

3) Using softmax

4) Adding Dropout at 0.5.

2.4.   The proposed CBAM-ResNet101 model

The convolutional block attention module (CBAM) combines channel and spatial attention mechanisms and has been applied to many common convolutional neural networks and it has been confirmed that it can effectively improve the performance of convolutional neural networks for image classification, target detection and other tasks. The CBAM module is mainly divided into a channel attention module and a spatial attention module. The details of the two modules will be explained below ^[25].

1) Channel attention module: first input the feature map to the global maximum pooling and global average pooling layers. Then, after processing by the shared Multilayer perceptron multi-layer perceptron add the two results and operate through the sigmoid activation function to generate each Channel weight values. Finally, multiply each channel weight value with the input feature map.

2) Spatial attention module: Take the maximum value and average value of each feature point of the feature map output by the channel attention mechanism to stack. Then, use the convolutional layer with a channel number of 1 to reduce the dimension and operate through the sigmoid activation function to generate Spatial attention feature map. Finally, multiply this feature map with the input feature map to get the final feature map.

Figure 8 presents the architecture of the propsosed CBAM-ResNet101 model. Adding a CBAM module after the last average pooling layer of ResNet101 can retain more effective features than the original ResNet101. Compared with the SE module, CBAM contains channel attention mechanism and spatial attention mechanism. CBAM combines two attention mechanisms can capture the important features more effectively in the image. The channel attention module (CAM) adaptively learns the correlation between channels, highlights important channels and suppresses secondary channels. The spatial attention module (SAM) learns key areas in space and improves the perception of areas of interest. CBAM can not only improve the feature representation ability of the model, but also help to capture the details and context information in the image.

2.5.   The proposed SE-CBAM-ResNet101 model

Figure 9 presents the architecture of the proposed SE-CBAM-ResNet101 model. The proposed SE-CBAM-ResNet101 model adds a CBAM attention mechanism module and an SE module. The CBAM attention mechanism module is added after the last average pooling layer of ResNet101 and the SE module is added after the CBAM module. The advantage of adding these two attention mechanism modules to ResNet101 is that both the SE module and the CBAM module can perform multi-layer feature fusion. The SE module can perform attention adjustment on the output feature map while the CBAM module performs feature fusion on different modules or different layers of the convolutional neural network. This multi-layer feature fusion can capture different levels of detailed information and contextual relations and enhance the expressive ability of the model. The combination of the two may obtain a more comprehensive and refined feature representation.

2.6.   Evaluation of model performance

A confusion matrix reports the true positive (TP), false positive (FP), true negative (TN) and false negative (FN) which is commonly used to evaluate the classification performance of models. TP represents the number of positive categories that are correctly classified as positive; FP represents the number of negative categories that are incorrectly classified as positive; TN represents the number of negative categories that are correctly classified as negative; and FN represents the number of positive categories that are incorrectly classified as negative. We selected the following performance indicators: accuracy, precision, recall and F1-score to evaluate the performance of each model. In addition, one more metric PR-AUC is calculated for the unbalanced dataset used in this study.

1) Accuracy

2) Precision

3) Recall

4) F1-score

3.   Results

In the experiment, five-fold cross validation is used to evaluate the performance of each model. Table 2 shows the parameter setting for the experiments. The input size is 224 × 224; batch size at 8; epochs count of 50; optimizer of Adam, learning rate of 0.00001 with cross-entropy loss function. The equipment used in the experiment is not the latest, we have an Intel(R) Core(TM) i7-10700F 2.81 GHz CPU, NVIDIA GeForce RTX 2070 8 G GPU using Python 3.7 [Python Software Foundation, Fredericksburg, Virginia, USA] which contains keras 2.6 and tensorflow 2.6.

As presented in Tables 3 and 4, the parameter, accuracy, specificity, sensitivity and F1-score of AlexNet were 62M, 71.28%, 73.65%, 71.28% and 71.35%, respectively. The parameter, accuracy, specificity, sensitivity and F1-score of InceptionV3 were 79.53%, 74.84%, 79.65% and 76.30%, respectively. The parameter, accuracy, specificity, sensitivity and F1-score of ResNet50 were 75.20%, 76.25%, 75.27% and 74.79%, respectively. The parameter, accuracy, specificity, sensitivity and F1-score of ResNet101 were 75.26%, 77.10%, 75.28% and 75.33%, respectively. Figure 10 displays the accuracy and model size for each model. Figure 11 shows the boxplots for all models. From the boxplot, obviously, the accuracy from AlexNet is the lowest while InceptionV3 achieves the highest accuracy of 79.53%. Although, the accuracy comes from InceptionV3 is the highest, the precision from ResNet101 is the highest. ResNet has a simpler, single scale processing unit. Inception focuses on computational cost while ResNet emphasizes on computational accuracy. Therefore, we select ResNet for the further experiments in the following sections.

4.   Ablation experiments

To demonstrate the effectiveness of the proposed model, we have competed several experiments to observe if there is synergy between the different changes by adding squeeze-and-excitation (SE) module, adding convolutional block attention module (CBAM) or using Bayesian optimization (BO). Table 5 presents the result of ablation experiments.

The best performance for accuracy, precision, recall, F1-score and PR-AUC are coming from ResNet101 model with Bayesian optimization (Experiment #3 in Table 5) which are 97.64%, 97.99%, 97.86%, 97.82%, 99.72%, respectively. The parameter setting for the proposed SE-ResNet101 (Experiment #1), CBAM-ResNet101 (Experiment #2), SE-CBAM-ResNet101 (Experiment #4) models are the same. The parameter setting for the experiment #3, #5, #6, and #7 Bayesian optimization is batch size from 8 to 32, epoch 30 to 70 and learning rate ranges 1e-5 to 1e-2. The best performance occurs at batch size 18; epoch count of 56; learning rate of 1.114e-05. From the result in Table 5, compared with adding convolutional block attention module (CBAM), adding squeeze-and-excitation (SE) module in ResNet101 model does not have a positive effect on the performances. In addition, ratio of accuracy to parameter quantity is calculated. Obviously, with the highest accuracy 97.64% and the smallest model size 42.68 M, the highest ratio 2.288 occurs in experiment #3.

In addition to ResNet101, we have conducted more experiments on VGG and DenseNet. As shown in Table 6, VGG16 and VGG19 with and without SE or CBAM are compared. The best accuracy is from VGG16 with SE. The accuracy achieves 97.06% and the PR-AUC is as high as 98.64% (Experiment #1 in Table 6). While the accuracy of Experiment #4 VGG19 with CBAM is 97.05% which is close to the highest accuracy in VGG16-SE model, the PR-ACU is lower than 98%. The accuracy from DenseNet121-SE is 97.06% while the PR-AUC from DenseNet121 with channel attention module achieves 99.62% (Table 7). Comparing the performance in Tables 5–7, the overall highest accuracy 97.64% and the PR-AUC 99.72% come from ResNet101 model with Bayesian optimization (Experiment #3 in Table 5).

5.   Conclusions

Chinese herbal medicine flowers have considerable economic value. Flowers are easily confused due to their similar appearance. The SE-ResNet101 model proposed in this study is used for image recognition and classification on traditional Chinese medicine flowers by using ImageNet for pre-training and add SENet to the trained ResNet101. The research results show that the best performance for accuracy, precision, recall, F1-score and PR-AUC are coming from ResNet101 model with Bayesian optimization which are 97.64%, 97.99%, 97.86%, 97.82%, 99.72%, respectively. Compared with four models including AlexNet, InceptionV3, ResNet50 and ResNet101, our proposed ResNet101 model has achieved the best performance while maintaining the total number of parameters favorable. Since the dataset is limited to twelve common Chinese medicinal flowers, the comprehensive review of Chinese medicinal flowers is suggested in the future research.

Use of AI tools declaration

The authors declare they have not used Artificial Intelligence (AI) tools in the creation of this article.

Acknowledgments

The authors gratefully acknowledge the financial support of the Ministry of Science and Technology of Taiwan through the grant MOST 111-2221-E-167-007-MY3.

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.