Tree species identification based on the fusion of bark and leaves

1.   Introduction

As one of the most important natural forest distribution areas in China, Northeast China is particularly important for the protection of trees. It is of great significance for plant diversity protection and botany research to realize automatic identification and classification of trees. Tree identification mainly uses leaves, flowers and bark ^[1,2,3]. There are also many difficulties in the study of tree species identification, such as ^[4]: (1) There is a lot of characteristic information to distinguish tree species: plant shape, texture, color, etc. (2) There are a lot of noise interference in the plant image under complex background. (3) The same tree will be different in different growth stages and seasons. (4) Tree species image database is incomplete. In order to solve this problem, researchers have done a lot of research on tree species identification and made some progress.

For tree species identification, many methods are proposed, most of which use leaves as recognition organs, and some scholars use bark as recognition organs. The main recognition methods are divided into two categories, which are recognition based on traditional algorithm and recognition based on deep learning.

Traditional algorithms are used to identify and classify trees, such as using AdaBoost, KNN, SVM, etc. ^[5,6,7]. In ^[8], using the combination features both texture features and shape feature and proposing a pre-training method based on the PID to improve the DBNs. In ^[9], the authors proposed a method of blade recognition based on the combination of clonal selection algorithm and support vector machine. In ^[10], the authors presented a leaf recognition system using orthogonal moments as shape descriptors and Histogram of oriented gradients (HOG) and Gabor features as texture descriptors. In ^[11], the authors proposed a novel statistical radial binary pattern (SRBP) descriptor to encode the between-scale texture information within large neighborhood areas using the statistical description of the grey scale intensity distribution. In ^[12], the authors proposed a hierarchical architectural design and another Feature based Shape Selection Template (FSST). Moreover, a novel approach is proposed by using the combination of fuzzy-color and edge-texture histogram in order to recognize fragmented leaf images in 2019 ^[13]. Traditional recognition algorithms need to select features manually, which has limitations and subjectivity. In addition, the recognition accuracy of traditional algorithms is sometimes lower than that of deep learning algorithms.

Due to boosting in data availability, and accompanying by substantial progress in machine learning algorithms, notably CNN, pushed these approaches to a stage where they are better, faster, cheaper and have the potential to significantly contribute to biodiversity and conservation research ^[14]. For example, MATHIEU et al. ^[15] show that there is no dataset with bark images, so a dataset called Baknet1.0 is provided, which contains 23 kinds of trees. And deep learning was used to demonstrate the feasibility of species identification through bark images. In ^[16], using CNN to realize tree species recognition through leaf images, and additional preprocessing steps are employed to increase the robustness of the identification results, this approach is evaluated based on the Leafsnap database and achieves satisfying performance. In ^[17], improving the structure of CNN, ELU excitation function with Maxout was used instead of the ReLU function to solve the model offset and zero gradient problem, and the model is verified on 5 kinds of 10, 000 bark images. Zhou et al. ^[18] provided an effective approach to automatically identify tree species using CNN. The work identifies tree species by analyzing tree leaves, which have multi-dimensional features such as color, shape, and leaf vein signatures. In ^[19], the authors proposed a plant recognition algorithm based on the optimized P-AlexNet model, the model training uses an image dataset with 206 plants, composed of Oxford102 and Ecust104 dataset, and the validation accuracy of the model is 86.7%. In ^[20], a deep learning framework is developed to enable path-based tree classifier training for supporting large-scale plant species recognition, where a deep neural network and a tree classifier are jointly trained in an end-to-end fashion. Compared with the traditional algorithm, Deep learning improves the recognition of accuracy and speed. To sum up, many scholars only use leaves or bark for identification. However, the appearance of a certain plant at different growth periods is different, which is called intra class difference, and the appearance similarity of different plants is called inter class similarity, as shown in Figure 1. And even for experienced botanists, sometimes it is impossible to provide a definite identification based on a single image.

At present, some scholars fuse multiple organs to identify plant species. Guo et al. ^[21] used the framework of CNN, and made the final plant decision based on multiple organs using Linear Weighted Classification and SVM on the dataset of 100 species. Sarah et al. ^[22] proposed a fusion method based on SVM including belief function and fusion of leaves and bark, compared them on a public database of 72 species of trees and shrubs. Rzanny et al. ^[23] collected a completely balanced dataset comprising images of leaf and flower for each of 101 species with an emphasis on groups of conspecific and visually similar species including twelve Poaceae species, using this dataset to train CNN and determining the prediction accuracy for each single perspective and their combinations via score level fusion.

Northeast China is chosen as the research area. Because trees of this area have a half year long leaf fall period, so we choose to collect leaf images in spring and summer. The collection of bark image follows the growth cycle of trees, in order to show that there is intra class difference between trees. The experiment is mainly conducted in two aspects. On the one hand, we train two classifiers for all the leaf images and bark images through CNN. On the other hand, we train a classifier for images of leaf and bark of each tree species through fusion. In the subsequent analysis, the results of the second method will be compared with the first method, and also with baseline approach, that is, the established plant identification system (such as Pl@ntNet ^[24], iNaturalist ^[25]).We combine fusion algorithm with CNN and improved, so that leaves and bark can be better fused. It can promote the research of forestry development, achieve rapid and efficient identification, and more widely applied to the public.

2.   Materials and methods

2.1.   Experiment data

The training model needs to have qualified training images. Today, people can capture, share images and complete recognition through smartphone APP. For example, Pl@ntNet and iNaturalist have a lot of image data. However, there is currently no image that contains both leaves and bark from the same tree, such images also inhibit a wide range of quality. A widely known example is the PlantCLEF dataset ^[26], which is used as benchmark for various computer vision tasks ^[27]. Yet, it is not clear how the results achieved on such a dataset are affected by data imbalance towards image number per species and organs, poor image quality and misidentified species ^[28]. Therefore, we choose self-photographing and web crawling to accomplish the collection of datasets.

Harbin, Heilongjiang Province, Northeast China, is selected as the research area, with abundant tree species. We selected representative trees and classified them according to type. The 10 kinds of trees were divided into 7 families. In order to ensure the diversity of data and better fit the actual situation. Firstly, we use different mobile phones to take photos, including Apple, Meizu and Huawei. Then we choose different scenes to take photos, including the campus and forest farm of Northeast Forestry University in China. Thirdly, we choose different angles and distances, including parallel and 45 degree elevation taking and between 20 cm and 40 cm from trees. Table 1 shows the species and number of images. The number of leaves and bark of each tree is 400 images respectively. We collect images in three ways, the number of images mainly includes: before expansion (original dataset): 2000 bark images and 2000 leaf images, after expansion (flip and translation): 14000 bark images and 14000 leaf images, web crawling: 838 bark images and 1157 leaf images. Figure 2 shows the images of bark and leaf.

2.2.   Methods

Based on the feature fusion of deep learning, ResNet50 ^[29] and DenseNet121 ^[30] are used as methods of tree species identification and cascade fusion, additive fusion and bilinear fusion are used as fusion algorithm. Before the fusion point, the same network model trains the bark and leaves separately, and fuse the feature of the leaves and bark at the fusion point. There are two fusion points, Fully connected layer and Convolution layer. The feature extraction network has ResNet50 and DenseNet121, and finally inputs them into softmax for recognition. In this fusion method, Fusion ResNet50 named FR50 and Fusion DenseNet121 named FD121, Fusion CNN named F-CNN.

2.2.1.   Devices

Those models were trained on the Ubuntu 16.04 LTS 64 system on an GeForce GTX 1080Ti GPU hardware platform, equipped with Intel CoreTM i7-7800XCPU@3.50GHz×12 processors, Pytorch1.0 framework based on deep learning framework, using the Python and MATLAB language.

2.2.2.   F-CNN

Features fusion belongs to the fusion of the intermediate level, and its basic principle is defined as the pattern space:

the sample set as $X$ and $Y$, representing the bark and leaf. It is stated that ${X}_{j}$ and ${Y}_{j}$ represent a feature set of $X$ and $Y$, where $j\in [1, p]$, $p$ is the total number of samples for $X$ and $Y$. The feature sets:

Indicates that the sample has $C$ features, and the sample feature vectors are:

${X}_{j{n}_{i}}^{1}$ ${\mathrm{x}}_{\mathrm{j}{\mathrm{n}}_{\mathrm{i}}}^{1}$ and ${Y}_{j{n}_{i}}^{1}$ represent one dimension of the i-th feature of the sample, and ${n}_{i}$ ${\mathrm{n}}_{\mathrm{i}}$ is the dimension of the feature. $N = \sum _{1}^{C}{n}_{i}$ represents the total dimensions of the sample. Therefore, the correspondence between the sample feature sets ${X}_{j}$, ${Y}_{j}$ and ${\omega }_{k}$.

The two feature vectors output by the two convolution layers are fused to obtain the fused feature vectors, thus connecting the two CNN models together, the connection point is the fusion point. The fusion function is defined as:

In Eq (4), $x$ and $y$ represent feature vectors of bark and leaf obtained by convolutional layer operations, $L$ represents the fusion feature vectors of bark and leaf, and $x$, $y$, $L\in {R}^{HWD}$, $H、W、D$ respectively the length, width and number of channels ^[30].

Cascade fusion function is as follows:

In Eq (5), the number of channels of the fused feature vectors is changed into two times of the original feature vector:

In Eq (6), the $L\in {R}^{H\times W\times 2D}$.

Additive fusion is to add the values of the corresponding position elements of the two feature maps. The number of channels in the fused feature map is constant. Function is as follows:

In Eq (7), the $i\in \left[1, H\right], j\in \left[1, W\right], d\in \left[1, D\right]$,

Bilinear fusion is the summation of the position elements corresponding to two characteristic graphs after the outer product operation. The formula is:

The number of channels in the fused feature map is the square of the number of channels in the original feature map, which is expressed as:

In Eq (9), the ${L}^{bil}\in {R}^{{D}^{2}}$. The fusion point is in the ReLu layer, and then the two feature images corresponding to the channel are fused.

Based on the fusion algorithm described above, we fuse leaves and bark on the ResNet50 and DenseNet121, the output of channel number of feature map is shown in the Table 2.

In this paper, leaves and bark were combined as the input of F-CNN. As shown in Figure 3, there are two fusion points. One is bilinear fusion in the Convolution layer, which is input to the Fully connected layer, and then the classifier output. The second is cascade fusion and additive fusion in the Fully connected layer, then the classifier output.

2.   Results and discussions

The original dataset has 4000 images, divided into train set and test set, 3520 images and 880 images respectively. After flipping and translating, there are 28000 images, 22400 images as training set and 5600 images as test set. One is to use a single organ (bark or leaf) to classify and recognize in ResNet50 and DenseNet121 respectively. The other is to add the fusion algorithm to two network models. The third is to get the confusion matrix of single organ and two organ fusion respectively. The fourth is to compare the organ fusion method using CNN with single organ recognition, SVM and other fusion methods.

In order to make a fair comparison between the test results, the super parameters are standardized in the experiment, as Table 3.

As shown in Table 4, it includes two network models, ResNet50 and DenseNet121, traditional classification algorithm SVM and other fusion algorithm. We give the recognition accuracy of each method. The first is single organ recognition, the recognition accuracy of bark is 75.75%, the recognition accuracy of leaves is 84.00%. The recognition accuracy of three fusion algorithms is 86.75%, 87.65%, 88.50%. For example, cascading fusion is 12.75% higher than bark recognition and 4.5% higher than leaf recognition by deep learning, 23.15% higher than bark recognition and 19.7% higher than leaf recognition by SVM, 12% higher than fusion recognition by SVM. Through the experimental verification, the fusion recognition of leaves and bark is better than single organ recognition and SVM recognition, which effectively solves the similarity and difference between species.

The following confusion matrix shows the effectiveness of the method used before the fusion, whether it has a good recognition effect on the image of leaves and bark, and what problems exist between trees, whether it will be recognized incorrectly and how many kinds of test sets are recognized correctly, and shows which classes are confused in the recognition process. In the matrix, the abscissa represents the prediction label and the ordinate represents the real label, diagonals represent the number of correctly predicted pictures. The darker the color is, the more correct pictures will be recognized. The following is the confusion matrix of ResNet50 and DenseNet121 of bark, leaf, the fusion of bark and leaf. As shown in Figure 4, the confusion matrix of bark, leaf, and fusion. Except for the first type of trees, the recognition accuracy of the remaining 9 kinds of trees has been improved. The confusion matrix shows the effectiveness of the proposed fusion algorithm for tree species identification.

Deep learning needs a large number of samples to form a suitable model. Due to the small amount of original data, the final recognition results are not very good, and it cannot objectively compare the fusion with a single organ, nor can it be shown that the fusion method is better than a single organ recognition, but also easy to be affected by over fitting. In order to solve the problem of sample number and over fitting, the data set is expanded by using the method of flipping and translation, in which the bark sample is expanded to 14000, and the leaf sample is expanded to 14000. Table 5 shows the recognition results after dataset expansion, the recognition accuracy of bark is 78.71%, the recognition accuracy of leaves is 86.75%. The recognition accuracy of three fusion algorithms is 93.02%, 90.43%, 93.17%.

Figure 5 is the confusion matrix of the extended dataset, which shows the single organ recognition and fusion recognition of each tree. Figure 5(a) is the confusion matrix under ResNet50 network, and Figure 5(b) is the confusion matrix under DenseNet121. Overall, more than half of the tree species recognition results improved.

The experiments of single organ recognition and fusion recognition are iterated 30 epochs, and the test results are shown in Figure 6. The red represents the recognition accuracy after the fusion of two organs (bark and leaf), the blue represents the recognition accuracy of leaf, and the green represents the recognition accuracy of bark. The red line is increasing and above the other two lines. The line graph also proves that the image fusion method based on bark and leaves has better effect on tree species recognition, and is obviously better than single organ recognition.

The images of leaf and bark of 7 kinds of trees were crawled through the network, as shown in Table 6. Through flipping and translation, the number of datasets is increased to obtain a more tree species recognition model to prevent overfitting due to data imbalance.

We integrate the trees collected in the field with the trees crawled by the network, and train new CNN models. The classification results are shown in Table 7. The training form is that the training set is 80% of the total and the test set is 20% of the total. The training network is DenseNet121, the accuracy of bark identification is 61.91%, and the accuracy of leaf identification is 78.04%. The recognition accuracy after fusion is 87.86%. For example, ^[31] used comparable methods and achieved an accuracy of 74% for the combination of flower and leaf images using species from the PlantCLEF 2014 dataset. He et al.^[32] extracted confidence scores for each single organ using a state-of-the-art DCNN, and deployed various schemes of the fusion approaches including not only conventional transformation-based approaches (sum rule, max rule, product rule), the accuracy of the proposed leaf and flower fusion technology reaches 82%. Therefore, it is proved that the recognition accuracy after fusion is better than single organ recognition.

Figure 7 is the confusion matrix for all tree species. (a) and (b) are the confusion matrix of bark, leaf, and fusion respectively. In (a), the correct recognition numbers of the 1, 2, 3, 6, 7, 10, 13, 14 and 15 of bark are 22, 30, 32, 34, 51, 40, 37, 13, 13 and 25, the correct recognition numbers of the 1, 2, 3, 6, 7, 10, 13, 14, and 15 of leaf are 42, 28, 59, 32, 76, 34, 35, 26 and 39, the correct recognition numbers of the 1, 2, 3, 6, 7, 10, 13, 14, and 15 of fusion are and 73, 30, 88, 73, 98, 40, 40, 40 and 40. It also shows that the fusion method can solve the problem of intra class differences and inter class similarities better than single organ recognition.

Based on the confusion matrix and according to the classification of families, the recognition accuracy of each family is shown in Table 8. The fusion results of AG, JR, QC, FA, PA and TA are higher than that of single organ recognition. After fusion, the accuracy of Fagaceae, Oleaceae and Aceraceae was improved obviously. It can be further proved that the fusion can improve the accuracy of trees between intra class differences and inter class similarities.

3.   Conclusions

For trees that are easily confused or difficult for people to differentiate, multiple organ recognition can be used to effectively improve. We used the fusion method of CNN and fusion algorithm to fuse leaves and bark. First, the networks of ResNet50 and DenseNet121 which are used to train the image of leaves and bark. Then the fusion algorithm is combined with CNN to realize the fusion of leaves and bark. The confusion matrix is used to illustrate and analyze the feasibility of the fusion method. Leaf and Bark provide quite different sources of information which, when used in combination, considerably improve the recognition result (Figures 4, 5 and 6). In addition, we also verified through three forms of data sets, one is the original dataset, which is 10 kinds of trees, the other is to expand on the basis of the original dataset, and the third is to increase the tree species, a total of 17 trees. The experimental results and confusion matrix show that the fusion method can improve the accuracy of tree species identification; it can effectively improve intra class differences and inter class similarities. This kind of fusion form also has reference significance for the recognition of other fields.

Acknowledgments

This work was supported by the Fundamental Research Funds for the Central Universities (Grants No. 2572017CB10), the Funding of Postdoctoral Research of Heilongjiang Province of China (Grants No. LBH-Z16006).

Conflict of interest

The author declares that he has no conflict of interest.

Availability of data and materials

The image datasets used and analyzed during the current study are available from the corresponding author on reasonable request and under copyright restrictions.