Segmenting plant organs is a crucial step in extracting plant phenotypes. Despite the advancements in point-based neural networks, the field of plant point cloud segmentation suffers from a lack of adequate datasets. In this study, we addressed this issue by generating Arabidopsis models using L-system and proposing the surface-weighted sampling method. This approach enables automated point sampling and annotation, resulting in fully annotated point clouds. To create the Arabidopsis dataset, we employed Voxel Centroid Sampling and Random Sampling as point cloud downsampling methods, effectively reducing the number of points. To enhance the efficiency of semantic segmentation in plant point clouds, we introduced the Plant Stratified Transformer. This network is an improved version of the Stratified Transformer, incorporating the Fast Downsample Layer. Our improved network underwent training and testing on our dataset, and we compared its performance with PointNet++, PAConv, and the original Stratified Transformer network. For semantic segmentation, our improved network achieved mean Precision, Recall, F1-score and IoU of 84.20, 83.03, 83.61 and 73.11%, respectively. It outperformed PointNet++ and PAConv and performed similarly to the original network. Regarding efficiency, the training time and inference time were 714.3 and 597.9 ms, respectively, which were reduced by 320.9 and 271.8 ms, respectively, compared to the original network. The improved network significantly accelerated the speed of feeding point clouds into the network while maintaining segmentation performance. We demonstrated the potential of virtual plants and deep learning methods in rapidly extracting plant phenotypes, contributing to the advancement of plant phenotype research.
Citation: Yuhui Zheng, Dongwei Wang, Ning Jin, Xueguan Zhao, Fengmei Li, Fengbo Sun, Gang Dou, Haoran Bai. The improved stratified transformer for organ segmentation of Arabidopsis[J]. Mathematical Biosciences and Engineering, 2024, 21(3): 4669-4697. doi: 10.3934/mbe.2024205
Segmenting plant organs is a crucial step in extracting plant phenotypes. Despite the advancements in point-based neural networks, the field of plant point cloud segmentation suffers from a lack of adequate datasets. In this study, we addressed this issue by generating Arabidopsis models using L-system and proposing the surface-weighted sampling method. This approach enables automated point sampling and annotation, resulting in fully annotated point clouds. To create the Arabidopsis dataset, we employed Voxel Centroid Sampling and Random Sampling as point cloud downsampling methods, effectively reducing the number of points. To enhance the efficiency of semantic segmentation in plant point clouds, we introduced the Plant Stratified Transformer. This network is an improved version of the Stratified Transformer, incorporating the Fast Downsample Layer. Our improved network underwent training and testing on our dataset, and we compared its performance with PointNet++, PAConv, and the original Stratified Transformer network. For semantic segmentation, our improved network achieved mean Precision, Recall, F1-score and IoU of 84.20, 83.03, 83.61 and 73.11%, respectively. It outperformed PointNet++ and PAConv and performed similarly to the original network. Regarding efficiency, the training time and inference time were 714.3 and 597.9 ms, respectively, which were reduced by 320.9 and 271.8 ms, respectively, compared to the original network. The improved network significantly accelerated the speed of feeding point clouds into the network while maintaining segmentation performance. We demonstrated the potential of virtual plants and deep learning methods in rapidly extracting plant phenotypes, contributing to the advancement of plant phenotype research.
[1] | R. Pieruschka, U. Schurr, Plant phenotyping: past, present, and future, Plant Phenomics, 2019 (2019). https://doi.org/10.34133/2019/7507131 doi: 10.34133/2019/7507131 |
[2] | C. Costa, U. Schurr, F. Loreto, P. Menesatti, S. Carpentier, Plant phenotyping research trends, a science mapping approach, Front. Plant Sci., 9 (2019), 1933. https://doi.org/10.3389/fpls.2018.01933 doi: 10.3389/fpls.2018.01933 |
[3] | A. K. Singh, B. Ganapathysubramanian, S. Sarkar, A. Singh, Deep learning for plant stress phenotyping: trends and future perspectives, Trends Plant Sci., 23 (2018), 883–898. https://doi.org/10.1016/j.tplants.2018.07.004 doi: 10.1016/j.tplants.2018.07.004 |
[4] | S. Arya, K. S. Sandhu, J. Singh, S. Kumar, Deep learning: as the new frontier in high-throughput plant phenotyping, Euphytica, 218 (2022), 47. https://doi.org/10.1007/s10681-022-02992-3 doi: 10.1007/s10681-022-02992-3 |
[5] | S. Bhagat, M. Kokare, V. Haswani, P. Hambarde, R. Kamble, Eff-UNet++: A novel architecture for plant leaf segmentation and counting, Ecol. Inf., 68 (2022), 101583. https://doi.org/10.1016/j.ecoinf.2022.101583 doi: 10.1016/j.ecoinf.2022.101583 |
[6] | K. Khan, R. U. Khan, W. Albattah, A. M. Qamar, End-to-end semantic leaf segmentation framework for plants disease classification, Complexity, 2022 (2022). https://doi.org/10.1155/2022/1168700 doi: 10.1155/2022/1168700 |
[7] | D. Zendler, N. Malagol, A. Schwandner, R. Töpfer, L. Hausmann, E. Zyprian, High-throughput phenotyping of leaf discs infected with grapevine downy mildew using shallow convolutional neural networks, Agronomy, 11 (2021), 1768. https://doi.org/10.3390/agronomy11091768 doi: 10.3390/agronomy11091768 |
[8] | J. Wu, C. Wen, H. Chen, Z. Ma, T. Zhang, H. Su, et al., DS-DETR: A model for tomato leaf disease segmentation and damage evaluation, Agronomy, 12 (2022), 2023. https://doi.org/10.3390/agronomy12092023 doi: 10.3390/agronomy12092023 |
[9] | Y. Wu, L. Xu, Crop organ segmentation and disease identification based on weakly supervised deep neural network, Agronomy, 9 (2019), 737. https://doi.org/10.3390/agronomy9110737 doi: 10.3390/agronomy9110737 |
[10] | Z. Li, R. Guo, M. Li, Y. Chen, G. Li, A review of computer vision technologies for plant phenotyping, Comput. Electron. Agric., 176 (2020), 105672. https://doi.org/10.1016/j.compag.2020.105672 doi: 10.1016/j.compag.2020.105672 |
[11] | Y. Jiang, C. Li, Convolutional neural networks for image-based high-throughput plant phenotyping: a review, Plant Phenomics, 2020 (2020). https://doi.org/10.34133/2020/4152816 doi: 10.34133/2020/4152816 |
[12] | W. D. Kissling, Y. Shi, Z. Koma, C. Meijer, O. Ku, F. Nattino, et al., Laserfarm–A high-throughput workflow for generating geospatial data products of ecosystem structure from airborne laser scanning point clouds, Ecol. Inf., 72 (2022), 101836. https://doi.org/10.1016/j.ecoinf.2022.101836 doi: 10.1016/j.ecoinf.2022.101836 |
[13] | J. Zhou, X. Fu, S. Zhou, J. Zhou, H. Ye, H. T. Nguyen, Automated segmentation of soybean plants from 3D point cloud using machine learning, Comput. Electron. Agric., 162 (2019), 143–153. https://doi.org/10.1016/j.compag.2019.04.014 doi: 10.1016/j.compag.2019.04.014 |
[14] | X. Ma, K. Zhu, H. Guan, J. Feng, S. Yu, G. Liu, Calculation method for phenotypic traits based on the 3D reconstruction of maize canopies, Sensors, 19 (2019), 1201. https://doi.org/10.3390/s19051201 doi: 10.3390/s19051201 |
[15] | S. Wu, W. Wen, Y. Wang, J. Fan, C. Wang, W. Gou, et al., MVS-Pheno: a portable and low-cost phenotyping platform for maize shoots using multiview stereo 3D reconstruction, Plant Phenomics, 2020 (2020). https://doi.org/10.34133/2020/1848437 doi: 10.34133/2020/1848437 |
[16] | H. You, Y. Liu, P. Lei, Z. Qin, Q. You, Segmentation of individual mangrove trees using UAV-based LiDAR data, Ecol. Inf., (2023), 102200. https://doi.org/10.1016/j.ecoinf.2023.102200 doi: 10.1016/j.ecoinf.2023.102200 |
[17] | P. Li, X. Zhang, W. Wang, H. Zheng, X. Yao, Y. Tian, et al., Estimating aboveground and organ biomass of plant canopies across the entire season of rice growth with terrestrial laser scanning, Int. J. Appl. Earth Obs. Geoinf., 91 (2020), 102132. https://doi.org/10.1016/j.jag.2020.102132 doi: 10.1016/j.jag.2020.102132 |
[18] | Y. Sun, Y. Luo, Q. Zhang, L. Xu, L. Wang, P. Zhang, Estimation of crop height distribution for mature rice based on a moving surface and 3D point cloud elevation, Agronomy, 12 (2022), 836. https://doi.org/10.3390/agronomy12040836 doi: 10.3390/agronomy12040836 |
[19] | F. Tardieu, Virtual plants: modelling as a tool for the genomics of tolerance to water deficit, Trends Plant Sci., 8 (2003), 9–14. https://doi.org/10.1016/S1360-1385(02)00008-0 doi: 10.1016/S1360-1385(02)00008-0 |
[20] | P. Prusinkiewicz, Graphical applications of L-systems, in Proceedings of Graphics Interface, Canadian Information Processing Society, Vancouver, Canada, 86 (1986), 247–253. |
[21] | R. Karwowski, P. Prusinkiewicz, Design and implementation of the L+ C modeling language, Electron. Notes Theor. Comput. Sci., 86 (2003), 134–152. https://doi.org/10.1016/S1571-0661(04)80680-7 doi: 10.1016/S1571-0661(04)80680-7 |
[22] | F. Boudon, C. Pradal, T. Cokelaer, P. Prusinkiewicz, C. Godin, L-Py: an L-system simulation framework for modeling plant architecture development based on a dynamic language, Front. Plant Sci., 3 (2012), 76. https://doi.org/10.3389/fpls.2012.00076 doi: 10.3389/fpls.2012.00076 |
[23] | R. Barth, J. IJsselmuiden, J. Hemming, E. J. V. Henten, Synthetic bootstrapping of convolutional neural networks for semantic plant part segmentation, Comput. Electron. Agric., 161 (2019), 291–304. https://doi.org/10.1016/j.compag.2017.11.040 doi: 10.1016/j.compag.2017.11.040 |
[24] | M. Cieslak, N. Khan, P. Ferraro, R. Soolanayakanahally, S. J. Robinson, I. Parkin, et al., L-system models for image-based phenomics: case studies of maize and canola, In Silico Plants, 4 (2021), diab039. https://doi.org/10.1093/insilicoplants/diab039 doi: 10.1093/insilicoplants/diab039 |
[25] | E. Fiestas, O. E. Ramos, S. Prado, RPA and L-system based synthetic data generator for cost-efficient deep learning model training, in 2021 IEEE 3rd Eurasia Conference on IOT, Communication and Engineering (ECICE), National Formosa University, Yunlin, Taiwan, (2021), 645–650. https://doi.org/10.1109/ECICE52819.2021.9645719 |
[26] | D. Ward, P. Moghadam, N. Hudson, Deep leaf segmentation using synthetic data, preprint, arXiv: 1807.10931. https://doi.org/10.48550/arXiv.1807.10931 |
[27] | R. Barth, J. IJsselmuiden, J. Hemming, E. J. V. Henten, Data synthesis methods for semantic segmentation in agriculture: A Capsicum annuum dataset, Comput. Electron. Agric., 144 (2018), 284–296. https://doi.org/10.1016/j.compag.2017.12.001 doi: 10.1016/j.compag.2017.12.001 |
[28] | J. Ubbens, M. Cieslak, P. Prusinkiewicz, I. Stavness, The use of plant models in deep learning: an application to leaf counting in rosette plants, Plant Methods, 14 (2018), 1–10. https://doi.org/10.1186/s13007-018-0273-z doi: 10.1186/s13007-018-0273-z |
[29] | K. Turgut, H. Dutagaci, D. Rousseau, RoseSegNet: An attention-based deep learning architecture for organ segmentation of plants, Biosyst. Eng., 221 (2022), 138–153. https://doi.org/10.1016/j.biosystemseng.2022.06.016 doi: 10.1016/j.biosystemseng.2022.06.016 |
[30] | A. Chaudhury, P. Hanappe, R. Azaïs, C. Godin, D. Colliaux, Transferring PointNet++ segmentation from virtual to real plants, in ICCV 2021-International Conference on Computer Vision, IEEE computer society, Montreal, (2021), 13. |
[31] | Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, M. Bennamoun, Deep learning for 3d point clouds: A survey, IEEE Trans. Pattern Anal. Mach. Intell., 43 (2020), 4338–4364. https://doi.org/10.1109/TPAMI.2020.3005434 doi: 10.1109/TPAMI.2020.3005434 |
[32] | H. Su, S. Maji, E. Kalogerakis, E. Learned-Miller, Multi-view convolutional neural networks for 3d shape recognition, in Proceedings of the IEEE International Conference on Computer Vision, IEEE computer society, Montreal, QC, Canada, (2015), 945–953. https://doi.org/10.1109/ICCV.2015.114 |
[33] | W. Shi, R. van de Zedde, H. Jiang, G. Kootstra, Plant-part segmentation using deep learning and multi-view vision, Biosyst. Eng., 187 (2019), 81–95. https://doi.org/10.1016/j.biosystemseng.2019.08.014 doi: 10.1016/j.biosystemseng.2019.08.014 |
[34] | X. Wang, C. Wang, B. Liu, X. Zhou, L. Zhang, J. Zheng, et al., Multi-view stereo in the deep learning era: A comprehensive review, Displays, 70 (2021), 102102. https://doi.org/10.1016/j.displa.2021.102102 doi: 10.1016/j.displa.2021.102102 |
[35] | D. Maturana, S. Scherer, Voxnet: A 3d convolutional neural network for real-time object recognition, in 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, (2015), 922–928. https://doi.org/10.1109/IROS.2015.7353481 |
[36] | R. Du, Z. Ma, P. Xie, Y. He, H. Cen, PST: Plant segmentation transformer for 3D point clouds of rapeseed plants at the podding stage, ISPRS J. Photogramm. Remote Sens., 195 (2023), 380–392. https://doi.org/10.1016/j.isprsjprs.2022.11.022 doi: 10.1016/j.isprsjprs.2022.11.022 |
[37] | C. R. Qi, H. Su, K. Mo, L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, IEEE computer society, Honolulu, HI, USA, (2017), 652–660. https://doi.org/10.48550/arXiv.1612.00593 |
[38] | C. R. Qi, L. Yi, H. Su, L. J. Guibas, Pointnet++: Deep hierarchical feature learning on point sets in a metric space, Adv. Neural Inf. Process. Syst., 30 (2017). https://doi.org/10.48550/arXiv.1706.02413 doi: 10.48550/arXiv.1706.02413 |
[39] | H. Kang, H. Zhou, X. Wang, C. Chen, Real-time fruit recognition and grasping estimation for robotic apple harvesting, Sensors, 20 (2020), 5670. https://doi.org/10.3390/s20195670 doi: 10.3390/s20195670 |
[40] | T. Masuda, Leaf area estimation by semantic segmentation of point cloud of tomato plants, in Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE computer society, Montreal, QC, Canada, (2021), 1381–1389. https://doi.org/10.1109/ICCVW54120.2021.00159 |
[41] | D. Li, G. Shi, J. Li, Y. Chen, S. Zhang, S. Xiang, et al., PlantNet: A dual-function point cloud segmentation network for multiple plant species, ISPRS J. Photogramm. Remote Sens., 184 (2022), 243–263. https://doi.org/10.1016/j.isprsjprs.2022.01.007 doi: 10.1016/j.isprsjprs.2022.01.007 |
[42] | M. Ghahremani, B. Tiddeman, Y. Liu, A. Behera, Orderly disorder in point cloud domain, in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, (2020), 494–509. https://doi.org/10.1007/978-3-030-58604-1_30 |
[43] | M. Ghahremani, K. Williams, F. M. K. Corke, B. Tiddeman, Y. Liu, J. H. Doonan, Deep segmentation of point clouds of wheat, Front. Plant Sci., 12 (2021), 608732. https://doi.org/10.3389/fpls.2021.608732 doi: 10.3389/fpls.2021.608732 |
[44] | M. H. Guo, J. X. Cai, Z. N. Liu, T. J. Mu, R. R. Martin, S. M. Hu, Pct: Point cloud transformer, Comput. Visual Media, 7 (2021), 187–199. https://doi.org/10.1007/s41095-021-0229-5 doi: 10.1007/s41095-021-0229-5 |
[45] | H. Zhao, L. Jiang, J. Jia, P. H. Torr, V. Koltun, Point transformer, in Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE computer society, Montreal, QC, Canada, (2021), 16259–16268. https://doi.org/10.1109/ICCV48922.2021.01595 |
[46] | N. Engel, V. Belagiannis, K. Dietmayer, Point transformer, IEEE Access, 9 (2021), 134826–134840. https://doi.org/10.1109/ACCESS.2021.3116304 doi: 10.1109/ACCESS.2021.3116304 |
[47] | J. Lin, M. Rickert, A. Perzylo, A. Knoll, Pctma-net: Point cloud transformer with morphing atlas-based point generation network for dense point cloud completion, in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic, (2021), 5657–5663. https://doi.org/10.1109/IROS51168.2021.9636483 |
[48] | L. Hui, H. Yang, M. Cheng, J. Xie, J. Yang, Pyramid point cloud transformer for large-scale place recognition, in Proceedings of the IEEE/CVF International Conference on Computer Vision, IEEE computer society, Montreal, QC, Canada, (2021), 6098–6107. https://doi.org/10.1109/ICCV48922.2021.00604 |
[49] | X. Yu, L. Tang, Y. Rao, T. Huang, J. Zhou, J. Lu, Point-bert: Pre-training 3d point cloud transformers with masked point modeling, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE computer society, New Orleans, LA, USA, (2022), 19313–19322. https://doi.org/10.48550/arXiv.2111.14819 |
[50] | D. Li, J. Li, S. Xiang, A. Pan, PSegNet: Simultaneous semantic and instance segmentation for point clouds of plants, Plant Phenomics, 2022 (2022). https://doi.org/10.34133/2022/9787643 doi: 10.34133/2022/9787643 |
[51] | E. Nezhadarya, E. Taghavi, R. Razani, B. Liu, J. Luo, Adaptive hierarchical down-sampling for point cloud classification, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE computer society, Seattle, WA, USA, (2020), 12956–12964. https://doi.org/10.1109/CVPR42600.2020.01297 |
[52] | X. Lai, J. Liu, L. Jiang, L. Wang, H. Zhao, S. Liu, et al., Stratified transformer for 3d point cloud segmentation, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE computer society, New Orleans, LA, USA, (2022), 8500–8509. https://doi.org/10.1109/CVPR52688.2022.00831 |
[53] | M. Tomkins, Towards modelling emergence in plant systems, Quant. Plant Biol., 4 (2023), e6. https://doi.org/10.1017/qpb.2023.6 doi: 10.1017/qpb.2023.6 |
[54] | A. Chaudhury, F. Boudon, C. Godin, 3D plant phenotyping: All you need is labelled point cloud data, in Computer Vision–ECCV 2020 Workshops, Glasgow, UK, 16 (2020), 244–260. https://doi.org/10.1007/978-3-030-65414-6_18 |
[55] | U. Krämer, Planting molecular functions in an ecological context with Arabidopsis thaliana, Elife, 4 (2015), e06100. https://doi.org/10.7554/eLife.06100 doi: 10.7554/eLife.06100 |
[56] | C. Wyman, The Alias Method for Sampling Discrete Distributions, Ray Tracing Gems Ⅱ: Next Generation Real-Time Rendering with DXR, Vulkan, and OptiX, (2021), 339–343. https://doi.org/10.1007/978-1-4842-7185-8_21 doi: 10.1007/978-1-4842-7185-8_21 |
[57] | S. Laine, T. Karras, Efficient sparse voxel octrees, in Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, Association for Computing Machinery, New York, NY, USA, (2010), 55–63. https://doi.org/10.1145/1730804.1730814 |
[58] | Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, et al., Learning semantic segmentation of large-scale point clouds with random sampling, IEEE Trans. Pattern Anal. Mach. Intell., 44 (2021), 8338–8354. https://doi.org/10.1109/TPAMI.2021.3083288 doi: 10.1109/TPAMI.2021.3083288 |
[59] | L. Li, L. He, J. Gao, X. Han, Psnet: Fast data structuring for hierarchical deep learning on point cloud, IEEE Trans. Circuits Syst. Video Technol., 32 (2022), 6835–6849. https://doi.org/10.1109/TCSVT.2022.3171968 doi: 10.1109/TCSVT.2022.3171968 |
[60] | R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, et al., On layer normalization in the transformer architecture, in International Conference on Machine Learning, Association for Computing Machinery, New York, NY, USA, (2020), 10524–10533. https://doi.org/10.48550/arXiv.2002.04745 |
[61] | C. Moenning, N. A. Dodgson, A new point cloud simplification algorithm, in Proc. Int. Conf. Visualization Imaging Image Proc., (2003), 1027–1033. |
[62] | M. Connor, P. Kumar, Fast construction of k-nearest neighbor graphs for point clouds, IEEE Trans. Visual Comput. Graphics, 16 (2010), 599–608. https://doi.org/10.1109/TVCG.2010.9 doi: 10.1109/TVCG.2010.9 |
[63] | J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization, Preprint. arXiv: 160706450. https://doi.org/10.48550/arXiv.1607.06450 |
[64] | O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, 18 (2015), 234–241. https://doi.org/10.1007/978-3-319-24574-4_28 |
[65] | I. Ziamtsov, K. Faizi, S. Navlakha, Branch-Pipe: Improving graph skeletonization around branch points in 3D point clouds, Remote Sens., 13 (2021), 3802. https://doi.org/10.3390/rs13193802 doi: 10.3390/rs13193802 |
[66] | M. Xu, R. Ding, H. Zhao, X. Qi, PAConv: Position adaptive convolution with dynamic kernel assembling on point clouds, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE computer society, Nashville, TN, USA, (2021), 3172–3181. https://doi.org/10.1109/CVPR46437.2021.00319 |
[67] | J. Morel, A. Bac, T. Kanai, Segmentation of unbalanced and in-homogeneous point clouds and its application to 3D scanned trees, Visual Comput., 36 (2020), 2419–2431. https://doi.org/10.1007/s00371-020-01966-7 doi: 10.1007/s00371-020-01966-7 |
[68] | J. Le Louëdec, G. Cielniak, 3D Shape sensing and deep learning-based segmentation of strawberries, Comput. Electron. Agric., 190 (2021), 106374. https://doi.org/10.1016/j.compag.2021.106374 doi: 10.1016/j.compag.2021.106374 |
[69] | H. Weiser, L. Winiwarter, J. Schäfer, F. E. Fassnacht, K. Anders, A. M. E. Pena, et al., Virtual laser scanning (VLS) in forestry-Investigating appropriate 3D forest representations for LiDAR simulations with HELIOS++, in EGU General Assembly Conference Abstracts, Vienna, Austria, (2021), EGU21-9178. https://doi.org/10.5194/egusphere-egu21-9178 |