Neural architecture search (NAS), a promising method for automated neural architecture design, is often hampered by its overwhelming computational burden, especially the architecture evaluation process in evolutionary neural architecture search (ENAS). Although there are surrogate models based on regression or ranking to assist or replace the neural architecture evaluation process in ENAS to reduce the computational cost, these surrogate models are still affected by poor architectures and are not able to accurately find good architectures in a search space. To solve the above problems, we propose a novel surrogate-assisted NAS approach, which we call the similarity surrogate-assisted ENAS with dual encoding strategy (SSENAS). We propose a surrogate model based on similarity measurement to select excellent neural architectures from a large number of candidate architectures in a search space. Furthermore, we propose a dual encoding strategy for architecture generation and surrogate evaluation in ENAS to improve the exploration of well-performing neural architectures in a search space and realize sufficiently informative representations of neural architectures, respectively. We have performed experiments on NAS benchmarks to verify the effectiveness of the proposed algorithm. The experimental results show that SSENAS can accurately find the best neural architecture in the NAS-Bench-201 search space after only 400 queries of the tabular benchmark. In the NAS-Bench-101 search space, it can also get results that are comparable to other algorithms. In addition, we conducted a large number of experiments and analyses on the proposed algorithm, showing that the surrogate model measured via similarity can gradually search for excellent neural architectures in a search space.
Citation: Yu Xue, Zhenman Zhang, Ferrante Neri. Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy[J]. Electronic Research Archive, 2024, 32(2): 1017-1043. doi: 10.3934/era.2024050
Neural architecture search (NAS), a promising method for automated neural architecture design, is often hampered by its overwhelming computational burden, especially the architecture evaluation process in evolutionary neural architecture search (ENAS). Although there are surrogate models based on regression or ranking to assist or replace the neural architecture evaluation process in ENAS to reduce the computational cost, these surrogate models are still affected by poor architectures and are not able to accurately find good architectures in a search space. To solve the above problems, we propose a novel surrogate-assisted NAS approach, which we call the similarity surrogate-assisted ENAS with dual encoding strategy (SSENAS). We propose a surrogate model based on similarity measurement to select excellent neural architectures from a large number of candidate architectures in a search space. Furthermore, we propose a dual encoding strategy for architecture generation and surrogate evaluation in ENAS to improve the exploration of well-performing neural architectures in a search space and realize sufficiently informative representations of neural architectures, respectively. We have performed experiments on NAS benchmarks to verify the effectiveness of the proposed algorithm. The experimental results show that SSENAS can accurately find the best neural architecture in the NAS-Bench-201 search space after only 400 queries of the tabular benchmark. In the NAS-Bench-101 search space, it can also get results that are comparable to other algorithms. In addition, we conducted a large number of experiments and analyses on the proposed algorithm, showing that the surrogate model measured via similarity can gradually search for excellent neural architectures in a search space.
| [1] |
C. Swarup, K. U. Singh, A. Kumar, S. K. Pandey, N. varshney, T. Singh, Brain tumor detection using CNN, AlexNet & GoogLeNet ensembling learning approaches, Electron. Res. Arch., 31 (2023), 2900–2924. https://doi.org/10.3934/era.2023146 doi: 10.3934/era.2023146
|
| [2] |
X. He, K. Zhao, X. Chu, AutoML: A survey of the state-of-the-art, Knowledge-Based Syst., 212 (2021), 106622. https://doi.org/10.1016/j.knosys.2020.106622 doi: 10.1016/j.knosys.2020.106622
|
| [3] | B. Zoph, Q. V. Le, Neural architecture search with reinforcement learning, in 5th International Conference on Learning Representations, (2017), 1–16. |
| [4] |
P. Ren, Y. Xiao, X. Chang, P. Huang, Z. Li, X. Chen, et al., A comprehensive survey of neural architecture search: Challenges and solutions, ACM Comput. Surv., 54 (2022), 1–34. https://doi.org/10.1145/3447582 doi: 10.1145/3447582
|
| [5] |
B. Lyu, S. Wen, K. Shi, T. Huang, Multiobjective reinforcement learning-based neural architecture search for efficient portrait parsing, IEEE Trans. Cybern., 53 (2023), 1158–1169. https://doi.org/10.1109/TCYB.2021.3104866 doi: 10.1109/TCYB.2021.3104866
|
| [6] |
J. Huang, B. Xue, Y. Sun, M. Zhang, G. G. Yen, Particle swarm optimization for compact neural architecture search for image classification, IEEE Trans. Evol. Comput., 27 (2023), 1298–1312. https://doi.org/10.1109/TEVC.2022.3217290 doi: 10.1109/TEVC.2022.3217290
|
| [7] |
Y. Xue, J. Qin, Partial connection based on channel attention for differentiable neural architecture search, IEEE Trans. Ind. Inf., 19 (2023), 6804–6813. https://doi.org/10.1109/TII.2022.3184700 doi: 10.1109/TII.2022.3184700
|
| [8] | E. Real, A. Aggarwal, Y. Huang, Q. V. Le, Regularized evolution for image classifier architecture search, in Proceedings of the AAAI Conference on Artificial Intelligence, AAAI Press, (2019), 4780–4789. https://doi.org/10.1609/aaai.v33i01.33014780 |
| [9] |
Y. Liu, Y. Sun, B. Xue, M. Zhang, G. G. Yen, K. C. Tan, A survey on evolutionary neural architecture search, IEEE Trans. Neural Networks Learn. Syst., 34 (2023), 550–570. https://doi.org/10.1109/TNNLS.2021.3100554 doi: 10.1109/TNNLS.2021.3100554
|
| [10] |
D. Floreano, P. Dürr, C. Mattiussi, Neuroevolution: From architectures to learning, Evol. Intell., 1 (2008), 47–62. https://doi.org/10.1007/s12065-007-0002-4 doi: 10.1007/s12065-007-0002-4
|
| [11] |
S. Liu, H. Zhang, Y. Jin, A survey on computationally efficient neural architecture search, J. Autom. Intell., 1 (2022), 100002. https://doi.org/10.1016/j.jai.2022.100002 doi: 10.1016/j.jai.2022.100002
|
| [12] | T. Elsken, J. H. Metzen, F. Hutter, Neural architecture search: A survey, J. Mach. Learn. Res., 20 (2019), 1997–2017. |
| [13] | C. White, A. Zela, R. Ru, Y. Liu, F. Hutter, How powerful are performance predictors in neural architecture search, in 35th Conference on Neural Information Processing Systems, (2021), 1–16. |
| [14] |
X. Xu, X. Zhao, M. Wei, Z. Li, A comprehensive review of graph convolutional networks: Approaches and applications, Electron. Res. Arch., 31 (2023), 4185–4215. https://doi.org/10.3934/era.2023213 doi: 10.3934/era.2023213
|
| [15] | W. Wen, H. Liu, Y. Chen, H. Li, G. Bender, P. J. Kindermans, Neural predictor for neural architecture search, Computer Vision – ECCV 2020, Springer, (2020), 660–676. https://doi.org/10.1007/978-3-030-58526-6_39 |
| [16] |
C. Wei, C. Niu, Y. Tang, Y. Wang, H. Hu, J. Liang, NPENAS: Neural predictor guided evolution for neural architecture search, IEEE Trans. Neural Networks Learn. Syst., 34 (2022), 8441–8455. https://doi.org/10.1109/TNNLS.2022.3151160 doi: 10.1109/TNNLS.2022.3151160
|
| [17] |
B. Wang, B. Xue, M. Zhang, Surrogate-assisted particle swarm optimization for evolving variable-length transferable blocks for image classification, IEEE Trans. Neural Networks Learn. Syst., 33 (2021), 3727–3740. https://doi.org/10.1109/TNNLS.2021.3054400 doi: 10.1109/TNNLS.2021.3054400
|
| [18] | Y. Xu, Y. Wang, K. Han, Y. Tang, S. Jui, C. Xu, et al., ReNAS: Relativistic evaluation of neural architecture search, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2021), 4409–4418. https://doi.org/10.1109/CVPR46437.2021.00439 |
| [19] | L. Xie, A. Yuille, Genetic CNN, in 2017 IEEE International Conference on Computer Vision (ICCV), IEEE, (2017), 1388–1397. https://doi.org/10.1109/ICCV.2017.154 |
| [20] |
Y. Sun, B. Xue, M. Zhang, G. G. Yen, Evolving deep convolutional neural networks for image classification, IEEE Trans. Evol. Comput., 24 (2020), 394–407. https://doi.org/10.1109/TEVC.2019.2916183 doi: 10.1109/TEVC.2019.2916183
|
| [21] |
Y. Xue, Y. Wang, J. Liang, A. Slowik, A self-adaptive mutation neural architecture search algorithm based on blocks, IEEE Comput. Intell. Mag., 16 (2021), 67–78. https://doi.org/10.1109/MCI.2021.3084435 doi: 10.1109/MCI.2021.3084435
|
| [22] | B. Deng, J. Yan, D. Lin, Peephole: Predicting network performance before training, preprint, arXiv: 1712.03351. |
| [23] | Y. Tang, Y. Wang, Y. Xu, H. Chen, B. Shi, C. Xu, et al., A semisupervised assessor of neural architectures, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, (2020), 1807–1816. https://doi.org/10.1109/CVPR42600.2020.00188 |
| [24] | Y. Chen, Y. Guo, Q. Chen, M. Li, W. Zeng, Y. Wang, et al., Contrastive neural architecture search with neural architecture comparators, in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2021), 9497–9506. https://doi.org/10.1109/CVPR46437.2021.00938 |
| [25] | M. Huang, Z. Huang, C. Li, X. Chen, H. Xu, Z. Li, et al., Arch-Graph: Acyclic architecture relation predictor for task-transferable neural architecture search, in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2020), 11871–11881. https://doi.org/10.1109/CVPR52688.2022.01158 |
| [26] | C. Ying, A. Klein, E. Christiansen, E. Real, K. Murphy, F. Hutter, NAS-Bench-101: Towards reproducible neural architecture search, in Proceedings of the 36th International Conference on Machine Learning, PMLR, (2019), 7105–7114. |
| [27] | X. Dong, Y. Yang, NAS-Bench-201: Extending the scope of reproducible neural architecture search, in International Conference on Learning Representations, (2020), 1–16. |
| [28] | Y. Liu, Y. Tang, Y. Sun, Homogeneous architecture augmentation for neural predictor, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2021), 12229–12238. https://doi.org/10.1109/ICCV48922.2021.01203 |
| [29] | B. Guo, T. Chen, S. He, H. Liu, L. Xu, P. Ye, et al., Generalized global ranking-aware neural architecture ranker for efficient image classifier search, in Proceedings of the 30th ACM Interna-tional Conference on Multimedia, ACM, (2022), 3730–3741. https://doi.org/10.1145/3503161.3548149 |
| [30] | X. Zhou, S. Liu, K. Wong, Q. Lin, K. Tan, A hybrid search method for accelerating convolutional neural architecture search, in Proceedings of the 2023 15th International Conference on Machine Learning and Computing, ACM, (2023), 177–182. https://doi.org/10.1145/3587716.3587745 |
| [31] | S. Yan, Y. Zheng, W. Ao, X. Zeng, M. Zhang, Does unsupervised architecture representation learning help neural architecture search, in Advances in Neural Information Processing Systems, Curran Associates, Inc., (2020), 12486–12498. |
| [32] |
Y. Sun, H. Wang, B. Xue, Y. Jin, G. G. Yen, M. Zhang, Surrogate-assisted evolutionary deep learning using an end-to-end random forest-based performance predictor, IEEE Trans. Evol. Comput., 24 (2020), 350–364. https://doi.org/10.1109/TEVC.2019.2924461 doi: 10.1109/TEVC.2019.2924461
|
| [33] | R. Luo, F. Tian, T. Qin, E. Chen, T. Y. Liu, Neural architecture optimization, in Advances in Neural Information Processing Systems, Curran Associates, Inc., (2018), 7827–7838. |
| [34] |
X. Xie, Y. Sun, Y. Liu, M. Zhang, K. C. Tan, Architecture augmentation for performance predictor via graph isomorphism, IEEE Trans. Cybern., 2023 (2023), 1–13. https://doi.org/10.1109/TCYB.2023.3267109 doi: 10.1109/TCYB.2023.3267109
|
| [35] |
G. T. Pereira, I. B. Santos, L. P. Garcia, T. Urruty, M. Visani, A. C. De Carvalho, Neural architecture search with interpretable meta-features and fast predictors, Inf. Sci., 649 (2023), 119642. https://doi.org/10.1016/j.ins.2023.119642 doi: 10.1016/j.ins.2023.119642
|
| [36] | Y. Li, C. Hao, P. Li, J. Xiong, D. Chen, Generic neural architecture search via regression, in Advances in Neural Information Processing Systems, Curran Associates, Inc., (2021), 20476–20490. |
| [37] | C. White, W. Neiswanger, Y. Savani, BANANAS: Bayesian optimization with neural architec-tures for neural architecture search, in Proceedings of the AAAI Conference on Artificial Intelligence, AAAI Press, (2021), 10293–10301. https://doi.org/10.1609/aaai.v35i12.17233 |
| [38] | B. Wu, K. Keutzer, X. Dai, P. Zhang, Y. Wang, F. Sun, et al., FBNet: Hardware-aware efficient ConvNet design via differentiable neural architecture search, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, (2019), 10726–10734. https://doi.org/10.1109/CVPR.2019.01099 |
| [39] | Z. Guo, X. Zhang, H. Mu, W. Heng, Z. Liu, Y. Wei, et al., Single path one-shot neural architecture search with uniform sampling, in Computer Vision – ECCV 2020, Springer, (2020), 544–560. https://doi.org/10.1007/978-3-030-58517-4_32 |
| [40] | X. Chu, B. Zhang, R. Xu, FairNAS: Rethinking evaluation fairness of weight sharing neural architecture search, in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, (2021), 12219–12228. https://doi.org/10.1109/ICCV48922.2021.01202 |
| [41] | J. Mellor, J. Turner, A. Storkey, E. J. Crowley, Neural architecture search without training, in Proceedings of the 38th International Conference on Machine Learning, PMLR, (2021), 7588–7598. |
| [42] |
L. Fan, H. Wang, Surrogate-assisted evolutionary neural architecture search with network embedding, Complex Intell. Syst., 9 (2023), 3313–3331. https://doi.org/10.1007/s40747-022-00929-w doi: 10.1007/s40747-022-00929-w
|
| [43] | J. Snoek, O. Rippel, K. Swersky, R. Kiros, N. Satish, N. Sundaram, et al., Scalable Bayesian optimization using deep neural networks, in Proceedings of the 32nd International Conference on Machine Learning, PMLR, (2015), 2171–2180. |
| [44] |
C. Wei, Y. Tang, C. N. C. Niu, H. Hu, Y. Wang, J. Liang, Self-supervised representation learning for evolutionary neural architecture search, IEEE Comput. Intell. Mag., 16 (2021), 33–49. https://doi.org/10.1109/MCI.2021.3084415 doi: 10.1109/MCI.2021.3084415
|
Figures(11) / Tables(5)
Yu Xue, Zhenman Zhang, Ferrante Neri. Similarity surrogate-assisted evolutionary neural architecture search with dual encoding strategy[J]. Electronic Research Archive, 2024, 32(2): 1017-1043. doi: 10.3934/era.2024050
DownLoad: