Performance metrics evaluation of multi-objective optimization for transportation

Hongyu Zhang; Wen Yi; Haoran Li; Shuaian Wang; Hongyu Zhang; Wen Yi; Haoran Li; Shuaian Wang

doi:10.3934/era.2025176

Electronic Research Archive

2025, Volume 33, Issue 6: 3968-3988. doi: 10.3934/era.2025176

Previous Article Next Article

Research article Special Issues

Performance metrics evaluation of multi-objective optimization for transportation

1.
Department of Logistics and Maritime Studies, The Hong Kong Polytechnic University, Hong Kong 999077, China
2.
Department of Building and Real Estate, The Hong Kong Polytechnic University, Hong Kong 999077, China
3.
College of Economics and Management, Shandong University of Science and Technology, Qingdao 266590, China

Received: 19 March 2025 Revised: 24 May 2025 Accepted: 04 June 2025 Published: 24 June 2025

Various multi-objective optimization methods are widely applied in the transportation field, enabling decision-makers to find solutions that balance trade-off objectives. Since comparing the performance of multi-objective optimization methods is generally difficult, many performance metrics are introduced to quantitatively evaluate the performance of multi-objective optimization methods. However, the effectiveness of the performance metrics needs to be further investigated. Thus, we first critically analysed a series of performance metrics, including number of solutions obtained (NOSO), overall nondominated solutions number (ONSN), normalized maximum spread (NMS), error ratio (ER), nearest ideal distance (NID), mean ideal distance (MID), spacing (SP), inverted generational distance (IGD), and hypervolume-based ratio (HR), which were extensively adopted to assess the performance of multi-objective optimization methods. We found that these performance metrics cannot always accurately reflect the quality of solutions obtained and may be misleading. Thereafter, two axioms were proposed to define the criteria for reliable performance metrics. Additionally, whether these performance metrics satisfied the two axioms was rigorously proved. The performance metrics that satisfied both axioms, i.e., NOSO, ONSN, NMS, ER, and HR, were considered reliable. Furthermore, a real-world cargo transportation case was investigated, indicating the unreliability of metrics MID and SP.
- multi-objective optimization,
- performance metrics,
- transportation,
- Pareto optimality
Citation: Hongyu Zhang, Wen Yi, Haoran Li, Shuaian Wang. Performance metrics evaluation of multi-objective optimization for transportation[J]. Electronic Research Archive, 2025, 33(6): 3968-3988. doi: 10.3934/era.2025176

Related Papers:

Abstract

Various multi-objective optimization methods are widely applied in the transportation field, enabling decision-makers to find solutions that balance trade-off objectives. Since comparing the performance of multi-objective optimization methods is generally difficult, many performance metrics are introduced to quantitatively evaluate the performance of multi-objective optimization methods. However, the effectiveness of the performance metrics needs to be further investigated. Thus, we first critically analysed a series of performance metrics, including number of solutions obtained (NOSO), overall nondominated solutions number (ONSN), normalized maximum spread (NMS), error ratio (ER), nearest ideal distance (NID), mean ideal distance (MID), spacing (SP), inverted generational distance (IGD), and hypervolume-based ratio (HR), which were extensively adopted to assess the performance of multi-objective optimization methods. We found that these performance metrics cannot always accurately reflect the quality of solutions obtained and may be misleading. Thereafter, two axioms were proposed to define the criteria for reliable performance metrics. Additionally, whether these performance metrics satisfied the two axioms was rigorously proved. The performance metrics that satisfied both axioms, i.e., NOSO, ONSN, NMS, ER, and HR, were considered reliable. Furthermore, a real-world cargo transportation case was investigated, indicating the unreliability of metrics MID and SP.

References

[1]	E. Demir, T. Bektaş, G. Laporte, A review of recent research on green road freight transportation, Eur. J. Oper. Res., 237 (2014), 775–793. https://doi.org/10.1016/j.ejor.2013.12.033 doi: 10.1016/j.ejor.2013.12.033
[2]	Y. J. Hu, L. Yang, H. Cui, H. Wang, C. Li, Developing a balanced strategy: A multi-objective model for emissions reduction and development of civil aviation in China, Energy, 307 (2024), 132597. https://doi.org/10.1016/j.energy.2024.132597 doi: 10.1016/j.energy.2024.132597
[3]	F. Mastroddi, S. Gemma, Analysis of Pareto frontiers for multidisciplinary design optimization of aircraft, Aerosp. Sci. Technol., 28 (2013), 40–55. https://doi.org/10.1016/j.ast.2012.10.003 doi: 10.1016/j.ast.2012.10.003
[4]	A. Ala, M. Yazdani, M. Ahmadi, A. Poorianasab, M. Y. N. Attari, An efficient healthcare chain design for resolving the patient scheduling problem: Queuing theory and MILP-ASA optimization approach, Ann. Oper. Res., 328 (2023), 3–33. https://doi.org/10.1007/s10479-023-05287-5 doi: 10.1007/s10479-023-05287-5
[5]	Z. Fei, B. Li, S. Yang, C. Xing, H. Chen, L. Hanzo, A survey of multi-objective optimization in wireless sensor networks: Metrics, algorithms, and open problems, IEEE Commun. Surv. Tutorials, 19 (2016), 550–586. https://doi.org/10.1109/COMST.2016.2610578 doi: 10.1109/COMST.2016.2610578
[6]	W. Deng, X. Zhang, Y. Zhou, Y. Liu, X. Zhou, H. Chen, et al., An enhanced fast non-dominated solution sorting genetic algorithm for multi-objective problems, Inf. Sci., 585 (2022), 441–453. https://doi.org/10.1016/j.ins.2021.11.052 doi: 10.1016/j.ins.2021.11.052
[7]	H. Zhang, Q. Ruan, Y. Jin, S. Wang, Bi-objective optimization for transportation: Generating near-optimal subsets of Pareto optimal solutions, Appl. Sci., 15 (2025), 2519. https://doi.org/10.3390/app15052519 doi: 10.3390/app15052519
[8]	S. Petchrompo, D. W. Coit, A. Brintrup, A. Wannakrairot, A. K. Parlikad, A review of Pareto pruning methods for multi-objective optimization, Comput. Ind. Eng., 167 (2022), 108022. https://doi.org/10.1016/j.cie.2022.108022 doi: 10.1016/j.cie.2022.108022
[9]	J. Zhao, Y. Li, J. Bai, L. Ma, C. Shi, G. Zhang, et al., Multi-objective optimization of marine nuclear power secondary circuit system based on improved multi-objective particle swarm optimization algorithm, Prog. Nucl. Energy, 161 (2023), 104740. https://doi.org/10.1016/j.pnucene.2023.104740 doi: 10.1016/j.pnucene.2023.104740
[10]	E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, V. G. da Fonseca, Performance assessment of multiobjective optimizers: An analysis and review, IEEE Trans. Evol. Comput., 7 (2003), 117–132. https://doi.org/10.1109/TEVC.2003.810758 doi: 10.1109/TEVC.2003.810758
[11]	E. Demir, T. Bektaş, G. Laporte, The bi-objective pollution-routing problem, Eur. J. Oper. Res., 232 (2014), 464–478. https://doi.org/10.1016/j.ejor.2013.08.002 doi: 10.1016/j.ejor.2013.08.002
[12]	R. H. Stewart, T. S. Palmer, B. DuPont, A survey of multi-objective optimization methods and their applications for nuclear scientists and engineers, Prog. Nucl. Energy, 138 (2021), 103830. https://doi.org/10.1016/j.pnucene.2021.103830 doi: 10.1016/j.pnucene.2021.103830
[13]	Z. Wang, A. Ala, Z. Liu, W. Cui, H. Ding, G. Jin, et al., A hybrid equilibrium optimizer based on moth flame optimization algorithm to solve global optimization problems, J. Artif. Intell. Soft Comput. Res., 14 (2024), 207–235. https://doi.org/10.2478/jaiscr-2024-0012 doi: 10.2478/jaiscr-2024-0012
[14]	A. Ala, A. Goli, Incorporating machine learning and optimization techniques for assigning patients to operating rooms by considering fairness policies, Eng. Appl. Artif. Intell., 136 (2024), 108980. https://doi.org/10.1016/j.engappai.2024.108980 doi: 10.1016/j.engappai.2024.108980
[15]	W. Chen, Q. Zhuo, L. Zhang, Modeling and heuristically solving group train operation scheduling for heavy-haul railway transportation, Mathematics, 11 (2023), 2489. https://doi.org/10.3390/math11112489 doi: 10.3390/math11112489
[16]	Z. Elmi, B. Li, B. Liang, Y. Y. Lau, M. Borowska-Stefańska, S. Wiśniewski, et al., An epsilon-constraint-based exact multi-objective optimization approach for the ship schedule recovery problem in liner shipping, Comput. Ind. Eng., 183 (2023), 109472. https://doi.org/10.1016/j.cie.2023.109472 doi: 10.1016/j.cie.2023.109472
[17]	C. A. Nallolla, V. P, D. Chittathuru, S. Padmanaban, Multi-objective optimization algorithms for a hybrid AC/DC microgrid using RES: A comprehensive review, Electronics, 12 (2023), 1062. https://doi.org/10.3390/electronics12041062 doi: 10.3390/electronics12041062
[18]	C. Zhao, J. Tang, W. Gao, Y. Zeng, Z. Li, Many-objective optimization of multi-mode public transportation under carbon emission reduction, Energy, 286 (2024), 129627. https://doi.org/10.1016/j.energy.2023.129627 doi: 10.1016/j.energy.2023.129627
[19]	R. Jia, K. Gao, S. Cui, J. Chen, J. Andric, A multi-objective reinforcement learning-based velocity optimization approach for electric trucks considering battery degradation mitigation, Transp. Res. Part E: Logist. Transp. Rev., 194 (2025), 103885. https://doi.org/10.1016/j.tre.2024.103885 doi: 10.1016/j.tre.2024.103885
[20]	Z. Huang, Z. Sheng, C. Ma, S. Chen, Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving, Commun. Transp. Res., 4 (2024), 100127. https://doi.org/10.1016/j.commtr.2024.100127 doi: 10.1016/j.commtr.2024.100127
[21]	N. Riquelme, C. Von Lücken, B. Baran, Performance metrics in multi-objective optimization, in 2015 Latin American Computing Conference (CLEI), IEEE, (2015), 1–11. https://doi.org/10.1109/CLEI.2015.7360024
[22]	Y. Tian, R. Cheng, X. Zhang, M. Li, Y. Jin, Diversity assessment of multi-objective evolutionary algorithms: Performance metric and benchmark problems, IEEE Comput. Intell. Mag., 14 (2019), 61–74. https://doi.org/10.1109/MCI.2019.2919398 doi: 10.1109/MCI.2019.2919398
[23]	Q. Fan, W. Wang, X. Yan, Multi-objective differential evolution with performance-metric-based self-adaptive mutation operator for chemical and biochemical dynamic optimization problems, Appl. Soft Comput., 59 (2017), 33–44. https://doi.org/10.1016/j.asoc.2017.05.044 doi: 10.1016/j.asoc.2017.05.044
[24]	K. Lapa, Meta-optimization of multi-objective population-based algorithms using multi-objective performance metrics, Inf. Sci., 489 (2019), 193–204. https://doi.org/10.1016/j.ins.2019.03.054 doi: 10.1016/j.ins.2019.03.054
[25]	J. G. Falcon-Cardona, C. A. C. Coello, Indicator-based multi-objective evolutionary algorithms: A comprehensive survey, ACM Comput. Surv., 53 (2020), 1–35. https://doi.org/10.1145/3376916 doi: 10.1145/3376916
[26]	T. Wagner, H. Trautmann, Online convergence detection for evolutionary multi-objective algorithms revisited, in IEEE Congress on Evolutionary Computation, IEEE, (2010), 1–8. https://doi.org/10.1109/CEC.2010.5586474
[27]	L. Marti, J. Garcia, A. Berlanga, J. M. Molina, A stopping criterion for multi-objective optimization evolutionary algorithms, Inf. Sci., 367 (2016), 700–718. https://doi.org/10.1016/j.ins.2016.07.025 doi: 10.1016/j.ins.2016.07.025
[28]	S. Mirjalili, A. Lewis, Novel performance metrics for robust multi-objective optimization algorithms, Swarm Evol. Comput., 21 (2015), 1–23. https://doi.org/10.1016/j.swevo.2014.10.005 doi: 10.1016/j.swevo.2014.10.005
[29]	C. Audet, J. Bigeon, D. Cartier, S. Le Digabel, L. Salomon, Performance indicators in multiobjective optimization, Eur. J. Oper. Res., 292 (2021), 397–422. https://doi.org/10.1016/j.ejor.2020.11.016 doi: 10.1016/j.ejor.2020.11.016
[30]	T. Okabe, Y. Jin, B. Sendhoff, A critical survey of performance indices for multi-objective optimisation, in 2003 Congress on Evolutionary Computation (CEC), IEEE, (2003), 878–885. https://doi.org/10.1109/CEC.2003.1299759
[31]	S. Zajac, S. Huber, Objectives and methods in multi-objective routing problems: A survey and classification scheme, Eur. J. Oper. Res., 290 (2021), 1–25. https://doi.org/10.1016/j.ejor.2020.07.005 doi: 10.1016/j.ejor.2020.07.005
[32]	B. Qian, L. Wang, D. X. Huang, W. L. Wang, X. Wang, An effective hybrid DE-based algorithm for multi-objective flow shop scheduling with limited buffers, Comput. Oper. Res., 36 (2009), 209–233. https://doi.org/10.1016/j.cor.2007.08.007 doi: 10.1016/j.cor.2007.08.007
[33]	H. Amirian, R. Sahraeian, Augmented $\epsilon$-constraint method in multi-objective flowshop problem with past sequence set-up times and a modified learning effect, Int. J. Prod. Res., 53 (2015), 5962–5976. https://doi.org/10.1080/00207543.2015.1033033 doi: 10.1080/00207543.2015.1033033
[34]	M. Rashidnejad, S. Ebrahimnejad, J. Safari, A bi-objective model of preventive maintenance planning in distributed systems considering vehicle routing problem, Comput. Ind. Eng., 120 (2018), 360–381. https://doi.org/10.1016/j.cie.2018.05.001 doi: 10.1016/j.cie.2018.05.001
[35]	G. G. Yen, Z. He, Performance metric ensemble for multiobjective evolutionary algorithms, IEEE Trans. Evol. Comput., 18 (2014), 131–144. https://doi.org/10.1109/TEVC.2013.2240687 doi: 10.1109/TEVC.2013.2240687
[36]	J. Wang, X. Hu, E. Demeulemeester, Y. Zhao, A bi-objective robust resource allocation model for the RCPSP considering resource transfer costs, Int. J. Prod. Res., 59 (2019), 367–387. https://doi.org/10.1080/00207543.2019.1695168 doi: 10.1080/00207543.2019.1695168
[37]	S. Mirjalili, S. Saremi, S. M. Mirjalili, L. D. S. Coelho, Multi-objective grey wolf optimizer: A novel algorithm for multi-criterion optimization, Expert Syst. Appl., 47 (2016), 106–119. https://doi.org/10.1016/j.eswa.2015.10.039 doi: 10.1016/j.eswa.2015.10.039
[38]	M. Tadaros, A. Migdalas, Bi- and multi-objective location routing problems: Classification and literature review, Oper. Res., 22 (2022), 4641–4683. https://doi.org/10.1007/s12351-022-00734-w doi: 10.1007/s12351-022-00734-w
[39]	J. S. Neufeld, S. Schulz, U. Buscher, A systematic review of multi-objective hybrid flow shop scheduling, Eur. J. Oper. Res., 309 (2023), 1–23. https://doi.org/10.1016/j.ejor.2022.08.009 doi: 10.1016/j.ejor.2022.08.009
[40]	Q. Liu, X. Li, H. Liu, Z. Guo, Multi-objective metaheuristics for discrete optimization problems: A review of the state-of-the-art, Appl. Soft Comput., 93 (2020), 106382. https://doi.org/10.1016/j.asoc.2020.106382 doi: 10.1016/j.asoc.2020.106382

Reader Comments

Your name:*

Email:*
© 2025 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)