This study investigates the potential of GenAI tools, specifically ChatGPT-4/4o, Wolfram GPT, and Tutor Me GPT, to function as accessible, on-demand 24/7 tutoring systems for engineering and mathematics education. With increasing interest in personalized learning, GenAI offers the promise of scalable and individualized academic support. However, concerns about hallucinations, erroneous or fabricated outputs common in GenAI, have hindered recommendations for unsupervised educational use. The application of GenAI in specialized tutoring contexts has not been rigorously evaluated for accuracy or pedagogical risk in engineering and mathematics. To address this gap, this research assesses the accuracy and instructional capability of GenAI through a human-led simulation involving three trained research assistants. This approach enables the systematic evaluation of how hallucinations may impact learning outcomes. The GenAI tools were tested across seven engineering and two mathematics subjects, encompassing 35 distinct topics. Results indicate that ChatGPT-4 and Wolfram GPT demonstrated strong performance in tutoring electrical engineering and mathematics, but exhibited limitations in mechanical engineering content. Minor inaccuracies were frequently observed, raising concerns about student reliance without oversight. Nevertheless, notable strengths include GenAI's adaptability to varying student proficiency levels and its structured, step-by-step problem-solving methodology. While GenAI shows promise as a supplementary learning tool, further research is required to improve accuracy and evaluate its long-term pedagogical impact in real-world educational settings. Based on current capabilities, GenAI is best regarded as a supportive aid rather than a replacement for human instruction. This study provides foundational insights for the future integration of GenAI into education, with potential to transform tutoring practices.
Citation: Sasha Nikolic, Bao Anh Vu, Yang Di, Ashley Heath, Son Lam Phung, Xiaoping Lu, David Hastie, Md Rabiul Islam, Le Chung Tran, Brad Stappenbelt. Generative Artificial Intelligence (GenAI) 24x7 Tutor: A simulation of the capability of ChatGPT, Wolfram GPT and Tutor Me GPT to accurately and effectively tutor engineering and math content[J]. STEM Education, 2026, 6(1): 56-83. doi: 10.3934/steme.2026004
This study investigates the potential of GenAI tools, specifically ChatGPT-4/4o, Wolfram GPT, and Tutor Me GPT, to function as accessible, on-demand 24/7 tutoring systems for engineering and mathematics education. With increasing interest in personalized learning, GenAI offers the promise of scalable and individualized academic support. However, concerns about hallucinations, erroneous or fabricated outputs common in GenAI, have hindered recommendations for unsupervised educational use. The application of GenAI in specialized tutoring contexts has not been rigorously evaluated for accuracy or pedagogical risk in engineering and mathematics. To address this gap, this research assesses the accuracy and instructional capability of GenAI through a human-led simulation involving three trained research assistants. This approach enables the systematic evaluation of how hallucinations may impact learning outcomes. The GenAI tools were tested across seven engineering and two mathematics subjects, encompassing 35 distinct topics. Results indicate that ChatGPT-4 and Wolfram GPT demonstrated strong performance in tutoring electrical engineering and mathematics, but exhibited limitations in mechanical engineering content. Minor inaccuracies were frequently observed, raising concerns about student reliance without oversight. Nevertheless, notable strengths include GenAI's adaptability to varying student proficiency levels and its structured, step-by-step problem-solving methodology. While GenAI shows promise as a supplementary learning tool, further research is required to improve accuracy and evaluate its long-term pedagogical impact in real-world educational settings. Based on current capabilities, GenAI is best regarded as a supportive aid rather than a replacement for human instruction. This study provides foundational insights for the future integration of GenAI into education, with potential to transform tutoring practices.
| [1] |
Bloom, B.S., The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher, 1984, 13(6): 4‒16. https://doi.org/10.3102/0013189x013006004 doi: 10.3102/0013189x013006004
|
| [2] |
Kim, W.H. and Kim, J.H., Individualized AI Tutor Based on Developmental Learning Networks. IEEE Access, 2020, 8: 27927-37. https://doi.org/10.1109/ACCESS.2020.2972167 doi: 10.1109/ACCESS.2020.2972167
|
| [3] |
Nwana, H.S., Intelligent Tutoring Systems: An Overview. Artificial Intelligence Review, 1990, 4(4): 251‒277. https://doi.org/10.1007/BF00168958 doi: 10.1007/BF00168958
|
| [4] | Mollick, E., Co-Intelligence: Living and Working with AI, London: WH Allen, 2024. |
| [5] |
Belkina, M., Daniel, S., Nikolic, S., Haque, R., Lyden, S., Neal, P., et al., Implementing Generative AI (GenAI) in Higher Education: A Systematic Review of Case Studies. Computers and Education: Artificial Intelligence, 2025, 100407. https://doi.org/https://doi.org/10.1016/j.caeai.2025.100407 doi: 10.1016/j.caeai.2025.100407
|
| [6] |
Ahmed, Z., Shanto, S.S. and Jony, A.I., Potentiality of Generative AI Tools in Higher Education: Evaluating ChatGPT's Viability as a Teaching Assistant for Introductory Programming Courses. STEM Education, 2024, 4(3): 165‒182. https://doi.org/10.3934/steme.2024011 doi: 10.3934/steme.2024011
|
| [7] |
Eddine, R.J., Gide, E. and Al-Sabbagh, A., Generative AI in Higher Education: A Cross-Sector Analysis of ChatGPT's Impact on STEM, Social Sciences, and Healthcare. STEM Education, 2025, 5(5): 757‒801. https://doi.org/10.3934/steme.2025035 doi: 10.3934/steme.2025035
|
| [8] |
Nikolic, S., Wentworth, I., Sheridan, L., Moss, S., Duursma, E., Jones, R.A., et al., A Systematic Literature Review of Attitudes, Intentions and Behaviours of Teaching Academics Pertaining to AI and Generative AI (GenAI) in Higher Education: An Analysis of GenAI Adoption Using the Utaut Framework. Australasian Journal of Educational Technology, 2024, 40(6): 56‒75. https://doi.org/10.14742/ajet.9643 doi: 10.14742/ajet.9643
|
| [9] |
Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G.M., Grundy, S., et al., ChatGPT Versus Engineering Education Assessment: A Multidisciplinary and Multi-Institutional Benchmarking and Analysis of This Generative Artificial Intelligence Tool to Investigate Assessment Integrity. European Journal of Engineering Education, 2023, 48(4): 559‒614. https://doi.org/10.1080/03043797.2023.2213169 doi: 10.1080/03043797.2023.2213169
|
| [10] |
Cong-Lem, N., Soyoof, A. and Tsering, D., A Systematic Review of the Limitations and Associated Opportunities of ChatGPT. International Journal of Human–Computer Interaction, 2025, 41(7): 3851‒66. https://doi.org/10.1080/10447318.2024.2344142 doi: 10.1080/10447318.2024.2344142
|
| [11] | Haynes, K. and Brendle, J., Cross-Age Math Tutoring of Kindergarten and First Grade Students by Middle School Tutors. International Journal of Education in Mathematics, Science and Technology, 2019, 7(3): 238‒250. |
| [12] |
Nikolic, S., Sandison, C., Haque, R., Daniel, S., Grundy, S., Belkina, M., et al., ChatGPT, Copilot, Gemini, Scispace and Wolfram Versus Higher Education Assessments: An Updated Multi-Institutional Study of the Academic Integrity Impacts of Generative Artificial Intelligence (GenAI) on Assessment, Teaching and Learning in Engineering. Australasian Journal of Engineering Education, 2024, 29(2): 126‒53. https://doi.org/10.1080/22054952.2024.2372154 doi: 10.1080/22054952.2024.2372154
|
| [13] | Kalai, A.T., Nachum, O., Vempala, S.S. and Zhang, E., Why Language Models Hallucinate, 2025. Available from: https://cdn.openai.com/pdf/d04913be-3f6f-4d2b-b283-ff432ef4aaa5/why-language-models-hallucinate.pdf. |
| [14] | Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic. Advances in Neural Information Processing Systems, 2024, 37: 46819‒46836. |
| [15] |
Nikolic, S., Heath, A., Vu, B.A., Daniel, S., Alimardani, A., Sandison, C., et al., Prompt Potential: A Pilot Assessment of Using Generative Artificial Intelligence (ChatGPT-4) as a Tutor for Engineering and Maths. 52nd Annual Conference of the European Society for Engineering Education (SEFI), 2024. http://dx.doi.org/10.2139/ssrn.5121997 doi: 10.2139/ssrn.5121997
|
| [16] |
Merrill, D.C., Reiser, B.J., Ranney, M. and Trafton, J.G., Effective Tutoring Techniques: A Comparison of Human Tutors and Intelligent Tutoring Systems. The Journal of the learning sciences, 1992, 2(3): 277‒305. https://doi.org/10.1207/s15327809jls0203_2 doi: 10.1207/s15327809jls0203_2
|
| [17] | Feng, S., Magana, A.J. and Kao, D., A Systematic Review of Literature on the Effectiveness of Intelligent Tutoring Systems in STEM. In 2021 IEEE Frontiers in Education Conference (FIE), 2021, 1‒9. https://doi.org/10.1109/FIE49875.2021.9637240 |
| [18] |
Lin, C.C., Huang, A.Y. and Lu, O.H., Artificial Intelligence in Intelligent Tutoring Systems toward Sustainable Education: A Systematic Review. Smart Learning Environments, 2023, 10(1): 41. https://doi.org/10.1186/s40561-023-00260-y doi: 10.1186/s40561-023-00260-y
|
| [19] |
Liu, H., Zhang, Y. and Jia, J., The Design of Guiding and Adaptive Prompts for Intelligent Tutoring Systems and Its Effect on Students' Mathematics Learning. IEEE Transactions on Learning Technologies, 2024, 17: 1379‒89. https://doi.org/10.1109/TLT.2024.3382000 doi: 10.1109/TLT.2024.3382000
|
| [20] |
VanLehn, K., Siler, S., Murray, C. and Baggett, W.B., What Makes a Tutorial Event Effective? Proceedings of the Twentieth Annual Conference of the Cognitive Science Society, 2022, 1084‒1089. https://doi.org/10.4324/9781315782416-195 doi: 10.4324/9781315782416-195
|
| [21] |
Kulik, J.A. and Fletcher, J.D., Effectiveness of Intelligent Tutoring Systems: A Meta-Analytic Review. Review of educational research, 2016, 86(1): 42‒78. https://doi.org/10.3102/0034654315581420 doi: 10.3102/0034654315581420
|
| [22] |
Singh, N., Gunjan, V.K. and Nasralla, M.M, A Parametrized Comparative Analysis of Performance between Proposed Adaptive and Personalized Tutoring System "Seis Tutor" with Existing Online Tutoring System. IEEE Access, 2022, 10: 39376‒39386. https://doi.org/10.1109/ACCESS.2022.3166261 doi: 10.1109/ACCESS.2022.3166261
|
| [23] |
Aleven, V., Rowe, J., Huang, Y. and Mitrovic, A., Domain Modeling for AIED Systems with Connections to Modeling Student Knowledge: A Review. Handbook of Artificial Intelligence in Education, 2023,127‒169. https://doi.org/10.4337/9781800375413.00017 doi: 10.4337/9781800375413.00017
|
| [24] | Romoser, M., An Autonomous Intelligent Driving Simulation Tutor for Driver Training and Remediation: A Concept Paper. In Driving Assessment Conference, 2011. https://doi.org/10.17077/drivingassessment.1438 |
| [25] |
Nikolic, S., Ros, M., Al-Abdeli, Y.M. and Fairweather, H., Beyond Assessment Security: A Critical Policy Analysis of Four Alternative Strategies to Uphold Academic Integrity and Adopt the GenAI Transformation of Teaching and Learning for an Accredited Engineering Degree. STEM Education, 2025, 5(4): 564‒586. https://doi.org/10.3934/steme.2025027 doi: 10.3934/steme.2025027
|
| [26] |
Rudolph, J., Tan, S. and Tan, S., ChatGPT: Bullshit Spewer or the End of Traditional Assessments in Higher Education?. Journal of Applied Learning and Teaching, 2023, 6(1): 342‒363. https://doi.org/https://doi.org/10.37074/jalt.2023.6.1.9 doi: 10.37074/jalt.2023.6.1.9
|
| [27] |
Vujinović, A., Luburić, N., Slivka, J. and Kovačević, A., Using ChatGPT to Annotate a Dataset: A Case Study in Intelligent Tutoring Systems. Machine Learning with Applications, 2024, 16: 100557. https://doi.org/https://doi.org/10.1016/j.mlwa.2024.100557 doi: 10.1016/j.mlwa.2024.100557
|
| [28] | Piro, L., Bianchi, T., Alessandrelli, L., Chizzola, A., Casiraghi, D., Sancassani, S., et al., Mylearningtalk: An LLM-Based Intelligent Tutoring System. In International Conference on Web Engineering, 2024,428‒431. https://doi.org/10.1007/978-3-031-62362-2_39 |
| [29] |
Mustapha, K.B., Yap, E.H. and Abakr, Y.A., Bard, ChatGPT and 3dGPT: A Scientometric Analysis of Generative AI Tools and Assessment of Implications for Mechanical Engineering Education. Interactive Technology and Smart Education, 2024, 21(4): 588‒624. https://doi.org/10.1108/ITSE-10-2023-0198 doi: 10.1108/ITSE-10-2023-0198
|
| [30] |
Ghassemi, M., Birhane, A., Bilal, M., Kankaria, S., Malone, C., Mollick, E., et al., ChatGPT One Year On: Who Is Using It, How and Why?. Nature, 2023,624(7990): 39‒41. https://doi.org/10.1038/d41586-023-03798-6 doi: 10.1038/d41586-023-03798-6
|
| [31] |
Chen, X., Mitrovic, A. and Mathews, M., Learning from Worked Examples, Erroneous Examples, and Problem Solving: Toward Adaptive Selection of Learning Activities. IEEE Transactions on Learning Technologies, 2020, 13(1): 135‒49. https://doi.org/10.1109/TLT.2019.2896080 doi: 10.1109/TLT.2019.2896080
|
| [32] |
Nikolic, S., Suesse, T.F., Grundy, S., Haque, R., Lyden, S., Hassan, G.M., et al., Laboratory Learning Objectives: Ranking Objectives across the Cognitive, Psychomotor and Affective Domains within Engineering. European Journal of Engineering Education, 2024, 49(3): 457‒473. https://doi.org/10.1080/03043797.2023.2248042 doi: 10.1080/03043797.2023.2248042
|
| [33] |
Honig, C., Rios, S. and Desu, A., Generative AI in Engineering Education: Understanding Acceptance and Use of New GPT Teaching Tools within a UTAUT Framework. Australasian Journal of Engineering Education, 2025, 30(1): 80‒92. https://doi.org/10.1080/22054952.2025.2467500. doi: 10.1080/22054952.2025.2467500
|
| [34] | OpenAI. "Hello GPT-4o." OpenAI. https://openai.com/index/hello-gpt-4o/ (accessed 13/05/2024. |
| [35] | Necesal, P. and Pospıšil, J., Experience with Teaching Mathematics for Engineers with the Aid of Wolfram Alpha. Proceedings of the World Congress on Engineering and Computer Science, 2012, 1: 271‒274. |
| [36] | ChatGPT Gets Its "Wolfram Superpowers"!, 2023, accessed 23/03. Available from: https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/. |
| [37] | Kelly, D.P. and Rutherford, T., Khan Academy as Supplemental Instruction: A Controlled Study of a Computer-Based Mathematics Intervention. The International Review of Research in Open and Distributed Learning, 2017, 18(4). http://orcid.org/0000-0002-6463-7022 |
| [38] |
Vidergor, H.E. and Ben-Amram, P., Khan Academy Effectiveness: The Case of Math Secondary Students' Perceptions. Computers & Education, 2020,157: 103985. https://doi.org/https://doi.org/10.1016/j.compedu.2020.103985 doi: 10.1016/j.compedu.2020.103985
|
| [39] | KhanLabs. "World-class AI for education." https://www.khanacademy.org/khan-labs (accessed 21/03/23). |
| [40] | Khan Academy. "Tutor Me." https://chatgpt.com/g/g-hRCqiqVlM-tutor-me (accessed 01/07/2024). |
| [41] |
Scarlatos, A., Editorial Overview: Special Issue on Artificial Intelligence in Education. Journal of Educational Technology Systems, 2024, 52(3): 299‒300. https://doi.org/10.1177/00472395241236997 doi: 10.1177/00472395241236997
|
| [42] |
Pardos, Z.A. and Bhandari, S., ChatGPT-Generated Help Produces Learning Gains Equivalent to Human Tutor-Authored Help on Mathematics Skills. Plos one, 2024, 19(5): e0304013. https://doi.org/10.1371/journal.pone.0304013 doi: 10.1371/journal.pone.0304013
|
| [43] |
Frankford, E., Sauerwein, C., Bassner, P., Krusche, S. and Breu, R., AI-Tutoring in Software Engineering Education. Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training, 2024,309‒319. https://doi.org/10.1145/3639474.3640061 doi: 10.1145/3639474.3640061
|
| [44] |
Nahar, N., Exploring Student Perspectives on AI-Generated Feedback Using a Socratic Method Chatbot. Journal of learning development in higher education, 2025, 37: 1‒11. https://doi.org/10.47408/jldhe.vi37.1724 doi: 10.47408/jldhe.vi37.1724
|
| [45] | OpenAI. "Introducing study mode." https://openai.com/index/chatgpt-study-mode/ (accessed 29/07/25). |
| [46] | Mollick, E. and Mollick, L., General Tutor - GPT4. Available from: https://www.moreusefulthings.com/student-exercises (accessed 01/02/24). |
| [47] |
Gregory, S., O'Connell, J., Butler, D., McDonald, M., Kerr, T., Schutt, S., et al., New Applications, New Global Audiences: Educators Repurposing and Reusing 3D Virtual and Immersive Learning Resources. ASCILITE Annual Conference, 2015. https://doi.org/10.14742/apubs.2015.980 doi: 10.14742/apubs.2015.980
|
| [48] |
Friedman, M., The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance. Journal of the American Statistical Association, 1937, 32(200): 675‒701. https://doi.org/10.1080/01621459.1937.10503522 doi: 10.1080/01621459.1937.10503522
|
| [49] |
Liu, J., Wang, C., Liu, Z., Gao, M., Xu, Y., Chen, J., et al., A Bibliometric Analysis of Generative Ai in Education: Current Status and Development. Asia Pacific Journal of Education, 2024, 44(1): 156‒75. https://doi.org/10.1080/02188791.2024.2305170 doi: 10.1080/02188791.2024.2305170
|
| [50] |
Shein, E., The Impact of AI on Computer Science Education. Communications of the ACM, 2024, 13‒15. https://doi.org/10.1145/3673428 doi: 10.1145/3673428
|
| [51] |
Alimardani, A., Borderline Disaster: An Empirical Study on Student Usage of GenAI in a Law Assignment. IEEE Transactions on Technology and Society, 2025, 1‒10. https://doi.org/10.1109/TTS.2025.3540978 doi: 10.1109/TTS.2025.3540978
|
| [52] |
Nikolic, S., Quince, Z., Lindqvist, A.L., Neal, P., Grundy, S., Lim, M., et al., Project-Work Artificial Intelligence Integration Framework (PAIIF): Developing a CDIO-Based Framework for Educational Integration. STEM Education, 2025, 5(2): 310‒32. https://doi.org/10.3934/steme.2025016 doi: 10.3934/steme.2025016
|
| [53] |
Nikolic, S., Ritz, C., Vial, P.J., Ros, M. and Stirling, D., Decoding Student Satisfaction: How to Manage and Improve the Laboratory Experience. IEEE Transactions on Education, 2014, 58(3): 151‒58. https://doi.org/http://dx.doi.org/10.1109/TE.2014.2346474 doi: 10.1109/TE.2014.2346474
|
| [54] | Horvath, Jared Cooney, 3 Critical Problems GenAI Poses for Learning. Harvard Business Publishing, 2024. Available from: https://hbsp.harvard.edu/inspiring-minds/the-limits-of-gen-ai-educators-in-higher-ed?cid = organic-social%7Clinkedin%7C2024-08-19-4-the-limits-of-gen-ai-educators-in-higher-ed%7Cnone%7Cthought-lead-inspiring-minds%7Cunknown%7Cinspiring-minds-article%7Caug2024. |
Figures(7) / Tables(3)
Sasha Nikolic, Bao Anh Vu, Yang Di, Ashley Heath, Son Lam Phung, Xiaoping Lu, David Hastie, Md Rabiul Islam, Le Chung Tran, Brad Stappenbelt. Generative Artificial Intelligence (GenAI) 24x7 Tutor: A simulation of the capability of ChatGPT, Wolfram GPT and Tutor Me GPT to accurately and effectively tutor engineering and math content[J]. STEM Education, 2026, 6(1): 56-83. doi: 10.3934/steme.2026004
DownLoad: