Potentiality of generative AI tools in higher education: Evaluating ChatGPT's viability as a teaching assistant for introductory programming courses

Zishan Ahmed; Shakib Sadat Shanto; Akinul Islam Jony; Zishan Ahmed; Shakib Sadat Shanto; Akinul Islam Jony

doi:10.3934/steme.2024011

STEM Education

2024, Volume 4, Issue 3: 165-182. doi: 10.3934/steme.2024011

Previous Article Next Article

Research article Special Issues

Potentiality of generative AI tools in higher education: Evaluating ChatGPT's viability as a teaching assistant for introductory programming courses

Department of Computer Science, American International University-Bangladesh, Dhaka, Bangladesh

Academic Editor: Alberto Borboni

Received: 06 March 2024 Revised: 09 May 2024 Accepted: 17 May 2024 Published: 03 June 2024

With the advent of large language models like ChatGPT, there is interest in leveraging these tools as teaching assistants in higher education. However, important questions remain regarding the effectiveness and appropriateness of AI systems in educational settings. This study evaluated ChatGPT's potential as a teaching assistant for an introductory programming course. We conducted an experimental study where ChatGPT was prompted in response to common student questions and misconceptions from a first-year programming course. This study was conducted over a period of 2 weeks with 20 undergraduate students and 5 faculty members from the department of computer science. ChatGPT's responses were evaluated along several dimensions—accuracy, completeness, pedagogical soundness, and the ability to resolve student confusion by five course faculties through a survey. Additionally, another survey was administered to students in the course to assess their perception of ChatGPT's usefulness after interacting with the tool. The findings suggested that while ChatGPT demonstrated strengths in explaining introductory programming concepts accurately and completely, it showed weaknesses in resolving complex student confusion, adapting responses to individual needs, and providing tailored debugging assistance. This study highlighted key areas needing improvement and provided a basis to develop responsible integration strategies that harness AI to enrich rather than replace human instruction in technical courses. The results, based on the limited sample size and study duration, indicated that ChatGPT has potential as a supplemental teaching aid for core concepts, but also highlighted areas where human instruction may be particularly valuable, such as providing advanced support. Further research with larger samples and longer study periods is needed to assess the generalizability of these findings.
- ChatGPT,
- teaching assistant,
- higher education,
- educational technology,
- generative AI
Citation: Zishan Ahmed, Shakib Sadat Shanto, Akinul Islam Jony. Potentiality of generative AI tools in higher education: Evaluating ChatGPT's viability as a teaching assistant for introductory programming courses[J]. STEM Education, 2024, 4(3): 165-182. doi: 10.3934/steme.2024011

Related Papers:

Abstract

With the advent of large language models like ChatGPT, there is interest in leveraging these tools as teaching assistants in higher education. However, important questions remain regarding the effectiveness and appropriateness of AI systems in educational settings. This study evaluated ChatGPT's potential as a teaching assistant for an introductory programming course. We conducted an experimental study where ChatGPT was prompted in response to common student questions and misconceptions from a first-year programming course. This study was conducted over a period of 2 weeks with 20 undergraduate students and 5 faculty members from the department of computer science. ChatGPT's responses were evaluated along several dimensions—accuracy, completeness, pedagogical soundness, and the ability to resolve student confusion by five course faculties through a survey. Additionally, another survey was administered to students in the course to assess their perception of ChatGPT's usefulness after interacting with the tool. The findings suggested that while ChatGPT demonstrated strengths in explaining introductory programming concepts accurately and completely, it showed weaknesses in resolving complex student confusion, adapting responses to individual needs, and providing tailored debugging assistance. This study highlighted key areas needing improvement and provided a basis to develop responsible integration strategies that harness AI to enrich rather than replace human instruction in technical courses. The results, based on the limited sample size and study duration, indicated that ChatGPT has potential as a supplemental teaching aid for core concepts, but also highlighted areas where human instruction may be particularly valuable, such as providing advanced support. Further research with larger samples and longer study periods is needed to assess the generalizability of these findings.

References

[1]	Abedi, M., Alshybani, I., Shahadat, M.R. and Murillo, M., Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education. Qeios, 2023. https://doi.org/10.32388/md04b0 doi: 10.32388/md04b0
[2]	Lund, B.D., Wang, T., Mannuru, N.R., Nie, B., Shimray, S. and Wang, Z., ChatGPT and a new academic reality: Artificial Intelligence‐written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, 2023, 74(5): 570‒581. https://doi.org/10.1002/asi.24750 doi: 10.1002/asi.24750
[3]	Horvatić, D. and Lipic, T., Human-Centric AI: The Symbiosis of Human and Artificial Intelligence. Entropy, 2021, 23(3): 332. https://doi.org/10.3390/e23030332 doi: 10.3390/e23030332
[4]	Yang, S.J., Ogata, H., Matsui, T. and Chen, N.S., Human-centered artificial intelligence in education: Seeing the invisible through the visible. Computers and Education: Artificial Intelligence, 2021, 2: 100008. https://doi.org/10.1016/j.caeai.2021.100008 doi: 10.1016/j.caeai.2021.100008
[5]	Rastogi, C., Zhang, Y., Wei, D., Varshney, K.R., Dhurandhar, A. and Tomsett, R., Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making. Proceedings of the ACM on Human-Computer Interaction, 2022, 6: 1–22. https://doi.org/10.1145/3512930 doi: 10.1145/3512930
[6]	Zhang, K. and Aslan, A.B., AI technologies for education: Recent research & future directions. Computers and Education: Artificial Intelligence, 2021, 2: 100025. https://doi.org/10.1016/j.caeai.2021.100025 doi: 10.1016/j.caeai.2021.100025
[7]	Rane, N.L., Choudhary, S.P., Tawde, A. and Rane, J., ChatGPT is not capable of serving as an author: ethical concerns and challenges of large language models in education. International Research Journal of Modernization in Engineering Technology and Science, 2023, 5(10): 851‒874. https://doi.org/10.56726/irjmets45212 doi: 10.56726/irjmets45212
[8]	Romero, M., Lepage, A. and Lille, B., Computational thinking development through creative programming in higher education. International Journal of Educational Technology in Higher Education, 2017, 14: 1‒15. https://doi.org/10.1186/s41239-017-0080-z doi: 10.1186/s41239-017-0080-z
[9]	Alasadi, E.A. and Baiz, C.R., Generative AI in Education and Research: Opportunities, Concerns, and Solutions. Journal of Chemical Education, 2023,100(8): 2965–2971. https://doi.org/10.1021/acs.jchemed.3c00323 doi: 10.1021/acs.jchemed.3c00323
[10]	Orrù, G., Piarulli, A., Conversano, C. and Gemignani, A., Human-like problem-solving abilities in large language models using ChatGPT. Frontiers in artificial intelligence, 2023, 6: 1199350. https://doi.org/10.3389/frai.2023.1199350 doi: 10.3389/frai.2023.1199350
[11]	Berendt, B., Littlejohn, A. and Blakemore, M., AI in education: learner choice and fundamental rights. Learning, Media and Technology, 2020, 45(3): 312–324. https://doi.org/10.1080/17439884.2020.1786399 doi: 10.1080/17439884.2020.1786399
[12]	Xu, W. and Ouyang, F., A systematic review of AI role in the educational system based on a proposed conceptual framework. Education and Information Technologies, 2022, 27(3): 4195‒4223. https://doi.org/10.1007/s10639-021-10774-y doi: 10.1007/s10639-021-10774-y
[13]	van Dijk, L.J., AI as the assistant of the teacher: an adaptive math application for primary schools. MS thesis, University of Twente, 2021. Available from: https://essay.utwente.nl/88893/.
[14]	Borthwick, K., Bradley, L. and Thouësny, S., CALL in a climate of change: adapting to turbulent global conditions – short papers from EUROCALL 2017. Research-publishing.net, 2017.
[15]	Kim, J., Merrill, K., Xu, K. and Sellnow, D.D., My Teacher Is a Machine: Understanding Students' Perceptions of AI Teaching Assistants in Online Education. International Journal of Human–Computer Interaction, 2020, 36(20): 1902–1911. https://doi.org/10.1080/10447318.2020.1801227 doi: 10.1080/10447318.2020.1801227
[16]	Zhai, X., Chu, X., Chai, C.S., Jong, M.S.Y., Istenic, A., Spector, M., et al., A Review of Artificial Intelligence (AI) in Education from 2010 to 2020. Complexity, 2021, 2021: 1–18. https://doi.org/10.1155/2021/8812542 doi: 10.1155/2021/8812542
[17]	Aggarwal, D., Exploring the Scope of Artificial Intelligence (AI) for Lifelong Education through Personalised & Adaptive Learning. Journal of Artificial Intelligence, Machine Learning and Neural Network (JAIMLNN), 2024, 4(01): 21–26. https://doi.org/10.55529/jaimlnn.41.21.26 doi: 10.55529/jaimlnn.41.21.26
[18]	Osmanoglu, B., Forms of Alliances between Humans and Technology: The Role of Human Agency to Design and Setting up Artificial Intelligence-based Learning Tools. Training, Education, and Learning Sciences, 2023,109. https://doi.org/10.54941/ahfe1003154 doi: 10.54941/ahfe1003154
[19]	Ansor, F., Zulkifli, N.A., Jannah, D.S.M. and Krisnaresanti, A., Adaptive Learning Based on Artificial Intelligence to Overcome Student Academic Inequalities. Journal of Social Science Utilizing Technology, 2023, 1(4): 202–213. https://doi.org/10.55849/jssut.v1i4.663 doi: 10.55849/jssut.v1i4.663
[20]	Pedro, F., Subosa, M., Rivas, A. and Valverde, P., Artificial intelligence in education : challenges and opportunities for sustainable development. MINISTERIO DE EDUCACIÓN, 2019. Available from: https://repositorio.minedu.gob.pe/handle/20.500.12799/6533.
[21]	Shanto, S.S., Ahmed, Z. and Jony, A.I., PAIGE: A generative AI-based framework for promoting assignment integrity in higher education. STEM education, 2023, 3(4): 288–305. https://doi.org/10.3934/steme.2023018 doi: 10.3934/steme.2023018
[22]	Shanto, S.S., Ahmed, Z. and Jony, A.I., Enriching Learning Process with Generative AI: A Proposed Framework to Cultivate Critical Thinking in Higher Education using Chat GPT. Tuijin Jishu/Journal of Propulsion Technology, 2024, 45(1): 3019–3029.
[23]	Yilmaz, R. and Yilmaz, F.G.K., Augmented intelligence in programming learning: Examining student views on the use of ChatGPT for programming learning. Computers in Human Behavior: Artificial Humans, 2023, 1(2): 100005. https://doi.org/10.1016/j.chbah.2023.100005 doi: 10.1016/j.chbah.2023.100005
[24]	Taherdoost, H., What Is the Best Response Scale for Survey and Questionnaire Design; Review of Different Lengths of Rating Scale / Attitude Scale / Likert Scale. papers.ssrn.com, 2019. Available from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id = 3588604.
[25]	Ho, G.W.K., Examining Perceptions and Attitudes. Western Journal of Nursing Research, 2016, 39(5): 674–689. https://doi.org/10.1177/0193945916661302 doi: 10.1177/0193945916661302
[26]	Braun, V. and Clarke, V., Using Thematic Analysis in Psychology. Qualitative Research in Psychology, 2006, 3(2): 77–101. https://doi.org/10.1191/1478088706qp063oa doi: 10.1191/1478088706qp063oa
[27]	Deng, Q., Zheng, B. and Chen, J., The Relationship Between Personality Traits, Resilience, School Support, and Creative Teaching in Higher School Physical Education Teachers. Frontiers in Psychology, 2020, 11: 568906. https://doi.org/10.3389/fpsyg.2020.568906 doi: 10.3389/fpsyg.2020.568906
[28]	da S. Fernandes, P.R., Jardim, J. and de Sousa Lopes, M.C., The Soft Skills of Special Education Teachers: Evidence from the Literature. Education Sciences, 2021, 11(3): 125. https://doi.org/10.3390/educsci11030125 doi: 10.3390/educsci11030125

Author's biography Zishan Ahmed an enthusiastic undergraduate pursuing a Bachelor of Science in Computer Science and Engineering. In data science, natural language processing (NLP), and machine learning, he sees the greatest potential for innovation and influence. He is well-versed in several programming languages, including Python, Java, and C++, and is always keen to acquire new tools and technologies. His education has included data structures and algorithms, database management, artificial intelligence, and computer vision; Shakib Sadat Shanto an undergraduate currently pursuing a Bachelor of Science in Computer Science and Engineering at American International University-Bangladesh. He is extremely passionate about artificial intelligence and the data science domain. He wants to do further research on Educational Technology, Natural Language Processing, and Cyber Security; Akinul Islam Jony currently works as an Associate Professor of Computer Science at American International University-Bangladesh (AIUB). His current research interests include AI, machine learning, e-Learning, and educational technology

Reader Comments

Your name:*

Email:*
© 2024 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)