EVALUATING THE QUALITY OF RESPONSES GENERATED BY CHATGPT
Abstract
Traditional educational models are becoming outdated; we are facing the challenge of using artificial intelligence, which involves adapting teaching methods, embracing new emerging tools, and utilizing new technologies. In prior research, it has been established that new technologies can lead to benefits related to personalization of education, teaching quality enhancement, monitoring of the learning progress, learning tools development, and increasing motivation. The introduction of ChatGPT, an OpenAI company, has attracted the attention of a diverse population since its inception. We believe that a modern primary school teacher must stay updated with innovations and adapt teaching to new generations of pupils. Therefore, we have conducted research to evaluate the quality of responses in the field of education generated by this application supported by artificial intelligence. Based on the knowledge gained from relevant literature, we have developed a questionnaire that was answered by final-year students of the Faculty of Education - University of Belgrade (Serbia). Firstly, we were interested in determining previous experiences future teachers had with using ChatGPT; subsequently, we assessed to what extent the use of ChatGPT contributed to their work. We have provided opportunities for them to critically evaluate the quality of responses to the questions asked through this application, and finally to express their views on the quality of the responses generated by ChatGPT. The obtained results are presented and discussed. Although most of our interviewees have heard about this application, the experiences they had were not planned for their professional development, making this an area for improving the program of their institutional education. The students expressed their motivation and willingness to evaluate the possibilities of using this language model and learn how to use it, with the awareness that artificial intelligence cannot replace their role, but that it has become a key tool in education, which opens new opportunities for improving the teaching and learning process.
References
Adams, C., Pente, P., Lemermeyer, G., & Rockwell, G. (2023). Ethical principles for artificial intelligence in K-12 education. Computers and Education: Artificial Intelligence, 4. https://doi.org/10.1016/j.caeai.2023.100131
Álvarez-Herrero, J. F. (2024). Opinion of Spanish Teachers About Artificial Intelligence and Its Use in Education. In IoT, AI, and ICT for Educational Applications: Technologies to Enable Education for All (pp. 163-172). Cham: Springer Nature Switzerland.
Amaro, I., Della Greca, A., Francese, R., Tortora, G., & Tucci, C. (2023, July). AI unreliable answers: A case study on ChatGPT. In International Conference on Human-Computer Interaction (pp. 23-40). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-35894-4_2
Aziz, N., Nordin, M.J., Jadid Abdulkadir, S., & Salih, M.M.M. (2021). Digital Addiction: Systematic Review of Computer Game AddictionImpact on Adolescent Physical Health. Electronics, 10 (9).
https://doi.org/10.3390/electronics10090996
Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. Ieee Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510
Chiu, T. K., Xia, Q., Zhou, X., Chai, C. S., & Cheng, M. (2023). Systematic literature review on opportunities, challenges, and future research recommendations of artificial intelligence in education. Computers and Education: Artificial Intelligence, 4, 100118. https://doi.org/10.1016/j.caeai.2022.100118
Chounta, I. A., Bardone, E., Raudsep, A., & Pedaste, M. (2022). Exploring teachers’ perceptions of Artificial Intelligence as a tool to support their practice in Estonian K-12 education. International Journal of Artificial Intelligence in Education, 32(3), 725-755. https://doi.org/10.1016/j.caeai.2022
Celik, I., Dindar, M., Muukkonen, H., & Järvelä, S. (2022). The promises and challenges of artificial intelligence for teachers: A systematic review of research. TechTrends, 66(4), 616-630. https://doi.org/10.1007/s11528-022-00715-y
Celik, I. (2023). Towards Intelligent-TPACK: An empirical study on teachers’ professional knowledge to ethically integrate artificial intelligence (AI)-based tools into education. Computers in Human Behavior, 138, 107468. https://doi.org/10.1016/j.chb.2022.107468
Clark, D. (2020). Artificial ıntelligence for learning: How to use AI to support employee development. Kogan Page Publishers.
Firat, M. (2023).What ChatGPT means for universities: Perception of scholars and students. Journal of Applied Learning & Teaching, 6(1), 1-22. doi: https://doi.org/10.37074/jalt.2023.6.1.22
Fernández-Batanero, J. M., Román-Graván, P., Reyes-Rebollo, M. M., & Montenegro-Rueda, M. (2021). Impact of educational technology on teacher stress and anxiety: a literature review. Int. J. Environ. Res. Public Health 18:548. https://doi.org/10.3390/ijerph18020548
Gill, S. S., Xu, M., Patros, P., Wu, H., Kaur, R., Kaur, K., ... & Buyya, R. (2024). Transformative effects of ChatGPT on modern education: Emerging Era of AI Chatbots. Internet of Things and Cyber-Physical Systems, 4, 19-23. https://doi.org/10.1016/j.iotcps.2023.06.002
Gupta, B., Mufti, T., Sohail, S.S., & Madsen, D. Ø. (2023). ChatGPT: A Brief Narrative Review. Cogent Business & Management, Vol. 10(3). https://doi.org/10.1080/23311975.2023.2275851
Ignjatović, A., & Stevanović, L. (2023). Efficacy and limitations of ChatGPT as a biostatistical problem-solving tool in medical education in Serbia: a descriptive study. Journal of Educational Evaluation for Health Professions, 20. https://doi.org/10.3352/jeehp.2023.20.28-
Jauhiainen, J. S., & Guerra, А. G. (2023). Generative AI and ChatGPT in School Children’s Education: Evidence from a School Lesson. Sustainability, 15 (18), 14025. https://doi.org/10.3390/su151814025
Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., ... & Wheless, L. (2023). Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Research square.https://doi.org/10.21203/rs.3.rs-2566942/v1
Jeong, E., Kim, D., & Lee, D. (2016). Why Do Some People Become Addicted to Digital Games More Easily? A Study of Digital Game Addiction from a Psychosocial Health Perspective. International journal of human–computer interaction, 33(3), 199–214. https://doi.org/10.1080/10447318.2016.1232908
Kelly, S., Kaye, S. A., & Oviedo-Trespalacios, O. (2023). What factors contribute to the acceptance of artificial intelligence? A systematic review. Telematics and Informatics, 77, https://doi.org/10.1016/j.tele.2022.101925
Kim, N. J., & Kim, M. K. (2022, March). Teacher’s perceptions of using an artificial intelligence-based educational tool for scientific writing. In Frontiers in Education (Vol. 7, p. 755914). Frontiers. https://doi.org/10.3389/feduc.2022.755914
Klarin, Z. i Livaić, T. (2023). Primjena chatgpt jezičnog modela umjetne inteligencije u visokom obrazovanju. U Dragan Zlatović (ur.), Zbornik radova Veleučilišta u Šibeniku (str. 99-112). Veleučilište u Šibeniku.
Khoso, J. F., Ali, N., & Aslam, N. (2023). Use of Chat-GPT and AI Tools by Undergraduates: Students andTeachers’ Perspective. Spry Contemporary Educational Practice. 2(2), 215-238.
Lemmens, J. S., Valkenburg, P.M., & Peter, J. (2011). Psychosocial causes and consequences of pathological gaming. Computers in Human Behavior, 27 (1), 144–152. https://doi.org/10.1016/j.chb.2010.07.015
Li, L., Fengchao, Y., & Zhang, E. (2024). A systematic review of learning task design for K-12 AI education: Trends, challenges, and opportunities. Computers and Education: Artificial Intelligence, 6. https://doi.org/10.1016/j.caeai.2024.100217
Lindner, A., Romeike, R., Jasute, E., & Pozdniakov, S. (2019). Teachers’ perspectives on artificial intelligence. In 12th International conference on informatics in schools,“Situation, evaluation and perspectives”, ISSEP.
Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410
Luo, X., Tong, S., Fang, Z., & Qu, Z. (2019). Frontiers: Machines vs. humans: The impact of artificial intelligence chatbot disclosure on customer purchases. Marketing Science, 38(6), 937–947. https://doi.org/10.1287/mksc.2019.1192
Ma, Z. H., Hwang, W. Y., & Shih, T. K. (2020). Effects of a peer tutor recommender system (PTRS) with machine learning and automated assessment on vocational high school students’ computer application operating skills. Journal of Computers in Education, 7(3), 435–462. https://doi.org/10.1007/s40692-020-00162-9
Mandic, D. (2023). Report on Smart Education in the Republic of Serbia. In: Zhuang, R., et al. Smart Education in China and Central & Eastern European Countries. Lecture Notes in Educational Technology. Springer, Singapore. https://doi.org/10.1007/978-981-19-7319-2_11
Miščević Kadijević, G., Mandić, D., & Bojanić, J. (2019). Preschool teachers' assessment of the integrative approach to Environmental Education. Inovacije u nastavi-časopis za savremenu nastavu, 32(3), 90-99. https://doi.org/10.5937/inovacije1903090M
Ng, D. T. K., Tan, C. W., & Leung, J. K. L. (2024). Empowering student self‐regulated learning and science education through ChatGPT: A pioneering pilot study. British Journal of Educational Technology.https://doi.org/10.1111/bjet.13454
Ray, P. P. (2023). ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3, 121-154. https://doi.org/10.1016/j.iotcps.2023.04.003
Robledo, D. A. R., Zara, C. G., Montalbo, S. M., Gayeta, N. E., Gonzales, A. L., Escarez, M. G. A., & Maalihan, E. D. (2023). Development and Validation of a Survey Instrument on Knowledge, Attitude, and Practices (KAP) Regarding the Educational Use of ChatGPT among Preservice Teachers in the Philippines. International Journal of Information and Education Technology, 13(10), 1582-1590. https://doi.org/10.18178/ijiet.2023.13.10.1965
Rospigliosi, P. A. (2023). Artificial intelligence in teaching and learning: what questions should we ask of ChatGPT?. Interactive Learning Environments, 31(1), 1-3. https://doi.org/10.1080/10494820.2023.2180191
Stolpe, K., & Hallström, J. (2024). Artificial intelligence literacy for technology education. Computers and Education Open, 6.https://doi.org/10.1016/j.caeo.2024.100159
Senol, Y., ̧Senol, F.B., Can Ya ̧sar, M. (2023). Digital game addiction of preschool children in the Covid-19 pandemic: Social emotionaldevelopment and parental guidance. Current Psychology, 43(23), 839-847. https://doi.org/10.1007/s12144-023-04323-8
Strzelecki, A. (2024). Students’ acceptance of ChatGPT in higher education: An extended unified theory of acceptance and use of technology. Innovative higher education, 49(2), 223-245. https://doi.org/10.1007/s10755-023-09686-1
Stoković, G., Matović, S. & Ristić, M. (2023).Osposobljavanje vaspitača i učitelja za imeplentaciju obrazovnih robota. U Vladimir Katić (ur.), Trendovi razvoja: Univerzitet pod novim izazovima (str. 218-221). Univerzitet u Novom Sadu, Fakultet tehničkih nauka.
Su, J., & Yang, W. (2023a). Unlocking the power of ChatGPT: A framework for applying generative AI in education. ECNU Review of Education, 6(3), 355-366.
https://doi.org/10.1177/20965311231168423
Su, J., & Yang, W. (2023b). Powerful or mediocre? Kindergarten teachers’ perspectives on using ChatGPT in early childhood education. Interactive Learning Environments, 1-13. https://doi.org/10.1080/10494820.2023.2266490
Yu, H. (2024). The application and challenges of ChatGPT in educational transformation: New demands for teachers' roles. Heliyon, 10, e24289. https://doi.org/10.1016/j.heliyon.2024.e24289
Vigotski. L. (1974). Mišljenje i govor. Beograd: Nolit.
Von Garrel, J., & Mayer, J. (2023). Artificial Intelligence in studies—use of ChatGPT and AI-based tools among students in Germany. Humanities and social sciences communications, 10(1), 1-9. https://doi.org/10.1057/s41599-023-02304-7
Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial intelligence applications in higher education–where are the educators?. International Journal of Educational Technology in Higher Education, 16(1), 1-27. https://doi.org/10.1186/s41239-019-0171-0
