Acceptability and Effectiveness Analysis of Large Language Model-Based Artificial Intelligence Chatbot Among Arabic Learners

Authors

  • Nely Rahmawati Zaimah Arabic Education Study Program Sekolah Tinggi Agama Islam Al-Anwar Rembang, Indonesia.
  • Eko Budi Hartanto Arabic Education Study Program Institut Agama Islam Negeri Kediri, Indonesia.
  • Fatchiatu Zahro Arabic Education Study Program Institut Ummul Qurro Al-Islami Bogor, Indonesia.

DOI:

https://doi.org/10.25217/mantiqutayr.v4i1.3951

Keywords:

Artificial Intelligence Chatbot, Large Language Model, Non-Native Arabic Learners

Abstract

This research stems from the broad use of AI based on Large Language Models (LLMs), which many academics find relevant and effective in higher education Arabic language learning. The goal is to confirm these views.This research is a mixed reseach that employs a both of qualitative and quantitative methodologies. The qualitative segment involves observations and literature reviews. Observations involved reviewing how participants used chatbots and carefully checking the accuracy and consistency of platform responses. The quantitative facet utilizes a paired experimental design, encompassing both classical and Bayesian Paired Sample t-Tests analysis. The research encompasses 45 individuals with a proficient understanding of Modern Standard Arabic and no hindrances in comprehending the material. These individuals are enrolled as students at Islamic College (STAI) Al-Anwar Rembang, Indonesia. The results show increased motivation and ease of use with the chatbot in Arabic language learning. However, concerns about the consistency of chatbot content have arisen, affecting participants' confidence in response accuracy of AI. This prompts an evaluation of effectiveness through classical and Bayesian tests, which fail to demonstrate statistically significant variances, even in the adaptive Bayesian probability analysis. These outcomes deviate from previous research on relevance and effectiveness and corroborate preceding studies on academic apprehensions and accuracy enhancements. The researchers advocate for further investigations, especially concerning the accuracy analysis of AI chatbots in Arabic pedagogical contexts.

References

Abdulkader, Zena, and Yousra Al-Irhayim. “A Review of Arabic Intelligent Chatbots: Developments and Challenges.” Al-Rafidain Engineering Journal (AREJ) 27, no. 2 (September 1, 2022): 178–89. DOI: https://doi.org/10.33899/rengj.2022.132550.1148

Anwar, Saiful, Guntur Cahaya Kesuma, and Koderi. “Development of Al-Qawaid an-Nahwiyah Learning Module Based on Qiyasiyah Method for Arabic Language Education Department Students | Pengembangan Modul Pembelajaran al-Qawaid an-NahwiyahBerbasis Metode QiyasiyahUntukMahasiswaJurusan Pendidikan Bahasa Arab.” Mantiqu Tayr: Journal of Arabic Language 3, no. 1 (January 2, 2023): 11–24. DOI: https://doi.org/10.25217/mantiqutayr.v3i1.2830

Attaqiana, Mita Rosyda, Saptorini Saptorini, and Achmad Binadja. “Pengembangan Media Permainan Truth and Dare Bervisi Sets Guna Memotivasi Belajar Siswa.” Jurnal Inovasi Pendidikan Kimia 10, no. 2 (2016): 1798 – 1806. DOI: https://doi.org/10.15294/jipk.v10i2.9533

Azmi, Aqil M, Abdulaziz O Al-Qabbany, and Amir Hussain. “Computational and Natural Language Processing Based Studies of Hadith Literature: A Survey.” Artificial Intelligence Review 52 (2019): 1369–1414. DOI: https://doi.org/10.1007/s10462-019-09692-w

Brundage, Miles, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, and Ruth Fong. “Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims.” arXiv Preprint arXiv:2004.07213, 2020. DOI: https://doi.org/10.48550/arXiv.2004.07213

Chan, Cecilia Ka Yuk. Assessment for Experiential Learning. (1st ed.). Routledge London 29 September (2022): 378. DOI: https://doi.org/10.4324/9781003018391

Chiu, Thomas KF, Benjamin Luke Moorhouse, Ching Sing Chai, and Murod Ismailov. “Teacher Support and Student Motivation to Learn with Artificial Intelligence (AI) Based Chatbot.” Interactive Learning Environments, 2023: 1–17. DOI: https://doi.org/10.1080/10494820.2023.2172044

De Angelis, Luigi, Francesco Baglivo, Guglielmo Arzilli, Gaetano Pierpaolo Privitera, Paolo Ferragina, Alberto Eugenio Tozzi, and Caterina Rizzo. “ChatGPT and the Rise of Large Language Models: The New AI-Driven Infodemic Threat in Public Health.” Frontiers in Public Health 11 (April 25, 2023): 1166120. DOI: https://doi.org/10.3389/fpubh.2023.1166120

Fiantika, Feny Rita, Mohammad Wasil, Sri Jumiyati, Leli Honesti, Sri Wahyuni, Erland Mouw, Imam Mashudi, Nur Hasanah, Anita Maharani, and Kusmayra Ambarwati. Metodologi Penelitian Kualitatif. Padang: Get Press, 2022.

Fitriani, Fitriani, Muhammad Akmansyah, Ahmad Basyori, Erlina Erlina, & Koderi Koderi. "Manajemen Pembelajaran Bahasa Arab di SMP Qur’an Darul Fattah (SQDF) Bandar Lampung." Al Maghazi : Arabic Language in Higher Education, 1.2 (2023): 47-60. DOI: http://dx.doi.org/10.51278/al.v1i2.786

Fuad, Ahlam, and Maha Al-Yahya. “Recent Developments in Arabic Conversational AI: A Literature Review.” IEEE Access Volume: 10 (2022): 23842 - 23859. DOI: https://doi.org/10.1109/ACCESS.2022.3155521

Gilbert, Stephen, Hugh Harvey, Tom Melvin, Erik Vollebregt, and Paul Wicks. “Large Language Model AI Chatbots Require Approval as Medical Devices.” Nature Medicine 29, no. 10 (October 2023): 2396–98. DOI: https://doi.org/10.1038/s41591-023-02412-6

Hatherley, Joshua James. “Limits of Trust in Medical AI.” Journal of Medical Ethics 46, no. 7 (2020): 478–81. DOI: https://dx.doi.org/10.1136/medethics-2019-105935

Hidayatullah, Achmad, and Csaba Csíkos. “The Role of Students’ Beliefs, Parents’ Educational Level, and The Mediating Role of Attitude and Motivation in Students’ Mathematics Achievement.” The Asia-Pacific Education Researcher, March 30, 2023. DOI: https://doi.org/10.1007/s40299-023-00724-2

Hong, Matthew K, Adam Fourney, Derek DeBellis, and Saleema Amershi. “Planning for Natural Language Failures with the Ai Playbook,” CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021: 1-11. DOI: https://doi.org/10.1145/3411764.3445735

Johnson, Douglas, Rachel Goodman, J Patrinely, Cosby Stone, Eli Zimmerman, Rebecca Donald, Sam Chang, Sean Berkowitz, Avni Finn, and Eiman Jahangir. “Assessing the Accuracy and Reliability of AI-Generated Medical Responses: An Evaluation of the Chat-GPT Model,” Research Square 2023. DOI: https://doi.org/10.21203/rs.3.rs-2566942/v1

Kjell, Oscar NE, Sverker Sikström, Katarina Kjell, and H Andrew Schwartz. “Natural Language Analyzed with AI-Based Transformers Predict Traditional Subjective Well-Being Measures Approaching the Theoretical Upper Limits in Accuracy.” Scientific Reports 1 12, (2022): 3918. DOI: https://doi.org/10.1038/s41598-022-07520-w

Liang, Wei, and Hongsheng Dai. “Bayesian Inference.” In Quantum Chemistry in the Age of Machine Learning, 233–50. Elsevier, 2023. DOI: https://doi.org/10.1016/B978-0-323-90049-2.00005-6

Mohamed, Amr M. “Exploring the Potential of an AI-Based Chatbot (ChatGPT) in Enhancing English as a Foreign Language (EFL) Teaching: Perceptions of EFL Faculty Members.” Education and Information Technologies, 2023: 1–23. DOI: https://dx.doi.org/10.1007/s10639-023-11917-z

Munir, Munir. “Pendekatan Struktural dalam Pelajaran Bahasa Arab.” Shaut al Arabiyyah 6, no. 1 (August 24, 2018): 13. DOI: https://doi.org/10.24252/saa.v6i1.5644

Mustaqim, Mustaqim. “Metode Penelitian Gabungan Kuantitatif Kualitatif/Mixed Methods Suatu Pendekatan Alternatif.” Intelegensia: Jurnal Pendidikan Islam 4, no. 1 (2016): 1-9. DOI: https://doi.org/10.34001/intelegensia.v6i1.1351

Nita Amelia, (second) Noval Mulya Dava S., and (third) Muhammad Minan Chusni, “Pemanfaatan Artificial Intelligence dalam Pembelajaran di Masa Pandemi| Prosiding Amal Insani Foundation,” September 6, 2023. https://prosiding.amalinsani.org/index.php/semnas/article/view/10

Perkins, Mike. “Academic Integrity Considerations of AI Large Language Models in the Post-Pandemic Era: ChatGPT and Beyond.” Journal of University Teaching and Learning Practice, British University, Vietnam 20, no. 2 (February 22, 2023). DOI: https://doi.org/10.53761/1.20.02.07

Rahmah, Risa, Azizatul Muzdalifah, & Mu’alim Wijaya. "Penggunaan Thariqah Mubasyarah sebagai Pembelajaran Bahasa Arab yang Efektif." Al Maghazi : Arabic Language in Higher Education, 1.1 (2023): 23-29. DOI: https://doi.org/10.51278/al.v1i1.706

Ritonga, Mahyudin, Rizka Widayanti, Fitri Alrasi, Julhadi, and Syaflin Halim. “Analysis of Arabic Language Learning at Higher Education Institutions with Multi-Religion Students.” Universal Journal of Educational Research 8 (September 1, 2020): 4333–39. DOI: https://doi.org/10.13189/ujer.2020.080960

Rumaisa, Fitrah, Yan Puspitarani, Ai Rosita, Azizah Zakiah, and Sriyani Violina. “Penerapan Natural Language Processing (NLP) di Bidang Pendidikan.” Jurnal Inovasi Masyarakat 1, no. 3 (2021): 232–35. DOI: https://doi.org/10.33197/jim.vol1.iss3.2021.799

Shao, Sicong, Saleem Alharir, Salim Hariri, Pratik Satam, Sonia Shiri, and Abdessamad Mbarki. “AI-Based Arabic Language and Speech Tutor.” In 2022 IEEE/ACS 19th International Conference on Computer Systems and Applications (AICCSA), 1–8. Abu Dhabi, United Arab Emirates: IEEE, 2022. DOI: https://doi.org/10.1109/AICCSA56895.2022.10017924

Shneiderman, Ben. “Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy.” International Journal of Human–Computer Interaction 36, no. 6 (April 2, 2020): 495–504. DOI: https://doi.org/10.1080/10447318.2020.1741118

Sulaeman, Islamiyah, Syuhadak Syuhadak, and Insyirah Sulaeman. “ChatGPT as a New Frontier in Arabic Education Technology.” Al-Arabi: Jurnal Bahasa Arab Dan Pengajarannya= Al-Arabi: Journal of Teaching Arabic as a Foreign Language 7, no. 1 (2023): 83–105. DOI: https://dx.doi.org/10.17977/um056v7i1p83-105

Van Doorn, Johnny, Don Van Den Bergh, Udo Böhm, Fabian Dablander, Koen Derks, Tim Draws, Alexander Etz, et al. “The JASP Guidelines for Conducting and Reporting a Bayesian Analysis.” Psychonomic Bulletin & Review 28, no. 3 (June 2021): 813–26. DOI: https://doi.org/10.3758/s13423-020-01798-5

Wekke, Ismail Suardi, and Maimun Aqsha Lubis. “A Multicultural Approach in Arabic Language Teaching: Creating Equality at Indonesian Pesantren Classroom Life.” Sosiohumanika 1, no. 2 (2008): 296-310. DOI: https://doi.org/10.2121/sosiohumanika.v1i2.337

Wiranto, Erham Budi, and Sri Suwartini. “Artificial Intelligence and Trustworthy Principles in Global Islamic Education.” Ushuluddin International Conference (USICON) 6 (2022): 64–87. https://vicon.uin-suka.ac.id/index.php/USICON/article/view/1252

Downloads

Published

2023-12-05

How to Cite

Nely Rahmawati Zaimah, Eko Budi Hartanto, & Fatchiatu Zahro. (2023). Acceptability and Effectiveness Analysis of Large Language Model-Based Artificial Intelligence Chatbot Among Arabic Learners. Mantiqu Tayr: Journal of Arabic Language, 4(1), 1–20. https://doi.org/10.25217/mantiqutayr.v4i1.3951