Evaluasi Aplikasi Pembelajaran Berbasis Web Menggunakan Generative Artificial Intelligence dengan Metode ROUGE
DOI:
https://doi.org/10.47065/bulletincsr.v6i3.1032Keywords:
System Evaluation; Generative Artificial Intelligence; PPKN; ROUGE; Black-Box TestingAbstract
This study aims to evaluate the functionality and answer quality of a web-based learning application that uses Generative Artificial Intelligence (GenAI) for the Pancasila and Civic Education (PPKN) course. The primary focus of this research lies in the system evaluation process, while the application development was carried out solely as a means of generating test data. The system was evaluated in two stages: functional testing using the black-box testing method and answer quality assessment using the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) method. Black-box testing was conducted to ensure that all core system features operated according to specifications. The results of the black-box testing showed a 100% success rate across all test scenarios. Furthermore, answer quality evaluation was performed on 50 test data pairs consisting of GenAI-generated answers and reference texts (gold standards) prepared by PPKN lecturers using the ROUGE method. The evaluation results showed an average F1-score of 97% on the ROUGE-1, ROUGE-2, and ROUGE-L metrics. A total of 49 out of 50 answers were categorized as “Very Good” (? 0.75), while 1 answer was categorized as “Good.” These findings indicate that the application is capable of generating answers with a very high level of textual similarity to academic references. This study contributes to filling the gap in empirical evidence and provides a standardized evaluation benchmark for web-based GenAI applications in education, while also offering an evaluation approach that integrates system functional testing and ROUGE-based answer quality measurement. However, this evaluation is still limited to linguistic aspects based on n-grams and does not yet fully represent semantic depth.
Downloads
References
A. Verma and N. V, “Harnessing Generative AI in Education: From Theory to Real-World Impact,” Preprints, May 30, 2025, doi: 10.20944/preprints202505.1177.v3.
S. Harahap and Z. N. Napitupulu, “Pengaruh Teknologi terhadap Pendidikan di Indonesia: Systematic Literature Review,” Rekognisi: Jurnal Pendidikan dan Kependidikan, vol. 8, no. 2, Dec. 2023. [Online]. Available: https://jurnal.unusu.ac.id/index.php/rekognisi/article/view/162
D. Hermawan, C. Dermawan, and P. Bestari, “Transforming Citizenship Education in the Digital Era: Challenges and Opportunities for the Indonesian Millennial Generation,” Unnes Political Science Journal, vol. 8, no. 1, pp. 30–38, Jun. 2024, doi: 10.15294/upsj.v8i1.5783.
Fatimah and D. A. Nugroho, “Strengthening Digital Citizenship Values in Pancasila dan Civics Learning in the 21st Century,” in Proceedings of the 4th Annual Civic Education Conference (ACEC 2022), 2023, pp. 948–954, doi: 10.2991/978-2-38476-096-1_99.
M. R. Hadwiriantor, F. Hamami, and O. N. Pratiwi, “Extractive Text Summarization terhadap Artikel Berita Indonesia Berbasis Machine Learning,” eProceedings of Engineering, vol. 11, no. 4, Jul. 2024. [Online]. Available: https://openlibrarypublications.telkomuniversity.ac.id/index.php/engineering/article/view/23804
M. Akter, N. Bansal, and S. K. Karmaker, “Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE?,” in Findings of the Association for Computational Linguistics: ACL 2022, May 2022, pp. 1547–1560, doi: 10.18653/v1/2022.findings-acl.122.
I. D. Saputra, N. S. Harahap, S. Agustian, M. Fikry, and L. Oktavia, “Aplikasi Web Question Answering Menggunakan Langchain OpenAI tentang Peraturan Perundang-undangan Bidang Pendidikan,” Journal of Computer System and Informatics (JoSYC), vol. 6, no. 1, pp. 293–304, Nov. 2024, doi: 10.47065/josyc.v6i1.6182.
A. Yuniarti et al., “Aplikasi Konsultasi Psikologi Berbasis Flutter dan ChatGPT Menggunakan Metode Extreme Programming,” Journal of Digital Business and Technology Innovation (DBESTI), vol. 2, no. 1, pp. 14–20, 2025. [Online]. Available: https://journal.nurulfikri.ac.id/index.php/DBESTI
R. Darman, “Peran ChatGPT Sebagai Artificial Intelligence dalam Menyelesaikan Masalah Pertanahan dengan Metode Studi Kasus dan Black Box Testing,” Tunas Agraria, vol. 7, no. 1, pp. 18–46, Jan. 2024, doi: 10.31292/jta.v7i1.256.
J. Zhang and D. Sun, “A Systematic Review of Generative Artificial Intelligence in Education,” in 2025 7th International Conference on Computer Science and Technologies in Education (CSTE), Apr. 2025, pp. 552–556, doi: 10.1109/CSTE64638.2025.11092288.
R. D. Agustin, S. Wiyono, and R. Yamanto, “Analysis of Value Alignment and Ethical Guardianship of Learning with AI in Civic Education,” Jurnal Moral Kemasyarakatan, vol. 9, no. 2, pp. 255–265, Nov. 2024, doi: 10.21067/jmk.v9i2.10650.
Z. Chen and W. Zhou, “Ethical Shifts and Innovative Approaches to Civic Education under Generative Artificial Intelligence,” in Proceedings of the 2024 2nd International Conference on Language, Innovative Education and Cultural Communication (CLEC 2024), 2024, pp. 181–187, doi: 10.2991/978-2-38476-263-7_25.
I. Jurenka et al., “Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach,” arXiv preprint arXiv:2407.12687, Dec. 2025. [Online]. Available: http://arxiv.org/abs/2407.12687
Q. Xia, X. Weng, F. Ouyang, T. J. Lin, and T. K. F. Chiu, “A Scoping Review on How Generative Artificial Intelligence Transforms Assessment in Higher Education,” International Journal of Educational Technology in Higher Education, Dec. 2024, doi: 10.1186/s41239-024-00468-z.
D. Lee et al., “The Impact of Generative AI on Higher Education Learning and Teaching: A Study of Educators’ Perspectives,” Computers and Education: Artificial Intelligence, vol. 6, Jun. 2024, doi: 10.1016/j.caeai.2024.100221.
M. Klesel and H. F. Wittmann, “Retrieval-Augmented Generation (RAG),” Business and Information Systems Engineering, vol. 67, no. 4, pp. 551–561, Aug. 2025, doi: 10.1007/s12599-025-00945-3.
G. D. Albert and A. Voutama, “Pengembangan Chatbot Berbasis PDF Menggunakan Local Retrieval-Augmented Generation (RAG) dan Ollama,” Jurnal Informatika dan Teknik Elektro Terapan, vol. 13, no. 2, Apr. 2025, doi: 10.23960/jitet.v13i2.6361.
A. Fajriati, W. Wisroni, and C. Handrianto, “Pemanfaatan Teknologi Artificial Intelligence (AI) dalam Pembelajaran Berbasis Peserta Didik di Era Digital,” Wahana Pedagogika, vol. 6, no. 2, pp. 71–85, Dec. 2024.
E. R. P. Astuti and M. H. Baysha, “Evaluasi Efektivitas Sistem Umpan Balik Berbasis AI dalam Meningkatkan Hasil Belajar Mahasiswa,” EDUTECH: Jurnal Inovasi Pendidikan Berbantuan Teknologi, vol. 4, no. 3, pp. 122–136, 2024, doi: 10.51878/edutech.v4i3.3142.
A. S. Wiradinata and V. C. Mawardi, “Abstractive Text Summarization Berita Bahasa Indonesia Menggunakan Retrieval-Augmented Generation,” Jurnal Ilmu Komputer dan Sistem Informasi, vol. 13, no. 1, 2025, doi: 10.24912/jiksi.v13i1.32861.
Firdaus, I. E. Putra, F. Kesumaningtyas, N. Sahrun, and T. Hadyanto, “Perancangan Sistem Cashless Payment Berbasis Aplikasi Mobile dan Web Menggunakan Teknologi QR Code,” Jurnal Sains Informatika Terapan, vol. 4, no. 3, pp. 547–553, Oct. 2025, doi: 10.62357/jsit.v4i3.786.
Z. Idhafi, S. Agustian, F. Yanto, and N. Safaat H, “Peringkas Teks Otomatis pada Artikel Berbahasa Indonesia Menggunakan Metode Maximum Marginal Relevance,” Jurnal CoSciTech (Computer Science and Information Technology), vol. 4, no. 3, pp. 609–618, Dec. 2023, doi: 10.37859/coscitech.v4i3.6311.
Halimah, S. Agustian, and S. Ramadhani, “Peringkasan Teks Otomatis (Automated Text Summarization) pada Artikel Berbahasa Indonesia Menggunakan Algoritma LexRank,” Jurnal CoSciTech (Computer Science and Information Technology), vol. 3, no. 3, pp. 371–381, Dec. 2022, doi: 10.37859/coscitech.v3i3.4300.
Bila bermanfaat silahkan share artikel ini
Berikan Komentar Anda terhadap artikel Evaluasi Aplikasi Pembelajaran Berbasis Web Menggunakan Generative Artificial Intelligence dengan Metode ROUGE
ARTICLE HISTORY
How to Cite
Issue
Section
Copyright (c) 2026 Rusmanto Rusmanto, Nuranisah Nuranisah
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under Creative Commons Attribution 4.0 International License that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (Refer to The Effect of Open Access).
