A Case Study on English Test Item Development Training for Secondary School Teachers Using AI Tools: Focusing on ChatGPT

Shin, Dongkwang

doi:10.30961/lr.2023.59.1.21

Lang. Res. 59(1):21-42

pISSN: 0254-4474, eISSN: 2586-7113

DOI: https://doi.org/10.30961/lr.2023.59.1.21

Article

AI 도구를 활용한 중등교사 영어 출제 연수 사례 연구: ChatGPT를 중심으로

신동광 ¹ ^, ^†

A Case Study on English Test Item Development Training for Secondary School Teachers Using AI Tools: Focusing on ChatGPT

Dongkwang Shin ¹ ^, ^†

Author Information & Copyright ▼

¹광주교육대학교

¹Gwangju National University of Education

^†Corresponding Author: 부교수 영어교육과 광주교육대학교 61204 광주광역시 북구 필문대로 55 E-mail: sdhera@gmail.com

ⓒ Copyright 2023 Language Education Institute, Seoul National University. This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received: Feb 28, 2023 ; Revised: Apr 06, 2023 ; Accepted: Apr 20, 2023

Published Online: Apr 30, 2023

ABSTRACT

This study analyzed the use of various AI tools (e.g., ChatGPT) by ten teachers who participated in a secondary teacher training program hosted by University A in January 2023. The training program was conducted over three days, and after learning how to use AI tools, the teachers were asked to apply the AI tools in the actual practice of test item development, particularly for reading test items in the College Scholastic Ability Test (CSAT). The study divided the test item development process into five stages: 1) passage creation; 2) test item creation; 3) test item selection; 4) vocabulary level control; and 5) refining details such as styling, choice length, and so on. At each stage, the cases showed that Al tools could significantly contribute to improving the efficiency of developing test items. The teachers also responded very positively to the training program, and the conclusion section discussed future challenges and further research.

Keywords: generative AI; test item development; ChatGPT; ATG; AIG; teacher training

References

2020). Language models are few-shot learners. arXiv (preprint). Retrieved from https://arxiv.org/abs/2005.14165.

2013). AntWordProfiler Version 1.4.0w [Computer Software]. Waseda University. Retrieved from https://www.laurenceanthony.net/software/antwordprofiler/releases/Ant WordProfiler140/AntWordProfiler.exe.

2010). Immediate feedback and opportunity to revise answers to open-ended questions. Educational and Psychological Measurement, 70, 22-35.

2005). Computer-based testing and the Internet: Issues and advances. Hoboken, NJ: JohnWiley & Sons.

2020). Language models are few-shot learners. arXiv (preprint). Retrieved from https://arxiv.org/abs/2005.14165.

2006). Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

2013). Automatic item generation: Theory and practice. New York, NY: Routledge.

2015). 2015 compentency-building training for process-oriented assessment (2015-183). Changwon: Elementary Education Division, Gyeongnam Office of Education.

2021). Transformer-based deep neural language modeling for construct-specific automatic item generation. Psychometrika, 87, 1-24.

10.

2020). Sphinx: An automated generation system for English reading comprehension assessment. Paper presented at the International Conference on Learning Analytics and Knowledge (Online).

11.

Feburary). 'AI textbooks' in which the level of learning content goes up as the learner studies well are coming soon. THE AI. Retrieved from https://newstheai. com/site/data/html_dir/2023/02/24/2023022480178.html.

12.

2023). GENQUE [Computer Software]. Artificial Society. Retrieved from https:// www.genque.ai/.

13.

2023). Can ChatGPT be an innovative tool?: The use cases and prospects of ChatGPT (NIA_The AI Report 2023-1). NIA AI Future Strategy Center.

14.

2018). Automating reading comprehension by generating question and answer pairs. Paper presented at the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Melbourne, Australia.

15.

2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30, 121-204.

16.

2018). AI Test Maker (ATM) [Computer Software]. LXPER Inc. Retrieved from https://atm.lxper.ai/.

17.

2020). A study on the online assessment using artificial intelligence for distance education. Journal of Learner-Centered Curriculum and Instruction, 20(14), 389-407.

18.

2022). YouWrite [Computer Software]. You.com. Retrieved from https://you.com/search?q=how+to+write+well&&tbm=youwrite&cfr=write.

19.

Making and using word lists for language learning and teaching. Amsterdam, Netherland: John Benjamins Publishing Company.

20.

1992). Methods and procedures for developing test items to assess academic achievement. Seoul: Far Eastern Cultural History.

21.

1994). On structuring probabilistic dependences in stochastic language modelling. Computer Speech and Language, 8, 1-38.

22.

2022). ChatGPT: Optimizing language models for dialogue [Computer Software]. Retrieved from https://chat.openai.com/.

23.

2019). Language models are unsupervised multitask learners. OpenAI Blog, 1, 9.

24.

2013). Handbook of automated essay evaluation: Current applications and new directions. New York, NY: Routledge.

25.

2020). An empirical study on paraphrasing patterns of Korean pre-service teachers in the process of controlling words that make up the passage for test items in English reading assessment. Multimedia-Assisted Language Learning, 23(3), 157-179.

26.

2022). What is the limit for using AI in English education?: Feasibility of using automated text generation and automated item generation. Journal of the Korea English Education Society, 21(3), 147-163.

27.

May). It's time to "push the envelope" on AI and prepare for the AGI era. InfoWorld. Retrieved from https://www.itworld.co.kr/opinion/235494#csidx7b60556 267fb24a9b47cec45b44c91c.

28.

2006). Innovative item formats in computerbased testing: In pursuit of improved construct representation. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of test development (pp. 329-347). Mahwah, NJ: Erlbaum.

29.

2014). Student assessment support materials for education that nurtures dreams and aspirations (2014-62). Ulsan: Ulsan Metropolitan Office of Education, Curriculum Operation Division.

30.

2010). Elements of adaptive testing. New York, NY: Springer.

31.

2018). Automated item generation with recurrent neural networks. Psychometrika, 83, 847-857.

32.

2020). CopyAI [Computer Software]. CopyAI Inc. Retrieved from https:// www.copy.ai/.