Development of a Diagnostic Tool to Verify the Adequacy of Multiple-Choice Fill-in-the-Blank Programming Learning Tasks and the Appropriateness of Their Difficulty Level
Abstract
In programming education at higher education institutions, closed-ended assignments are often presented to learners. Closed-ended tasks are used as a means of checking the learner’s understanding level. A typical closed-ended problem in programming is a multiple-choice fill-in-the-blank learning task. When this learning task is applied to a class, the teacher must usually design the problem-solving procedure by themself in advance and verify its validity. Preventing unintended correct answers due to the teacher’s oversight is essential, but this task is not easy. Therefore, we developed a diagnostic tool in this study to verify the validity
of multiple-choice fill-in-the-blanks programming learning tasks. Using the proposed tool, the teacher can enumerate patterns of input/output according to to answer combinations when creating questions. Since variations in inputs and outputs and overlaps of inputs and outputs are revealed in advance, the teacher can find the difficulty of the learning task and the existence of unintended correct answers before the question is posed. In this paper, we asked teachers to use the proposed tool on a trial basis and obtained feedback from them. As a result, the proposed tool revealed correct answers that the teachers had not noticed before. This defect was derived from a behavior peculiar to the C language and would have been difficult to detect under normal circumstances. Furthermore, the teacher could easily adjust the difficulty level of the problem by knowing the input/output variations results in advance. These results demonstrate the usefulness of the proposed tool.
References
Shinkai, J., Hayase, Y., Miyaji, I., A study of generation and utilization of fill-in-the-blank questions for programming education on Moodle. IEICE Tech. Report, et, pp.7-10 (2010), In Japanese.
Funabiki, N., Korenaga, Y., Nakanishi, T., Watanabe, K., An extension of fill-in-the-blank problem function in Java programming learning assistant system. In 2013 IEEE Region 10 Humanitarian Technology Conference, pp.85-90, IEEE (2013), In Japanese.
Kashihara A., Soga M., Toyoda J., A Support for Program Understanding with Fill-in-Blank Problems, Transactions of Japanese Society for Information and Systems in Education 15(3), pp.129-138 (1998), In Japanese.
Pehkonen E., Use of Open-Ended Problems in Mathematics Classroom. Research Report 176, 1997.
Okimoto K., Matsumoto S., Yamagishi S., Kashima T., Developing a source code reading tutorial system and analyzing its learning log data with multiple classification analysis, Journal of Artificial Life and Robotics, Vol.22, Issue 2, pp 227-237 (2017).
Morinaga S., Matsumoto S., Hayashi Y., Hirashima T., A Study on Programming Learning Method to Reduce Non-Essential Cognitive Load, 2018 ISCEAS, International Scientific Conference on Engineering and Applied Sciences, pp.414-423 (2018).
Iwamoto T., Matsumoto S., Hayashi Y., Hirashima T., Examining Presentation Method of Question’s Requirement for Game Development-Based Programming Learning Support System, Proc. of AROB 24th, GS5-3, pp.130-133 (2019).
Kashihara A., Kumei K., Umeno K., Toyoda J., How to Make Fill-in-Blank Program Problems and Its Evaluation, Transactions of the Japanese Society for Artificial Intelligence : AI, Vol.16, pp. 384-391 (2001), In Japanese.
Horwitz S., Interprocedural Slicing Using Dependence Graphs, ACM Transactions on Programming Languages and Systems 12(1), 26-60 (1990).
Goshima R., Asai S., Shimakawa Hl, Extraction of Poor Learning Items with Automatic Labeling in Fill-in-the-blank Test. Proceedings of Annual Conference of the Institute of Systems, Control and Information Engineers, Vol. 63, pp. 1503-1510 (2019).
Asai S., Shimakawa H., Automatic scoring system of fill-in-the-blank tests to measure programming skills. In Proc. of the 6th the International Conference on Information Technology and Its Applications, pp.23-29 (2017), In Japanese.
Crow T., Luxton-Reilly A.,Wunsche B., Intelligent tutoring systems for programming education: a systematic review. pp.53-62. 10.1145/3160489.3160492 (2018).
Kakeshita T., Yanagita R., Ohta K., Development and Evaluation of Programming Education Support Tool pgtracer Utilizing Fill-in-the-Blank Question, Transactions of Information Processing Society of Japan, Computer and Education, Vol.2, No.2, pp.20-36 (2016), In Japanese
Ta Na., A proposal of blank element selection algorithm for Java programming learning support system, Doctoral dissertation, Okayama University (2017), In Japanese.
Uchida Y., An Automatic Question and Answer System for Learning Elementary Programming, Information Processing Society of Japan, Computer and Education (CE), 2007(123 (2007-CE-092)), pp.109-113 (2007), In Japanese.
Ariyasu K., Ikeda E., Okamoto T., Kunishita T., Yokota, K., Automatic Generation of Fill-in-the-Blank Exercises in Adaptive C Language Learning System, DEIM Forum, D9-5 (2009), In Japanese.
Van der Maaten, L., Hinton, G., Visualizing data using t-SNE, Journal of machine learning research, 9 (11), pp.2579-2605 (2008).
Shute, V. J., Psotka, J. (1994). Intelligent Tutoring Systems: Past, Present, and Future. ARMSTRONG LAB BROOKS AFB TX HUMAN RESOURCES DIRECTORATE.
