Towards Automatic Competency Assessment of Programming Skills - Preliminary results of a current development

Author(s):

Barkmin, Mike; Kramer, Matthias

Name of Event:

IZfB "Scientist in Residence" Symposium: "Digitalisation and the Future of Higher Education and Work"

Location:

Essen

Date:

05.12.2017

Language:

englisch

Abstract

Background

Due to the results of international student assessment studies like PISA, the normative determination and empirical verification of competencies are topics in the focus of educational and psychological research during the last years. In the area of object-oriented programming the project COMMOOP focuses on the identification of such competencies. Currently, in the framework of this project a test instrument for comparative studies of larger cohorts as well as for individual diagnostics is under construction.

Following a comparing analysis of various definitions of the term *competency*, we understand competencies in the area of object-oriented programming as domain specific cognitive and metacognitive abilities and skills that enable individuals to solve problems in the area of object-oriented programming. During an intensive literature research a four-dimensional competency model could be derived. First steps towards verification were done by interviewing experts and integrating competency formulations from national and international curricula. Furthermore 12 test items were derived and piloted in a first pen and paper test. This format has been shown resulted in a quite inefficient and time-consuming procedure due to the process of digitalization and evaluation as well as due to the reach-out for further test subjects. It was thus considered detrimental. Hence, it was reasonable to seek an alternative approach in a digital solution.

Implementation & current status

Considering the situation that teachers often do not have administrative rights and furthermore to avoid installation issues we decided to develop a web-based application, which could be run in any web browser on any device with internet access. The web application is independent from the field of computer science and could be used in other fields as well. So there are simple generic item formats, like multiple choice and questionnaire. But there are also special item formats, which were custom-built for accessing programming competencies. Because some item formats are more complex than others requiring more than one step to complete, it would not been enough to use the final submission for enhancing the items. So we implemented a generic way to record all actions the user has taken during the process. These recordings could be helpful to explore cognitive processes while solving the tasks. Given an inside into difficulties, the improvement of the item pool will be facilitated. To make it easy to create new items and tests, we have implemented a graphical user interface for this purpose.

Potential and outlook

The tool has the potential to let interested researchers assess and evaluate the competency level of students with regard to a competency model. Especially the recording of all actions could be very helpful to get an insight into how students deal with the item, to know where they struggle. In the future, when we have collected enough data, we will implement adaptive tests to assess the competency of our pupil with a far better accuracy.