Type of Publication: Article in Collected Edition

Analysis of Programming Assessments — Building an Open Repository for Measuring Competencies

Author(s):
Barkmin, M.; Brinda, T.
Editor:
Falkner, N.; Seppala, O.
Title of Anthology:
Koli Calling '20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research
Publisher:
Association for Computing Machinery
Location(s):
New York, NY, USA
Publication Date:
2020
ISBN:
9781450389211
Language:
Englisch
Keywords:
assessment, upper secondary education, programming, higher education, competency framework
Digital Object Identifier (DOI):
doi:10.1145/3428029.3428039
Citation:
Download BibTeX

Abstract

Within different approaches and aims to teach programming, context-specific languages are used which might support different paradigms. Therefore, we are developing a framework for modeling programming competencies regardless of the used language or paradigm. In this paper, we present an open repository for measuring competencies to support our theoretical model. Our goal is to make use of already existing assessments for programming by evaluating their quality and fit to our competency framework. We conducted a systematic literature review to find assessments present in the ACM DL, develop a scheme for evaluating the quality of the assessments following three criteria (objectivity, reliability, and validity) and a scheme for evaluating their fit to the competency framework. An in-depth analysis of 13 assessments showed that all fit to our competency framework with an average coverage of 39% of all concepts. Regarding the quality of the assessments, three reported the reliability by evaluating Cronbach’s alpha and five the validity by using different methods. To expand our open repository and to improve our framework we plan a five-step program: analyze more, develop a guide, fill gaps, specialize and replicate assessments. We hope that providing this framework will foster the development of competency models in the field of programming.