Validating integrated writing tasks - A mixed-method approach to investigate the construct of summarization

Abstract

The present study reports on validating an integrated writing task in the context of the new digital version of the Test of German as a foreign language (Test Deutsch als Fremdsprache; TestDaF) that became operational in late 2020. Following recent approaches, the present study builds on a mixed-methods design to shed light on the construct underlying the integrated writing task by looking. In Strand 1 of this study, the cognitive processes of international study applicants were examined, using a combination of eye-tracking and stimulated recall techniques. Findings from the quantitative and qualitative data revealed, that test takers engaged in a variety of cognitive processes related to basic processes of reading and writing, but also employed processes that integrated reading and writing in so-called shared processes. The written performances were analyzed in Strand 2 to investigate the relevance and accuracy of information included from the two sources, and to look into the transformation of the language of the input material. Findings not only showed differences between participants at distinct levels of proficiency in relation to source use and integration style, but also revealed differences in reproducing information between the written source text and the graphical input. Strand 3 addressed the scoring of written performances. Applying many-facet Rasch measurement (MFRM), data of examinees and raters were analyzed. Results showed that raters were able to apply the rating scale as intended and rated consistently. Following argument-based approaches to validation, the comprehensive empirical data gathered in all three strands contributed to establishing a validity argument for the integrated writing task of the TestDaF.