July 2015; Vol. 32, No. 3

Diagnosing diagnostic language assessment
Yong-Won Lee
Language Testing  2015 32: 299-316
Diagnostic language assessment (DLA) is gaining a lot of attention from language teachers, testers, and applied linguists. With a recent surge of interest in DLA, there seems to be an urgent need to assess where the field of DLA stands at the moment and develop a general sense of where it should be moving in the future. The current article, as the first article in this special issue, aims to provide a general theoretical background for discussion of DLA and address some fundamental issues surrounding DLA. More specifically, the article (a) examines some of the defining characteristics of DLA and its major components, (b) reviews the current state of DLA in conjunction with these components, and (c) identifies some promising areas of future research and development of DLA where important breakthroughs can be made in the future. Some of the major obstacles and challenges facing DLA are identified and discussed, along with some possible solutions to them.


Computerized dynamic assessment (C-DA): Diagnosing L2 development according to learner responsiveness to mediation
Matthew E. Poehner, Jie Zhang, and Xiaofei Lu
Language Testing  2015 32: 337-357
Dynamic assessment (DA) derives from the sociocultural theory of mind as elaborated by Russian psychologist L. S. Vygotsky. By offering mediation when individuals experience difficulties and carefully tracing their responsiveness, Vygotsky (1998) proposed that diagnoses may uncover abilities that have fully formed as well as those still in the process of developing. This insight has led to numerous assessments, collectively referred to as DA, that have been pursued primarily in the domains of special education and general cognitive abilities measurement (Feuerstein, Feuerstein, & Falik, 2010; Haywood & Lidz, 2007). To date, L2 DA work has been primarily conducted in classroom settings (Ableeva, 2010; Lantolf & Poehner, 2011; Poehner, 2007, 2008). This paper discusses a recent project concerning the design of online multiple-choice tests of L2 reading and listening comprehension that leverage the principle that mediation is indispensable for diagnosing development. Specifically, each test item is accompanied by a set of prompts graduated from implicit to explicit. In this way, resultant diagnoses include not only whether learners answered correctly (their actual score) but also the amount of support they required (mediated score) during the test. We argue that the set of scores automatically generated by the tests, together with a breakdown of learner performance on items targeting particular component features of comprehension, provide a fine-grained diagnosis of their L2 development while also offering information relevant to subsequent teaching and learning.


How do young students with different profiles of reading skill mastery, perceived ability, and goal orientation respond to holistic diagnostic feedback?
Eunice Eunhee Jang, Maggie Dunlop, Gina Park, and Edith H. van der Boom
Language Testing  2015 32: 359-383
One critical issue with cognitive diagnostic assessment (CDA) lies in its lack of research evidence that shows how diagnostic feedback from CDA is interpreted and used by young students. This mixed methods research examined how holistic diagnostic feedback (HDF) is processed by young learners with different profiles of reading skills, goal orientations, and perceived ability. HDF provides three learner profiles: learners’ current skill mastery levels; self-assessed skill proficiency; and goal orientations. It also has a section for plans for future learning. A total of 44 Grades 5 and 6 students (aged 11–12) from two classrooms, their parents and teacher received individually customized HDF reports. Students’ reading skill mastery profiles were determined based on the application of cognitive diagnostic modeling to their performance on a provincial reading achievement measure, while their perceived ability and goal orientation profiles were created by using self-assessment and goal-orientation questionnaires. Students and parents provided written responses to their HDF reports.
The study findings show the dynamic influence of young students’ profiles on the ways in which they perceive, interpret and use HDF. Students’ responses to diagnostic feedback did not differ substantially across reading mastery levels; however, psychological factors most strongly impacted the efficacy of learner feedback processing. Furthermore, the result that it was not students’ actual goal orientations but their perceived parent goal orientations that showed significant relationships with their skill mastery levels strongly indicates that young students’ responses to HDF are likely to be influenced by broader learning environments, and such influences are further filtered through their own perceptions. Understanding students’ interactions with diagnostic feedback is critical for maximizing its effect because their perceptions about ability and orientations to learning strongly influence the ways in which they process diagnostic feedback on their learning.


Validity arguments for diagnostic assessment using automated writing evaluation
Carol A. Chapelle, Elena Cotos, and Jooyoung Lee
Language Testing  2015 32: 385-405
Two examples demonstrate an argument-based approach to validation of diagnostic assessment using automated writing evaluation (AWE). Criterion ?, was developed by Educational Testing Service to analyze students’ papers grammatically, providing sentence-level error feedback. An interpretive argument was developed for its use as part of the diagnostic assessment process in undergraduate university English for academic purposes (EAP) classes. The Intelligent Academic Discourse Evaluator (IADE) was developed for use in graduate EAP university classes, where the goal was to help students improve their discipline-specific writing. The validation for each was designed to support claims about the intended purposes of the assessments. We present the interpretive argument for each and show some of the data that have been gathered as backing for the respective validity arguments, which include the range of inferences that one would make in claiming validity of the interpretations, uses, and consequences of diagnostic AWE-based assessments.