This small-scale qualitative study considers feedback and results reported for two major large-scale English language tests administered in Japan: the Global Test of English Communication for Students (GTECfS) and the Eiken Test in Practical English Proficiency (Eiken). Specifically, it examines current score-reporting practices in student and teacher score reports and supplementary materials as well as students' and teachers' perceptions and uses of reported information. First, to summarize key features of performance feedback provided, two independent judges coded the information that was included as feedback in the score reports and supplementary materials as well as how the feedback was presented for both tests, based on Roberts and Gierl's (2010) test score report analysis framework. Second, 16 students and five teachers at three upper secondary schools (Grades 10-12) in Japan were interviewed to explore how they perceived the assessment results and used them in their learning and teaching. Key study results showed that the documents generally followed principles of good score-reporting practice suggested in previous studies, while student and teacher interviews identified various issues to be considered to improve the communication of test performance feedback to its intended audiences.

The present study examined the factor structures across features of 446 examinees' responses to a writing task that integrates reading and listening modalities as well as reading and listening comprehension items of the TOEFL iBT® (Internet-based test). Both human and automated scores obtained for the integrated essays were utilized. Based on a series of preliminary factor analyses, a confirmatory factor analysis (CFA) identified a model that specified a higher order factor for comprehension. In the model, the Comprehension factor underlay factors representing content of the written essay as well as reading and listening comprehension. The Comprehension factor correlated with two writing factors-Productive Vocabulary and Sentence Conventions. Furthermore, follow-up CFA models with covariates (multiple indicators multiple causes models, or MIMIC models) were tested to compare performance between a group of 190 examinees scoring above a frequently used TOEFL iBT Total score requirement for international student admission and the other group of 128 examinees scoring below the requirement. The higher ability group performed significantly better than the lower ability group on all three constructs: Comprehension, Productive Vocabulary, and Sentence Conventions. The identification of the multiple distinct factors in this study may hold promise for obtaining writing profiles that inform instruction in contexts such as test preparation. © 2013 Taylor &amp
Francis Group, LLC.

This study examined the relationship among word reading efficiency, text reading fluency, and reading comprehension for adult English as a Foreign Language (EFL) learners. Data from 185 adult Chinese EFL learners preparing to take the Test-of-English-as-a-Foreign-Language™(TOEFL®) were analyzed in this study. The participants completed a computer-based test battery of word reading efficiency and text reading fluency in addition to TOEFL iBT (Internet-Based Test) reading comprehension. The findings showed a stronger relationship between text oral reading fluency and reading comprehension than between word reading efficiency and reading comprehension. The authors recommended that attention be paid to text oral reading fluency in adult EFL reading research and instruction. © Taylor &amp
Francis Group, LLC.

In this study, confirmatory factor analyses were used to examine the interrelationships among latent factors of the simple view model of reading comprehension (word recognition and language comprehension) and hypothesized additional factors (vocabulary and reading fluency) in a sample of 476 adult learners with low literacy levels. The results provided evidence for reliable distinctions between word recognition, fluency, language comprehension, and vocabulary skills as components of reading. Even so, the data did not support the hypothesis that the simple view needs to be expanded to include vocabulary or fluency factors, as has been posited in a few prior studies of younger and more able readers. Rather, word recognition and language comprehension alone were found to account adequately for variation in reading comprehension in adults with low literacy.

This study compared the underlying factors measured by a state standards-based grade 4 English-Language Arts (ELA) assessment given to several groups of students. The focus of the research was to gather evidence regarding whether or not the tests measured the same construct or constructs for students without disabilities who took the test under standard conditions, students with learning disabilities who took the test under standard conditions, students with learning disabilities who took the test with accommodations as specified in their Individualized Educational Program (IEP) or 504 plan, and students with learning disabilities who took the test with a read-aloud accommodation/modification. The ELA assessment contained both reading and writing portions. A total of 75 multiple-choice items were analyzed. A series of nested hypotheses were tested to determine if the ELA measured the same factors for students with disabilities who took the assessment with and without accommodations and students without disabilities who took the test without accommodations. The results of these analyses, although not conclusive, indicated that the assessment had a similar factor structure for all groups included in the study.

This construct validation study investigated the factor structure of the Test of English as a Foreign Language (TM) Internet-based test (TOEFL (R) iBT). An item-level confirmatory factor analysis was conducted for a test form completed by participants in a field study. A higher-order factor model was identified, with a higher-order general factor (ESL/EFL ability) and four first-order factors for reading, listening, speaking and writing. Integrated Speaking and Writing tasks, which require language processing in multiple modalities, defined the target modalities (speaking and writing). These results broadly supported the current practice of reporting a total score and four scores corresponding to the modalities for the test, as well as the test design that permits the integrated tasks to contribute only to the scores of the target modalities.

A variety of cognitive diagnosis approaches (CDAs) have been used by researchers to extract fine-grained information about the linguistic knowledge states and skill mastery levels of test takers based on their performance on language tests. Although some of these approaches have been in use in the psychometric community for quite some time, many of these are relatively new to language testers and have not been widely researched or discussed in the field. The goal of this article is to describe how a CDA framework could be applied to language assessment. We start with a brief history and an overview of the major components of CDAs and then compare the characteristics of psychometric models for CDAs and discuss score interpretation and reporting issues. We conclude this article by outlining some of the challenges ahead and future research avenues for the application of CDAs in language assessment.

In cognitive diagnosis a Q-matrix (Tatsuoka, 1983, 1990), which is an incidence matrix that defines the relationships between test items and constructs of interest, has great impact on the nature of performance feedback that can be provided to score users. The purpose of the present study was to identify meaningful skill coding categories that reflect core language skills and processes assessed in the Reading and Listening sections of the Test of English as a Foreign Language (TM) Internet-based Test (TOEFL (R) iBT). The study was conducted as part of a research activity to explore the possibility of developing a detailed score report for low-stakes use by taking a cognitive diagnosis approach. Content experts conducted a test content analysis to develop draft Q-matrices, while measurement experts empirically analyzed the draft Q-matrices with examinee performance data using a cognitively diagnostic psychometric model called the fusion model (DiBello, Stout, & Roussos, 1995; Hartz, 2002). The draft Q-matrices were refined by repeating fusion model analysis and revision of skill definitions and item coding. This resulted in a set of Q-matrices that represented substantively meaningful score reporting categories of a suitable grain size for score reporting, while maintaining an acceptable level of examinee classification consistency.

The present study investigated the functioning of three psychometric models for cognitive diagnosis-the general diagnostic model, the fusion model, and latent class analysis-when applied to large-scale English as a second language listening and reading comprehension assessments. Data used in this study were scored item responses and incidence matrices, or Q-matrices (Tatsuoka, 1983, 1990), for the Listening and Reading sections of two field test forms of the TOEFL (R) iBT (Test of English as a Foreign Language T). Results of empirical data analyses based on the three cognitive diagnosis models above revealed that (a) all three models adequately separated examinees into two mastery levels on most of the reading and listening skills, (b) a moderate level of across-form consistency of examinee skill mastery classification was achieved for most of the reading and listening skills as defined in this study, but (c) a large proportion of examinees were classified into "flat" profiles (i.e., "nonmasters of all skills" and "masters of all skills"). Overall, the three models produced similar results in terms of examinee classification, but some subtle differences across them were identified as well.