Study Purpose

The purpose of this study was to compare the consequential validity properties of CORE and a traditional oral reading fluency (ORF) assessment (easyCBM) for students in Grades 2 through 4. We examine the predictive and concurrent validity of CORE and easyCBM using (a) state reading test scores for students in Grades 3 and 4, and (b) CBM comprehension scores for all students Grades 2 - 4. We also use a longitudinal design with four repeated measurement occasions (waves) to model the within-year growth separately for both the CORE scale scores and the easyCBM words correct per minute (WCPM) raw scores to compare: (a) the estimates of the standard error (SE) of the CORE and easyCBM intercept and slope estimates, and (b) the reliability of the growth slopes, and the reliability of each measurement occasion, for the CORE and easyCBM growth models.

Method

This study was conducted in the 2017-18 and 2018-19 school years. We were unable to meet the targeted sample size in 2017-18, so we conducted a replication study in 2018-19 to accrue the remaining student participants. These studies ran concurrently with the Calibration, Equating, Linking Study.

The table below shows the open and closing dates for the four waves of testing for each year of the study.

Reading Windows for the Consequential Validity Study

Wave	2017-18		2018-19
Wave	Open	Close	Open	Close
1	2017-10-24	2017-11-13	2018-10-22	2018-11-09
2	2017-11-29	2018-02-12	2018-12-03	2018-12-21
3	2018-02-12	2018-03-23	2019-02-11	2019-03-08
4	2018-05-01	2018-06-13	2019-05-13	2019-06-07

Sample

In each of the 2017-18 and 2018-19 school years, we recruited three school districts for participation. District A has less than 3,000 students, about 3% EL students and 19% students with IEPs (Town: Distant). District B has about 11,000 students, about 6% EL students and 17% students with IEPs (Suburb: Midsize). District C has about 17,400 students, about 3% EL students and 15% students with IEPs (City: Midsize). District D has about 4,600 students, about 4% EL students and 16% students with IEPs (City: Midsize). District E has about 4,400 students, about 14% EL students and 16% students with IEPs (Town: Distant). We worked with Districts A and B in the prior two years of the project (2014-15 and 2015-16).

We targeted the largest elementary schools within two districts because (a) we needed such a large number of student participants that it was desirable to have all Grade 2 – 4 classrooms in a school participate, and (b) because the budget called for schools to share a set of 30 headsets with noise-cancelling microphones, it was most beneficial to share one set of headphones across as many classrooms as possible.

Data for this study to date include approximately 2,094 students and approximately 26,272 audio files (where each passage read is a file).

The table below details the number of schools, teachers, and the approximate number of students who participated in this study.

Participants in the Consequential Validity Study

School District	Schools	Teachers	Approximate Students
2017-18
District A	1	10	158
District B	6	55	662
District C	4	18	414
2018-19
District D	1	11	256
District E	2	25	462
District F	3	21	569
District A	1	9	142

Total	18	149	2,663

Participating districts received a reduction in the annual cost of the district version of easyCBM. Schools were compensated $100 per participating classroom to maximize the number of teacher participants within each school (in order to maximize the number of students who could share a set of headsets and minimize the number of headsets purchased). In addition, all participating teachers were paid a research incentive ($50 gift card) for completing a survey about the assessment process.

For teacher participants we used the active informed consent procedure, by which we provided the teachers with a written document containing all the required elements of informed consent that gives teachers the opportunity and sufficient time to provide permission. For student participants we used (a) the passive parental consent procedure, by which caregivers were provided with a written document containing all the required elements of informed notification within two weeks of the study start date (giving caregivers the opportunity and sufficient time to opt-out of providing permission), and (b) assent from student participants, by which an assent form appeared to each student before online reading began and students were asked to click whether they were willing to participate in the study before they could continue.

Measures

For more detailed information about CORE and easyCBM passage development, please see the Study Procedures for the Content & Convergent Evidence Study.

CORE assessment

Passages were written with the following specifications. Each passage was to be an original work of fiction, and be ±5 words of the target length (i.e., short = 25 words, medium = 50 words, long = 85 words). Each passage was to have a beginning, middle, and end; this broad specification was intended to give the passage writer freedom in meeting the word constraint specification, which was crucial in this project.

Final passages included 330 passages total, 110 at each of Grades 2-4, with 20 long passages, 30 medium passages, and 60 short passages for each grade.

We decided to exclude the short passages (≈ 25 words) from this study for three reasons. First, the results of our Teacher Survey of the Accessibility and Text Features of the Computerized Oral Reading Evaluation (CORE) suggested that our the CORE short passages are most appropriate for Grade 2 students and the CORE long and medium passages were preferred by teachers for students in Grades 3 and 4. Second, our preliminary psychometric analyses revealed that more words read by a student will increase reliability of the scale score, and this does not depend on passage length (i.e., 200 words from short passages or from medium passages are equally reliable). Third, we determined that our original plan that consisted of administering 4 long, 7 medium, and 12 short passages would take too much time and be too burdensome on teachers and students, particularly low-performing students.

Traditional ORF assessment

The easyCBM ORF measures were the traditional ORF assessments administered for the purpose of comparison to the CORE system. Developed in 2006, easyCBM is an online screening and progress monitoring assessment system for use in schools working under an RTI framework, available for an annual fee for district-wide adoption. easyCBM is currently used by over 143,000 teachers, representing over 1 million students in schools across every state in the country.

The ORF passages used in easyCBM were developed to assess students’ ability to fluently read narrative text. During instrument development, each form was created to be consistent in length and the readability of each form was verified to fit appropriate grade-level, initially using the Flesch-Kincaid index feature available on Microsoft Word (Alonzo & Tindal, 2007). The passages were developed to be of equivalent difficulty for each grade level following word-count, grade-level guidelines (e.g., Flesch-Kincaid readability estimates), and form equivalence empirical testing using repeated measures ANOVA to evaluate comparability of forms (Alonzo & Tindal, 2007).

Procedures

The project website was programmed to accommodate the study design. This consists of a log-in for teachers to upload their classroom roster.

Teachers gave students a quick overview of what they would be doing.

“Today we are going to have you read some short passages on a computer. You are going to help researchers at the University of Oregon see if computers can be used to listen how students read. You will be wearing head sets so the computer can listen to the readings. Just do your best reading.”

Teachers made sure the mute buttons were not switched on, that the volume was turned up so students could hear the directions, and that the microphones were about an inch away from the students’ mouths, and asked them not to touch it while they read.

Students were directed to go to the study website. The first page showed a student assent form, where students clicked whether they would like to participate. If they clicked Yes the testing continued. If they clicked No they were opted-out of the study.

The study instructions were then be presented via audio as well as print.

“Get ready, [student name]! You are about to do some reading! After pressing start, read the story on the screen. When you are finished click done. Do your best reading, and have fun!”

Students clicked Start to begin reading, and were randomly assigned to a fixed set of passages. Student then read each passage aloud, and progressed through each of the study’s 10 to 12 passages. Students clicked Done after reading a passage, and the system automatically advanced a student to the next screen after 90 seconds (with a 10 second warning). Students clicked Continue to read the next passage, and the system automatically advanced a student to the next passage after 10 seconds. The purpose of automatically advancing students was to ensure students, particularly low-performing readers, were not burdened with the task and did not take an excessive amount of time to complete the task.

Collected data for the purpose of the study included, for all passages: duration and digital audio recording (which will be scored by the automated speech recognition engine). Audio recordings of fluency readings are used for the purposes of scoring, data analysis, improving the scoring engine, and equating all passages within each grade, as well as linking all passages across grades.

Form Assembly Design

We were unable to meet the targeted sample size for our 2015-16 Calibration, Equating, Linking Study, so we used Wave 3 of the Consequential Validity Study (2017-18 and 2018-19) as a replication to accrue the remaining student participants for the Calibration, Equating, Linking Study. Thus, Wave 3 of each year of the Consequential Validity Study also served as a replication of the Calibration, Equating, Linking Study, such that the sample and data for the Wave 3 measurement occasion served the purpose of both studies. The form assembly design is as follows.

The participating students were randomly assigned to one of the 10 forms at the beginning of the study. Forms for Waves 1, 2, and 4 consisted of one easyCBM passage and nine passages from the CORE system (6 medium passages and 3 long passages). Ten forms were constructed such that Wave 3 forms were identical to those delivered in 2016-17 and 2017-2018, and CORE medium and long passages were randomly selected from the remaining passages such that no passage was duplicated at any point across the four waves. Forms for Wave 3 consisted of one easyCBM passage and: 10 CORE passages for Grades 2 and 4, and 12 CORE passages for Grade 3. Because Wave 3 also served as a replication of the Passage Calibration, Equating, Linking Study, forms were identical to those created in 2016-17 (meaning that the same CORE passages were associated with the same form id). However, there were two caveats: (1) one traditional ORF (easyCBM) passage was included in each form (to meet the needs of the Consequential Validity Study), and (2) passages within forms, including the traditional ORF passage, were randomly ordered (so as to control for fatigue and order effects, as well as reduce the chance that two adjacent students read the same passage at the same time) so that the passage delivery order was likely different across years.

For Wave 3, we administered one calibration form and one equating/linking form for each examinee, where a calibration form consisted of 3 long and 3 medium passages, and an equating/linking form consisted of 1 long and 1 medium passage for Grades 2 and 4, and 2 long and 2 medium passage for Grade 3.

All calibration forms had common passages to two other forms within each version (1 and 2 common passages for long and medium, respectively). Each calibration form had 2 unique passage for long and 2 unique passages for medium. Each equating/linking forms was assembled with long and medium passages, but fewer passages from each type. Each equating/linking forms contained one (Grade 2 and 4) or two (Grade 3) long passage(s), and one (Grade 2 and 4) or two (Grade 3) medium passage(s). In addition, each equating/linking form had off-grade-level passages. All Grade 2 and 4 students received Grade 3 passages as off-grade-level passages, and all Grade 3 students received Grade 3 passages as off-grade-level passages. For each grade level, the CORE reading passages were assembled into 10 calibration/ equating/linking forms. Passages were randomly ordered within forms. The tables below shows form assembly design.

Form Assembly Design for Grades 2 and 4

Form	On-Grade Passages¹								Off-Grade Passages²
Form
1	1L	2L	3L	1M	2M	3M	4M	5M	2L	3M
2	3L	4L	5L	4M	5M	6M	7M	8M	4L	6M
3	5L	6L	7L	7M	8M	9M	10M	11M	6L	9M
4	7L	8L	9L	10M	11M	12M	13M	14M	8L	12M
5	9L	10L	11L	13M	14M	15M	16M	17M	10L	15M
6	11L	12L	13L	16M	17M	18M	19M	20M	12L	18M
7	13L	14L	15L	19M	20M	21M	22M	23M	14L	21M
8	15L	16L	17L	22M	23M	24M	25M	26M	16L	24M
9	17L	18L	19L	25M	26M	27M	28M	29M	18L	27M
10	19L	20L	1L	28M	29M	30M	1M	2M	20L	30M
¹ Note. L = long passage; M = medium passage. ² All Grade 2 and 4 students received Grade 3 off-grade passages.

Form Assembly Design for Grade 3

Form	On-Grade Passages¹								Off-Grade Passages²
Form									Grade 2	Grade 4
1	1L	2L	3L	1M	2M	3M	4M	5M	2L 3M	2L 3M
2	3L	4L	5L	4M	5M	6M	7M	8M	4L 6M	4L 6M
3	5L	6L	7L	7M	8M	9M	10M	11M	6L 9M	6L 9M
4	7L	8L	9L	10M	11M	12M	13M	14M	8L 12M	8L 12M
5	9L	10L	11L	13M	14M	15M	16M	17M	10L 15M	10L 15M
6	11L	12L	13L	16M	17M	18M	19M	20M	12L 18M	12L 18M
7	13L	14L	15L	19M	20M	21M	22M	23M	14L 21M	14L 21M
8	15L	16L	17L	22M	23M	24M	25M	26M	16L 24M	16L 24M
9	17L	18L	19L	25M	26M	27M	28M	29M	18L 27M	18L 27M
10	19L	20L	1L	28M	29M	30M	1M	2M	20L 30M	20L 30M
¹ Note. L = long passage; M = medium passage. ² All Grade 3 students received two Grade 2 and two Grade 4 off-grade passages.

Acknowledgments

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A140203 to the University of Oregon. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Study Procedures: Consequential Validity

Table of Contents