Teacher Focus Group Survey

The purpose of this post was to summarize teacher feedback about the utility and interpretability of the CORE data compared to Traditional ORF scores to help collect consequential evidence of validity for CORE.

Joseph F. T. Nese https://education.uoregon.edu/people/faculty/jnese (University of Oregon)https://www.uoregon.edu , Makayla Whitney (University of Oregon)https://www.uoregon.edu
2021-06-24

Introduction

The purpose of this survey was to solicit teacher feedback about the utility and interpretability of the CORE data compared to Traditional ORF scores to help collect consequential evidence of validity for CORE.

We conducted an online survey in summer of 2020 with institutional IRB approval with nine teachers, three in each of Grades 2 to 4, who participated in our 2018-19 study.

We provided participating teachers with reports and graphs for individual students using de-identified, longitudinal student results from the 2018-19 study to compare the differentiation of teachers’ decisions about instruction between Traditional ORF (easyCBM) results and CORE results.

Summary

Nearly all teachers reported the student sample as reading “below grade level”, but many teachers wanted more information (besides ORF scores) before labeling a student “at risk”.

In general, most teachers agreed that the CORE data was more trustworthy based on its trajectory, and was better for progress monitoring as it displayed steady growth, but several teachers noted that both CORE and Traditional ORF showed consistent trends. Although the teachers reported it was difficult to interpret whether an intervention was working based on the available data, many teachers noted that it was easier to differentiate that the intervention was or was not working based on the CORE data. It was commonly difficult for teachers to determine if the intervention was effective based on the Traditional ORF data.

Teachers unanimously agreed that having measurement error displayed on the graphs was helpful when making data-based decisions. Many teachers agreed that the CORE data was better for progress monitoring and data-based decisions because it appeared to have a smaller measurement error and more consistent trend than the Traditional ORF data. Some teachers reported increased usability of the CORE data as it gave them more reliable and accurate data. Common applications of the measurement error included early identification, understanding the range of a student’s score, and as a guide to evaluate whether their interventions were effective or not.

Sample

All teachers who participate in the 2018-19 study were invited to participate in the study. The first three teachers per grade that responded to the invitation to participate were selected to participate.

Of the 1,008 student from the 2018-19 study, only 430 students had complete ORF data across the four measurement occasions for both measures, CORE and Traditional ORF. For each grade, we randomly sampled four students that scored at/above the 10th percentile and at/ below the 25th percentile.

Procedures

For each student, we created two figures. The first figure showed the CORE and Traditional ORF scores across the four measurement occasions. The second figure showed the same data but included a shaded region representing the standard error of measurement (SEM) at each measurement occasion. The Traditional ORF SEM was a classical SEM taken from the easyCBM technical documentation, and the CORE SEM was a conditional SEM. A more thorough description can be found here.

For each student, teachers responded to the following items.

Teachers were provided with the student’s fall easyCBM oral reading fluency score in words correct per minute (WCPM) and were asked to:

  1. Describe the student’s reading level based on this score alone.
  2. Describe if the student was above, at, or below grade level.
  3. Describe if the student was at risk of poor reading outcomes.

The teacher then needed to refer to the figure without the SEM to answer the following questions.

  1. Which line might be better for progress monitoring?

  2. Based on the trajectory of the lines - that is, the shape of the lines - which line might be more trustworthy?

  3. Let’s say the figure shows data of a student receiving a reading intervention. For both the the YELLOW and BLACK lines separately: Look at the first three data points only (ignore the last data point from May or June). What might you say about the intervention? Why?

Teachers were told:

The figure below shows the same data you have just seen, but in a different way. Now, the lines are in separate graphs, and each is surrounded by gray shading. The shaded area represents the area that we think contains the students true score. This is sometimes called the measurement error.

The less measurement error, the better, because that means the score is more accurate. That is, the score better reflects the students “true ORF ability.” So the smaller the gray shaded area, the better. How familiar are you with this concept?

  1. When you and your peers discuss student data (for example, in school teams), do you discuss measurement error, or whether the data is accurate, or whether the data reflects what the student can do?

  2. Which line might be better for progress monitoring and data-based decisions? Why?

  3. Does the shaded area in the graphs give you useful information if you were using the graphs to make data-based decisions?

  4. How might you use this figure to make a data-based decision?

Teachers in Grades 3 and 4 were also given the students’ year-end Smarter Balanced ELA (reading) performance level and were asked the following questions.

  1. Which line might better correspond with the SBAC performance level?

  2. Based on the last data point from May or June ONLY, which point might better match what you expect from a student at this performance level?

  3. Based on the first data point from Oct or Nov ONLY, which point might better match what you expect from a student at this performance level?

Results

Grade 2

Student Figures

The below figures are representative of 2nd grade student scores. Please refer back to these figures when observing the data collected from teacher surveys.



Reading Level


Based on the fall ORF score, the student is reading:


Below are descriptions of the student’s reading level based on fall ORF score alone.
Teacher ORF Response
Student A
Teacher 1 36 This student is below grade level. We want students coming in to second grade in the fall at around 63 cwpm to be at the 50th %ile or above. This students falls under the 25th percentile.
Teacher 2 36 Based on this score alone I would said this student is reading below grade level, HOWEVER I occasionally have a student who reads slowly (below grade level in ORF) yet has great comprehension and does not fit into any of our reading intervention programs. It is also fairly typical for kids to come into fall a little low after having the summer off.
Teacher 3 36 In my class, I would have not placed this student in an intervention group for this score alone. Nevertheless, this is also a score that would have caused me to take note of this student.
Student B
Teacher 1 34 This students is below grade level and coming into second grade below the 25th percentile.
Teacher 2 34 This student's score is below the fall target for second graders. Although it is possible that this student is coming out of summer and will pop back up, this student will likely need/benefit from interventions.
Teacher 3 34 I would want to say they were below grade level, but I wouldn't have enough information to know what support to give them.
Student C
Teacher 1 34 This student is coming in below the 25th percentile in the fall of 2nd grade.
Teacher 2 34 This student is reading below the fall benchmark target and will likely need/benefit from interventions.
Teacher 3 34 The student is probably below grade level, but I don't know what services to offer the student based on that score alone.
Student D
Teacher 1 30 This student is far below grade level coming into second grade.
Teacher 2 30 This student is pretty significantly below the fall target for second grade. He/she would be placed into an intervention.
Teacher 3 30 Student seems to be below grade level, however I do not have enough data to make an informed decision about supports for the student.


Results: Based on all of the second grade ORF scores, all three teachers unanimously agreed that these students fell below the 2nd grade reading level and would be considered at risk.



At Risk?


Is this student at risk of poor reading outcomes?


If you answered “not enough information,” please specify what other data you would like to see for this student.
Teacher Student Response
Teacher 3 Student A Reading Fluency
Teacher 3 Student B Reading Fluency, Oral samples
Teacher 3 Student C Reading Fluency, Oral Samples
Teacher 3 Student D Reading Fluency, Oral Sample


Results: Many teachers believe these students are at risk of poor reading outcomes. When asked to further explain, many teachers felt they didn’t have enough information. To make a conclusive decision about the student, teachers asked to see an oral sample and data on reading fluency.



Progress Monitoring


Which line might be better for progress monitoring? And why?

CORE (Black)
Teacher Student
Teacher 2 Student A It has a more steady and realistic incline. The yellow line does a lot of bouncing around. I don't typicall trust a huge outlying score (like the 87) until I have seen several more scores like it.
Teacher 2 Student B It shows steady growth without outlying data points. However, if you're asking which "looks" better, I suppose yellow, because it is ultimately higher.
Teacher 2 Student D The growth is more steady and reliable and the final data point is higher than the yellow.
Teacher 3 Student B They show progress coming back from summer and then level off at the end of the school year. This seems typical of many students.
Teacher 3 Student D The data seems more indicative of student progress.
No difference between the two lines
Teacher Student
Teacher 2 Student C Both lines show a pretty rapid rate of growth with one outlier date point.
Teacher 3 Student A They both show me WCPM, I have seen both types of progress in my students.
Teacher 3 Student C They both show typical student data.
I don't know
Teacher Student
Teacher 1 Student A I don't understand what you mean by "better for progress monitoring." Are you asking which student would better qualify for progress monitoring? Or which line looks like more accurate progress monitoring? I would definitely consider the May score on the yellow line to be an outlier and would reassess this student on 3 different passages.
Teacher 1 Student B Again, not understanding what this question is asking. The yellow line is what we typically see as students go up and down based on passages.
Teacher 1 Student C I don't understand what the question is asking.
Teacher 1 Student D NA


Results: For most of the plots, teachers agreed that the black (CORE) line was better for progress monitoring. Common aspects pointed out included steady and realistic inclines displayed as well as the data being indicative of progress.



Trustworthy Trajectory


Based on the trajectory of the lines - that is, the shape of the lines - which line might be more trustworthy? And why?

Why?
CORE (Black)
Student A Teacher 1 It looks like more typical growth because of the last data point on the yellow line. Although, it is also very normal to see students go up and down depending on the passage.
Student A Teacher 2 The black line shows steady growth, wich is probably more realistic of improvement.
Student A Teacher 3 Typically students show a steady progress rather that a sudden increase (shown in yellow).
Student B Teacher 2 The growth is more steady.
Student B Teacher 3 This trajectory is common for the majority of my students.
Student D Teacher 1 It shows more typical growth without the large spike that the yellow line has.
Student D Teacher 2 It has no outlying data points like the yellow line does.
Student D Teacher 3 The data is consistent.
I don't know
Student B Teacher 1 Either of these lines could represent a typical student. Ideally, they'd all show growth like the black line, but that's not reality.
No difference between the two lines
Student C Teacher 1 They follow a similar pattern.
Student C Teacher 2 Both lines show a pretty rapid rate of growth with one outlier date point.
Student C Teacher 3 I am used to seeing data represented in both student samples.


Results: Many teachers agreed that the black (CORE) line is more trustworthy based on its trajectory. Some noted the steady growth that the line follows which is indicative of progress.



Intervention Working?



Let’s say the figure shows data of a student receiving a reading intervention. For the Traditional ORF (YELLOW) line, look at the first three data points only (ignore the last data point from May or June). What might you say about the intervention? And why?
Teacher
I cannot tell whether the intervention is working or not
Student A Teacher 1 For a student this low, they would probably need to be progress monitored at a first grade level. We'd want to see bi-weekly data points to see if they're making progress at their instructional level. Progress on a second grade level passage may not be visible even though an intervention is working because the second grade passage is still too difficult.
Student A Teacher 2 The student seems to be progressing in an upward trend. BUT, since the student is bouncing up and down, I would need to see more data points showing growth. Otherwise, moving from 21 to 28 WPM is not adequate growth.
Student A Teacher 3 The data points are close together and there is some reading regression from time to time.
Student B Teacher 2 While the third data point looks really good, it is not super trustworthy on its own after the prior data point that was so low. I would need to see more data points showing a continuation of the upward trend before I would trust the growth.
Student B Teacher 3 The data points are all over.
Student D Teacher 1 That last data point being so much higher than the first two makes me think there was an error. I would want more data to tell whether the intervention was working or not.
Student D Teacher 2 This student appears to be showing miraculous growth, however, I wouldn't trust this data without additional data points supporting/proving the steady growth.
It appears the intervention is working for the student
Student B Teacher 1 They ended higher than they were. Again, I'd like to see a lot more data points between Nov-Mar to make this decision.
Student C Teacher 1 They are showing a lot of growth in ORF from Oct-Mar.
Student C Teacher 2 This student appears to be making great growth. All date points are showing growth. BUT, I would still love to see more data points to see this student maintaining the growth, as well as accuracy.
Student C Teacher 3 There is a steady increase in student achievement.
Student D Teacher 3 There is a large increase in student ability at the third point.


Let’s say the figure shows data of a student receiving a reading intervention. For the CORE (BLACK) line, look at the first three data points only (ignore the last data point from May or June). What might you say about the intervention? And why?
Teacher
I cannot tell whether the intervention is working or not
Student C Teacher 1 If I'm ignoring the last data point, I'd want to see at least one other data point near the 67 cwpm mark from March before deciding whether the intervention was working.
It appears the intervention is working for the student
Student A Teacher 1 They increased from 18 to 33, so I assume it's working, but again I'd want to see more information and data points at their instructional level before making this assumption.
Student A Teacher 2 The black line shows steady growth in an upward trend with bouncing up and down. Although this nowhere near the amount of growth I would like to see from this student in this amount of time, this tells me the student is steadily progressing/showing growth. I would hope to see more date points in that amount of time (we progress monitor student receiving interventions every two weeks). I would also like to see student accuracy. For a student who started out reading so low, it would be important to me that their accuracy is improving, even if their fluency isn't making huge gains.
Student A Teacher 3 There is a steady increase in WCPM,
Student B Teacher 1 It shows a steady increase in their ORF.
Student B Teacher 2 This student appears to be making steady growth with no outlying data points.
Student B Teacher 3 There is a constant growth demonstrated throughout the year.
Student C Teacher 2 This student appears to be making great growth, although the number of data points is limited!
Student C Teacher 3 There is very little regression and then student achievement is demonstrated.
Student D Teacher 1 It shows realistic growth.
Student D Teacher 2 This student appears to be making steady, realistic growth with now crazy outlying data points.
Student D Teacher 3 There is a steady performance increase throughout the three data points.


Results: Most teachers noted that is was easier to differentiate that the intervention was working based on the black (CORE) line. It was commonly difficult to determine if the intervention was working when only observing the yellow line.




Student Figures with Measurement Error