DETECTING IMPROVEMENTS IN ACHIEVEMENT IS POSSIBLE, BUT NOT WITH SQRP OR NWEA MAP (Back to blog)

Optional: This section describes experimental data work to identify schools where successful instructional improvement efforts were underway. See good news and bad news below for a few quick points. This method is a reasonable starting point for identifying schools to learn more about instructional practices under churn conditions.

SQRP used NWEA MAP data. NWEA MAP is a consideration when understanding differences in SQRP ratings/status for high-churn schools. Several relevant observations about this data.

  • The Inspector General for the Chicago Public Schools found “gaming” patterns due to non-standard NWEA MAP test administration procedures. Schools took advantage of these procedures, for example, by giving students excessive amounts of time to complete the test.

  • When I worked with the data, I observed sizeable jumps in NWEA MAP scores, mostly in the early years of the SQRP system. It seemed that schools were learning how to get the best possible results on the exam. This “learning” may have been unrelated to improvements in instruction. Achievement data tended to drift/meander after big jumps, suggesting schools encountered limits to a strategy of chasing the data. This is anecdotal evidence but consistent with the findings above.

We did not trust NWEA MAP enough to use it in our analysis. (CPS has canceled its contract with NWEA MAP going forward.) Hence, we used IAR/PARCC data. As noted earlier, we did not find changes in achievement for the four school improvement patterns identified above using IAR/PARCC. We aimed to detect improvements in achievement among high-churn schools to learn more about instructional practices under churn conditions. We took a novel approach created by Paul Zavitkovsky. We analyzed shifts in distributions (i.e., percentages of students below, at, and above grade level) at the upper and lower ends of student achievement over time. For example, were the percentage of students at the upper end of achievement increasing in a school? Were they decreasing at the lower end? What do scores tell us when we categorize schools based on their patterns of increases and decreases?

We hypothesized that increases in achievement at the upper end of the distribution and decreases at the lower end meant a school was engaged in improving its instructional practices consistent with the Common Core. IAR/PARCC is a much better measure of Common Core learning than NWEA MAP.

GOOD NEWS:

  • Through this analysis, we found that about 31% of high-churn schools demonstrate substantial positive shifts in reading, about 23% demonstrate these shifts in math, and 15% demonstrate these shifts in BOTH reading and math.

  • Scale scores are best for schools that improve both the upper AND lower end of the achievement distribution, as shown below. When high-churn schools provide instruction that challenges higher-performing students, this supports the work of helping lower-performing students close gaps in their learning.

BAD NEWS:

  • Over twice as many high-churn schools (42%) showed ongoing declines in math achievement compared with ongoing declines in reading (19%).

  • We can’t tell from SQRP 1.0 data which schools are/are not improving their distribution patterns. Schools may rate well on the accountability system, but demonstrate declines in achievement using our method. Likewise, schools may not rate well on the accountability system, but demonstrate substantial improvement using our method. For example, while not-improving schools decline the most in ELA, the maintaining schools decline in math as much as the not-improving group, and the not-sustaining schools are as likely to see strong improvements in ELA as improving schools.

3rd grade achievement in CPS high-churn elementary schools: Schools that positively shift the achievement distribution over time at both the upper and lower ends demonstrate strong outcomes
3rd grade achievement in CPS high-churn elementary schools: Schools unable to positively shift the achievement distribution over time or that see a decline demonstrate weak outcomes

SUMMARY: We were able to detect improvements in achievement among high-churn schools. Improvements and declines in student achievement (based on our method) cut across the SQRP school improvement groups described earlier. There is a disconnect between achievement distributions reflecting progress on the Common Core and SQRP accountability data. This analysis shows that some high-churn schools are doing good work to improve instruction, but not being recognized for it and may in fact experience a downgrade in accountability ratings. Other schools are performing well on the accountability system at the same time they are oversee declines in achievement.

(Back to blog)