Variability in the impacts of COVID-19 on student achievement
VoxEU Column COVID-19 Education

Variability in the impacts of COVID-19 on student achievement

There is robust evidence that children lost ground academically as a consequence of the pandemic, but less research into how the effects varied by school district and household demographics. Using test data from 2.1 million students across the US, this column finds that remote instruction was a primary driver of widening achievement gaps by race and poverty. The authors argue that low-income districts that taught remotely in 2020-21 will need to spend nearly all of their federal aid on academic recovery to help students recover from pandemic-related achievement losses.

The COVID-19 pandemic has led to a critical moment in the history of US education. There is now robust evidence that the disruptions dramatically impacted students’ learning and wellbeing and exacerbated pre-existing educational inequities for historically marginalised students (Curriculum Associates 2021, Darling-Aduana et al. 2021, Dorn et al. 2020, Grewenig et al. 2020, Kogan and Lavertu 2021, Lewis and Kuhfeld 2021). District leaders, teachers, and communities preparing for the coming school year will need to know just how far behind their students are to provide them with targeted academic support. But while we know that students have lost ground academically as a consequence of the pandemic, much of the existing research has not documented how or why these effects varied across districts, making it difficult to provide universal recommendations about the scale of additional support needed, or the subjects and students to target for recovery efforts in any one district.

To that end, we recently released two reports that provide a comprehensive examination of the academic impact of the pandemic, as well as differences in those impacts across districts (Goldhaber et al. 2022a and 2022b). We use NWEA MAP Growth test data from more than two million students across 49 states to examine changes in achievement and growth from the autumn of 2017 to the autumn of 2021. To examine the impact of the pandemic on growth, we estimate the effect size of the difference between students’ academic growth from the autumn 2019 to autumn 2021 pandemic period to the most recent pre-pandemic period, from autumn 2017 to autumn 2019.

Our reports make four primary contributions. First, we add to the growing body of evidence that the pandemic was devastating for student achievement and growth. Relative to autumn 2019, median test scores in autumn 2021 decreased by roughly 0.20 standard deviations in maths and 0.10 standard deviations in reading. For perspective, these drops are larger than those observed in Louisiana in the aftermath of Hurricanes Katrina and Rita (Sacerdote 2012) and larger than the COVID-19 drops predicted by researchers in the spring of 2020 (Burgess and Sievertsen 2020). We also estimate reductions in two-year achievement growth during the pandemic period. We used average NWEA test score gains per week within a school year (across grades 3–8) to translate our estimates into lost weeks of instruction. Nationwide, students were an average of three months behind their expected achievement in maths and over two months behind in reading. 

Second, we show that the impacts of the pandemic on test scores were not the same across different districts. Using data from the American Enterprise Institute’s 'Return to Learn Tracker' on school districts’ remote learning status, we find that students attending schools that were primarily remote for the 2020–21 school year, students at high-poverty schools,1 students in elementary school, and students of colour tended to be more negatively impacted. Our analysis shows that the impact of remote schooling on historically marginalised students was two-fold. High-poverty schools, which have higher percentages of students of colour, were more likely to operate remotely for more of the year, and the negative impact of being remote was larger for the subgroups of students that tend to be served by these schools. We estimate students in high-poverty schools that were remote for 50% or more of 2020-21 were 5.5 months – or over half of a school year – behind in maths in the autumn of 2021. 

Our analysis of the incidence and impact of remote schooling allows us to make a third contribution, documenting the role of remote instruction in the widening of achievement gaps by race/ethnicity and by school poverty status. As displayed in Figure 1, drops in maths test scores were similar among high-poverty and low-poverty districts that did not operate remotely for the great majority of the 2020-21 school year. But, among districts that were remote for over half of the year, math-score drops in high-poverty schools were about 1.7 times the size of those in low-poverty schools. 

Figure 1 Pandemic achievement effects by remote schooling and school poverty (maths)

Figure 1

Fourth, we demonstrate that in spite of broad trends showing patterns in the districts, students, and subjects most impacted by the pandemic, district demographics and the amount of time a district spent remotely don’t tell the whole story. The spread of the dots in Figure 2 shows that the pandemic's impact on test scores varied widely across districts. Though nearly 90% of districts experienced lower than expected achievement (the dots below the zero on the Y-axis), not all districts did. Districts serving lower achieving students, who would already be expected to have lower achievement in autumn 2021, tended to be further behind (all the dots in the bottom-left quadrant of the figure). But in some cases, districts with similar pre-pandemic achievement, enrolments, student demographics, income levels, and amounts of remote instruction in 2020-21 had quite different autumn 2021 maths results (this is also true for reading tests). District A and District B are similar on all those counts. But students in District A were about two weeks behind in maths than would have been expected from a pre-pandemic year, whereas students in District B were about 14 weeks behind. This suggests that districts wishing to accurately target students for pandemic-related academic recovery will need to carefully assess local data rather than relying on national trends to infer local recovery needs.

Figure 2 Variation in median autumn 2021 district maths achievement (grades 3–5)

Figure 2

Taken together, our findings support calls for urgent implementation of additional supports for students at scale, tailored to a district’s needs. Fortunately, districts have access to about $190 billion in federal aid through the Elementary and Secondary School Emergency Relief (ESSER) funds, which is roughly $3,850 per pupil. To put this sum in perspective, according to the US Census, spending per pupil in public schools was nearly $13,500 in FY 2020 (US Census Bureau 2022). Thus, the additional funding provided through ESSER (which can be obligated through September of 2024) is over 20% of average per pupil spending.2 Thus, districts have an extraordinary opportunity to invest in academic recovery interventions. 

Districts are implementing a variety of strategies, including but not limited to reduced class sizes, tutoring programmes, summer learning programmes, Saturday academies, virtual learning programmes, extended school days and years, double-dose math and reading blocks (FutureEd 2022). Unfortunately, we have limited evidence on the effectiveness of many of these strategies; even those most promising for recovery, such as tutoring, may not yield effects large enough to add up to a full recovery in many districts (e.g. Lynch et al. 2022, Filges et al. 2018). 

High-dosage tutoring (HDT) – tutoring administered by a qualified tutor in one-on-one or very small group settings for at least 30 minutes several times a week – stands out as an evidence-based strategy with great potential. HDT has been shown (Nickow et al. 2020) to have large effects on maths scores for elementary school students (+0.44 standard deviations, or 16 weeks) and middle school students (+0.20 standard deviations, or 14 weeks). However, challenges to implementation this year, such as finding available tutors, have prevented or stalled HDT in many districts. Furthermore, based on our calculations, for the hardest hit districts, even providing HDT to all students may not close their recovery gap. 

Full academic recovery from the pandemic will almost certainly take multiple years and multiple strategies. Unfortunately, we know from an abundance of research that conceptually well-grounded programmes often fail to improve student outcomes (e.g. Heinrich et al. 2010). Timely monitoring and evaluation of districts’ recovery initiatives will be essential, so we can adapt our strategies over time and give our children the best chance at recovery.

References

Burgess, S and H H Sievertsen (2020), “Schools, skills, and learning: The impact of COVID-19 on education”, VoxEU.org, 1 April. 

Curriculum Associates (2021), “Academic achievement at the end of the 2020–2021 school year: Insights after more than a year of disrupted teaching and learning”, June. 

Darling-Aduana, J, H T Woodyard, T R Sass and S S Barry (2022), “Learning mode choice, student engagement, and achievement growth during the COVID-19 pandemic”, CALDER Working Paper 260-0122. 

Dorn, E, B Hancock and J Sarakatsannis (2021), “COVID-19 and education: The lingering effects of unfinished learning”, McKinsey & Company, 27 July. 

Filges, T, C S Sonne‐Schmidt and B C V Nielsen (2018), “Small class sizes for improving student achievement in primary and secondary schools: a systematic review”, Campbell Systematic Reviews 14(1): 1–107.

FutureEd (2022), “How local educators plan to spend billions in federal Covid aid”, 7 June. 

Goldhaber, D, T J Kane, A McEachin, E Morton, T Patterson and D O Staiger (2022), “The consequences of remote and hybrid instruction during the pandemic”, NBER Working Paper 30010. 

Goldhaber, D, T J Kane, A McEachin and E Morton (2022), “A comprehensive picture of achievement across the COVID-19 pandemic years: Examining variation in test levels and growth across districts, schools, grades, and students”, CALDER Working Paper 266-0522.

Grewenig, E, P Lergetporer, K Werner, L Woessmann and L Zierow (2020), “COVID-19 school closures hit low-achieving students particularly hard”, VoxEU.org, 15 November. 

Heinrich, C J, R H Meyer and G Whitten (2010), “Supplemental education services under No Child Left Behind: Who signs up, and what do they gain?”, Educational Evaluation and Policy Analysis 32(2): 273–298. 

Kogan, V and S Lavertu (2021), “The COVID-19 pandemic and student achievement on Ohio’s third-grade English language arts assessment”, 27 January. 

Lewis, K and M Kuhfeld (2021), “Learning during COVID-19: An update on student achievement and growth at the start of the 2021–22 school year”, Northwest Evaluation Association (NWEA), December. 

Lynch, K, L An and Z Mancenido (2022), “The Impact of Summer Programs on Student Mathematics Achievement: A Meta-Analysis”, EdWorkingPaper 21-379.  

Nickow, A J, P Oreopoulos and V Quan (2020), “The Impressive Effects of Tutoring on PreK-12 Learning: A Systematic Review and Meta-Analysis of the Experimental Evidence”, EdWorkingPaper 20-267.  

Sacerdote, B (2012), “When the saints go marching out: Long-term outcomes for student evacuees from Hurricanes Katrina and Rita”, American Economic Journal: Applied Economics 4(1): 109–135. 

US Census Bureau (2022), “Per Pupil Spending Continues to Increase in 2020”, May.

Endnotes

1 We defined high-poverty schools as schools with over 75% of students eligible for free or reduced-price lunches, and low-poverty schools as schools with less than 25% of students eligible for free or reduced-price lunches.

2 ESSER funds were allocated to districts based on the Title 1 funding formula, so the amount of funds received by each district varied from the average $3,850 per pupil.

525 Reads