Scaling up effective programmes is a classic problem in social policy. Interventions that succeed in small-scale demonstrations often fail to sustain their effects when implemented at larger scales. For example, recent studies of early childhood programmes, class size reductions, and the Success For All curriculum show effects that fall short of the impressive gains seen in earlier evaluations of similar interventions (Heckman et al. 2010, Heckman et al. 2013, Puma et al. 2012, Krueger 1999, Jepsen and Rivkin 2009, Borman et al. 2007, Quint et al. 2015). This suggests that in some cases the success of programmes may be due to unique features such as special teachers, school leaders, peer environments, or other factors that cannot be easily replicated.
“No Excuses” charter schools in the US share a set of practices that includes high expectations, strict discipline, increased time in school, frequent teacher feedback, high-intensity tutoring, and data-driven instruction. Evidence based on randomised lotteries shows that No Excuses charter schools produce test score gains large enough to close achievement gaps between racial and socioeconomic groups in a short time, as well as improvements in longer-run outcomes like teen pregnancy and four-year postsecondary attendance (Abdulkadiroglu et al. 2011, 2017, Angrist et al. 2012, 2013, 2016, Dobbie and Fryer 2011, 2013, 2015, Tuttle et al. 2013, Walters 2018). It is an open question whether these effects can be replicated at a larger scale, however. Replicability is of particular interest in the charter school context, since by design charters are intended to serve as laboratories of innovation and spread successful practices.
A policy experiment in Massachusetts
In a recent paper (Cohodes et al. 2019), we address the replicability question using a recent policy change that expanded the charter school sector in Boston, Massachusetts, a city where most charter schools operate according to the No Excuses model. In 2010, Massachusetts passed an education reform law that raised the state’s cap on the share of funding dedicated to charter schools in low-achieving districts. Charter operators that the state labelled “proven providers” with track records of success were allowed to expand existing campuses or open new campuses in these districts. As a result, the number of charter schools in Boston increased from 16 to 32 between 2010 and 2014, with most of the new openings linked to existing No Excuses charter schools. As a result, the fraction of Boston sixth graders attending charter schools increased from 15% to 31% between 2010 and 2015 (see Figure 1).
Figure 1 Charter school enrolment in Boston
We use records from randomised charter school admission lotteries to study changes in the effectiveness of Boston’s charter middle schools during this period. Our research design compares outcomes for students who receive lottery offers for outcomes to those that do not. Since lottery offers are random, these comparisons are free of selection bias, allowing us to estimate the causal effects of charter school attendance.
Consistent with previous studies, our estimates for students applying before the 2010 policy change show large positive effects of charter attendance on test scores. During this period, a year of attendance at a Boston charter middle school boosted maths achievement by 0.2 to 0.3 standard deviations and increased English achievement by 0.1 standard deviations on average. To put these effects in context, in Boston, black students score about 0.8 standard deviations below white students in math and 0.7 standard deviations lower in English (Abdulkadiroglu et al. 2011). Therefore, a few years in Boston charter schools generate test score gains equivalent to a large reduction in the black–white achievement gap.
Our results also demonstrate that policymakers selected more effective schools for expansion: proven providers produced larger effects (0.3 standard deviations in maths) than other charter schools (0.2 standard deviations in math) before the reform (see Figure 2).
Estimates for the post-reform period reveal that Boston’s charter sector maintained effectiveness while doubling in size. Specifically, proven providers and other existing charters generated test score gains similar to those prior to the reform: over 0.3 standard deviations and 0.2 standard deviations, respectively, in maths.
At the same time, new expansion charters generated achievement gains comparable to those of their parent schools, boosting maths and English scores by 0.3 standard deviations and 0.2 standard deviations. Moreover, expansion charters produce these large impacts while enrolling a mix of students more representative of Boston as a whole than students at other charters. On average, students applying for expansion charters were more economically disadvantaged and lower-achieving at the time of application than those who applied to other charter schools. These results imply an increase in overall effectiveness of the charter sector despite the substantial enrolment growth and changing demographics after the 2010 reform.
Figure 2 Effect of one year of attendance at a charter school (maths)
Why are proven providers successful at replicating?
We explored several candidate explanations for the Boston charter sector’s unusually successful scale-up. While the characteristics of the charter student population evolved during expansion, these demographic changes were not large enough to impact the effectiveness of new campuses. Charter lottery losers attend a similar mix of traditional public schools before and after the reform, suggesting that the large achievement gains experienced by lottery winners are not due to a change in the quality of fallback options. Instead, the success of new charters appears to be due to successful transmission of a highly standardised, replicable school model from parent campuses to expansions. Charters share educational practices throughout their networks and limit variation in environments and practices across classrooms. This suggests that organisational practices in Boston’s charter sector may be an important component of its success at scale.
Several states, as well as New York City, have reached their overall cap on charter schools or have limited remaining growth, including Connecticut, Maine, Massachusetts, and Rhode Island, setting these states up for policy decisions around further growth (Ziebarth and Palmer 2018). The federal government also supports charter school replication, with several charter school networks receiving very large grants to replicate their models, including 2019 awards of over $100 million to IDEA Public Schools and over $85 million to KIPP.
Our findings point to the potential of scaling up successful school models as one strategy to address persistent achievement gaps in the United States. The efficacy of this strategy requires schools selected for expansion to sustain their success in new locations and for new student populations. Our study documents that Boston’s No Excuses charter schools replicated their effectiveness during a period of rapid expansion and demographic change.
Abdulkadiroglu, A, J Angrist, S Dynarski, T J Kane, and P Pathak (2011), “Accountability and flexibility in public schools: Evidence from Boston’s charters and pilots”, The Quarterly Journal of Economics 126(2): 699–748.
Abdulkadiroglu, A, J D Angrist, Y Narita, and P A Pathak (2017), “Research Design Meets Market Design: Using Centralized Assignment for Impact Evaluation”, Econometrica 85 (5): 1373–1432.
Angrist, J, P Pathak, and C Walters (2013), “Explaining Charter School Effectiveness”, American Economic Journal: Applied Economics 5(4):1–27.
Angrist, J D, S R Cohodes, S M Dynarski, P A Pathak, and C R Walters (2016), “Stand and Deliver: Effects of Boston’s Charter High Schools on College Preparation, Entry, and Choice”, Journal of Labor Economics 34(2): 275–318.
Angrist, J D, S M Dynarski, T J Kane, P A Pathak, and C R Walters (2012), “Who Benefits from KIPP?” Journal of Policy Analysis and Management 31(4): 837–860.
Borman, G D, R E Slavin, A C K. Cheung, A M Chamberlain, N A Madden, and B Chambers (2007), “Final Reading Outcomes of the National Randomized Field Trial of Success for All”, American Educational Research Journal 44(3): 701–731.
Cohodes, S, R, E M Setren, and C R Walters (2019), “Can Successful Schools Replicate? Scaling Up Boston’s Charter Sector”, NBER Working Paper 25796.
Dobbie, W and R Fryer (2013. “Getting Beneath the Veil of Effective Schools: Evidence from New York City”, American Economic Journal: Applied Economics 5(4): 28–60.
Dobbie, W and R Fryer (2015), “The Medium-Term Impacts of High-Achieving Charter Schools”, Journal of Political Economy 123(5): 985–1037.
Heckman, J, S H Moon, R Pinto, P A Savelyev, and A Yavitz (2010), “Analyzing Social Experiments as Implemented: A Reexamination of the Evidence from the High/ Scope Perry Preschool Program”, Quantitative Economics 1(1): 1–46.
Heckman, J, R Pinto, and P Savelyev (2013), “Understanding the Mechanisms Through Which an Influential Early Childhood Program Boosted Adult Outcomes”, American Economic Review 103(6): 2052–2086.
Jepsen, C and S Rivkin (2009), “Class Size Reduction and Student Achievement: The Potential Tradeoff between Teacher Quality and Class Size”, The Journal of Human Resources 44(1): 223–250.
Krueger, A (1999), “Experimental Estimates of Education Production Functions”, Quarterly Journal of Economics 114(2): 497–532.
Puma, M, S Bell, and C Heid (2012), Third Grade Follow-up to the Head Start Impact Study, OPRE Report 2012-45. U.S. Department of Health and Human Services.
Quint, J, P Zhu, R Balu, S Rappaport, and M DeLaurentis (2015), Scaling Up the Success for All Model of School Reform, MDRC Report.
Tuttle, C C, B Gill, P Gleason, V Knechtel, I Nichols-Barrer, and A Resch (2013), KIPP Middle Schools: Impact on Achievement and Other Outcomes, Mathematica Policy Research Report 06441.910.
Walters, C R (2018), “The Demand for Effective Charter Schools”, Journal of Political Economy 126(6).
Ziebarth, T and L B Palmer (2018), Measuring up to the model: A ranking of state public charter school laws, Technical Report, National Alliance for Public Charter Schools.