1 NBER WORKING PAPER SERIES WHAT CAN WE LEARN FROM CHARTER SCHOOL LOTTERIES? Julia Chabrier Sarah Cohodes Philip Oreopoulos Working Paper 22390 http://www.nber.org/papers/w22390 NATIONAL BUREAU OF ECONOMIC RESEARCH 1050 Massachusetts Avenue Cambridge, MA 02138 July 2016 For sharing the data from and answering question about their studies, we thank Ira Nichols- Barrer, Brian Gill, Phil Gleason, Josh Furgeson, and Christina Clark Tuttle of Mathematica Policy Research; Danielle Eisenberg of the KIPP Foundation; Josh Angrist, Parag Pathak, and Chris Walters; and Will Dobbie and Roland Fryer. We also thank Carrie Conaway of the Massachusetts Department of Elementary and Secondary Education and the National Center for Education Statistics for additional data access, and Josh Angrist, Will Dobbie, and Mike Gilane, as well as seminar participants at Teachers College Columbia University and IFN-Stockholm for of the authors and do not necessarily helpful comments. The views expressed herein are those reflect the views of the National Bureau of Economic Research. At least one co-author has disclosed a financial relationship of potential relevance for this research. Further information is available online at http://www.nber.org/papers/w22390.ack NBER working papers are circulated for discussion and comment purposes. They have not been peer-reviewed or been subject to the review by the NBER Board of Directors that accompanies NBER publications. official © 2016 by Julia Chabrier, Sarah Cohodes, and Philip Oreopoulos. All rights reserved. Short sections of text, not to exceed two paragraphs, may be quoted without explicit permission provided that full credit, including © notice, is given to the source.
2 What Can We Learn from Charter School Lotteries? Julia Chabrier, Sarah Cohodes, and Philip Oreopoulos NBER Working Paper No. 22390 July 2016 JEL No. I20,I24,I28,J18 ABSTRACT We take a closer look at what we can learn about charter schools by pooling data from lottery- based impact estimates of the effect of charter school attendance at 113 schools. On average, each year enrolled at one of these schools increases math scores by 0.08 standard deviations and scores by 0.04 standard deviations. There is wide variation in impact English/language arts this variation, we link these effects to school practices, inputs, estimates. To glean what drives and characteristics of fallback schools. In line with the earlier literature, we find that schools that adopt an intensive “No Excuses” attitude towards students are correlated with large gains in class size playing no role in explaining charter academic performance, with traditional inputs like school effects. However, we highlight that “No Excuses” schools are also located among the most disadvantaged neighborhoods in the country. After accounting for performance levels at fallback schools, the relationship between the remaining variation in school performance and the entire “No Excuses” package of practices weakens. “No Excuses” schools are effective at raising performance in neighborhoods with very poor performing schools, but the available data have less to say on whether the “No Excuses” approach could help in nonurban settings or whether other practices would similarly raise achievement in areas with low-performing schools. We find intensive tutoring is the only “No Excuses” characteristic that remains significant (even for that schools) once the performance levels of fallback schools are taken into account. nonurban Julia Chabrier Philip Oreopoulos J- PAL Department of Economics Massachusetts Institute of Technology University of Toronto 150 St. George Street 30 Wadsworth Street, E53-320 Cambridge, MA 02142 Toronto, ON M5S 3G7 CANADA [email protected] and NBER [email protected] [email protected] Sarah Cohodes 525 West 120th Street, Box 11 Teachers College Columbia University New York, NY 10027 [email protected]
3 I. Introduction Publicly funded charter schools, which set their own curriculum, financial management, and staffing, have been spreading rapidly throughout the United States. Originally designed as testing grounds for trying out new and innovative approaches for improving academic achievement, charter schools have since captured a growing share of enrollment and become competitors to traditional public schools for students, funding, and facilities . Beginning with the first charter in 1993, enrollment has increased from a few dozen students to about 3 million 1 That ’s more than 5 percent of all public elementary and secondary across 7,000 schools. students in the country. Some lar ge urban districts, like Indianapolis, Philadelphia, Detroit, and Washington, D.C. have shares of students at charters greater than 30 percent, and in the 2014- 2015 school year, the New Orleans Recovery School District became the first U.S. school district to be comprised entirely of charter schools (National Alliance for Public Charter Schools 2015a ; Abdulkadiroğlu et al. 2014, 1). Anyone residing in a All charter schools are free to students. given geography (depending on state law, ) is eligible to attend. the district, region, or state where the charter school is located Increasingly, however, more students apply than there are spots available. When faced with too 2 Some lotteries are held in large many applicants , charters must admit students by lottery. auditoriums in front of anxious parents and children, leading to heartbreaking scenes of 1 National Center for Education Statistics. 2015. “The Condition of Education 2015: Charter School Enrollment.” (accessed January 11, 2016). U.S. Department of Education. http://nces.ed.gov/programs/coe/indicator_cgb.asp 2 The authors of a national evaluation of charter school impacts (Gleason et al., 2010) estimated that about 26% of 7 charter middle schools were likely to be oversubscribed (Clark Tuttle, Gleason, and Clark, 2012) in the 2006-0 school year. 2
4 disappointment like those in the documentary, “Waiting for Superman,” in which administrators pick only a minority of names from a box. To avoid public letdown, some schools discourage although many still do). parents from showing up to watch the live admissions process ( While most charter schools do not face oversubscribed demand, some in disadvantaged 3 urban neighborhoods have a dmission rates 20 percent. To remove the incentive for of less than to apply separately to multiple schools and to maximize the number of students who get parents 4 Results from the into at least one school, a few districts now centralize the lottery process. Distric t of Columbia's most recent common lottery provide an indicator of oversubscribed demand: of the 17,000 students that entered the unified lottery, 71 percent of students received an offer from at least one school on their lists, but only 60 percent received an offer from one of 5 Space is limited in high -demand areas due to legislation that caps the their top three choices. total number of schools or because the authorization process, limited funding , or other regulations make the process of opening new charter schools difficult. Chart er school authorizers (typically state or local education agencies, higher education institutions, or independent boards created for the purpose of charter authorization ) choose which charters to grant and which to renew. 3 For some examples of oversubscribed demand at popular charter schools see: Wiltenburg, Mary. 2015. WYPR. http://news.wypr.org/post/uncertain - “Uncertain Future for Thousands in Charter School Lottery” February 13. -school future -thousands -charter New York Daily -lottery#stream/0 . Chapman, Ben and Stephen Rex Brown. 2014. News. ” Success Academy Charter Schools Admissions Rate is Only 20%, Lower Than NYU.” April 4. . Pisano, Chris. 2015. “Hundreds Turn Out for Holyoke Charter School Enrollment http://nydn.us/1fJapci WGGB. http://www.masscharterschools.org/media/news/hundreds -turn -out -holyoke -charter -school - Lottery.” March 4. -lottery . Rahman, Fauzeya. 2015. Houston Chronicle . “Word of Mouth Major Factor for Parents Charting enrollment Charter School Course.” November 7. http://www.houstonchronicle.com/news/houston- texas/houston/article/Word - of-mout h- major -factor -for -parents -charting -6617645.php . 4 Many centralized school assignment mechanisms draw upon Alan Roth’s work on market design, and most notably his design of the National Resident Matching Program. For example, both New Orleans and Denver u se matching algorithms to assign students to schools. 5 Brown, Emma. 2014. “Joy and Anguish for Parents as D.C. Releases School Lottery Results.” Washington Post, https://www.washingtonpost.com/local/education/joy -and -anguish -for -parents -as-dc-releases -school - March 31. c73 9f29ccb08_story.html lottery -results/2014/03/31/a04aea1e -b8ff -11e3- 9a05- 3
5 The required random assignment from oversubscribed demand in the charter schools’ admissions process generates worry and letdown for those who lose, but it also generates opportunity for exceptional research. Over the past decade, a relatively small number of studies have been able to gat her data from lottery results and match them to administrative records to allow for rigorous evaluation of the impact of charter school attendance on student outcomes. Most of these studies look at 3 to 30 schools at a time and estimate the average impact of charter level standardized test scores. They find a wide dispersion of effect school attendance on state- sizes: some schools are estimated to increase performance (especially math scores) by more than half a standard deviation per year of attendance, while others are estimated to have substantial negative effects. Whether students gain from winning any charter school lottery is by no means certain. A general conclusion seems to be emerging, however, that what distinguishes schools with the largest positive effect sizes is that they adopt an intensive “ No Excuses ” approach with strict and clear disciplinary policies, mandated intensive tutoring, longer instruction times, 6 There has even been frequent teacher feedback, and a relentless effort to help all students . evidence that reinventing traditional public schools in urban settings with these characteristics leads to similarly large performance improvement s (Fryer 2014). 6 This pattern seems to have emerged from a number of studies looking at the association between charter school effectiveness and school characteristics. In a study of New York City public schools, Dobbie and Fryer (2013) identify an index of five policies driven instruction, high -dosage tutoring, —frequent teacher feedback, data- increased instructional time, and relentless focus on academic achievement —that explains roughly half of their observed variation in school effectiveness. In a study of Massachusetts charter schools, Angrist, Pathak, and Walters (2013) identify five variables that are highly correlated with so -called “No Excuses” charter schools and also highly correlated with charter effectiveness: strict discipline, cold -calling, Teach for America alumni, and videotaping of lessons for teacher feedback. In a national study of Charter Management Organization s, Furgeson et al. (2012) find that comprehensive behavioral policies, intensive teacher coaching, and the use of Teach for America and teaching fellow teachers are positively associated with student impacts. Hoxby, Muraka, and Kang (2009) find that a long school year, more minutes of English instruction, a discipline system with small rewards/penalties, a teacher pay scale that incorporates merit, and a mission statement focused on academic performance all are associated with larger charter school gains. 4
6 In this paper, we take a closer look at what can be learned from charter school lott eries by pooling data from 113 schools. We refer to previous studies to summarize the state of the literature so far, but we also take a more school- specific approach in analyzing what charter schools actually do. We use both qualitative and quantitative approaches for trying to disentangle why some schools consistently out- or under- perform. Isolating specific effective school strategies is challenging because no one school can be considered a simple derivative of another. A more ethnographic look at effective schools reveals multiple potential factors for explaining why some charters are more successful than others. In line with the earlier literature, we also find that schools that have adopted a “No Excuses” approach are correlated with large gains in academic performance. However, “No Excuses” schools are concentrated in urban neighborhoods with very poor performing schools and scarce in nonurban areas. Thus, one reason for the gains achieved by “No Excuses” urban schools is that the fallback traditional public schools for urban students have such poor performance. Neal (2009) makes a similar point that private school returns are largest for urban minority students. Once the performance levels of fallback schools are taken into account, and we look at the individual components of a “No Exc uses” approach using charter -school- level data, w e find tha t intensive tutoring is the only characteristic that remain s significant in improving student performance. Tutoring offered at charter schools is typically more intense than tutoring offered at traditional public schools. Charter schools often use paid tutors, add tutoring on top of already long school days, and require all students to participate. This finding about the importance of tutoring is in line with other recent evidence pointing to dramatic gains ntensive tutoring on its own, suggesting a good place to start for effective and practical from i reform at traditional public schools. 5
7 7 Our intent is not to enter the debate on whether charter public schools should exist at all. Rather, in keeping with the original intent of charter schools to serve as laboratories for best practices, we focus on what researchers and policymakers can learn from charter . In lotteries Section 2, we present a brief history of the charter school movement, with a particular focus on led to new avenues for research. We als o discuss issues around external how admissions lotteries validity in determining the extent to which conclusions from lottery studies can carry over to In Section 3, we present an analysis of estimated charter school traditional public school settings. impacts from pooling multip le datasets. We describe the distribution of these estimates, how they vary by student characteristics, their degree of precision, their relation to counterfactual school characteristics, and their implied cumulative impacts over time. In Section 4, we explore what school factors might explain the wide distribution using a combination of methodologies. S ection 5 concludes , noting the exciting possibilities for further research with more lotteries and more support behind matching them to administrative data. II. Oversubscribed Charter Schools and Their Lotteries A. A Brief History of Charter Schools in the United States An interesting chain of events led to the birth of charter schools and the need for admissions lotteries (Junge 2014). In the early 1980s, the Secretary of Education created the National Commission on Excellence in Education and directed it to write a report about “the 7 For a good overview of the research on charter schools that more directly addresses this debate, see: “Charter Schools in Perspective: A Guide to Research” (2015). The Spencer Foundation and Public Agenda. www.in - perspective.org. 6
8 widespread public perception that something is seriously remiss in our educational system” (National Commission on Exc A Nation at Risk, was ellence in Education 1983). The report, released in 1983 and concluded that declines in educational performance in the United States were due to a diluted curriculum with more students taking intermediate or basic -level classes, a decline in expectations in terms of what skills graduates should possess, less time studying, and “too many teachers being drawn from the bottom quarter of graduating high school and college students” (National Commission on Excellence in Education 1983). Alb ert Shanker, president of the American Federation of Teachers, acknowledged many points in the report and a need for reforms. In a remarkable 1988 speech to the National Press Club, he also lamented a constant back and forth between making curricula too easy or too tough. He asserted that raising the bar in terms of harder questions, more homework, or a faster pace was only helpful to the top 20 percent of students and that neither raising standards nor lowering them helped the bottom 80 percent (“students who are not able to sit still and listen for that many hours, and are not able to read that long”). Shanker argued that teachers and principals need to be allowed to innovate and try alternative strategies for improving performance, recognizing that not al l students learn the same. He proposed that: The school district and the teacher union develop a procedure that would encourage any group of six or more teachers to submit a proposal to create a new school. Do not think of a school as a building, and you can see how it works. Consider six or seven or twelve teachers in a school who say, 'We've got an idea. We've got a way of doing something very different. We've got a way of reaching the kids that are now not being reached by what the school is doing.' That group of teachers could set up a school within that school which ultimately, if the procedure works and it's accepted, would be a totally autonomous school within that district. (Shanker 1988) Shortly following this speech, the Citizens League of Minnesota, which had supported school choice reforms in the past, organized a seminar around K-12 education and invited Shanker to present. Several attendees, including Ember Reichgott, then a Minnesota state 7
9 senator, subsequently began to work on developing specifics to Shanker's proposal. A few weeks later, the League produced a report, “Chartered Schools = Choices for Educators + Quality for All Students,” which included the idea of teachers applying for ‘charters’ to set up small schools—not within existing traditional public schools as Shanker envisioned, but separate (Citizens League of Minnesota 1998). For logistical and practical reasons, it was easier for the report’s authors to imagine experimen tal schools operating apart from the existing regulatory systems. The new charter schools would still be publicly funded, without tuition or admissions requirements, but they would be allowed to operate with autonomy from some of the rules and regulations governing traditional public schools. Senator Reichgott tried two times unsuccessfully to introduce a bill allowing for charter schools. Critics argued that charters would drain public resources away from traditional schools and raised concerns that higher-ability students, white students, and students without special needs would disproportionately shift out of the traditional public school system (Frankenberg and Lee 2003; Fiske and Ladd 2000; Cobb and Glass 1999). The sentiment among Minnesota's Governor and many state politicians, however, was that the idea was at least worth trying, and an amended bill capping the program at eight schools and making districts the only sponsors narrowly passed in 1991 (Kolderie 1998). Since then, the number of charter schools has grown rapidly, to about 7,000 schools, and 8 continues to grow. All but eight states now have legislation authorizing charter schools. Support from legislators for the expansion of charter schools appears to be more a response to increasing 8 National Center for Education Statistics. 2015. “The Condition of Education 2015: Charter School Enrollment.” (accessed January 11, 2016). U.S. Department of Education. http://nces.ed.gov/programs/coe/indicator_cgb.asp 8
10 paren tal demand for alternatives to traditional public schools , rather than a commitment to the 9 idea that charters can serve as laboratories to improve public education. Charter school authorizers, designated by state law, choose which charters to grant. They also provide ongoing oversight of charter schools and make renewal decisions at the end of the charter contract term, typically every five years. Decisions to renew are often based on relative test score measures or financial health (including enrollment) . Schools do close—sometimes suddenly—compelling students to find another charter school option or revert back to their local 10 In Texas and North Carolina, respectively, Baude at al. (2014) and traditional public school. Ladd et al. (2015) conclude that charters that close are disproportionately less effective, while open improve in value- added over time. those that remain B. The Methodology of Lottery Studies In broad terms, the methodology of studies is to compare those who charter school lottery complexities arise . One challenge won a charter school lottery with those who did not. Of course, researchers must take into account that not all winners attend charter schools and not all is that 9 There have been some attempts to learn from what has worked for charter schools and what has not . At the national level, the U.S. Department of Education’s Charter School Exemplary Collaboration grants supported s and traditional public schools and school districts. The Bill collaboration and lesson sharing between charter school & Melinda Gates Foundation has awarded grants to 16 cities that have signed District -Charter Collaboration Compacts detailing plans for collaboration between charter and traditional public schools. Many state laws authorizing charter schools include the sharing of best practices as one of their aims, but generally this is encouraged rather than mandated. 10 According to the National Alliance for Public Charter Schools, 216 charter schools, or about 3 percent of the total charter schools, closed in 2014 (National Alliance for Public Charter Schools 2015b , 2). For some examples of charter school closures see: Bush, Bill. 2015. “Closure of North Side Charter School Strands About 300 Students.” The Columbus Dispatch, August 26. . Jacobsen, http://www.dispatch.com/content/stories/local/2015/08/26/FCI_Academy_charter_school_closed.html 2015. “Two Charter Schools Closing Within Days of Starting Dates.” Deseret News , August 20. Morgan. http://www.deseretnews.com/article/865634957/Alianza -Academy -charter -school -closing -only- days -after -school - begins.html?pg=all . Vevea, Becky. 2015. “Four Charter Schools Push Back Against Sudden Closings.” WBEZ , . -113767 November 13. http://www.wbez.org/news/four -charter -schools -push- back -against -sudden- closings 9
11 losers end up at traditional public schools. In Boston, for example, Abdulkadiroğlu et al. (2011) find one-fifth of lottery winners never attend a charter school and some lottery losers eventually or gaining sibling end up in one (by moving off a waitlist, entering a future admissions lottery, preference when a sibling wins the lottery ). Most of the studies of how charter schools affect test in two stages: first, estimating how winning a lottery scores therefore measure the effects predicts increased attendance at charter schools; and second , estimating how this predicted 11 Because the effects of attending a charter school are increased attendance affects achievement. identified based on differences between initial lottery winners and losers, selection in who enrolls or persists in charter schools does not bias the causal estimates. While this approach addresses internal validity, external validity concerns may arise if the potential impact of charte rs is weaker for those who do not apply or do not adhere to their lottery assignment , a topic we return to below. usually Fixed effects are added to the estimating equation for each group of students that applied to the same set of school lotteries to ensure that winner -loser comparisons are between In many those who had an equal chance of being selected (to the set of schools they applied). cases, test score data from different grade levels are stacked together, implicitly assuming that attendance effects increase equally for each year spent in a charter school versus not. Pooling data from multiple test results while clustering standard error estimates by grouping at the student level may also help increase precision. 11 In other words, winning a charter school lottery is used as an instrumental variable for charter school attendance. Conceptually, researchers estimate the “intention -to-treat” (ITT) effect of winning a lottery for a charter school seat on the outcome of interest (e.g. student test scores) by calculating the difference in average outcomes between lottery winners and los ers. The “local average treatment effect” (LATE) of charter school attendance on the outcome of interest is calculated by scaling up the ITT estimate by the difference in charter school attendance between lottery winners and lottery losers (this is sometim es called the treatment on the treated (TOT) when no or few lottery losers . gain entry to charter schools) 10
12 C. An Overview of the Studies 12 When charter schools are oversubscribed, they must admit students using lotteries, although some states allow charters to give preference to siblings (National Alliance for Charter Public Schools 2015c arate applications for each charter ). In most cases, families must submit sep school, though this is changing with an increasing number of districts using centralized applications that allow students to rank schools, which maximiz e the chances that each student gets into at least one school. 13 The Table 1 lists the set of charter schools with lotteries that have been studied to date. studies described in Table 1 do not include all charter schools that have held lotteries. To do research on outcomes of winners and losers in a charter school lottery; records must be in suitable condition; enough time must elapse to observe student outcomes of interest; researchers must obtain permission from schools to work with their lottery records; and, because of federal privacy law, matching lottery records to student test scores often requires either individual consent from study participants or collaboration with state or school district administrators who can conduct or supervise the match. In cases of multiple studies working with the same data, we focus here in the on the most recent published academic study or report. In some cases discussion that follows to be comparable , we will rescale the estimates of charter school effects 14 across studies. 12 The first charter school law anticipated the possibility of oversubscribed demand. In 1991, Minnesota legislation stated that if “the number of applications exceeds the capacity of a program, class, grade level, or building...pupils shall be accepted by lot” (Minnesota Statute § 120.064). 13 Some of these studies are primarily observational; for the purpose of this paper, we focus on the lotter y- based findings and do not report observational findings. 14 More specifically, in cases where a study reported only the “intention -to-treat” effect (the outcome effect from winning a lottery) and no first stage estimate (the effect of winning a lottery on attendance) , we noted this in Table -to-treat are reported but a local average treatment effect is not, we divide by the 1. If the first stage and intention 11
13 Hoxby and Rockoff (2004) collected ” “No Excuses admissions lottery data from three style Chicago International Charter Schools (CICS), which deliberately locate in disadvantaged urban communitie s to target low -income families. Hoxby and Rockoff had admissions lottery to Chicago Public School administrative data on test score outcomes. They find data matched small positive changes, not statistically significant at standard levels. Around the same time as Hoxby and Rockoff’s study, another team of economists began collecting charter school lottery data from Massachusetts and , with support from s tate officials , obtained access to administrative public school data for matching. Abdulkadiroğlu et al. (2011) focus on students in Boston who applied to at least one of five charter middle schools or one of three charter high schools where high demand causes the schools to be oversubscribed. They find very large average effects : charter school attendance increases state- level English /language arts and math test scores by 0.2 and 0.35 standard deviations per year, respectively. Given that that and white students in Massachusetts is about 0.7 to 0.8 the achievement gap between black , these estimates standard deviations attendance for charter school suggest that three years of black black students would eliminate the . Angrist et al. (2013) update -white performance gap setts, along with this analysis to include urban and nonurban schools across Massachu additional They continue to find positive average test score effects , but these effects years of test score data. best estimate of the first sta ge. In cases where a study reported only cumulative estimates, we divided the final year estimate by the number of years observed to obtain a per year estimate. When we convert estimates to per year or second stage estimates, we also divide the standard errors by the same factors we divide the coefficients. In the cas es where we are converting intention -to-treat estimates to second stage estimates, this will not correct the standard errors as a typical procedure would in a statistical software program. Thus our standard two -stage least squares errors are likely slightl y too small for a subset of the charter school impact estimates that are based on intention -to- treat estimates ( see Table 1 for details ). We follow these conventions in our data analysis as well. Means and , both here and deviations are weighted by the inverse of the standard error of the relevant point estimates standard . throughout our study 12
14 appear in urban schools only and with wide variance across schools —a finding we revisit later in this paper. ducation also facilitated matching charter school The New York City Department of E lottery data with standardized test scores in English/language arts Dobbie and Fryer and math. (2013) collected data from 19 elementary and 10 middle schools with oversubscribed demand. They find large average effects from each year of charter school attendance, especially for math , though again with large variance across schools. In an earlier lottery -based study of N ew scores Hoxby et al. (2009) also found large and significant results for middle York City charter schools, schools and report even larger effects for charter high schools. Studies that use survey data for a national sample of charter schools tend to find positive but not statistically significant overall impacts. Both Gleason et al (2010) and Furgeson et al. (2012) contacted charter schools asking for permission to survey lottery applicants and obtain consent prior to randomization, and the Furgeson et al. group also collected retrospective data to match directly Among the 77 charter middle schools that agreed to with administrative data. participate in Gleason et al. (2010), only 32 ended up with a sufficiently large enough waiting list to use in their study. On average, lottery winners performed no better and no worse in math and reading scores than lottery losers two years after students applied. Furgeson et al. (2012) identified 16 of 109 charter schools run by Charter Management Organizations (CMOs ) with adequate records and also find insignificant overall test score effects from winning the lottery . Estimates from survey data, however, are generally more imprecise than those using administrative data . studies estimate charter impacts for specific schools Seven additional lottery -based or organizations. Three of these studies examine the Knowledge Is Power Program (KIPP) charter 13
15 schools. KIPP is the largest network of charter schools in the country and often described as the source of the “No Excuses” movement ( as reported in Rotherham 2011). In KIPP schools, principals and teachers have high behavioral and academic expectations for all students. Further, parents, students, and teachers sign a ‘learning pledge’ and follow a strict disc iplinary code. School hours are extended typically to between 7:30AM and 5:00PM and include occasional Saturdays and summer weeks, and tutoring is also offered during these time s. In the 2014-2015 school year, KIPP’s network included 162 schools serving 58,495 students in prekindergarten through grade 12 (Clark Tuttle et al. 2015). All three KIPP lottery studies listed in Table 1 find significant achi evement effects from attendance (Angrist et al. 2012; Clark Tuttle et al. 2013; Clark Tuttle et al. 2015). In addition to the test score results, Clark Tuttle et al. (2013) also find that KIPP attendance increases the amount of homework per night by about 45 minutes and increases school satisfaction but does not affect effort or engagement. contain The Promise Academy charter schools in the Harlem Children's Zone (HCZ) many similar . Dobbie and Fryer (2011) estimate that attendance at the “No Excuses” elements Promise Academy raises test scores by about 0.20 standard deviations per year, although e ffects on Engli sh/language were not significant. The study also finds that attendance at the Promise Academy reduces absenteeism. Two other charter No Excuses” model have been evaluated. schools aligned with the “ The Unlocking Potential (UP) Network focuses on in-district school turnaround for chronically underperforming schools. In 2011, UP Academy Charter School of Boston replaced a failing traditional public school in Boston; within a year , the school was required to hold a lottery to address oversubscribed de mand (as reported in Nix 2015). Abdulkadiroğlu et al. (2014) find UP attendance effects of 0.1 2 standard deviations per year for English/language arts scores and 0.27 14
16 standard deviations for math. SEED schools are “No Excuses” boarding schools in Baltimore and Washington, D.C. for students from disadvantaged backgrounds in grades 6 through 12. At the Washington, D.C. school, Curto and Fryer (2014) estimate increases in math scores of 0.23 standard deviations and in reading scores of 0.21 standard deviations per year of attendance. Many of the estimated effects in Table 1 are impressive. Attendance at some charter schools leads to test score gains of more than half a standard deviation after two years of attendance. Most educational interventions such as class size reductions, teacher or student quarter of this incentives, more resources, or extended time, generate gains that are less than one- from attending “No Excuses amount (Fryer, 2016). However, while the large impacts ” schools , some of the other charters like KIPP, UP Academy, and the Promise Academy are encouraging generate no effect or even negative effects. Overall , the per- year mean effect of attending a charter school in our sample of 113 schools is 0.080 standard deviations in math and 0.046 standard deviations in English/language arts. Our real interest from these papers, therefore, is not whether charter schools are effective on average , but rather what makes an effective charter school. To investigate this, we dig a little deeper. III. School- Specific Effects A. Wide variance in outcomes -specific charter effects and their corresponding standard We collected data for school errors and sample sizes from the lottery studies in Table 1, with the exception of schools used in 15
17 15 Hoxby and Rockoff (2004) and Hastings et al. (2012). Figure 1A plots a histogram of the Figure 1B plots the same, standardized math score effects from a year of attending these schools. The range in estimates is wide, with outliers ranging from scores. - but for English/language arts 0.57 to 1.16 for math and -0.78 to 1.06 f or English/language arts . The estimates are slightly -year math test score effect is 0.080 and its standard deviation is skewed to the left. The mean per 16 The mean English/language arts effect is 0.046 with a standard deviation of 0.21. 0.23. Some of this variation is likely due to imprecision. Not surprisingly, larger effects are related to larger standard errors. Figures 2A and 2B plot the math and English/language arts effect sizes against their corresponding standard errors, respectively. The largest math effect of 1.16 standard deviations from a year of charter attendance at a New York City school has an associated standard error of 0.66. On the other end, the lowest English/language arts effect of - 0.78 from a school in Massachusetts has a standard error of 0.75. Estimates from the two national studies using survey data from Gleason et al. (2010) and Furgeson (2012) generally have higher standard errors than those from the other studies. To get a better sense as to the range of charter school effects, Figure 3 plots the math and English/language arts score impacts for cases with standard errors less than or equal to 0.1. -0.27 to 0.57, with a mean Among these more precise results, effects on math scores range from effects are of 0.092 and standard deviation of 0.18. The separate math and English/language arts 15 Table 2 3 school estimates and their availability of corresponding school characteristic lists the sources of these 11 data. We thank the authors of these studies for their willingness to share data and results . The reasons for excluded studies are as follows. Hoxby and Rockoff (2004) do not report results on a scale we can convert to standard deviations. Hastings et al. (2012) do not report estimates that we can convert to per- year estimates. We also exclude studies where the schools are included in a more comprehensive or a more recent study. The KIPP Lynn and Promise Academy studies are not included as single studies since those schools are included in studies of the same regions with larger school samples. Hoxby , Muraka , and Kang (2009) estimate effects on New York City schools ; we use the more recent estimates from Dobbie and Fryer (2013). The same reasoning applies to estim ates of Boston schools – these are included within the Massachusetts study. 16 Means and standard deviations are weighted by the inverse of the standard error of the relevant point estimates , both here and throughout our study . 16
18 strongly correlated , with m being generally larger than English/language arts in ath effects . The slope from regressing English/language arts absolute terms on math scores is 0.64. B. Larger Effects in Poor Performing Urban Neighborhoods students who would have otherwi A charter sch ool which attracts se attended a particularly poor-performing traditional public school would appear more effective than an identical charter school that draws students who would have otherwise attended a better hool. Estimates of charter school impacts should therefore be performing sc relative to interpreted 17 the experience of students who lose the lottery at that school. 2013) find stark differences in the gain that can be As mentioned earlier, Angrist et al. ( the school is located in an urban or nonurban attributed to a charter school according to whether 18 The large positive effects from the studies are concentrated among the Massachusetts setting. urban charter schools, while nonurban charters are generally ineffective and may even reduce achievement for some. We show this pattern in Figure 4, Panel I , which plots the Massachusetts estimates by average achievement levels at the fallback schools for lottery losers. The fallback chievement school a level is measured as the average test score at the noncharter school that 19 lottery losers attend the following year, weighted by the number of students that attend . 17 Hastings et al. (2012) exam ine whether winning a school choice lottery impacts students’ academic achievement even before they enroll in their chosen schools by raising their intrinsic motivation. They find that charter and magnet school lottery winners in an anonymous urban school district had truancy rates that were 7 percent lower than lottery losers in the period after the lottery was held but before winners enrolled in their new schools. 18 Angrist et al. (2013) define urban schools as those located in areas in which the district superintendent participates in the Massachusetts Urban Superintendents Network. This includes Boston, as well as smaller districts such as Cambridge, Holyoke, Lawre nce, and Worcester. In Massachusetts, urban charter schools are almost uniformly located in areas with high poverty rates and high minority enrollment. We follow the definitions of variables as defined in their original studies. See Appendix Table 2 for a full list of variable definitions across studies. 19 Results are very similar, though less precise, when we use control complier test scores rather than the average school outcome of lottery losers. Additionally, to address the concern that these findings reflect a mechanical correlation due to the presence of lottery losers’ outcome s in the fallback school scores , we recalculate the fallback school scores using the prior year’s scores (which the lottery losers do not contribute to). The findings are essentially identical, likely due to the relatively small proportion of lottery losers in any given school. 17
19 Students at urban fallback schools score well below average i n test scores while the students at The black circles from nonurban schools generally score above average. indicate effects s, which are almost all uniformly positive. Larger attendance at urban charter school circles indicate more precise estimates ( that is, smaller standard errors). The average urban charter school math effect is 0.25 (se=0.044). The grey, open circles that indicate nonurban effects are mostly close to zero or even negative. The average math impact at nonurban charters is -0.07 (se=0.092). Regressing performance gains in math at the charter school on the average test scores at 2 is more negative relationship (-0.629, se=0. 086). The R the fallback schools, we get a strong than half (0.513). An indicator for whether a school is in an urban area has no additional 20 Panel B of Figure 4 shows a qualitatively similar pattern, although less explanatory power. extreme, for the gains by charters in English/language arts scores compared to the fallback schools. Clearly , the most impressive charter school effects are found where fallback schools performance. have the least impressive academic The national charter school study by Gleason et al. (2010) also displays a noticeable ionship between charter school effects and conditions at fallback schools. In this negative relat e use their dummy variable indicating “Large City ” to define urban versus nonurban case, w areas. For the performance level of fallback schools, we use the average proficiency rate of the traditional public schools attended by lottery applicants in the year and grade level after losing a charter lottery ( not on the same scale as the Massachusetts variable). Figure 4, Panel C which is shows that the slope from regressing charter impacts on performance levels at fallback schools is also negative in this data (-0.227, se=.073). Again, the slope remains essentially the same when 20 2 Specifically, the slope and R remain about the same when adding a dummy variable for the school being in an urban city ( -0.658, se=0.375). 18
20 adding the urban dummy (-0.191, s.e.=0.088). The slope for English/language arts test score school performance is also negative, but less steep impacts regressed on fallback and insignificant. The importance of the fallback school in affecting the gains from enrolling in a charter school can also been seen in Figure 5, Panel I which traces the accumulation of charter school effects over time for applicants to urban and nonurban middle schools in Massachusetts . We 21 for urban charter attendees (solid, on the state standardized exam calculate percent proficient dark line) and noncharter attendees (solid, lighter line) at each grade level, with similar calculations for the nonurban charter applicants (dashed lines), using the methods from Abadie (2002, 2003) as described in Angrist et Al. (2016). In both subjects in middle school, applicants at baseline. Then the proficiency to urban and nonurban charters have very low proficiency rates rates diverge, with lottery winners that attend charter schools increasing their qualifications over time substantially, from about 30 to 70 percent . In math, lottery losers that attend noncharter th grade . In urban schools actually have proficiency rates lower than their baseline rate by 8 nonurban schools (dashed lines), the opposite is true: noncharter schools (the light dashed lines) improve over time, and charter schools (the dark dashed lines) do worse. The figure also shows th grade , proficiency levels for urban charter school attendees are in the range of that , by 8 children in the suburbs. The pattern for urban high schools shown in Figure 5, Panel II also indicates a large proficiency gap of about 20 percentage points that opens up between charter and noncharter schools after for both math and English/language arts . two years C. Larger E ffects for Black , Hispanic, and Previously Poor Performing Students 21 We use percent proficient as opposed to mean scores, so that we are making comparisons to a set standard rather than the state mean. However, mean scores show a very similar pattern. 19
21 The urban charter school advantage is fairly consistent across subgroups. Table 2 reports -stage least squares regressions as discussed earlier, but for different per -year results of two subgroups of students after combining data from the Massachusetts and national charter school studies mentioned in the previous section. The dependent variables for the four columns are school-specific gains for students in charter schools in math and English/language arts test scores for urban and nonurban students. positive and statistically significant For urban charter schools, the coefficients reveal each of the subgroups we examine, with the exception of white students, for whom effects across the charter school effect is positive and marginally significant in math and essentially zero in English/language arts . Effects are generally larger for less advantaged students, including black and Hispanic students, those with low baseline scor es, those who receive subsidized lunch, and English language learners. Special education and non-special education students in urban (for more details on impacts on charters have essentially the same test score impact estimates English language learners and s pecial education students, including effects on classification, see Setren 2015). For nonurban charter schools, we find negative and statistically significant effects for femal e students, white students, and those without low baseline test scores, who do not receive subsidized lunch, who are not in special education, or who are not English language learners. There are marginally positive effects in math in nonurban schools for black students and those 22 with low baseline scores. D. Similar Estimated Effects for Students Who Do Not Apply 22 See Appendix Table 1 for results for subgroups by each study. 20
22 outcomes to estimate charter school effects provides a useful estimate of the lottery Using gains from charter schools for those who students who applied to oversubscribed charter schools. However , the lottery studies cannot clearly tell us whether adopting approaches practiced by the oversubscribed charter schools would help the type of students who don't apply. most successful For example, charter schools often try to engage parents in their child’s learning; if students who do not apply to charter schools have less involved parents, these types of parental engagement strategies may not work for these students. In fact, there are a few studies suggesting that charters also benefit those who end up in them without applying. Abdulkadiroğlu et al. (2014) examine charter takeovers in New Orleans and Boston, where chronically poor performing schools were replaced with charters, most of By comparing students at schools not yet taken over which follow the “No Excuses” pedagogy. with student s a t schools that were taken over and turned into charter schools , the authors estimate chart er school effects for students who passively enroll. T hey calculate estimates of charter school impacts at New Orleans takeover charters of 0.21 standard deviations in math and 0.14 standard deviations in English/language arts per year of charter school attendance. These estimates are similar to or larger than lottery estimates for the sample of Massachusetts urban charters schools in Angrist et al. (2013). At Unlocking Potential Academy Boston, Abdulkadiroğlu et al. (2014) find that students who passively enroll in UP due to being grandfathered into the school have substantially larger gains in English/language arts test scores than students who attend due to winning an admissions lottery. Students who have been English/language arts grandfathered have baseline achievement 0.23 standard deviations below that o f their lottery counterparts; attendance at UP effectively closes this gap. 21
23 Indeed, evidence from the lottery studies suggests that charter schools may actually be In more effective at increasing the achievement of students who are less likely to apply. prior to 2011, charter applicants were slightly less likely to participate in special Massachusetts education programs or to qualify for a subsidized lunch and had slightly higher test scores at baseline, compared to their traditional public school counterparts (Angrist et al. 2013). However, these subgroups tend to have larger gains in test scores Angrist et . In their study of KIPP Lynn, al. (2012) find that students with special needs or those who have limited English proficiency experience larger gains in reading (0.42 and 0.27 standard deviations for students with special needs and with limited English proficiency, respectively, compared to an average of 0.12 standard deviations) and math (0.47 and 0.42 standard deviations, respectively, compared to an average of 0.35 standard deviations) for each year of attendance. They also find that the effects of attendance at KIPP Lynn are larger for students with lower baseline scores. In Boston, Walters (2014) finds that high income families are more likely to apply -achieving students from higher- to charter schools, but charter schools generate larger gains for disadvantaged, low-achieving, and non- white applicants. These results are promising because they suggest these charter schools may be good at helping the most disadvantaged among the group of disadvantaged students. Evidence is mixed as to whether charter schools for which lotte ry estimates are not available—either because the schools are not oversubscribed or because lottery records are not available—are more or less effective than the charter schools included in lottery-based studies. Angrist et al. (2011) find that for Massachusetts urban charter middle and high schools, observational estimates, calculated using a combination of matching and regression, and lottery- based estimates are very similar. However, for nonurban charter middle schools, the observational and lottery- based estimates are not as close, with the observational estimates 22
24 seeming to overstate the effect of charter schools . Using observational estimates, they find that in the lottery sample relative to the set of schools that for urban charter schools, gains are larger are undersubscribed or have poorly documented lotteries. Following Angrist et al. (2011), also find that the observational estimates for the lottery sample are Dobbie and Fryer (2013) somewhat higher than for the full sample of New York City charter schools, but the difference is quite sm all . In their study of KIPP middle schools, Clark Tuttle et al. (2013) find t hat matching - based estimates for the 10 schools in their lottery sample are similar to the matching -based estimates for all 41 study schools. E.. Effects on Non- Test Score Outcomes focuses on how charter school attendance affects scores Most of the available research on state -mandated tests , but some studies look at subsequent educational attainment and other outcomes likely linked to adult well-being (e.g. Oreopoulos and Salvanes 2011). Angrist et al. (2016) find that charter attendance increases pass rates on the state high school graduation exam (which also qualifies students for state -sponsored college scholarships), as well as increasing SAT scores, advanced placement exam test -taking, and advanced placement scores. While charter school attend ance does not result in a statistically significant increase in overall college enrollment, it seems to shift enrollment from two -year to four- year colleges: charter school attendance decreases immediate enrollment in a two -year college by 11 percentage po ints and increases immediate enrollment in a four -year college by 17 percentage points. Dobbie and Fryer (2015) collect longer -term survey and administrative data for the admissions lottery, earliest cohorts of the Promise Academy middle school. Six years after the the authors estimate a 0.075 standard deviation increase in math achievement among youth 23
25 offered admission to Promise Academy, higher college enrollment immediately following high school graduation, higher rates of immediate enrollment in a four-year college, a 10.1 percentage point drop in female pregnancy, and a 4.4 percentage point drop in male incarceration. IV. Why Are Some Charter Schools Effective, But Not Others? A. “No Excuses” Studies Lottery studies that use admissions data from identifiable schools, like KIPP Lynn , UP Academy , SEED, and the Promise Academy charter schools , allow for a more in -depth analysis of the mechanisms behind why some types of charter schools are more effective than others. All four of these charters boost student performance substantially (especially in math) compared to schools that lottery losers attend the low- performing urban . Because each of these charter , they also have a competitive advantage against surrounding s disadvantaged areas schools target public schools. Because these charters are traditional all trying to turn around the prospects of youth from disadvantaged neighborhoods, it is perhaps not surprising that they have adopted similar “No Excuses” strategies, which have been cited for decades by qualitative researchers as important for improving student performance (Dobbie and Fryer 2013). As noted earlier , these strategies include uniforms, high expectations from principals and teachers, a tightly enforced discipline code, along with intensive tutoring, longer instruction time, regular feedback, college preparation services, and an energetic commitment to ensuring the academic success of all re of these schools are empowered and inspiring principals, students. Another featu , flexible, whose presence may be necessary to implement “No Excuses” schools successfully (Carter 2000). 24
26 There is some question about the extent to which the “No Excuses” framework captures what is different about these schools. While these schools share many similarities, they also exhibit distinct differences in curricula and culture—for example, KIPP schools follow a particularly unique setup, with middle schools starting in Grade 5 instead of 6, students receiving 'paychecks' for exhibiting good behavior that can be used for participation in school activities, assrooms requiring students to ‘SLANT' (Sit up straight, Listen, Ask questions, Nod, and and cl Harlem Children’s Zone’s 's Promise Academy, Track the person speaking with your eyes). At and regular instruction on character and social emotional students receive a free daily breakfast issues in gender -based groups, and all classrooms are equipped with smart boards. Suspension UP Academy and SEED report relativ ely high suspension rates (33.5 percent in rates also differ. and 52 percent for SEED compared to a 23 2013 for UP compared to a 2.8 percent state average, city average) KIPP Lynn and Harle m Children’s Zone 's Promise Academy report percent , while 23 low suspension rates that are close to state averages (4.7 and 2.5 percent, respectively). Moreover, some evidence suggests that these four charter schools may spend more per student than the traditional public schools, because they receive additional funding from charitable foundations. KIPP, for example states that 15 percent of its annual operation expenses 24 Goldman Sachs Gives made a $20 million donation is covered by philanthropic contributions. 23 For UP Academy: “2015 Massachusetts School Report Card Overview: UP Academy Charter School of Boston,” Massachusetts Department of Elementary and Secondary Education, accessed January 21, 2016, http://profiles.doe.mass.edu/reportcard/SchoolReportCardOverview.aspx?linkid=105&orgcode=04800405&fycode= code=6&. For SEED: “SEED PCS of Washington, DC: 2014- 2015 Equity Report,” District of 2015&orgtype . For KIPP Lynn: http://learndc.org/schoolprofiles/view?s=0174#equityreport Columbia, accessed January 21, 2016, “2015 Massachusetts School Report Card Overview: KIPP Academy Lynn Charter School,” accessed January 21, 2016, http://profiles.doe.mass.edu/reportcard/SchoolReportCardOverview.aspx?linkid=105&orgcode=04290010&fycode= 2015&orgtypecode=6& . For Promise Academy: “Charter School Suspension Rates: Way Above District Averages,” United Federation of Teachers, accessed January 21, 2016, http://www.uft.org/files/charter -school -suspension- rates - way -above -most -district- averages. Note that, according to the UFT report, suspension rates for KIPP schools in New York City vary widely, from 0% (KIPP NYC Washington Heights Academy Charter School) to 23% (KIPP AMP). 24 “Frequently Asked Questions,” KIPP, accessed January 21, 2016, http://www.kipp.org/faq. 25
27 25 to aid in the construction of the Promise Academy's new school building. UP Academy relies 26 primarily on philanthropy to support new school startup costs and head- office administration. Capital and start -up operating costs at SEED are als o funded by donations from foundations and 27 private individuals. -student are pursued do to less per The extent to which these revenues 28 KIPP schools, at least, appear to spend funding from public sources remains a source of debate. significantly mor e in per -student spending compared to traditional schools (Miron et al. 2011, Baker et al. 2012), though this pattern is not observed in Boston’s charter high schools (Angrist et al. 2016). Extensive research would be needed to document and appreciate th e detailed differences across these schools (for an example, see Merseth et al. 2009). However, the similarity in effectiveness of these charter schools suggests that it is their common set of “No Excuses” characteristics that matter most in boosting performance. One exception might be the higher Curto and Fryer (2014) suggest that this may be due to reading score effects for SEED Academy. 29 the fact that SEED is a boarding school. 25 “Supporting the Harlem Children’s Zone,” Goldman Sachs, acces sed January 21, 2016, http://www.goldmansachs.com/citizenship/goldman -sachs -gives/building -and -stabilizing -communities/hcz/. 26 “UP Education Network (Unlocking Potential Inc),” The Giving Common, accessed January 21, 2016, ile/1108725/up- -network -unlocking -potential- inc/. https://www.givingcommon.org/prof education 27 “FAQs,” The SEED Foundation, accessed January 21, 2016, http://www.seedfoundation.com/index.php/about - seed/faqs. 28 For more on charter school finances, see: “Charter Schools in Perspective: A Guide to Research” (2015). The Spencer Foundation and Public Agenda. www.in -perspective.org. 29 For UP Academy: “2015 Massachusetts School Report Card Overview: UP Academy Charter School of Boston,” Massachusetts Department of Elementary and Secondary Education, accessed January 21, 2016, http://profiles.doe.mass.edu/reportcard/SchoolReportCardOverview.aspx?linkid=105&orgcode=04800405&fycode= 2015&orgtypecode=6&. For SEED: “SEED PCS of Washington, DC: 2014- 2015 Equity Report,” District of . For KIPP Lynn: Columbia, accessed Janu ary 21, 2016, http://learndc.org/schoolprofiles/view?s=0174#equityreport “2015 Massachusetts School Report Card Overview: KIPP Academy Lynn Charter School,” accessed January 21, 2016, http://profiles.doe.mass.edu/reportcard/SchoolReportCardOverview.aspx?linkid=105&orgcode=04290010&fycode= 2015&orgtypecode=6& . For Promise Academy: “Charter School Suspension Rates: Way Above District Averages,” United Federation of Teachers, accessed January 21, 2016, http://www.uft.org/files/charter -school -suspension- rates - -averages. way -above -most -district Note that, according to the UFT report, suspension rates for KIPP schools in New York City vary widely, from 0% (KIPP NYC Washington Heights Academy Charter School) to 23% (KIPP AMP). 26
28 B. ? What Relationships Exist Between Charter School Characteristics and Effectiveness We combine data from three studies for which school- specific charter effects and school characteristics are available in order to explore the relationship between school characteristics and effectiveness . We first use Dobbie and Fryer ’s (2013) definition of school characteristics . These include five “non -traditional” inputs that are measured on a binary basis: teacher feedback, data-driven instruction, instructional time, high dosage tutoring, and high expectations, as well as also include an index based on four a standardized index of the five characteristics. They traditional inputs: class size, per pupil expenditures, highly qualified teachers (as measured by masters degrees), and teacher certification. We create equivalent variables for schools in the Massachusetts and Gleason et al. (2010) survey data. In general, our method for creating equivalent dummy variables t o the New York City data is to estimate the med ian of a school characteristic—for example, per pupil expenditure—and assign values of one for schools that were above the median and zero for schools that were below. We are able to create fairly similar measures in the Massachusetts study . Unlike Angrist et al. (2013) and Dobbie and Fryer (2013), based rather when we combine the three studies our sample size is large enough to use lottery- 30 than observational estimates as our outcome of interest. In Table 3, we present results from regressing the estimated charter school effects from the studies themselves on the school characteristics defined above. All regressions include study fixed effects and are weighted by the and a control for school level (elementary, middle, high) 30 In Appendix Table 2, we describe in detail the variables and our adaptations across the underlying studies. See Appendix Tables 4a and 4b for the individual study results, which tend to be similar though less precisely estimated. available survey variables do not map well to The most dissimilar study is IES, which is not surprising given that the the constructs from the New York City study. 27
29 inverse of the outcome’s standard error. We also cluster standard errors by school to account for a handful of the charter schools in this sample have campuses serving multiple the fact that school levels. Columns 1 and 5 include results from single variable regressions, while all other columns include multiple school characteristics. We also present results using an index of “No Excuses” school practices, equal to a standardized sum of each characteristic employed, as well as an index for school resource inputs summarized by a second standardized index. When each characteristic is considered separately, in both math (in Colu mn 1) and English/language arts (in Column 5), all of the school practice inputs but one are positive and statistically significant (excluding data -driven instruction for math and teacher feedback for English/language arts ). The coefficient on the index summarizing “No Excuses” practices is positive and precise. In math, none of the school resource variables have predictive power for charter school effects . In English/language arts , there appear s to be a positive association between the coefficient on class size is per pupil expenditures and school level impacts, and significant but in the ‘wrong’ direction. For both subjects, the summary index of resource inputs in Columns 1 and 5 has no explanatory power. The other columns include multiple characteristics and generally show that school practices remain important even when controlling for resource inputs. These findings are consistent with the results from Angrist et al. (2013) and Dobbie and Fryer (2013) . C. Taking Location into Account We pointed out earlier that the charters with the highest value- added locate in areas where lottery losers end up in some of the worst performing schools; conversely, charter schools -added with the lowest value are in more suburban areas, where neighboring traditional public 28
30 more likely to locate in highly schools that are schools do relatively well. Also, charter segregated and disadvantaged areas tend to be “No Excuses” schools, while nonurban charter schools, in contrast, tend to emphasize other priorities, such as performing arts, interdisciplinary group projects, field work, or customized instruction. In Massachusetts, for example, no charter with a identify “No Excuses” philosophy schools in nonurban areas , while two -thirds of charter schools in urban areas identify as “No Excuses ” (Angrist et al. 2013). Thus, we condition on test performance at fallback schools to explore whether the remaining variance in estimated charter school effectiveness still relates to “No Excuses” 31 We drop data from New York City (Dobbie and Fryer 2013), for which we have no practices. e, leaving us with a sample of 57 schools. In information about fallback school performanc Column 1 of Table 4 we regress charter school effect estimates on a dummy variable for whether the charter is located in an urban area, while also including study fixed effects and school level dummies , again weighted by the inverse of the school effect standard error. Urban charters increase annual math scores by 0.28 standard deviations more than nonurban charters per year of attendance, on average. In bivariate relationships shown in Columns 2-4, we see that the counterfactual mean —that is, the test scores in the fallback school if not chosen in the charter school lottery—as well as school practice inputs also have explanatory power for charter school impacts Beginning i n Column 5, we combine the additional variables with the urban indicator. When we include average test performance at fallback schools as a conditioning variable, along 31 Several others have also pointed out the importance of the fallback, or counterfactual, option in estimating program effects. See, for example, Heckman, Hohmann, S mith, and Khoo (2000) for evidence from job training, -secondary decisions, and Kline and Walters (2015) Kirkebøen, Leuven, and Mogstad (2014) for evidence from post for evidence on Head Start. 29
31 with an urban indicator and index variables for practice and resource inputs, the variables for . This finding is consistent with remain strongly significant in math fallback school performance Figure 4, which shows a strong relationship between char ter effects and test scores at fallback , “No Excuses” characteristics are strongly related schools, even within urban areas. Remember t when including controls for urban areas and fallback school with charter school effects. Bu 32 A similar performance, the coefficient falls by about half (to 0.065) and becomes insignificant. pattern holds for English/language arts scores: No the coefficient on the index variables of “ Excuses ” practices falls by about half (from 0.077 to 0.047) and loses statistical significance. Of course, in both cases the loss of statistical significance could be due, in part, to an increase in the standard error. The estimated impact of more resource inputs remains negligible, with or without additional controls. We rerun these results in Table 5, this time breaking up the school practice index variable into specific charter school characteristics and adding high suspension rates to the list of variables , following the method used by Dobbie and Fryer (2013). When including urban and fallback school performance controls, the characteristics that remain significantly correlated with charter school math effects are teacher feedback, intensive tutoring, and above average suspension rates (significant at the 10 percent level). These variables may serve as proxies for other underlying characteristics. Notably, the importance of the high expectations variable disappears once both urban status and fallback performance is taken into account. When all of the school characteristics variables are included together in Column 7, the point estimates for the tutoring and high suspension rate variables remain about the same, while the others drop or 32 In other specifications we tried , with an additional squared and cubic fallback school quality term, and without the characteristics also falls by about half. For the model without the urban dummy, the coefficient for “No Excuses” urban dummy, the coefficient is significant. 30
32 remain Charter school s that offer intensive tutoring have math test scores 0.15 negligible. standard deviations higher, on average, for each year of charter attendance. This value is large and significant at the 10 percent level. After three years of attendance, students at these schools have test scores almost half a st andard deviation higher than lottery losers at fallback schools. Charter schools with high suspension rates have math test scores that are 0.12 standard deviations higher, on average, though this measure is not statistically significant. For English/language arts test outcomes, only differentiated instruction is significant when included with urban and fallback school performance controls, and none of the school characteristics variables are significant when they are included together in the s ame regression . Overall, o nce one accounts for surrounding neighborhood and school characteristics, are no longer associated with student improvement. many of the specific charter school practices The main exception is intensive tutoring. Its estimated impact remains large and relatively stable, especially in math, even when conditioning on other charter school characteristics. Nonurban schools account for most of the variation in charter school effects that identifies the importance of tutoring after conditioning on fallback school quality. When the model in Column 7 of Table 5 is estimated for the 21 urban schools only (conditioning on fallback quality), the coefficients for all school characteristics are statistically insignificant with large standard errors. The characteristic coefficients when using only the 28 nonurban schools are also insignificant except the one for intensive tutoring (0.254, with a standard error of 0.112). This evidence in support of tutoring is of course only suggestive, based on analysis of correlations rather than on the randomized provision of tutoring services. However, t he potential experimental and experimental importance of intensive tutoring is in line with recent quasi- t of studies that find large increases in student performance from tutoring, delivered either as par 31
33 a package of school reforms or on its own. Kraft (2015) uses two quasi- experimental methods to estimate the impact of implementing individualized tutoring classes four days a week at MATCH Charter Public High School in Boston and finds large and stati stically significant impacts on English /language arts achievement. In his review of randomized experiments in education, Fryer (2016) distinguishes high-dosage tutoring—which he defines as being tutored in groups of 6 or fewer for more than three days per week, or being tutored at a rate that would equate to 50 hours week period or more over a 36- -dosage tutoring. Consistent with our findings, Fryer —from low finds that high-dosage tutoring programs have, on average, statistically significant positive treatment effects on math and reading achievement . In contrast, the meta- coefficient on low- dosage tutoring is not statistically significant for either subject. Some examples of recent randomized experiments showing gains from intensive tutoring include Lee et al. (2010), who study the Experience Corps® (EC) program for placing older volunteers in elementary schools to tutor reading; Fryer ), who studied the use of intensive tutors in fourth, sixth and ninth (2014 grades in Houston public schools ; Markovitz et al. (2014), who evaluated the Minnesota Reading Corps, a literacy tutoring program for kindergarten through third grade students ; C ook et al. (2015), who studied an intensive tutoring serving male ninth and tenth graders in 12 public high schools in Chicago; and M ay et al. (2014), who evaluated an early -intervention literacy tutoring program called Reading Recovery. Conclusions Charter schools were originally intended to serve as research laboratories for learning about best practices in education. They have since become viewed more as competitive 32
34 alternatives to traditional public schools. But with many charters now receiving more applications than spots available, the requirement that oversubscribed charter schools admit students through lottery has unintentionally created the research setting that ool the charter sch movement’s originators were seeking. Our purpose in this paper is not to enter the debate on whether charter scho ols should exist or expand: we have not discussed issues like how increased competition from charters affects traditional public schools over time, or the possible effects of charter schools on the racial/ethnic or socioeconomic mixture of students (for an example of discussion of this point in North Carolina, see Ladd et al. 2015). Instead, our purpose is to gather existing evidence from charter lotteries to learn more about the education production function. the We confirm a finding from previous studies that a sharp divide exists between effectiveness of charter schools in urban and nonurban settings. However, there are two urban and nonurban charters that have been studied. One is important differences between the that almost all the charter school alternatives that have been the subject of lottery studies in ” disadvantage d urban areas use a “No Excuses ” approach, while there are few “No Excuses schools in nonurban settings. The other main diffe rence is that students who attend charter schools in disadvantaged urban areas are usually being compared to students who end up in very poor performing schools, while students in charter schools in nonurban areas are being compared to students who attend better performing schools. This pattern arises because the charter schools aim ing to attract students from the worst performing traditional public schools often find them residing in highly segregated and disadvantaged urban neighborhoods. school s in disadvantaged urban schools have proven to be impressively Many charter effective, often raising average test score performance by more than half a standard deviation 33
35 after just two years of attendance. For less advantaged students, including black and Hispanic students, those with low baseline scores, and English language learners , impacts are similar or higher than for the more- advantaged . Other studies find corresponding improvements to longer- term outcomes, such as reductions in incarceration rates and teen pregnancies and increases in enrollment in four- year colleges (Dobbie and Fryer 2015, Angrist et al. 2016). I t is unclear , however, if schools would deliver similarly impr essive results in areas with other types of charter very poor performing traditional schools, since there are currently not enough of them in these areas to tell. It is also unclear if “No Excuses” schools would deliver similar results in nonurban areas; again, there are currently not enough of them to tell. For now, the kinds of charters that have been created in nonurban areas—such as those emphasizing performing arts, exploratory learning , or instruction tailored to different learning styles—may offer other benefits but do not appear to be improving standardized test scores. After accounting for the charter school effect variation explained by urban status and performance levels at fallback schools, we examine which charter school characteristics most strongly correlate with the little remaining variation. In line with previous studies, we find no evidence that differences in class size, per pupil expenditures, or teacher certification explain charter school effectiveness. The “No Excuses” explanatory factor that remains significant after controlling for fallback school performance (even for nonurban schools only) is whether a intensive tutoring program ( though the effect of high suspension rates is close to charter has an significant). Of course, this variabl e could be a proxy for other school differences, and the relationship s between effectiveness and several other associations are estimated imprecisely. But and convenient than currently a push for intensive tutoring—more frequent provided at 34
36 —may serve as an important traditional public schools, and in some cases mandatory complement to instruction in many different kinds of classrooms. . 35
37 Acknowledgements For sharing the data from and answering question about their studies, we thank Ira Nichols -Barrer , Brian Gill, Phil Gleason, Josh Furgeson, and Christina Clark Tuttle of Mathematica Policy Research; Danielle Eisenberg of the KIPP Foundation; Josh Angrist, Parag Pathak, and Chris Walters; and Will Dobbie and Roland Fryer. We also thank Carrie Conaway of the Massachusetts Department of Elementary and Secondary Education and the National Center for Education Statistics for additional data access , as well as , and Josh Angrist, Will Dobbie, and Mike Gilane seminar participants at Teachers College Columbia U niversity and IFN -Stockholm for helpful comments. 36
38 References Abadie, Alberto. 2002. "Bootstrap Tests for Distributional Treatment Effects in Instrumental Variables Models." 97 (457):284-292. Journal of the American Statistical Association http://www.hks.harvard.edu/fs/aabadie/dtep.pdf ———. 2003. "Semiparametric Instrumental Variable Estimation of Treatment Response Models." Journal of Econometrics 113 (2):231-263. http://www.hks.harvard.edu/fs/aabadie/gtep.pdf Abdulkadiroğlu, Atila, Joshua D. Angrist, Susan M. Dynarski, Thomas J. Kane, and Parag A. Pathak. 2011. “Accountability and Flexibility in Public Schools: Evidence from Boston’s Charter s and Pilots.” Quarterly Journal of Economics 126(2): 669-748. doi: http://dx.doi.org/10.1093/qje/qjr017 Abdulkadiroğlu, Atila, Joshua D. Angrist, Peter D. Hull, and Parag A. Pathak. 2014. “Charters Without Lotteries: Testing Takeovers in New Orleans and Boston.” National Bureau of Economic Research (NBER) Working Paper 20792. Abdulkadiroğlu, Atila, Joshua D. Angrist, Yusuke Narita and Parag A. Pathak. 2015. “Resea rch Design Meets Market Design: Using C entralized Assignment for Impact Evaluation.” National Bureau of Economic Research (NBER) Working Paper 21705. Angrist, Joshua D., Sarah R. Cohodes, Susan M. Dynarski, Parag A. Pathak, and Christopher R. Walters. 20 16. “Stand and Deliver: Effects of Boston’s Charter High Schools on College Journal of Labor Economics Preparation, Entry, and Choice.” 34(2). http://dx.doi.org/10.1086/683665 Angrist, Joshua D., Susan M. Dynarski, Thomas J. Kane, Parag A. Pathak, and Christopher R. Journal of Policy Analysis and Management Walters. 2012. “Who Benefits from KIPP?” 31(4): 837-860. doi: http://dx.doi.org/10.1002/pam.21647 Angrist, Joshua D., Parag A. Pathak, and Christopher R. Walters. 2011. “Explaining Charter School Effectiveness.” National Bureau of Economic Research (NBER) Working Paper 17332. ———. 2013. “Explaining Charter School Effectiveness.” American Economic Journal: Applied Economics 5(4) : 1-27. doi: http://dx.doi.org/10.1257/app.5.4.1 Baker, Bruce D., Ken Libby, and Kathryn Wiley. 2012. “Spending by the Major Charter Management Organizations: Comparing Charter School and Local Public District Finan cial Resources.” Boulder, CO: National Education Policy Center. http://nepc.colorado.edu/publication/spending- major -charter 37
39 Baude, Patrick L., Marcus Casey, Eric A. Hanushek, and Steven G. Rivkin. 2014. “The Evolution of Charter School Quality. df http://harris.uchicago.edu/sites/default/files/Rivkin.paper_.p Brown, Emma. 2014. “Joy and Anguish for Parents as D.C. Releases School Lottery Results.” March 31. Washington Post, https://www.washingtonpost.com/local/education/joy- and - for- parents -as-dc-releases -school- lottery -results/2014/03/31/a04aea1e-b8ff- anguish- 11e3-9a05-c739f29ccb08_story.html Carter, Samuel Casey. 2000. No Excuses: Lessons from 21 High-Performing, High- Poverty Schools . Washington, DC: Heritage Foundation. http://eric.ed.gov/?id=ED440170 Chapman, Ben and Stephen Rex Brown. 2014. New York Daily News. ” Success Academy Charter Schools Admissions Rate is Only 20%, Lower Than NYU.” April 4. http://nydn.us/1fJapci . Chetty, Raj, John N. Friedman, Nathaniel Hilger, Emmanuel Saez, Diane Whitmore Schanzenbach, and Danny Yagan. 2011. “How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star.” Quarterly Journal of Economics 126(4): 1593-1660. doi: http://dx.doi.org/10.1093/qje/qjr041 Citizens League of Minnesota. 1998. “Chartered Schools = Choices for Educators + Quality for - All Students.” http://citizensleague.org/wp Quality - content/uploads/2013/05/424.Report.Chartered-Schools- Choices -for-Education- for- All -Students.pdf Clark Tuttle, Christina, Brian Gill, Philip Gleason, Virginia Knechtel, Ira Nichols -Barrer, and Alexandra Resch. 2013. “KIPP Middle Schools: Impacts on Achievement and Other http://www.mathematica - Outcomes.” Washington, DC: Mathematica Policy Research. mpr.com/our-publications- and -findings/publications/kipp- middle -schools- impacts -on- achievement -and -other- outcomes -full -report Clark Tuttle, Christina, Philip Gleason, and Melissa Clark. 2012. “Using Lotteries to Eva luate Schools Of Choice: Evidence from a National Study of Charter Schools.” Economics of Education Review 31(2): 237-253. doi: http://dx.doi.org/10.1016/j.econedurev.2011.07.002 Clark Tu ttle, Christina, Philip Gleason, Virginia Knechtel, Ira Nichols -Barrer, Kevin Booker, Gregory Chojnacki, Thomas Coen, and Lisbeth Goble. 2015. “Understanding the Effect of KIPP as It Scales: Volume I, Impacts on Achievement and Other Outcomes.” http://www.mathematica -mpr.com/our- Washington, DC: Mathematica Policy Research. publications- and -findings/publications/executive- summary -understanding-the- effect -of- kipp- as-it- scales -volume-i- impacts -on- achievement Cobb, Casey D. and Gene V. Glass. 1999. “Ethnic Segregation in Arizona Charter Schools.” ves 7(1). doi: http://dx.doi.org/10.14507/epaa.v7n1.1999 Education Policy Analysis Archi 38
40 Cohodes, Sarah R., Elizabeth M. Setren, Christopher R. Walters, Joshua D. Angrist, and Parag fectiveness: A Boston Update.” The A. Pathak. 2013. “Charter School Demand and Ef Boston Foundation and New Schools Venture Fund. http://seii.mit.edu/research/study/charter -school-demand- -a-boston- and -effectiveness update/ Cook, Philip J., Kenneth Dodge, George Farkas, Roland G. Fryer, Jr., Jonathan Guryan, Jens Ludwig, Susan Mayer, Harold Pollack, and Laurence Steinberg. 2015. “Not Too Late: Improving Academic Outcomes for Disadvantaged Youth.” Institute for Policy Research Northwestern University Working Paper WP-15-01. http://www.ipr.northwestern.edu/publications/papers/2015/ipr- wp-15-01.html Curto, Vilsa E. and Roland G. Fryer, Jr. 2014. “The Potential of Urban Boarding Schools for the Journal of Labor Econoics 32(1): 65-93. doi: Poor: Evidence from SEED.” http://dx.doi.org/10.1086/671798 Dobbie, Will and Roland G. Fryer, Jr. 2011. “Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone.” American Eco nomic Journal: Applied Economics 3(3): 158-87. doi: http://dx.doi.org/10.1257/app.3.3.158 ———. 2013. “Getting Beneath the Veil of Effective Schools: Evidence from New York City.” 28-60. doi: American Economic Journal: Applied Economics 5(4) : http://dx.doi.org/10.1257/app.5.4.28 Term Impacts of High -Achieving Charter Schools.” Journal of ———. 2015. “The Medium- Political Economy 123(5): 985-1037. doi: http://dx.doi.org/10.1086/682718 . Dougherty, Chris. “Getting FERPA Right: Encouraging Data Use While Protecting Student Privacy.” In A Byte at the Apple: Rethinking Education for the Post- NCLB Era, edited by Marci Kanstoroom and Eric C. Osberg. Washington, D.C.: Thomas B. Fordham Institute, . 2008. http://files.eric.ed.gov/fulltext/ED504444.pdf Fiske, Edward B. and Helen F. Ladd. 2000. When Schools Compete: A Cautionary Tale. Washington, D.C.: Brooking Institute Press. Frankenburg, Erica and Chungmei Lee. 2003. “Charter Schools and Race: A Lost Opportunity for Integrated Education.” Education Policy Analysis Archives 11(32). doi: http://dx.doi.org/10.14507/epaa.v11n32.2003 Fryer, Roland G. 2014. “Injecting Charter School Best Practices into Traditional Public Schools: Evidence from Field Experiments.” The Quarterly Journal of Economics 129(3): 1355- http://dx.doi.org/10.1093/qje/qju011 1407. doi: 39
41 ———. 2016. “The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments.” Elsevier. Handbook of Field Experiments. field -experiments https://www.povertyactionlab.org/handbook- Furgeson, Joshua, Brian Gill, Joshua Haimson, Alexandra Killewald, Moira McCullough, Ira Barrer, Bing Nichols- -Savi tz, Melisa Bowen, Allison -Ru The, Natalya Verbitsky Demerrit, Paul Hill, and Robin Lake. 2012. “Charter School Management Organizations: Diverse Strategies and Diverse Student Impacts.” Princeton, NJ: Mathematica Policy Research. http://www.mathematica -mpr.com/our-publications- and - findings/publications/charterschool- -organizations-diverse- strategies -and- management diverse-student- impacts Gleason, Philip, Melissa Clark, Christina Clark Tuttle, and Emily Dwoyer. 2010. The Evaluation of Charter School Impacts: Final Report (NCEE 2010-4029). Washington, DC: U.S. Department of Education, National Center for Educati on Evaluation and Regional Assistance, Institute of Education Sciences. http://ies.ed.gov/ncee/pubs/20104029/ Hastings, Justine S., Christopher A. Nielson, and Seth D. Zimmerman. 2012. “The Effect of School Choice on Intrinsic Motivation and Academic Outcomes.” National Bureau of Economic Research (NBER) Working Paper 18324. Heckman, James, Neil Hohmann, Jeffrey Smith, and Michael Khoo. 2010. “Substitution and Dropout Bias in Social Experiments: A Study of an Influential Social Experiment.” The Quarterly Journal of Economics : 115 (2): 651-694 doi:10.1162/003355300554764 Hoxby, Caroline M., Sonali Murarka, and Jenny Kang. 2009. “How New York City's Charter Schools Affect Achievement.” Cambridge, MA: The New York City Charter Schools Evaluation Project. http://users.nber.org/~schools/charterschoolseval/ Hoxby, Caroline M. and Jonah E. Rockoff. 2004. “The Impact of Charter Schools on Student Achievement.” https://www0.gsb.columbia.edu/faculty/jrockoff/hoxbyrockoffcharters.pdf Junge, Ember Reichgott. 2014. “Charter School Path Paved with Choice, Compromise, Common Sense.” Phi Delta Kappan 95(5): 13-17. doi: http://dx.doi.org/10.1177/003172171409500504 Kirkebøen, Lars and Edwin Leuven, and Magne Mogstad. 2015. “Field of Study, Earnings, and Self -Selection .” National Bureau of Economic Research (NBER) Working Paper 20816 .Kline, Patrick and Christopher Walters. 2015. “ Evaluating Public Programs with Close Substitutes: The Case of Head Start” National Bureau of Economic Research (NBER) Working Paper 21658. 40
42 Kolderie, Ted. 2008. “How the Idea of ‘Chartering’ Schools Came About: What Role Did the Citizens League Play?” 25(5): 5-6. Minnesota Journal Chartering -League-Role.pdf http://www.educationevolving.org/pdf/Origins-of- -Citizens Kraft, Matthew A. 2015. “How to Make Additional Time Matter: Integrating Individualized Education Finance and Policy Tutorials into an Extended Day.” 81-116. doi: 10(1): http://dx.doi.org/10.1162/EDFP_a_00152 Krueger, Alan B. 1999. “Experimental Estimates of Education Production Functions.” Quarterly Journal of Economics 114(2): 497-532. doi: http://dx.doi.org/10.1162/003355399556052 ., Charles T. Clotfelter, and John B. Holbein. 2015. “The Growing Segmentation Ladd, Helen F of the Charter School Sector in North Carolina.” National Bureau of Economic Research (NBER) Working Paper 21078. Lee, Yung Soo, Nancy Morrow- Howell, Melissa Jonso n-Reid, and Stacey McCrary. 2010. “The Effect of the Experience Corps Program on Student Reading Outcomes.” Education and Urban Society 44(1): 97-118. doi: http://dx.doi.org/10.1177/0013124510381262 Markovitz, Carrie E., Marc W. Hernandez, Eric C. Hedberg, and Benjamin Silberglitt. 2014. “Impact Evaluation of the Minnesota Reading Corps K-3 Program.” Chicago, IL. http://www.nationalservice.gov/documents/research- - evaluation and -reports/2014/impact- minnesota- -corps-k-3- program reading May, Henry, Heather Goldsworthy, Michael Armijo, Abigail Gray, Philip Sirinides Toscha J. Blal ock, Helen Anderson-Clark, Andrew J. Schiera, Horatio Blackman, Jessica Gillespie, and Cecile Sam. 2014. “Evaluation of the i3 Scale-up of Reading Recovery: Year Two Report, 2012-2013.” Philadelphia, PA: Consortium for Policy Research in Education. doi: http://dx.doi.org/10.12698/cpre.2014.RR79 Merseth, Katherine K., Kristy Cooper, John Roberts, Mara Casey Tieken, Jon Valant, and Chris Wynne. 2009. Inside Urban Charter Schools: Promising Practices and Strategies in Five High -Performing Schools. Cambridge, MA: Harvard Education Press. Miron, Gary, Jessica L. Urschel, and Nicholas Saxton. 2011. “What Makes KIPP Work? A Study of Student Characteristics, Attrition, and School Finance.” New York, NY: Na tional Center for the Study of Privatization in Education. http://www.ncspe.org/readrel.php?set=pub&cat=253 National Alliance for Public Charter Schools. 2015a. “A Growing Movement: America ’s Largest http://www.publiccharters.org/publications/enrollment- Charter School Communities.” share-10/ National Alliance for Public Charter Schools. 2015b. “Estimated Number of Public Charter close-2015/ Schools & Students.” http://www.publiccharters.org/publications/open- 41
43 National Allia nce for Public Charter Schools. 2015c. “State Laws on Weighted Lotteries and Enrollment Practices.” lotteries - http://www.publiccharters.org/publications/weighted- paper/ Neal, Derek. 2009. "The Role of Private Schools in Education Markets,” Handbook of Research on School Choice, edited by Mark Berends, Matthew G. Springer, Dale Ballou, and Herbert J. Walberg, Lawrence Erlbaum Associates/Taylor & Francis Group. National Center for Education Statistics. 2015. “The Condition of Education 2015: Charter School Enrollment.” U.S. Department of Education. http://nces.ed.gov/programs/coe/indicator_cgb.asp (accessed January 11, 2016). National Commission on Excellence in Education. 1983. “A Nation at Risk: The Imperative for Educational Reform.” http://www2.ed.gov/pubs/NatAtRisk/index.html Nix, Naomi. 2015. “A Boston Breakthrough: UP Academy Goes from Failing to First.” The Seventy Four : August 10. https://www.the74million.org/article/a -boston-breakthrough- up-academy -goes -from -failing -to-first Oreopoulos , Philip and Kjell G. Salvanes . 2011. “Priceless: The Nonpecuniar y Benefits of 159-184. doi: Schooling.” Journal of Economic Perspectives 25(1): http://dx.doi.org/10.1257/jep.25.1.159 Pisano, Chris. 2015. “Hundreds Turn Out for Holyoke Charter School Enrollment WGGB. http://www.masscharterschools.org/media/news/hundreds- turn -out- Lottery.” March 4. charter -school-enrollment- lottery . holyoke- Rahman, Fauzeya. 2015. Houston Chronicle . “Word of Mouth Major Factor for Parents Charting Charter School Course.” November 7. http://www.houstonchronicle.com/news/houston- texas/houston/article/Word -of-mouth- major -factor -for- parents -charting -6617645.php Time , Rotherham, Andrew J. 2011. “KIPP Schools: A Reform Triumph, or Disappointment.” April 27. http://content.time.com/time/nation/article/0,8599,2067941,00.html Setren, Elizabeth. 2015 “Special Education and English Language Learners in Boston Charter Schools: Impact and Classification.” School Effectiveness and Inequality Institute (SEII) Discussion paper 2015.05. http://seii.mit.edu/wp -content/uploads/2015/12/SEII- Discussion -Paper -2015.05-Setren1.pdf Shanker, Albert. “National Press Club Speech.” Washington, DC, March 31, 1998. https://reuther.wayne.edu/files/64.43.pdf Walters, Christopher R. 2014. “The Demand for Effective Charter Schools.” National Bureau of Economic Research (NBER) Working Paper 20640. 42
44 Wiltenburg, Mary. 2015. WYPR. “Uncertain Future for Thousands in Charter School Lottery” February 13. http://news.wypr.org/post/uncertain- charter -thousands- future -school- lottery#stream/0 . 43
45 Data Appendix Overview We are grateful to the authors of the various studies who have generously shared data and answered questions about their charter school projects. This data appendix details the sources of the data that goes into our analyses and details any modifications we needed to make to the Our goal was to generate second- stage (LATE), original studies in order to combine materials. per -year estimates of charter school impacts in both math and ELA, which were directly obtained from the Massachusetts, NYC, SEED, and UP studies. We were able to estimate per -year local average treatment effects from the student level data for the IES and KIPP 2015 studies. However, for the CMO and KIPP 2013 studies, we needed to -year, LATEs (student level data for modify reported estimates to get our best estimate of the per these studies was not available). When studies reported estimates for a specific year (e.g. year two), we divided reported estimates by the number of years to generate a per- year impact. When studies reported intention- to-treat estimates, we divided by our best estimate of the associated first stage. We also divide the standard errors by the same factors we divide the coefficients. In the cases where we are converting intention -to-treat estimates to second stage estimates, this will not correct the standard errors as a typical 2SLS procedure would in a statistical software program. Thus our standard errors are likely slightly too small for a subset of the charter school impact estimates. Appendix Table on the studies and Appendix Table 2 provides 1 summarizes the information information about the creation of comparable school characteristic variables across studies. Note that means and regressions using school- level data are weighted by the inverse of the standard error of the relevant estimate. Study- Specific Details Site level experimental estimates CMO: of charter school effects are in the appendix of the report; researchers at Mathematica shared information about the number of schools per site. The experimental estimates are not linked to school characteristic information. The estimates of charter school impacts were intent ion- to-treat with varying outcome years. We converted these estimates to per -year, second stage effects by dividing by the participation rate (also available in the appendix) and dividing by the number of years before the exam (provided by Mathematica researchers). We followed the same procedure for standard errors, which likely makes the standard errors too small, since we cannot make the appropriate correction for IV estimation. IES: We accessed the student -level data from the federal study of charter school impacts through a National Center for Education Statistics (NCES) restricted use data license. Using this data, we calculated per year, second stage school level estimates of charter school effects and their associated 2SLS standard errors. The IES report focuses on intent ion- to-treat estimates, whereas we use the second stage estimates for our analysis. The IES data also included survey information that we used to generate school characteristic variables. However, given the limits of the survey questions asked, it was sometimes difficult to generate variables consistent with those 44
46 from Dobbie and Fryer (2013), which we used as the basis of our descriptive school characteristics. KIPP 2013 : Mathematica researchers provided school level intention- to-treat estimates. To convert these estimates to per -year, second stage effects, we divided the year 2 estimates by 2 and also by 0.6, the overall first stage. First stage effects by school were not available, and we must assume that there was some variation in the first stages. Thus these estimates are our best guess of the per -year, second stage impacts. It was not possible to link these estimates to school characteristics for privacy reasons. KIPP 2015 : The KIPP Foundation provided access to the student-level data for this report. Due to data use agreements, the only lottery- based data available was for elementary schools. We calculated the second stage impacts using student level data, however years of attendance was not in the data. Thus we divided the year 3 impacts by 3 for per year estimates. At this time, we cannot link the school level estimates to school characteristic information. Note that the two overlap ping samples as the 2013 study is limited to middle schools and KIPP studies have non- only elementary school experimental results were available in the 2015 study. Massachusetts: Angrist, Pathak, and Walters provided the school level lottery estimates and school characteristic information from surveys and administrative from their 2013 paper for 23 schools in Massachusetts. The estimates of charter school impacts were per year, second stage effects, with associated standard errors and sample sizes. We use the exact same estimates from the 2013 paper. We also have access to the student- level lottery, demographic, and test score data through the Massachusetts Department of Elementary and Secondary Education (DESE) to conduct additional analyses that require the micro data, such as the analysis of subgroup effects and F igure 2 which shows the accumulation of effects over time. NYC: The AEJ: Applied website hosts school level estimates and summary dummy variable school characteristic data from Dobbie and Fryer (2013). The estimates of charter school impacts were per year, second stage effects. This database also provided information for 9 school characteristics. 45
47 Table 1: Summary of Lottery-Based Charter School Estimates of Reading and Math Test Score Impacts Per year of charter attendance second stage impacts (unless otherwise noted; all effects significant at 5% level unless otherwise noted) Setting Sample Paper (3) (4) (1) (2) MS: 0.198sd ELA, 0.359sd math Boston (8 schools) Abdulkadiroglu, Angrist, Dynarski, Kane, and HS: 0.265sd ELA, 0.364sd math Pathak (QJE, 2011) Boston (13 schools) Cohodes, Setren, Walters, Angrist, and Pathak MS: 0.138sd ELA, 0256sd math (The Boston Foundation, 2013) HS: 0.271sd ELA, 0.354sd math MS: 0.075sd ELA, 0.213sd math Angrist, Pathak, and Walters (AEJ: Applied Massachusetts (26 schools) HS: 0.206sd ELA, 0.273sd math Economics, 2013) Massachusetts Angrist, Dynarski, Kane, Pathak, and Walters KIPP Lynn MS: 0.133sd ELA, 0.352sd math (JPAM, 2012) MS: 0.118sd ELA, 0.270sd math UP Academy Charter School of Boston Abdulkadiroglu, Angrist, Hull, and Pathak (NBER Working Paper, 2014) MS: -0.04sd reading, -0.04sd math (not significant). Gleason, Clark, Clark Tuttle, Dwoyer, & 15 states (36 schools) Silverberg (IES, 2010) *Year2 impacts divided by 2 to get a per year estimates KIPP schools (24 schools) Clark Tuttle, Gleason Knechtel, Nichols- ES: 0.11sd on letter-word identification and 0.10sd on passage comprehension test in reading, 0.14sd on Barrer, Booker, Chojnacki, Coen, and Goble (Mathematica, 2015) calculation, 0.02sd (not significant) on applied problems in math. From study-administered Woodcock-Johnson exam. *Year 3 impacts divided by 3 to get a per year estimate MS: 0.08sd reading, 0.12sd math. *Year 2 impacts divided by 2 to get a per year estimate National 0.08 reading (not significant), 0.18 math. Clark Tuttle, Gill, Gleason, Knechtel, Nichols- KIPP middle schools (12 schools) *Year 2 impacts divided by 2 to get a per year Barrer, Resch (Mathematica, 2013) estimate Furgeson, Gill, Haimson, Killewald, Intention-to-treat estimates: MS/HS: -0.02 reading Charter schools that were members of (not significant), -0.05 math (not significant). Charter Management Organizations (CMOs) McCullough, Nichols-Barrer, Teh, Verbitsky- in 14 states (16 schools in 6 sites; estimates Savitz, Bowen, Demeritt, Hill, and Lake (Mathematica, 2012) aggregated by site) Hoxby, Murarka, Kang (2009) New York City (42 schools) ES/MS: 0.09sd ELA, 0.12sd in math HS: 0.18sd ELA, 0.19sd math New York City (29 schools) ES: 0.058sd ELA, 0.113sd math Dobbie and Fryer (AEJ: Applied Economics, 2013) MS: 0.048 ELA (not significant), 0.126 math Dobbie and Fryer (Journal of Political Economy, 0.031sd (not significant) reading, 0.075sd math. Harlem Children's Zone Promise Academy 2015) From study-administered Woodcock-Johnson exam. middle school New York City Dobbie and Fryer (AEJ: Applied Economics, Harlem Children's Zone Promise Academy ES: 0.114sd ELA (not significant), 0.191sd math (not middle and elementary schools significant) 2011) MS: 0.047sd ELA (not significant), 0.229sd math Hoxby and Rockoff (Unpublished working No significant impacts on math or reading Chicago Chicago International Charter School schools (3 schools) (dependent variable is percentile score on Iowa Test paper, 2004) of Basic Skills) Anonymous "No Excuses" charter schools run 0.346sd reading, -0.092sd math (not significant), Unknown Hastings, Nielson, Zimmerman (NBER Working by prominent CMO in mid-sized urban school Paper, 2012) estimates are a mix of different years district (4 schools) 0.211sd reading, 0.229sd math SEED School Washington, DC Curto and Fryer (Journal of Labor Economics, 2014) Notes: This table only includes studies that use charter school lotteries to estimate effects on test scores. Some of these studies also include or focus on observational results, which are not reported here. In some cases where there are multiple studies of the same setting, we focus on published academic studies, adding studies when it appears that a substantial number of additional schools have been added. All impacts are second stage estimates reported in standard deviations and are statistically significant unless noted otherwise. Citations in boldface type indcate that this study contributes to the analyses presented in this paper. See Appendix Table 1 for more details on the studies indicated in boldface. Key to abbreviations: ES = elementary school, MS = middle school, HS = high school, sd = standard deviation, ELA = English/Lanugage Arts.
48 Figure 1: Distribution of School-Level Charter Effects A. Math B. ELA 40 40 Mean = 0.080 Mean = .046 30 30 20 20 Percent Percent 10 10 0 0 -1 -.5 0 .5 1 -1 -.5 0 .5 1 Math effect ELA effect Notes: This graph shows the distribution of school-level lottery-based charter school effects, where the effects are per-year school-level second stage point estimates for the 113 schools that contribute to our analysis. The means are weighted means of the school-level estimates, weighted by the inverse of the standard error of each estimate. The following studies are included in this figure: CMO, IES, KIPP 2013, KIPP 2015, Massachuestts, NYC, UP and SEED. See Table 2 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale.
49 Figure 2: School-Level Charter School Effects vs. their Standard Errors A. Math B. ELA 1 1 .8 .8 .6 .6 .4 .4 Standard Error Standard Error .2 .2 0 0 -1 -.5 0 .5 1 -1 -.5 0 .5 1 Math Effect ELA Effect CMO UP SEED NYC MA IES KIPP 2013 KIPP 2015 Notes: This graph shows school-level lottery-based charter school effects, where the effects are per-year school-level second stage point estimates, plotted against the standard error of the school-level estimate. The following studies are included in this figure: CMO, IES, KIPP 2013, KIPP 2015, Massachuestts, NYC, SEED, and UP. See Table 2 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale.
50 Figure 3: Distribution of School-Level Charter Effects, Restricted to More Precise Estimates .6 .4 .2 ELA mean = .066 0 ELA effect -.2 Math mean = .092 -.4 -.2 .6 .4 .2 0 Math effect KIPP 2013 SEED UP CMO IES NYC KIPP 2015 MA Notes: The notes for this figure are the same as those for Figure 1, except that the school level estimates that contribute to this figure are restricted to those with standard errors less than or equal to 0.1 in either subject, which restricts to 42 schools.
51 Figure 4: School-Level Charter School Effects by Scores of Counterfactual Schools Notes: This graph shows school-level lottery-based charter school effects, where the effects are per- year school-level second stage point estimates, plotted against the average scores of counterfactual schools attended by noncharter students that applied to the charter school. The size of the point is weighted by the inverse of the standard error (larger points are more precise estimates). The following studies are included in this figure: IES (Gleason et al. 2010) and Massachuestts (Angrist et al. 2013). See Appendix Table 1 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale. See Appendix Table 2 for description of the calculation of the fallback school scores.
52 Figure 5: Middle School Urban and Nonurban Charter School Effects over Time Notes: This graph shows charter school effects for urban (solid lines) and nonurban charters (dashed lines) in Massachusetts over time. The darker line in each pair shows mean scores for charter school compliers over time and the light line shows mean scores for noncharter compliers. Scores for compliers were calcuated using the methods from Abadie (2002, 2003). The gap between the lines is the second stange charter school effect at that grade level, using a dummy variable endogenous variable for charter school attendance. Percent proficient or above is the percentage of students who score at least 240 or higher on the scaled score of their state adminstered standadardized test (MCAS). Due to data availability, the following study is included in this figure: Massachusetts.
53 Table 2: Per-Year Lottery Estimated Charter School Attendance Effects for Subgroups Nonurban Urban ELA Math Math ELA (4) (2) (3) (1) 0.122*** -0.039 -0.046 0.228*** Male (0.049) (0.043) (0.042) (0.046) 8310 N 4020 4050 8180 -0.126*** 0.117*** 0.299*** -0.097** Female (0.045) (0.040) (0.045) (0.039) 4260 4230 8690 8800 N 0.126*** 0.107* 0.003 0.337*** Black/Hispanic (0.055) (0.062) (0.042) (0.046) 9460 9220 1140 1150 N -0.097*** 0.098* -0.005 -0.128*** White (0.032) (0.051) (0.059) (0.036) 7190 7130 3790 3830 N 0.289*** 0.022 0.003 0.123** Low Baseline Score (0.051) (0.050) (0.051) (0.052) 2090 N 4370 4380 2030 -0.130*** 0.250*** 0.100*** -0.180*** Not Low Baseline Score (0.029) (0.034) (0.034) (0.030) N 12200 5780 6080 11730 0.126* 0.075 0.315*** 0.156*** Subsidized Lunch (0.035) (0.066) (0.039) (0.062) 1320 11650 N 1340 11500 -0.107*** -0.130*** 0.042 0.171*** Not Subsidized Lunch (0.051) (0.057) (0.032) (0.037) N 5370 6930 6970 5460 0.246*** 0.117 0.025 -0.117 Special Education (0.074) (0.095) (0.073) (0.093) 1310 N 3120 3090 1330 -0.074** -0.108*** 0.123*** 0.277*** Not Special Education (0.032) (0.035) (0.030) (0.036) 6940 N 13990 13790 6990 0.204** -0.123 0.382*** 0.166 English Language Learner (0.090) (0.168) (ELL) (0.088) (0.142) 250 240 1390 1400 N 0.101*** -0.105*** -0.081*** 0.253*** Not-ELL (0.029) (0.035) (0.032) (0.033) 8000 15710 N 8070 15480 Notes: This table shows per-year two-stage least squares estimates of charter school impacts for various subgroups, by urban and nonurban schools. Standard errors are clustered by student and school by grade by year. The following studies are included in this figure: IES (Gleason et al. 2010) and Massachuestts (Angrist et al. 2013). Individual study results are estimated with the microdata. Since data security restrictions preclude combining the microdata from these two studies, the combined estimates are the inverse variance weighted average. Sample sizes are rounded to the nearest 10. *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level.
54 Table 3: Correlation between Lottery-Based Charter School Math Effects and Key Variables from Dobbie & Fryer (2013) IES, Massachusetts, and NYC ELA Math Single Single Variable Variable Multivariable Regressions Multivariable Regressions Regression Regression (6) (3) (7) (2) (1) (4) (5) (8) Teacher Feedback 0.023 0.050 0.104** 0.140** (0.047) (0.048) (0.047) (0.062) N 86 86 0.055 0.093 0.081* 0.106** Differentiated Instruction (Data Driven) (0.049) (0.055) (0.046) (0.072) N 82 82 0.027 0.146*** 0.071 0.078** Instructional Time (0.041) (0.038) (0.051) (0.049) N 86 86 0.136*** 0.073 0.260*** 0.153** High Quality Tutoring (0.050) (0.064) (0.069) (0.056) 86 N 86 0.072* 0.145** 0.100** 0.080* High Expectations (0.042) (0.057) (0.042) (0.047) N 86 86 0.109*** 0.142*** 0.110*** 0.064*** 0.067*** 0.064*** Index of Practice Inputs (0.020) (0.023) (0.020) (0.027) (0.027) (0.026) N 87 87 -0.079* 0.015 0.063 -0.053 Class Size (0.045) (0.037) (0.066) (0.047) 85 85 N 0.086** 0.089 0.030 -0.015 Per Pupil Expenditures (0.045) (0.041) (0.054) (0.055) 81 N 81 0.039 0.088*** 0.049 0.126*** Teachers with Masters (0.043) (0.040) (0.062) (0.034) 84 84 N -0.012 -0.034 0.034 -0.020 Teachers with Certification (0.043) (0.037) (0.044) (0.061) N 85 85 0.002 0.028 0.021 0.023 0.000 0.007 Index of Resource Inputs (0.019) (0.019) (0.026) (0.028) (0.025) (0.041) N 81 87 87 81 87 78 78 87 Notes: This table shows estimates from regressions of school characteristics on school-level charter school effect estimates. Columns (1) and (5) show results from single variable regressions; each coefficient comes from its own regression. Columns (2)-(4) and (6)-(8) show results from multivariate regressions, with the school characteristics included as indicated. Regressions are weighted by the inverse of the school-level standard error. Regressions include dummies for school levels (elementary, middle) as well as study fixed effects, and standard errors are clustered by the school level to account for schools with campuses at multiple grade levels. The following studies are included in this figure: IES (Gleason et al. 2010), Massachuestts (Angrist et al. 2013), and NYC (Dobbie and Fryer 2013). See Appendix Table 1 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale. See Appendix Table 2 for variable definitions across studies. *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level.
55 Table 4: Correlation between Lottery-Based Charter School Effects and Urbanicity, Counterfactual Mean, and School Inputs IES and Massachusetts (5) (6) (7) (2) (1) (3) (4) Panel A: Math Urban 0.111 0.113 0.280*** 0.170* (0.088) (0.116) (0.121) (0.076) -0.327*** -0.238*** -0.197** -0.197** Counterfactual Mean (0.080) (0.080) (0.076) (0.090) 0.131*** 0.064 0.065 Index of Practice Inputs (0.045) (0.047) (0.032) 0.015 0.008 Index of Resource Inputs (0.030) (0.047) 57 58 57 N 58 58 57 57 0.076 0.357 0.391 0.392 R-Squared 0.272 0.299 0.283 Panel B: ELA 0.052 0.145*** 0.090 0.048 Urban (0.072) (0.060) (0.054) (0.070) -0.084 -0.083 -0.120 -0.169** Counterfactual Mean (0.068) (0.080) (0.080) (0.076) 0.048 0.077*** 0.047 Index of Practice Inputs (0.033) (0.024) (0.034) -0.010 -0.007 Index of Resource Inputs (0.021) (0.028) 58 57 57 57 N 58 57 58 0.147 0.052 0.217 0.220 R-Squared 0.183 0.154 0.187 Notes: This table shows estimates from regressions of school characteristics on school-level charter school effect estimates. Columns (1) and (5) show results from single variable regressions; each coefficient comes from its own regression. Columns (2)-(4) and (6)-(8) show results from multivariate regressions, with the school characteristics included as indicated. Regressions are weighted by the inverse of the school-level standard error. Regressions include dummies for school levels (elementary, middle) as well as study fixed effects, and standard errors are clustered by the school level to account for schools with campuses at multiple grade levels. The following studies are included in this figure: IES (Gleason et al. 2010) and Massachuestts (Angrist et al. 2013). See Appendix Table 1 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale. See Appendix Table 2 for variable definitions across studies. *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level.
56 Table 5: Correlation between Lottery-Based Charter School Effects and Urbanicity, Counterfactual Mean, and Detailed School Inputs IES + Massachusetts (1) (3) (4) (5) (6) (7) (2) Panel A: Math Teacher Feedback 0.131** 0.066 (0.059) (0.064) 0.046 0.066 Differentiated Instruction (Data Driven) (0.066) (0.070) 0.072 -0.011 Instructional Time (0.078) (0.071) 0.185*** 0.153* High Quality Tutoring (0.091) (0.068) -0.013 -0.021 High Expectations (0.076) (0.079) 0.144* 0.120 High Suspensions (0.076) (0.083) 0.181* 0.114 0.104 0.184** 0.114 0.097 0.091 Urban (0.105) (0.082) (0.112) (0.088) (0.084) (0.089) (0.080) -0.223*** -0.250*** -0.204*** -0.220*** -0.272*** -0.240*** -0.242*** Counterfactual Mean (test scores in the fallback school) (0.088) (0.074) (0.080) (0.083) (0.086) (0.086) (0.092) 56 57 50 49 N 56 55 56 0.460 0.358 0.469 0.546 0.401 0.403 R-Squared 0.411 Panel B: ELA -0.063 0.017 Teacher Feedback (0.057) (0.071) 0.124** 0.071 Differentiated Instruction (Data Driven) (0.064) (0.054) 0.040 -0.009 Instructional Time (0.046) (0.064) 0.084 0.101 High Quality Tutoring (0.067) (0.105) 0.109 0.073 High Expectations (0.076) (0.071) 0.095 0.111 High Suspensions (0.062) (0.077) 0.055 0.047 -0.046 0.025 0.052 0.092 0.047 Urban (0.056) (0.069) (0.056) (0.062) (0.053) (0.053) (0.063) -0.117 -0.099 -0.185*** -0.154* -0.148** -0.124 -0.112 Counterfactual Mean (test scores in the fallback school) (0.083) (0.070) (0.087) (0.076) (0.079) (0.068) (0.078) 56 50 49 56 57 N 56 55 0.226 0.199 0.284 0.371 R-Squared 0.182 0.250 0.198 Notes: This table shows estimates from regressions of school characteristics on school-level charter school effect estimates. Regressions are weighted by the inverse of the school-level standard error. Regressions include dummies for school levels (elementary, middle) as well as study fixed effects, and standard errors are clustered by the school level to account for schools with campuses at multiple grade levels. The following studies are included in this figure: IES (Gleason et al. 2010) and Massachuestts (Angrist et al. 2013).. See Appendix Table 1 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale. See Appendix Table 2 for variable definitions across studies. *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level.
57 Appendix Table 1: Details on Studies that Contribute School-Level Data to Our Analyses Notes on Conversion of Lottery Estimates to Per-Year 2SLS Effects Paper Lottery Estimates Available School Characteristic Data Setting Abbreviation (3) (2) (6) (4) (1) (5) Panel A: Multi-Site Studies MA Massachusetts n/a Extensive survey and administrative Angrist, Pathak, and Walters (AEJ: 23 schools data Applied Economics, 2013) Extensive survey and administrative KIPP elementary 2SLS estimates calculated from raw KIPP 2015 8 schools Clark Tuttle, Gleason Knechtel, Nichols-Barrer, Booker, Chojnacki, data accessed through a data use schools data, currently not possible to link to school effects agreement with the KIPP Coen, and Goble (Mathematica, Foundation, divided by 3 to convert 2015) to per year estimates KIPP middle schools 12 schools Year 2 estimates divided by 2, then Clark Tuttle, Gill, Gleason, n/a KIPP 2013 divided by the overall first stage of Knechtel, Nichols-Barrer, Resch 0.6 (Mathematica, 2013) Dobbie and Fryer (AEJ: Applied NYC 29 schools n/a School characteristics summarized in 9 New York City Economics, 2013) dummy variables Furgeson, Gill, Haimson, Killewald, Index of 2 charter school practices: (1) Year 1 impacts divided by individual Charter schools that 6 sites (2 sites have 1 CMO intensive teacher coaching and (2) McCullough, Nichols-Barrer, Teh, were members of first stages, and in the case of one school each, 3 sites comprehensive behavior policy; only CMO's in 14 states have 3 schools each, Verbitsky-Savitz, Bowen, estimate that was a year 3 estimate, and 1 site has 8 in observational data Demeritt, Hill, and Lake divided by 3 schools) (Mathematica, 2012) Per year 2SLS estimates calculated Gleason, Clark, Clark Tuttle, from raw data accessed through an Extensive survey and administrative 15 unidentified data NCES restricted used data licsence states IES 35 schools Dwoyer, & Silverberg (IES, 2010) Panel B: Single-Site Studies UP 1 school n/a n/a Abdulkadiroglu, Angrist, Hull, and UP Academy Boston Pathak (NBER Working Paper, 2014) SEED 1 school n/a n/a Curto and Fryer (Journal of Labor SEED School Economics, 2014) Notes: Lottery estimates refers to per-year two-stage least squares (2SLS) estimates of charter school impacts, with calculation differences noted in the table. In the 2013 KIPP study, for the 3 schools missing year 2 outcomes, year 3 outcomes were substituted, dividing by 3 instead of 2 for the per year effect. Studies from Table 1 that are not included in our analyses are either studies that are superseded by another paper (for example, the Boston schools are included in the Massachusetts study, and the NYC 2009 study is replaced by the more recent 2013 study) or have characteristics that are incompatible with our analysis (Hoxby and Rockoff (2004) cannot be converted to standard deviations and Hastings et al. (2012) cannot be converted to per year effects).
58 Appendix Table 2: Definition of Variables across Studies IES Massachusetts NYC (3) (1) (2) Panel A: Variables for Table 3 = 1 if school has at least two requirements for =1 if new teachers are observed at least twice a teacher hires (temporary certification, full = 1 if school gives teachers feedback >=10 times month and veteran teachers are observed at certification, relevant major, graduate of per semester least once a month education program, and/or pass a test) Teacher Feedback = 1 if school administers >= 5 interim assessments & uses >= 4 differentiation =1 if school uses informal tests to gauge =1 if school uses ability grouping for some or all understanding (5 on scale of 1 to 5) strategies students in math or English Instruction (Data Driven/Differentiated) = 1 if school is held for more than 1260 hours = 1 if school is held for more than 1347 hours = 1 if school has 25% more instructional time (days*minutes per day) (days*minutes per day) than a traditional public school Instructional Time = 1 if all students in school participate in in- = 1 if tutoring >=4 times per week & tutoring school or after-school tutoring or school hires paid staff exclusively as tutors groups <= 6 =1 if school reports having a tutoring program High Quality Tutoring = 1 if school has uniforms & school informs = if school self reports being "No Excuses" (>=4 = 1 if school prioritizes high academic and on scale of 1 to 5) parents of bad behavior behavioral expectations for all students High Expectations Index of Practice Inputs Sum of the above 5 variables standardized to be mean 0, standard deviation 1. = 1 if class size < 13.61 = 1 if class size < 13 = 1 if class size < 11.65 Class Size = 1 if PPE > $15,000 = 1 if PPE > $7,160 = 1 if PPE > $12,345.5 Per Pupil Expenditures = 1 if if percent of teachers highly qualified > = 1 if percent of teachers with advanced degree > = 1 if if percent of teachers highly qualified > 96.83% 11% 84.95% Highly Qualified Teachers = 1 if percent of teachers certified > 65.95% = 1 if percent of teachers certified >85.71% = 1 if percent of certified teachers > 89%* Teachers with Certification Index of Resource Inputs Sum of the above 4 variables standardized to be mean 0, standard deviation 1. Panel B: Additional Variables for Tables 4 and 5 =1 if town is any of: Boston, Brockton, Cambridge, Chelsea, Chicopee, Everett, Fall River, Fitchburg, Framingham , Haverhill, Holyoke, Lawrence, Leominster, Lowell, Lynn, Malden, New Bedford, Pittsfield, Quincy, Revere, Urban n/a Somerville, Springfield, Taunton, or Worchester = 1 if NCES location code indicates "Large City" Average test score of the noncharter schools Average test score of the noncharter schools attended by lottery applicants in the year and attended by lottery applicants in the year and grade level after charter school application; grade level after charter school application; school averages are weighted by number of school averages are weighted by number of students attending and the test score is the students attending and the test score is the mean of standardized math and reading average mean of the math and ELA z-scores n/a proficiency levels Counterfactual Mean =1 if in top quartile of suspensions (suspension =1 if in top quartile of suspensions (suspension n/a rate > 17%) rate > 10%) High Suspensions Notes: *Reversed from Dobbie & Fryer so that all resource inputs are in the same direction. Inflection points are determined by the within-sample mean.
59 Appendix Table 3: Per-Year Lottery Estimates for Subgroups, Study-Specific Effects Nonurban Urban IES + MA MA IES IES + MA MA IES ELA Math ELA Math Math ELA Math ELA Math ELA Math ELA (10) (8) (7) (11) (6) (5) (4) (3) (2) (1) (9) (12) -0.073 -0.032 -0.042 0.122*** 0.228*** 0.168*** 0.265*** -0.039 -0.050 0.070 -0.034 -0.046 Male (0.049) (0.062) (0.037) (0.042) (0.043) (0.046) (0.042) (0.036) (0.038) (0.070) (0.053) (0.080) 1510 N 4050 2540 4020 2520 770 1500 8180 8310 7410 7540 770 -0.097** 0.085 0.351*** 0.129*** 0.299*** 0.117*** -0.117*** -0.070** -0.142*** -0.140*** 0.167*** -0.126*** Female (0.039) (0.061) (0.055) (0.038) (0.034) (0.045) (0.040) (0.040) (0.036) (0.054) (0.045) (0.045) 4230 7810 N 880 890 7920 8800 8690 1610 1630 2610 2630 4260 0.126*** 0.107* 0.003 -0.230 0.012 0.119** -0.241 0.337*** 0.211*** 0.421*** -0.040 0.155*** Black/Hispanic (0.055) (0.052) (0.040) (0.037) (0.046) (0.042) (0.054) (0.048) (0.285) (0.251) (0.058) (0.062) 240 1140 1150 N 1050 1050 8420 8180 9460 9220 900 910 240 -0.132*** 0.137** -0.097*** 0.042 -0.039 0.018 0.098* -0.005 -0.145*** -0.076*** -0.099** -0.128*** White (0.037) (0.032) (0.051) (0.059) (0.047) (0.054) (0.036) (0.058) (0.064) (0.029) (0.042) (0.032) 2500 7130 4670 7190 4630 2520 N 3790 3830 2850 2890 940 940 0.174*** 0.022 -0.039 0.370*** 0.264*** 0.289*** 0.123** 0.075* 0.093** -0.174** -0.154** 0.003 Low Baseline Score (0.052) (0.051) (0.056) (0.051) (0.047) (0.048) (0.051) (0.050) (0.044) (0.044) (0.070) (0.069) 1260 3680 2090 N 690 770 3610 4370 4380 840 840 1190 2030 0.100*** -0.105*** -0.106*** -0.180*** -0.148*** -0.224*** -0.130*** 0.250*** 0.110*** 0.278*** 0.046 0.118* Not Low Baseline Score (0.034) (0.029) (0.057) (0.028) (0.025) (0.034) (0.030) (0.031) (0.027) (0.041) (0.061) (0.032) 3510 3780 5780 6080 N 970 880 11230 10850 12200 11730 2270 2310 -0.028 0.156*** 0.126* 0.075 0.262*** 0.071 0.331*** 0.181*** 0.315*** 0.152*** 0.096* -0.014 Subsidized Lunch (0.133) (0.062) (0.035) (0.106) (0.039) (0.030) (0.032) (0.055) (0.058) (0.052) (0.055) (0.066) 540 800 790 11500 11650 10550 10700 1320 950 950 N 520 1340 0.042 -0.130*** -0.107*** -0.084* -0.106*** -0.157*** -0.108*** 0.171*** 0.088** 0.243*** -0.065 -0.003 Not Subsidized Lunch (0.032) (0.067) (0.049) (0.044) (0.057) (0.051) (0.034) (0.030) (0.044) (0.037) (0.076) (0.037) 4640 6930 6970 N 710 710 4760 4670 5460 5370 2320 2340 4610 -0.131 0.244*** -0.117 0.250*** 0.045 0.155** 0.246*** 0.117 0.035 -0.111 0.007 0.025 Special Education (0.117) (0.116) (0.095) (0.082) (0.084) (0.093) (0.073) (0.065) (0.064) (0.090) (0.092) (0.074) N 1310 1330 410 400 2710 2680 3120 3090 410 420 890 910 -0.097** -0.090*** -0.108*** -0.074** 0.141*** 0.030 0.321*** 0.152*** 0.277*** 0.123*** -0.114*** -0.064** Not Special Education (0.043) (0.035) (0.030) (0.032) (0.054) (0.048) (0.031) (0.027) (0.036) (0.031) (0.027) (0.034) 4240 2700 6990 13790 4270 13990 12530 12750 1250 1250 N 2720 6940 0.204** 0.213 0.618 0.166 -0.124 0.164 -0.123 0.382*** 0.258*** 0.415*** 0.069 0.264* ELL (0.168) (0.142) (0.121) (0.074) (0.077) (0.088) (0.090) (0.158) (0.136) (2.187) (0.139) (2.727) 50 50 240 250 N 210 210 1190 1180 1400 1390 190 190 -0.091** 0.101*** -0.105*** -0.081*** 0.148*** 0.014 0.291*** 0.132*** 0.253*** -0.112*** -0.065** -0.110*** Not-ELL (0.050) (0.029) (0.026) (0.035) (0.042) (0.033) (0.045) (0.030) (0.027) (0.035) (0.032) (0.029) 5080 5120 8000 N 1440 1450 14270 14030 15710 15480 2920 2950 8070 Notes: This table shows per-year 2SLS estimates of charter school impacts for various subgroups, by urban and nonurban schools. Standard errors are clustered by student and school by grad by year. The following studies are included in this figure: IES and Massachuestts. Individual study results are estimated with the microdata. Since data security restrictions preclude combining the microdata from these two studies, the combined estimates are the inverse variance weighted average. Sample sizes are rounded to the nearest 10. *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level.
60 Appendix Table 4A: Correlation between Lottery-Based Charter School Math Effects and Key Variables from Dobbie & Fryer, Study Specific Effects Multivariable Regression: Practices + Resource Index Single Variable Regressions Multivariable Regression: Both Indicies Multivariable Regression: Resources + Practice Index MA NYC All IES MA NYC All IES MA NYC All IES NYC All IES MA (6) (4) (9) (10) (11) (12) (13) (14) (15) (16) (3) (2) (1) (7) (5) (8) 0.162* 0.216** 0.140** 0.141** 0.265** -0.013 Teacher Feedback 0.059 0.104** (0.047) (0.086) (0.107) (0.062) (0.058) (0.127) (0.105) (0.090) N 35 22 86 29 0.055 0.253 0.093 0.017 0.016 0.012 -0.045 0.138 Differentiated Instruction (Data Driven) (0.091) (0.163) (0.062) (0.072) (0.087) (0.134) (0.073) (0.055) N 82 26 22 34 -0.124 0.324*** 0.046 0.039 0.146*** 0.081 0.167 0.071 Instructional Time (0.119) (0.049) (0.095) (0.096) (0.043) (0.051) (0.112) (0.123) N 34 23 29 86 0.153** -0.003 0.237** 0.260*** 0.212** 0.085 -0.069 0.332*** High Quality Tutoring (0.094) (0.100) (0.064) (0.047) (0.092) (0.099) (0.069) (0.142) N 34 86 29 23 0.070 0.080* 0.075 0.145** 0.281** 0.057 0.121 0.118 High Expectations (0.104) (0.047) (0.092) (0.110) (0.060) (0.057) (0.097) (0.147) N 35 86 28 23 0.169*** 0.071 0.142*** 0.040 0.157*** 0.048* 0.110*** 0.042 0.050** 0.109*** 0.124 0.157*** Index of Practice Inputs (0.061) (0.045) (0.027) (0.105) (0.027) (0.025) (0.099) (0.027) (0.023) (0.026) (0.086) (0.027) 87 N 35 23 29 0.147* 0.015 0.063 -0.095 0.113 -0.055 0.153* -0.019 Class Size (0.045) (0.066) (0.137) (0.081) (0.089) (0.080) (0.068) (0.135) N 33 23 29 85 0.219* -0.015 -0.005 0.089 -0.033 -0.032 0.053 -0.037 Per Pupil Expenditures (0.080) (0.054) (0.099) (0.055) (0.056) (0.118) (0.098) (0.094) 22 N 81 30 29 0.182** 0.058 0.126*** 0.060 0.034 0.025 0.039 0.040 Teachers with Masters (0.082) (0.089) (0.040) (0.093) (0.135) (0.065) (0.090) (0.062) 29 23 32 N 84 0.096 0.128 0.082 0.034 -0.247** 0.105* -0.020 -0.019 Teachers with Certification (0.143) (0.067) (0.044) (0.061) (0.088) (0.112) (0.061) (0.082) . 29 23 33 N 85 0.023 0.017 0.028 0.040 0.003 0.046 0.051 0.057* -0.013 0.021 0.048 0.040 Index of Resource Inputs (0.044) (0.061) (0.026) (0.040) (0.038) (0.047) (0.028) (0.041) (0.084) (0.046) (0.041) (0.034) 29 34 25 81 27 22 29 78 35 23 22 87 35 23 29 87 N Notes: This table shows estimates from regressions of school characteristics on school-level charter school effect estimates. Columns (1) and (5) show results from single variable regressions; each coefficient comes from its own regression. Columns (2)-(4) and (6)-(8) show results from multivariate regressions, with the school characteristics included as indicated. Regressions are weighted by the inverse of the school-level standard error. Regressions include dummies for school levels (elementary, middle) as well as study fixed effects, and standard errors are clustered by the school level to account for schools with campuses at multiple grade levels. The following studies are included in this figure: IES, Massachuestts, and NYC. See Appendix Table 1 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale. See Appendix Table 2 for variable definitions across studies.
61 Appendix Table 4B: Correlation between Lottery-Based Charter School ELA Effects and Key Variables from Dobbie & Fryer, Study Specific Effects Multivariable Regression: Both Indicies Multivariable Regression: Resources + Practice Index Multivariable Regression: Practices + Resource Index Single Variable Regressions MA NYC All IES MA NYC All IES MA NYC All IES NYC All IES MA (6) (4) (9) (10) (11) (12) (13) (14) (15) (16) (3) (2) (1) (7) (5) (8) 0.118** 0.201** 0.050 0.115** 0.153* -0.121 Teacher Feedback -0.117 0.023 (0.048) (0.052) (0.099) (0.047) (0.055) (0.092) (0.085) (0.094) N 35 22 86 29 0.081* 0.244*** 0.106** 0.069 0.080 0.013 -0.047 0.070 Differentiated Instruction (Data Driven) (0.075) (0.080) (0.057) (0.049) (0.081) (0.123) (0.082) (0.046) N 82 26 22 34 -0.173 0.177** 0.024 0.024 0.078** -0.005 0.038 0.027 Instructional Time (0.132) (0.041) (0.078) (0.071) (0.039) (0.038) (0.092) (0.070) N 34 23 29 86 0.073 -0.005 0.142 0.136*** 0.136 -0.018 -0.059 0.164** High Quality Tutoring (0.078) (0.101) (0.050) (0.033) (0.066) (0.105) (0.056) (0.094) N 34 86 29 23 0.025 0.072* 0.015 0.100** 0.264*** 0.194*** 0.141 0.052 High Expectations (0.104) (0.042) (0.079) (0.055) (0.050) (0.042) (0.082) (0.053) N 35 86 28 23 0.106*** 0.056 0.067*** 0.002 0.104*** 0.029 0.064*** 0.002 0.030 0.064*** 0.094 0.103*** Index of Practice Inputs (0.024) (0.046) (0.023) (0.063) (0.024) (0.026) (0.062) (0.022) (0.026) (0.020) (0.074) (0.020) 87 N 35 23 29 -0.095 -0.079* -0.053 -0.081 -0.007 -0.055 -0.060 -0.093 Class Size (0.037) (0.047) (0.092) (0.081) (0.081) (0.075) (0.062) (0.066) N 33 23 29 85 0.130 0.030 -0.033 0.086** 0.228*** -0.042 0.012 0.123 Per Pupil Expenditures (0.076) (0.045) (0.039) (0.041) (0.054) (0.081) (0.080) (0.085) 22 N 81 30 29 0.040 0.040 0.088*** 0.108 0.017 0.029 0.049 0.051 Teachers with Masters (0.057) (0.074) (0.034) (0.083) (0.083) (0.054) (0.086) (0.043) 29 23 32 N 84 -0.152* -0.064 0.072 -0.012 -0.131* 0.095** -0.034 0.000 Teachers with Certification (0.069) (0.061) (0.037) (0.043) (0.083) (0.075) (0.049) (0.083) . 29 23 33 N 85 0.002 0.004 0.007 0.009 -0.016 0.031 0.060 -0.001 -0.026 0.000 0.032 0.009 Index of Resource Inputs (0.035) (0.053) (0.019) (0.026) (0.028) (0.040) (0.019) (0.025) (0.051) (0.040) (0.025) (0.025) 29 34 25 81 27 22 29 78 35 23 22 87 35 23 29 87 N Notes: This table shows estimates from regressions of school characteristics on school-level charter school effect estimates. Columns (1) and (5) show results from single variable regressions; each coefficient comes from its own regression. Columns (2)-(4) and (6)-(8) show results from multivariate regressions, with the school characteristics included as indicated. Regressions are weighted by the inverse of the school-level standard error. Regressions include dummies for school levels (elementary, middle) as well as study fixed effects, and standard errors are clustered by the school level to account for schools with campuses at multiple grade levels. The following studies are included in this figure: IES, Massachuestts, and NYC. See Appendix Table 1 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale. See Appendix Table 2 for variable definitions across studies.
62 Appendix Table 5: Correlation between Lottery-Based Charter School Effects and Urbanicity, Counterfactual Mean, and School Inputs, Study Specific Effects Multivariable Regression: Urban, Multivariable Regression: Urban, Counterfactual Mean, Index of Practice Counterfactual Mean, and Index of Multivariable Regression: Urban and Inputs, and Index of Resource Inputs Practice Inputs Counterfactual Mean Single Variable Regressions MA All IES All IES MA All IES MA All IES MA (5) (11) (8) (12) (7) (6) (9) (4) (3) (2) (1) (10) Panel A: Math Urban 0.111 0.149 0.113 -0.061 0.155 0.170* -0.020 0.155 0.280*** 0.315*** 0.234** -0.061 (0.105) (0.088) (0.255) (0.136) (0.116) (0.076) (0.121) (0.120) (0.257) (0.124) (0.111) (0.248) 23 58 35 N 0.272 0.361 0.117 R-Squared -0.227*** -0.629*** -0.327*** -0.191** -0.658* -0.238*** -0.194** -0.304 -0.197** -0.194** -0.303 -0.197** Counterfactual Mean (0.390) (0.375) (0.073) (0.080) (0.394) (0.086) (0.080) (0.086) (0.085) (0.090) (0.088) (0.076) 34 57 23 N R-Squared 0.168 0.513 0.299 0.115** 0.042 0.157*** 0.131*** -0.012 0.064 -0.011 0.115** 0.065 Index of Practice Inputs (0.091) (0.052) (0.047) (0.032) (0.051) (0.045) (0.099) (0.027) (0.087) N 35 23 58 R-Squared 0.010 0.573 0.283 0.014 0.015 0.008 0.002 0.040 -0.013 Index of Resource Inputs (0.041) (0.084) (0.047) (0.042) (0.038) (0.030) 23 57 34 57 34 57 23 34 58 23 35 N 23 0.595 0.392 R-Squared 0.026 0.027 0.076 0.217 0.513 0.357 0.218 0.595 0.391 0.221 Panel B: ELA 0.148 0.022 0.245*** 0.145*** -0.020 0.188 0.090 0.148 0.048 -0.018 -0.019 0.052 Urban (0.072) (0.070) (0.114) (0.083) (0.060) (0.144) (0.083) (0.054) (0.058) (0.082) (0.090) (0.121) 35 23 58 N R-Squared 0.002 0.440 0.147 0.168 -0.083 -0.115 -0.115 0.167 -0.120 -0.084 -0.169** -0.100 -0.373*** -0.110 -0.106 Counterfactual Mean (0.063) (0.068) (0.078) (0.193) (0.076) (0.083) (0.234) (0.080) (0.085) (0.236) (0.080) (0.078) 57 23 34 N 0.048 0.154 0.384 R-Squared 0.002 0.048 0.093* -0.023 0.092* 0.047 -0.023 0.077*** 0.104*** Index of Practice Inputs (0.068) (0.050) (0.033) (0.069) (0.051) (0.034) (0.062) (0.022) (0.024) 35 N 23 58 R-Squared 0.000 0.516 0.187 -0.010 0.001 0.009 -0.026 -0.007 -0.017 Index of Resource Inputs (0.051) (0.021) (0.026) (0.028) (0.028) (0.025) 57 34 57 34 23 34 57 58 23 35 23 N 23 0.220 0.002 0.074 0.052 0.050 0.449 0.183 0.055 0.555 0.217 0.055 R-Squared 0.563 Notes: This table shows estimates from regressions of school characteristics on school-level charter school effect estimates. Columns (1) and (5) show results from single variable regressions; each coefficient comes from its own regression. Columns (2)-(4) and (6)-(8) show results from multivariate regressions, with the school characteristics included as indicated. Regressions are weighted by the inverse of the school-level standard error. Regressions include dummies for school levels (elementary, middle) as well as study fixed effects, and standard errors are clustered by the school level to account for schools with campuses at multiple grade levels. The following studies are included in this figure: IES and Massachuestts. See Appendix Table 1 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale. See Appendix Table 2 for variable definitions across studies.
63 Appendix Table 6: Correlation between Lottery-Based Charter School Effects and Urbanicity, Counterfactual Mean, and Detailed School Inputs, Study-Specific Results IES Massachusetts IES + Massachusetts (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (1) (4) (5) (2) (3) Panel A: Math 0.131** 0.142 0.060 0.072 0.066 0.080 Teacher Feedback (0.059) (0.064) (0.143) (0.109) (0.087) (0.094) 0.028 0.066 0.046 0.041 0.064 0.068 Differentiated Instruction (Data Driven) (0.074) (0.083) (0.101) (0.241) (0.070) (0.066) -0.075 -0.064 -0.011 0.153 0.072 0.079 Instructional Time (0.078) (0.146) (0.109) (0.122) (0.084) (0.071) 0.163** 0.156 0.147 0.141 0.185*** 0.153* High Quality Tutoring (0.196) (0.091) (0.068) (0.104) (0.095) (0.072) -0.009 -0.013 -0.021 -0.017 0.011 -0.043 High Expectations (0.088) (0.117) (0.108) (0.134) (0.079) (0.076) 0.182 0.075 0.144* 0.174 0.120 0.092 High Suspensions (0.076) (0.083) (0.131) (0.116) (0.114) (0.167) 0.134 0.138 0.080 -0.053 -0.014 0.039 -0.030 -0.007 0.126 0.184** 0.114 0.104 0.097 0.181* 0.173 0.091 0.080 0.100 0.072 0.164 0.114 Urban (0.161) (0.284) (0.241) (0.252) (0.231) (0.242) (0.270) (0.262) (0.088) (0.084) (0.089) (0.080) (0.105) (0.082) (0.112) (0.130) (0.123) (0.099) (0.105) (0.099) (0.110) -0.223*** -0.223** -0.670* -0.470 -0.413 -0.658* -0.552 -0.042 -0.220*** -0.272*** -0.240*** -0.425 -0.242*** -0.250*** -0.204*** -0.198** -0.224*** -0.235*** -0.194** -0.223*** -0.208** Counterfactual Mean (0.074) (0.438) (0.390) (0.382) (0.351) (0.387) (0.451) (0.579) (0.083) (0.086) (0.085) (0.080) (0.088) (0.086) (0.074) (0.085) (0.085) (0.081) (0.083) (0.089) (0.092) 34 29 22 22 23 23 23 21 20 56 29 56 56 57 50 49 33 33 33 34 N 55 0.222 0.349 0.422 0.573 0.529 0.563 0.550 0.513 0.456 0.567 0.411 0.403 0.401 0.460 0.358 0.469 0.546 R-Squared 0.230 0.245 0.250 0.286 Panel B: ELA 0.017 -0.120 -0.185* 0.124 0.083 -0.063 Teacher Feedback (0.071) (0.076) (0.099) (0.084) (0.114) (0.057) 0.071 0.124** 0.015 0.108 0.071 0.080 Differentiated Instruction (Data Driven) (0.054) (0.064) (0.073) (0.086) (0.164) (0.092) 0.008 -0.009 0.040 0.006 0.060 -0.059 Instructional Time (0.085) (0.074) (0.064) (0.072) (0.046) (0.091) 0.164 0.059 0.071 0.101 0.084 0.133 High Quality Tutoring (0.105) (0.067) (0.141) (0.179) (0.083) (0.120) 0.084 0.109 0.073 0.196 0.214* -0.016 High Expectations (0.118) (0.157) (0.088) (0.076) (0.092) (0.071) 0.111 0.045 0.049 0.124 0.100 0.095 High Suspensions (0.110) (0.077) (0.062) (0.109) (0.086) (0.107) 0.107 0.269* 0.128 0.187 0.211 0.005 0.198 -0.049 0.092 0.025 0.052 0.047 0.055 0.047 -0.046 -0.073 -0.074 -0.080 -0.018 -0.042 -0.087 Urban (0.076) (0.163) (0.145) (0.135) (0.134) (0.154) (0.130) (0.213) (0.063) (0.053) (0.078) (0.056) (0.069) (0.056) (0.062) (0.070) (0.069) (0.066) (0.086) (0.068) (0.053) -0.110 -0.190** 0.083 -0.125 -0.041 -0.013 -0.098 0.041 0.287 -0.117 -0.148** -0.124 -0.112 -0.099 -0.185*** -0.154* -0.105 -0.136* -0.125* -0.116 -0.197*** Counterfactual Mean (0.078) (0.196) (0.230) (0.213) (0.195) (0.221) (0.293) (0.079) (0.068) (0.076) (0.221) (0.083) (0.070) (0.087) (0.081) (0.095) (0.074) (0.086) (0.072) (0.073) (0.077) (29.000) (29.000) (22.000) (22.000) (23.000) (23.000) (23.000) (21.000) (20.000) (56.000) (55.000) (56.000) (56.000) (57.000) N (49.000) (34.000) (33.000) (33.000) (33.000) (34.000) (50.000) 0.051 0.353 0.524 0.478 0.463 0.460 0.510 0.456 0.581 0.182 0.250 0.198 0.226 0.172 0.284 0.371 R-Squared 0.105 0.105 0.078 0.118 0.199 Notes: This table shows estimates from regressions of school characteristics on school-level charter school effect estimates. Regressions are weighted by the inverse of the school-level standard error. Regressions include dummies for school levels (elementary, middle) as well as study fixed effects, and standard errors are clustered by the school level to account for schools with campuses at multiple grade levels. The following studies are included in this figure: IES and Massachuestts. See Appendix Table 1 for details on these studies and for notes on modifications of published point estimates which put estimates on the same scale. See Appendix Table 2 for variable definitions across studies. *** Significant at the 1 percent level. ** Significant at the 5 percent level. * Significant at the 10 percent level.