Basile Brief

Transcript

1 TUDY F MPRESSION : H OW A W IDELY C ITED S I V ASTLY O VERSTATES THE ALSE ENEFITS OF C HARTER S CHOOLS B B M ARCO B ASILE Y One significant change in American education in rec ent years has been the proliferation of charter sch ools throughout the country. Although charters are publi cly funded, they are allowed to operate independent ly from traditional public school systems while abidin g by rules that vary from state to state. Advocates of charters argue that their independence enables them to innovate and be more flexible in serving their students. Many charter supporters also believe that , by relying on teachers who in most cases are not unionized, better results will arise, in part becau se it is easier to fire ineffective non-unionized i nstructors es. unprotected by tenure and due process dismissal rul For all of the attention paid to charter schools, t hey still constitute only a very small segment of t he U.S. educational system. Just 1.5 million of the nation’ ugh in s 56 million students attend charter schools, altho some places—especially inner city communities—their penetration is greater: in New Orleans, for 1 example, 57 percent of students attend charter scho ols. The Obama administration strongly supports 2 4 billion in Its “Race to the Top” initiative, which provides $ expanding charter school attendance. additional federal funds for education, included a ely number of provisions intended to induce states to r more heavily on charter schools. President Obama has said that in all realms of publ ic policy, including education, he wants to build o n ideas that have demonstrated their effectiveness. B ut charter schools, which have been studied extensi vely, remain largely unproven. This issue brief focuses o n one particular report, released in September 2009 , that has been widely cited by charter school advoca In tes because it appears to show remarkable results. the report, “How New York City’s Charter Schools Af fect Achievement,” Stanford University professor Caroline Hoxby and her colleagues Sonali Murarka an d Jenny Kang probe the academic achievement of 30,000 New York City students who had applied to ch arter schools and had been randomly separated by lotteries into charter schools (the “lotteried-in” students) and traditional schools (the “lotteried-o ut” 3 students). Hoxby and her colleagues made the headline-grabbing assertion that, on average, for students that attended from kindergarten through grade eight, New York City charter schools could close the “Scarsdale-Harlem gap”—that is, the achievement gap between students in Harlem and students in the much more affluent suburb of Scarsdale—by 66 percen t in English and 86 percent in math. This is a shocking finding that, if true, would suggest that by’s charters could be a magic bullet after all. But Hox colleague at Stanford, Sean Reardon, the education y, researcher and expert in social sciences methodolog The Century Foundation conducts public policy resea olicy issues, including inequality, retirement secu rity, rch and analyses of economic, social, and foreign p sk books, reports, and other publications, convenes ta and international affairs. The foundation produces election reform, media studies, homeland security, nd Washington, D.C., The Century Foundation is mational Web sites. With offices in New York City a forces and working groups, and operates eight infor nonprofit and nonpartisan and was founded in 1919 b y Edward A. Filene. ) FAX ( 212.535.7534 – 212.535.4441 – 10021 NY , . Y EW N – TREET S TH 70 AST E 41 : EADQUARTERS H ORG . TCF WWW – ORG . TCF @ INFO – ORK TH INFO – ) FAX ( 202.483.9430 – 202.387.0400 – 20005 D.C. , ASHINGTON W – LOOR F TCF 10 – NW , TREET S H 1333 : OFFICE D.C. ORG – @ . . TCF . ORG WWW

2 4 scrutinized Hoxby’s report and uncovered serious de sign flaws in the study. Reardon’s analysis largely tracted so much media attention. undercuts the claims of dramatic gains that have at R TO -O RANGES : S TUDY ’ S D ESIGN A ESTROYS THE PPLES ANDOMIZATION OF I TS S AMPLE - D royed the way in which the lottery randomly Reardon found that the design of Hoxby’s study dest separated students into charter schools and traditi onal schools. This flaw means that the study design ultimately is self-defeating, because the randomiza tion of students is precisely what made the study placed each student into a charter school or a trad itional promising in the first place. The lottery randomly of students to compare whose only pertinent school and gave Hoxby and her colleagues two groups dy’s in or lotteried-out of a charter school. In the stu difference was whether the students were lotteried- pples’ comparison.” authors’ words, it constituted “a true ‘apples-to-a For the study’s analysis of kindergarten through th ird grade, which found relatively modest improvemen ts d in the charter schools, the comparison indeed works legitimately. The problems start with how Hoxby an and traditional schools from fourth through eighth her co-authors designed their comparison of charter group of students at each grade level while grades. The study compares test scores between each is means that it measures the difference in test sc ores controlling for the previous year’s test scores. Th between charter students and students attending tra ditional schools who had the same test score the and this is the critical point—is that the previous previous year. The problem that Reardon identifies— after the lottery. The assumption that the initial rando mization of the subjects of the year’s test took place study via lottery would persist in a comparison of charter school students and traditional school stud ents who scored the same on a test in a certain grade is a flawed assumption, because it ignores the fact t hat the t in the years that led up to that test score. As a result experiences of these two groups were quite differen and of these different experiences, it can no longer be assumed properly that the charter school students way—as they were at the time of the original rando m the traditional students are still similar in every separation by lottery. Charter students and traditi onal students who scored the same on a test in the ounterfactuals for one another”: we cannot assume t hat previous year are not, in Reardon’s words, “valid c d to traditional schools, would perform similarly a the students in charter schools, if they transferre s traditional students who began the year at the same level of academic achievement. A valid study would ional students since the moment of their original compare the progress of charter students and tradit random separation and not xposure to different educational since an arbitrary point in time years after the e environments. Imagine a medical experiment on two pills designed to cure the common cold. Upon arrival at the lab on Monday, subjects with similar symptoms are separate d randomly into two groups. One group is given square pills, and the other group is given round pi lls. In order to understand the overall comparative e experiment would want to observe the changes in t effects of the pills, the researchers conducting th he from the moment they split into two groups and took their respective pills . If, after an initial subjects’ symptoms make additional observations over the weekend, they observation on Thursday, the researchers wanted to would compare the progress of the two groups since Monday; they would not look at subjects who had similar symptoms on Thursday and then compare the s quare pill takers with the round pill takers within that group. Indeed, at the time of observation on T even if hursday, round pill takers and square pill takers, they have similar symptoms at that time, are no lon ger separated randomly—their experiences between Monday and Thursday have been different. To assume the randomization persists until Thursday is to ignore any factors that had come into play since Mo nday. For example, the square pill takers might hav e developed false confidence from their better-promot ed pill between Monday and Thursday, which led them to make riskier health decisions after Thursda y. To have as accurate an assessment of the pill’s n the researchers would compare the progress of one cumulative efficacy as possible, at each observatio . since Monday group to that of the other based on changes 2 www.tcf.org

3 Notably, to return to the New York City charter sch ools, Hoxby and colleagues write that the “two grou ps f the lottery. They are not identical just on dimen of students are essentially identical at the time o sions that we can readily observe, such as race, ethnicity, ge nder, poverty, limited English, and disability. They are also identical on dimensions that we cannot readily obse rve like motivation and their family’s interest in e ducation ” (emphasis sentially identical at the time of the lottery mine). Precisely: the two groups of students are es . However, when the scores of charter school students are compared de to the scores of traditional school students in gra students—as a result of having been divided into di fferent seven with identical test scores in grade six, the nvironments leading up to the grade six exam—will school environments and having developed in those e no longer be identical. If the charter school had a ny effect on achievement, motivation, confidence, o r any other factor prior to the grade six test, then comp aring students with similar achievement in grade si x is not the same as conducting a randomized experiment. As a result, Reardon shows, the observed effect of a ed, mostly because a student’s test score that year is a charter school during grade seven will be exaggerat result of exposure to the charter school not just d student uring grade seven, but for all the years since the was lotteried-in to the charter school. If charter schools have a positive effect on achievement, then a study with this design defect would exaggerate the positive effect (it also would exaggerate a negativ e effect). Reardon suggests that this might explain w hy Hoxby and colleagues’ results for fourth through eighth grades were so much more dramatic (that is, two to three times greater) than the kindergarten through third grade results, which do not suffer fr om this bias. In other words, if the charter school s have a modestly positive impact, this design flaw would exaggerate that impact into one that was seemingly more dramatic and significant. OTALED P A NNUAL S TUDENT A CHIEVEMENT G AINS N OT T ROPERLY A second problem Reardon finds with the design of t he Hoxby study for grades four through eight is its ill continue to make the same amount of academic ga implicit assumption that a charter school student w in er. But we know that this is not the case: a studen t who each year throughout the student’s educational care is not equally likely to make that same amount of makes significant academic progress in a given year progress the next year. Reardon cites data that sug gest that only 76 percent to 80 percent of a New Yo rk 5 ing academic year. (Other research suggests that the City student’s achievement is replicated the follow fade-out of academic gains might be even more drama tic: in a study on the persistence of academic gain s resulting from higher teacher quality, Brian Jacob and colleagues concluded that roughly 20 percent—an d no more than 33 percent—of achievement gains from t his educational intervention persisted into the nex t 6 year.) Thus, in order to estimate a charter school’s overa ll effect on a student’s academic career, a given y ear’s academic gain must be “discounted” before adding or multiplying it into the larger arithmetic of the overall estimate. Hoxby measured the annual h grades, effect of charter schools for fourth through eight ulative effect. Her study assumes that the progress a and then added these annual gains to estimate a cum ar in a charter school will be made each year that student given student makes in, say, the student’s first ye remains in the charter system, and thus that each g rade’s average gains simply can be added to get the cumulative effect. However, as noted, research sugg ests that a given student’s academic progress is no t constant. Especially because the study does not fol th low a single cohort of students from fourth to eigh grade, adding the average gains of each year does n ot offer a realistic picture of a particular studen t’s academic progress over four years. More realistical ly, students who make big gains one year find it mo re difficult to replicate this success in a later year . Reardon illustrates that this design problem migh t lead to an overestimation of the academic gains observed be tween fourth and eighth grades by as much as 50 percent. he the study was a truly longitudinal one, following t The above two concerns would not be problematic if initial lottery before kindergarten through eighth performance of a single cohort of students from an 3 www.tcf.org

4 grade. However, the data for the majority of subjec ts in the study were for students who had been in charter schools for only three or four years. ETHODOLOGICAL C F M URTHER ONCERNS hlights further points of concern: Reardon’s review of Hoxby and colleagues’ study hig Hoxby and colleagues’ study reports very large gain s in science and social studies as well as • or students in charter schools. (Again, as with increased probability of graduation by age twenty f the math and English gains, the study did not follo w a cohort of students through eighth grade, let alone graduation, so the gains are an extrapolated estimate based on adding each year’s average gains.) However, by the standard conventions of soc ial science, these findings are deemed not to be statistically significant. In other words, it is too likely that the observations leading to the extrapolated projections were due to chance rather than being caused by students’ attendance at charter schools. • It is likely that several ineffective charter schoo ls are not included in the study’s findings. This i s due to the fact that those schools whose effects ar e imprecisely measured, and hence are omitted, tend to be ones with small numbers of lotteried-in students, such as newer and smaller schools. • The model used by Hoxby would give more weight in i ts estimations to the academic performance of students in heavily oversubscribed charter schoo ls than students in less-oversubscribed schools. If the most effective charter schools tend to have more applicants (as market competition theory would predict), then the study’s findings may be di sproportionately weighted by these more 7 effective charter schools. s of subscription at the charter schools, the More generally, beyond the need to clarify the rate • study also lacks sufficiently detailed information about the students who participated in the have attended if not lotteried-in, the proportions lotteries, the schools that charter students would at which lotteried-in and lotteried-out students re main in their respective schools, and other key factors. Without more detailed information about th e students who participated in the lotteries, it ’s findings about New York City charter becomes more difficult to generalize from the study schools to the country at large. And if academicall y stronger students who were lotteried-out chose n the Hoxby study would have compared charter private schools instead of traditional schools, the et of lotteried-out students. Further, if school students only to an academically weaker subs students for whom charter schools are more effectiv e are more likely to remain in charter schools than their peers for whom charter schools are less effective, then the charter schools’ academic gains would be overstated because the estimates wou ld favor the former group disproportionately. In short, without more information about the source s of Hoxby’s data, it remains unclear what conclusions these data suggest. E XISTING C HARTER S CHOOL R ESEARCH S UGGESTS M ESSIER R ESULTS Reardon’s vigilant review constitutes a warning to policymakers and educators about rushing to Hoxby and colleagues’ study as a definitive account of ch arter schools’ effects. More information and that charter schools will still show a positive eff investigation is needed. And although it is likely ect on student achievement after these issues are addresse d, it seems that the estimated effect will be much heir ch on charter schools paints a messier picture of t smaller and more ambiguous. Indeed, existing resear results: 4 www.tcf.org

5 • mes (CREDO), also at Stanford University, In 2009, the Center for Research on Education Outco found that charter schools in fifteen states and th e District of Columbia had a positive impact on ools had no impact in 46 percent of the math gains only in 17 percent of cases. Charter sch observations, and had a negative impact 37 percent of the time. The study explored data from 70 attend one of 2,403 charter schools, roughly percent of all charter students in the country who 8 half of the country’s charter schools. A 2009 study by Thomas Kane and the Boston Foundati • on compared lotteried-in students at o attended traditional schools. Kane and Boston charter schools to lotteried-out students wh colleagues concluded that charter schools had a pos itive impact on student achievement in eighth and tenth grade math. However, because only seven o f 29 charter schools were popular enough to require a substantial lottery, the study include d only Boston’s most successful charter schools 9 (as suggested by the proxy of oversubscription rate s). A 2006 study comparing the performance of charter s chool students to public school students on • controlling for demographic factors— the 2003 NAEP math assessment concluded that—after charter school students performed at the same level or, in some cases, below the level of their 10 public school peers. The RAND Corporation determined in 2008 that there was no statistically significant difference • tudents in Philadelphia and their peers at between the academic gains made by charter school s 11 traditional schools. Even the achievement of the country’s seemingly mos • t successful charter network, the Knowledge Is Power Program (KIPP), is uncertain, gi ven research highlighting KIPP’s high attrition rate. Researchers found that 60 percent o f students who began attending a KIPP school 12 by the end of eighth grade. in the San Francisco Bay Area were no longer there Because these other studies indicate that charter s chools’ effects are, at best, mixed, and because of the methodological concerns raised by Reardon, readers should be highly skeptical of Hoxby and her colleagues’ astonishing claim that New York City ch arter school students who attended for kindergarten through eighth grade would close the “Scarsdale-Har lem gap” by 66 percent in English and 86 percent in math. W RITTEN B Y MARCO B ASILE , A P ROGRAM A SSISTANT AT T HE C ENTURY F OUNDATION M 14, 2010 AY T HE C ENTURY F OUNDATION IS PUBLISHING THE I SSUE B RIEF SERIES TO HELP EXPLAIN AND CALL ATTENTION TO P UBLIC POLICY IDEAS THAT ARE WORTHY OF DISCUSSION AND DEBATE . T HE VIEWS EXPRESSED IN THIS SERIES ARE SOLELY THOSE OF THE AUTHORS OF EACH ARTICLE . www.tcf.org : EB SITE HIS AND OTHER PUBLICATIONS CAN BE FOUND ON T HE C ENTURY F OUNDATION W T 5 www.tcf.org

6 1 “Top 10 Charter Communities by Market Share: Fourt h Annual Edition,” National Alliance for Public Cha rter Schools, October 2009, http://www.publiccharters.org/files/p ublications/MarketShare_P4.pdf. 2 for Reform: The Reauthorization of the Elementary and Secondary See the Obama administration’s recent “A Blueprint Education Act,” U.S. Department of Education, March 2010, http://www2.ed.gov/policy/elsec/leg/blueprin t/blueprint.pdf. Also see Lesli A. Maxwell, “Obama Team’s Advocacy B oosts Charter Momentum,” Education Week 28, no. 35 (June 17, 2009): 1 and 24–25, http://www.edweek.org/ew/articles/2009/ 06/17/35charter_ep.h28.html. 3 Caroline M. Hoxby, Sonali Murarka, and Jenny Kang, “How New York City’s Charter Schools Affect Achiev ement,” The New York City Charter Schools Evaluation Project, S eptember 2009, http://www.aeaweb.org/aea/conference/program/retrie ve.php?pdfid=532. 4 the Public Interest Sean F. Reardon, “Review of ‘How New York City’s C harter Schools Affect Achievement,’” Education and Center and Education Policy Research Unit, November 2009, http://epicpolicy.org/thinktank/review-How-N ew-York-City- Charter. 5 Donald Boyd, Pamela Grossman, Hamilton Lankford, S usanna Loeb, and James Wyckoff, “Measuring Effect S izes: the Effect of Measurement Error,” National Center for Analysis of Longitudinal Data in Education Research, Workin g Paper 19, June 2008, http://www.caldercenter.org/PDF/1001257_measu ring_effect_sizes.pdf. 6 Brian A. Jacob, Lars Lefgren, and David Sims, “The Persistence of Teacher-Induced Learning Gains,” NB ER Working Paper No. 14065, National Bureau of Economic Research, Ju ne 2008, http://www.nber.org/papers/w14065. 7 Although she did not report this information in he at New York City r study, Hoxby communicated privately to Reardon th charter schools are roughly equally oversubscribed. If this is true, then this particular instance of bias would not hold; but, as a corollary, it would suggest that parents are unable to determine which charter schools are the most ef fective and which are the least effective. After all, if parents could make a differentiation between better and worse charter s chools, would they not tend to enter lotteries for the better schools? Indeed, this notion that parental choice can act as a marke t mechanism to indicate high- quality schools while simultaneously highlighting l school philosophy. ow-performing ones is at the center of the charter 8 States,” Center for Research on Education “Multiple Choice: Charter School Performance in 16 6 www.tcf.org

7 Outcomes (CREDO), Stanford University, June 2009, http://credo.stanford.edu/reports/MULTIPLE_CHOICE_C REDO.pdf. 9 Thomas Kane et al., “Informing the Debate: Compari ng Boston’s Charter, Pilot and Traditional Schools, ” prepared for the Boston Foundation, January 2009, http://www.gse.har vard.edu/~pfpie/pdf/InformingTheDebate_Final.pdf. 10 Christopher Lubienski and Sarah Theule Lubienski, “Charter, Private, Public School and Academic Achie vement: New er for the Study of Privatization in Education, Tea Evidence from NAEP Mathematics Data,” National Cent chers College, Columbia University, January 2006, http://www.ncspe .org/publications_files/OP111.pdf. 11 Ron Zimmer, Suzanne Blanc, Brian Gill, and Jolley Christman, “Evaluating the Performance of Philadelp hia’s Charter Schools,” RAND Education Working Paper, March 2008, http://www.rand.org/pubs/working_papers/2008/RAND_W R550.pdf. 12 Katrina R. Woodworth, Jane L. David, Roneeta Guha, Haiwen Wang, and Alejandra Lopez-Torkos, “San Fran cisco Bay Area icy, SRI KIPP Schools: A Study of Early Implementation and A chievement: Final Report,” Center for Education Pol ublications/SRI_ReportBayAreaKIPPSchools_Final.pdf. International, 2008, http://policyweb.sri.com/cep/p 7 www.tcf.org

Related documents

Pedestrian Environmental Quality Index Part I

Pedestrian Environmental Quality Index Part I

Pedestrian Environmental Quality Index (P.E.Q.I.) San Francisco Department of Public Health The PEQI is a quantitative observational tool that was originally developed in 2008 by the to assess the qua...

More info »
bord ivashina taliaferro

bord ivashina taliaferro

Large Bank Small Firm s and Lending Vitaly M. Bord Harvard University Victoria Ivashina Harvard University and NBER Ryan D. Taliaferro Acadian Asset Management First draft: December 2014 This draft: O...

More info »
NRDC: The Issue with Tissue   How Americans are Flushing Forests Down the Toilet (PDF)

NRDC: The Issue with Tissue How Americans are Flushing Forests Down the Toilet (PDF)

FEBRUARY 2019 R: 19-01-A REPORT THE ISSUE WITH TISSUE: HOW AMERICANS ARE FLUSHING FORESTS DOWN THE TOILET AUTHOR Jennifer Skene, with significant contributions from Shelley Vinyard

More info »
Closing Achievement Gaps in Diverse and Low Poverty Schools

Closing Achievement Gaps in Diverse and Low Poverty Schools

Closing Achievement Gaps in Diverse and Low - Poverty Schools: An Action Guide for District Leaders By Public Impact

More info »
5209vpsTXT V3.indd

5209vpsTXT V3.indd

A report from The Century Foundation and the Poverty & Race Research Action Council A Better Start 200 18th St. NW #200, Why Classroom Diversity Washington, DC 20036 prrac.org Matters in Early Educati...

More info »
sol 02.PDF

sol 02.PDF

Chapter 2: The Basics of Supply and Demand CHAPTER 2 THE BASICS OF SUPPLY AND DEMAND EXERCISES 1. Consider a competitive market for which the quantities demanded and supplied (per year) at various pri...

More info »
eiawritingstyleguide

eiawritingstyleguide

EIA Writing Style Guide April 2015 U.S. Energy Information Administration Office of Communications This publication is available on the EIA employee intranet and at: www.eia.gov/eiawritingstyleguide.p...

More info »
DOD Dictionary of Military and Associated Terms, January 2019

DOD Dictionary of Military and Associated Terms, January 2019

DOD Dictionary of Military and Associated Terms As of February 2019

More info »
AffordableExcellencePDF

AffordableExcellencePDF

HASE LTINE This is the story of the Singapore healthcare system: how it works, how it is financed, its history, where it is going, and AFFORDABLE EXC what lessons it may hold for national health syste...

More info »
7340.2H  Bsc dtd 3 29 18

7340.2H Bsc dtd 3 29 18

ORDER JO 7340.2H Air Traffic Organization Policy Effective Date: March 29, 2018 Contractions SUBJ: contractions used by ed word and phrase This handbook contains the approv personnel of the Federal Av...

More info »
Microsoft Word   Final Report Draft 10 063016 1700.doc

Microsoft Word Final Report Draft 10 063016 1700.doc

Commission on Care Final Report June 30, 2016

More info »
Customer Suitability in the Retail Sale of Financial Products and Services

Customer Suitability in the Retail Sale of Financial Products and Services

Basel Committee on Banking Supervision The Joint Forum Customer suitability in the retail sale of financial products and services April 2008

More info »
calendar jan may 2019 version 1

calendar jan may 2019 version 1

INDIAN INSTITUTE OF TECHNOLOGY MADRAS EVEN Semester Jan May 201 9 Academic Calendar - – January 2019 February 2019 March 2019 April 2019 May Days June 2019 2019 Sat 1 2 Sun Enrollment of 1 Quiz - II G...

More info »
Has US shale gas redcued CO2 emissions Broderick Anderson 2012

Has US shale gas redcued CO2 emissions Broderick Anderson 2012

Emissions? Has US Shale Gas Reduced CO 2 Examining recent changes in emissions from the US power sector and traded fossil fuels October 2012 & Dr John Broderick Prof Kevin Anderson Tyndall Manchester ...

More info »
fschap24.fm

fschap24.fm

CalFresh Handbook page 24-1 Semi-Annual Reporting Requirements 24. Semi-Annual Reporting Requirements 24.1 Semi-Annual Reporting (SAR) Households As a result of Senate Bill (SB) 79, California elimin ...

More info »
sr880

sr880

Federal Reserve Bank of New York Staff Reports .S . Banks Complexity in Large U Linda Goldberg April Meehl Staff Report No. 880 February 2019 This paper presents preliminary findings and is being dist...

More info »
Approaches to State Pre K Eligibility Policy: Considerations for Policy Makers in Revising Policy to Increase Access for High Needs Children

Approaches to State Pre K Eligibility Policy: Considerations for Policy Makers in Revising Policy to Increase Access for High Needs Children

www.ceelo.org |[email protected] 1 Approaches to State Pre - K Eligibility Policy: Considerations for Policy Makers in Revising Policy to Increase Access for High Needs Children CEELO POLICY REPORT Megan...

More info »