Online Charter Study Final

Transcript

1 Online Charter School Study 2015

2 Online C harter School Study 2015 James L. Woodworth, Ph.D. – Lead Analyst Margaret E. Raymond, Ph.D. – Project Director Kurt Chirbas – Graphics and Figures Maribel Gonzalez – Data Collection Yohannes Negassi, M.A. – Research Analys t Will Snow, M.A. – Database Manager Christine V an Donge, Ph.D. – Research Analyst

3 © 201 5 CREDO Center for Research on Education Outcomes Stanford University Stanford, CA http://credo.stanford.edu CREDO, the Center for Research on Education Outcomes at Stanford University, was established to improve empirical evidence about education reform and student performance at the primary and . CREDO at Stanford University supports education organizations and policymakers in secondary levels rch and program evaluation to assess the performance of education initiatives using reliable resea . CREDO’s valuable insight helps educators and policymakers strengthen their focus on the results from innovative programs, curricula, policies and accountability practices . Ack nowledgements CREDO gratefully acknowledges the support of the State Education Agencies and School Districts who contributed their data to this partnership. Our data access partnerships form the foundation of CREDO's work, without which studies like this w ould be impossible . We strive daily to justify the confidence you in us. have placed CREDO a lso acknowledges the support of the Walton Family Foundation for this research. The views expressed herein do not necessarily represent the positions or policies o f the organizations noted above . No official endorsement of any product, commodity, service or enterprise mentioned in this publication is intended or should be inferred . The analysis and conclusions contained herein are exclusively those of the authors, a re not endorsed by any of CREDO’s supporting organizations, their governing boards, or the state governments, state education departments or school districts that . The conclusions of this research do not necessarily reflect the o pinions or participated in this study official position of the Texas Education Agency, the Texas Higher Education Coordinating Board, or the State of Texas. credo.stanford.edu i

4 Table of Contents ... 1 1. Introduction 1 Purpose of Study ... Need for the Study ... 1 ... Questions to Be Addressed 2 4 2. Methods and Data ... Identifying Online Charter Schools ... 4 ple States 5 Consolidating Student Data from Multi ... 6 ... Multiple Datasets Matched Data ... 6 Selection of Compar ... 6 ison Observations Brick ... 8 -District VCR Matched Sample ... 10 Mobility Study Data Set 11 ... Basic Analytic Model ... 14 Mixed Methods Analysis ... 15 Presentation of Results 3. Student Mobility 16 ... ... Characteristics of Online Charter Mobility 16 Mobility and Student Characteristic s ... 20 Mobility by Race -Ethnicity ... 20 Mobility by Student Sub -populations 21 ... ... 23 4. Impact Analysis ... 23 -District Students Online Charter Students Compared to Brick Results by State ... 25 -populations ... Sub 27 Race ... 27 -ethnicity ... Students in Poverty 28 English language learners ... 29 Special education students ... 30 Interpretation of Subpopulation E ffects 31 ... -District Schools 35 Online Charter Schools Compared to Brick ... ... 37 Network Affiliation Online Charter Students Compared to Brick -Charter Students ... 39 Mixed -Methods Analyses ... 40 Student Testing ... 40 Data and School Survey Data Self -Paced Delivery ... 41 credo.stanford.edu ii

5 42 Synchronous vs. Asynchronous ... ... 43 Class Size 43 ... School and Family Interactions Methods of Class Communication ... 47 ... School -Level Data and School Survey Data 49 -Wide Policies ... 49 School ... Student Support Activities 52 School and Family Interactions 55 ... ... Professional Development and Compensation 55 -Significant Findings ... 59 Non ... 59 Student Testing Data and Policy Changes Summary and Implications ... 61 Implications ... 63 Appendix A: DESCRIPTIVE PROFILE OF ONLINE CHARTER STUDENTS ... 64 dix B: TECHNICAL APPENDIX 68 Appen ... ... 69 Empirical Bayesian Shrinkage Alternative Specifications 71 ... Brick -and -Mortar Charter School VCR ... 71 Generalized OLS Model on Multi -Year Panel Data ... 73 Restricted OLS Model on Multi ... 74 -Year Panel Data Online Charter School Choice Analysis ... 76 Appendix C: CORRELATES OF SCHOOL -LEVEL EFFECTS WITH SURV EY REPSONSES ... 81 References 104 ... credo.stanford.edu iii

6 Table of Figures ... 8 Figure 1: CREDO VCR Methodology 22 ... Figure 2: Mobility Rates by Subpopulation Figure 3: Impact of Online Charter Attendance on Average Student Academic Growth, Reading and Math ... 23 24 Figure 4: Impact of Online Charter Attendance on Academic Growth by Year, Reading and Math ... 26 Figure 5: Online Charter Effect Size by State, Reading ... Figure 6: Online Charter Effect Size by State, Math ... 27 Figure 7: Overall Academ ic Growth for Students in Poverty Compared to Students Not in Poverty, Reading ... and Math 29 Figure 8: Overall Academic Growth for English Language Learners Compared to Non -English Language ... Learners, Reading and Math 30 Figure 9: Overall Academic Growth for Special Education Students Compared to Non -Special Education Students, Reading and Math ... 31 ... Figure 10: Expected Values of Effect Sizes by Student Profile, Reading 33 Figure 11: Expected Values of Effect Sizes by Student Profile, Reading ... 34 Figure 12: Online Charter School Quality Curve: Reading and Math ... 36 Figure 13: Relationship b etween Growth and Attending an Online Charter School with Self -Paced Classes ... 41 ... Figure 14: Count of Schools by Number of Synchronous Hours of Instruction 43 Figure 15: Relationship between Monitor ing Teacher/Family Interactions and Student Academic Growth ... 44 ... 46 Figure 16: Relationship between Expected Parental Roles and Academic Growth, Reading ntal Roles and Academic Growth, Math Figure 17: Relationship between Expected Pare 47 ... Figure 18: Race -District VCR Matched Sample Data Set, Math ... 65 -Ethnicity of Brick Figure 19: Brick -District VCR Matched Sample Sub -Populations by Year, Math ... 66 Figure 20: Pre ... 67 -Online Achievement Decile of Online Charter Students, Math Figure 21: Online Charter Effect Size by State for Online Charter vs. Brick -Charter, Reading ... 72 Figure 22: Online Charter Effect Size by State for Online Charter vs. Brick -Charter, Math ... 73 Figure 23: Average Growth for First Year in Online Charter and Subsequent Year by Stayer/Leaver Status ... 76 credo.stanford.edu iv

7 Table of Tables Students ... Table 1: States with Online- 5 9 Table 2: Student Population Demographics by TPS Sector ... 11 Table 3: Student Record Demograph ... ics for Mobility Study ... 11 Table 4: Number and Percentage of Records per Period 17 Table 5: Grade at Initial Enrollment in Online Charter School for New Entrants ... ble 6: Duration of Student Enrollment in Online Charter Schools by Entry Year ... Ta 17 18 ... Table 7: Percentage of Online Students Remaining in Online Charter Schools by State 19 Table 8: Mobility Rates for Students by Sector and State ... 20 ... Table 9: Annual Rates of Return from Online School to Traditional Schools ... Table 10: Grade on Return to TPS from Online Charter School 20 Table 11: Mobility Rates by Race- ... 21 Ethnicity and Sector Ethnic Group, Reading and Math ... Table 12: Effect Size of Attending Online Charter School by Racial- 28 ... 38 k Compared to Average VCR, Reading and Math Table 13: Effect Sizes by Networ Table 14: Effect Sizes by Network Compared to Independent Online Charter Schools, Reading and Math ... 39 Impacts by VCR Group ... Table 15: Summary of Significant Online Charter 40 Table 16: Reported Average and Maximum Class Size by School Level 43 ... ... 48 Table 17: Tools Used to Support Synchronous Instruction, Reading and Math Table 18: School- Wide Policies, Reading and Math ... 50 Table 19: Course Credits and Assessment Frequency, Reading and Math ... 50 Table 20: School- Wide Policies Relating to Curriculum and Instructio 52 n, Reading and Math ... 53 ... Table 21: Entry Assessment for New Enrollees on -One Support to Students, Reading and Math ... 54 Table 22: Providers of One- Provided Supports ... 54 Table 23: School- ... 55 Table 24: Relationship between Expected Parental Roles and Academic Growth, Reading and Math Table 25: Teacher Professional Development Activities, Reading and Math ... 56 ... 56 Table 26: School Leader Professional Development Activities, Reading and Math ... 57 Table 27: Influencing Factors for Teacher Compensation, Reading and Math ncing Factors for School Leader Compensation, Reading and Math ... 58 Table 28: Influe Table 29: Percent of School Leader Time by Task, Reading and Math ... 58 -Significant Corre Table 30: Survey Items of Interest with Non lations with Math and Reading Effect Sizes 59 ... Level Effects ... 60 Table 31: Correlations between Education Policies and School- Table 32: State- Level Policy Change Description ... 61 Table 33: Number of Matched Online Charter Students by State and Year, Math 64 ... Table 34: Empirical Bayesian Shrinkage of Effect Sizes by Network Compared to Independent Online Charter Schools, Reading and Math ... 70 Table 35: Student Population Demographics by Charter Sector ... 71 credo.stanford.edu v

8 Table 36: Effect Size by Subpopulations for Online Charter vs. Brick ... 71 -Charter, Reading and Math Panel Data Unrestricted OLS Regression Output, Reading and Math 74 Table 37: ... ... 75 Table 38: Panel Data Restricted OLS Regression Output, Reading and Math d to One Year Enrollees – Marginal Results, Table 39: Continuing Online Charter Enrollees Compare Reading ... 77 Table 40: Continuing Online Charter Enrollees Compared to One Year Enrollees – Marginal Results, Math ... 78 Table 41: Future Online Charter Choo sers, Reading ... 79 Table 42: Future Online Charter Choosers, Math ... 80 Table 43: Correlations of School- ... 82 Level Effects with Survey Responses, Math and Reading credo.stanford.edu vi

9 List o & Definitions f Acronyms CMOs Charter School Management Organizations Center for Research on Education Outcomes CREDO End -of -Course EOC Exam English Language Arts ELA ELLs English Language Learners FERPA Family Education Records Privacy Act Nat ional Assessment of Educational Progress NAEP TPS Traditional Public School VCR Virtual Control Record Asynchronous Learning that occurs when students complete assignments and learning on the ir own time and schedule without live interaction with a teacher -district A public school operated by a traditional school district which uses standard in -person Brick learning as its primary means of curriculum delivery (aka – TPS) -charter A public school operated under a charter as defined by the state which uses standard Brick person learning as its primary means of curriculum delivery in- A public school operated under a charter as defined by the state which uses online Online charter learning as its primary means of curriculum delivery district A public school operated by a traditional public school district which uses online Online learning as its primary means of curriculum delivery Growth -to -year change in academic performance relative to one’s peers . Growth can The year be positive or negative. Network A network i s defined as a single organization which oversees the operation of at least three charter schools . Not all the schools in a network must be online for the schools to be considered part of a network. Online School A school which offers a full- time online cu rriculum to the majority of its students credo.stanford.edu vii

10 Learning that occurs with all students in a class receiving instruction and completing Synchronous work at the same time . Students do not necessarily have to be in the same location for synchronous work. credo.stanford.edu viii

11 Online School Study Charter 2015 1. Introduction Purpose of Study The Center for Research on Education Outcomes (CREDO), Mathematica Pol icy Research, and the Center ha ve undertaken a collection of studies to contribute on Reinventing Public Education (CRPE) more on the landscape and operation of online charter schools and their impact on extensive information than has been available to date. Our aim was to deliver an unbiased, data - students’ academic growth schools . The intent of this report is to present to lay -reader s and driven examination of online charter policy decision makers information based on advanced statistical models of student growth in a manner which schools in the is accessible and useful for the promotion of deeper discussion of the role of online K-12 setting . This report presents the findings about impacts of online charter enrollment on the academic progress of students . Need for the Study Online schools, especially online charter schools, are a tiny, but rapidly growing sector in the education . Full- time realm schools are still a relatively new phenomenon, and some states have seen online enrollment growth which is literally exponential. While the overall percentage of students who attend online schools is small, only 0.5% of students in our da ta , based on increasing growth rates we should expect to see continued expansion of online educational services . The schools within our 18 state online set have increased their tested student 10 to over 65,000 data enrollment from 35,000 students in 2009- in 2012 . Based on even modest funding levels of $6,000 per student , 65,000 students students -13 . With the number of students expected to represents a public investment of $390,000,000 annually rapidly continue to grow n of the outcomes of public , good stewardship demands an examinatio investment. Online schools may be a good investment of these millions of dollars if they can provide quality education to students, especially those student s poorly served by the current education system . Online schooling option provide students a flexible, student -centered educational option . s have the potential to One of the desirable attributes of online schools is their adaptability for atypical students . Across the country , there are students who work to provide for their f amilies . There are other students who are who are already active in their chosen professions s, or Olympic hopefuls. These student s such as actors, artist could also benefit from a flexible, portable means of receiving their education . For migrant students or those in unstable households, the ability to sustain a consistent schooling environment could greatly credo.stanford.edu 1

12 boost educational outcomes . Likewise, students who learn at a greatly different rate from their age peers he self -paced nature of many online . (both slower and faster) might benefit from t programs , online . Only high these potential benefits learning may not be a good fit for many students Despite quality, rigorous research will provide the data necessary to address such policy questions. In spite of the rapid growth of the online sector, there have been few detailed longitudinal analyses on schools on academic achievement . Many states have little data on the number of the impact of online that online who they serve. Basic identification data on online operate within their state or programs to collect . Without reliable information on school performance, policy s out to be a challenge schools turn the future learning and career opportunities of students in an makers, school officials, and families risk uncha rted Since online learning at the K -12 level is still in its infancy, measures of the quality of arena. available school options can provide feedback to educational stakeholders, including the online authorizers and providers, about program performance that can shape the field as it evolves . Questions to Be Addressed charter schools and their This report presents the findings of an ambitious scope of analysis about online . The findings look at performance at several levels: at the individual st udent level, at the performance the organizational level of the online student population level, at schools and at the state policy level. Each facet of the analysis offers its particular insights about the influence of online charter schooling on end them. the students who att For this study, we examine the impact of attending an online charter school on the academic progress of students wh o attend them . We measure academic impact by comparing the annual academic growth of online students equivalent students who attend schools with traditional settings , i.e. with the growth of -and . This question, “What is the average impact of attending an online brick -mortar district schools school on the academic growth of students?” frames the analysis and drives the discussi on of charter the report assess how academic growth in results throughout online charter schools differs for . We different student backgrounds students with race -ethnicity, poverty status , and exceptional including needs. Online schools may be the best option for some students. Alternatively, it may not be the best option for every student. there student s who are better suited to the online school experience? Looking at the Are characteristics ine charter schools is of the students at the population level, we examine if success in onl more likely for some students than others . Attributes of the schools are also new territory for study . We studied differences in the makeup and operation of the schools themselves. Descriptions of these organizations provide a usefu l chart of the current landscape . Where possible, those differences were incorporated into the impact analyses to discern if school attributes varied with student results . To explore this aspect of the education equation, Mathematica Policy Research devel oped and administered to school principals a survey of online school characteristics . The survey covered many aspects of school operations including a range of students credo.stanford.edu 2

13 served, methods of curriculum delivery, teacher credentials, and parental involvement . In addition to - direct analyses of responses, we combined survey responses with student testing data for mixed . These analyses will allow providers to explore which methods analyses of these school characteristics student outcomes. d weaker relationships with services currently offered have stronger an Finally, under the terms of the Constitution , each state is free to implement public education policies as h, including the terms under which online schools operate. The Center on they wis Reinventing Public Edu schools . Their review included cation (CRPE) conducted a review of state policies related to online categorizing state policies and documenting policy changes which could be expected to have an impact on students . The policy findings from CRPE we re combined with educational outcomes for online school level data for mixed -methods analyses of policy implications on student academic growth . student- Policy makers should explore these results for policies they may wish to implement or eliminate from their states to maximize to student benefits of online school s. It is our intent that this study will serve as the foundation for constructive discussions on the role of online s in the K s exhaustive. school -12 sector. The findings presented in the rest of the report are by no mean . Are online school s the solution There are more questions policy makers and stakeholders need to ask for many of the educational challenges faced by families today or are they a niche option appropriate for only a small group of students with a specific set of characteristics? Is the current regulatory structure for online enrollment properly matching kids with services? Are online schools having a positive impact on students’ educational experiences ? What additional measures should be used to define “success” for online K -12 schools? Rather th is report aims to build a solid evidence base as the first of many analyses . The report provides a brief description of the approach to the analysis in the following section . The next chapter includes an analysis on the student -level, school- level, and network -level impact of attending an online charter school as well as a mixed methods analysis which combines impact data with school -level information gleaned from a survey of school leaders . The repor t concludes with a discussion of the implications of the study findings. credo.stanford.edu 3

14 Methods and Data 2. Online Charter Schools Identifying WHAT IS AN ONLINE CHARTER SCHOOL? charter Identifying students enrolled in online typically do not States schools was a challenge. online an indicator for stude record nts attending an One of the challenges faced by . Lists of the schools offering online enrollment school organizations which push beyond the - incomplete or non in each state proved to be familiar boundaries is the absence of the . existent common language needed to describe what it is they do. Online charter schools CREDO for information about online searched are not an exception to this problem. schools and programs from across the country using With the addition of online learning Internet searches the . Information from multiple -12 setting has come a options in the K -12 Online Learning International Association for K surplus of terms to describe these new (iNACOL) was the most complete directory we types of learning. Most problematically, . We extended the directory with additional located the virtual schooling sector is so new contacts with known online providers such as K12 . To many of the terms used have differing ine school s, we identify additional potential onl ons. definiti the National Center for Education Statistics searched We found many schools using terms like (NCES) website, the websites of state departments of online, virtual, digital, distance, etc. to es for terms education, and completed Google search describe very different types of services. es s. Our search included related to online school In some locations, a distance learning , “cyber” , and terms such as “online”, “virtual” school fit our definition of an online . “distance learning” among others school, in others distance learning had In creating this list of potential online school s, we nothing to do with online delivered found many of the identified schools were not education. independent schools, but were instead virtual For the purposes of this study, a n online education programs operating under the umbrella of school is a school which provides the -mortar school setting. -and nal brick a traditio For majority of classes (everything except PE , several reasons, we decided to exclude these band, or a similar elective) to full- time records of schools: it was impossible to isolate the through a computer via the students -programs which were students enrolled in online internet . Lessons may be synchronous or part of a larger brick and we were -mortar school -and asynchronous. Lessons may consist of concerned that the influence of traditional videos, live chat, bulletin boards, or any enrollment of students might influence the behavior other common means of electronic of either the operator or the students in the online . But the primary delivery communication setting . For the purposes of this study, a student was method must be online. online school considered to be attending if the an credo.stanford.edu 4

15 school’s enrollment consisted of full , online students only. -time online school s to verify the status of the program as a full- CREDO contacted each of the identified time ue online only school. The program also had to have a state school identification number which was uniq from any brick -mortar school . This means this study does not include the majority of students who -and take course while enrolled in traditional brick -and -mortar schools. one or more online offering a mixed or blended curriculum. Schools were also excluded as an online school if they reported -and -mortar school students taking online courses, the combination of classroom- based As with brick and online instruction creates a different educational environment from the one targeted in this study. , our data set for online school students is restricted to those students attending public, full - To be clear . After the multiple screens described above, time online schools data from 158 online schools was included in the report. 1: States with Online -Students Table Georgia Minnesota Arkansas Ohio Texas Colorado Arizona DC Louisiana New Mexico Oregon Utah California Michigan Nevada Pennsylvania Wisconsin Florida Consolidating Student Data from Multiple States is type, CREDO worked with the state departments In order to create a national data set for studies of th in states and the District of Columbia . Because each state used its own standards and of education 17 tests to evaluate student academic achievement, it was necessary for CREDO to standardize the values to m . CREDO did this by creating a "Bell curve" for each test -- by subject, grade and ake them comparable year --where the average student score on the test becomes the central value, and all other scores are distributed around it. The transformation places each students’ performance in relation to all other equivalent tested students, making it ready for comparison with other students . By comparing each student’s performance relative to the other students from one year to that same student’s relative performance in the next year, CREDO could estimate if the student was growing academically at a rate which was faster, similar, or slower than the rate of their peers . CREDO was able to c growth results from multiple grades, states, and years . Even though average ombine academic performance in state A may represent a difference in achievement from the average academic performance in state B, a change in academic performance (growth) of .05 standard deviations in state A and . 05 standard deviation change in performance in state B both represent the same level of improvement relative to their peers in the students’ home state. This is one of the reasons measurement of academic growth is superior to simple measures of academic achievement; the level of which can vary greatly from state to state. credo.stanford.edu 5

16 Multiple Datasets Matched Data . The first step in conducting a CREDO conducted analyses using its Virtual Control Record (VCR) method VCR analysis is to create a matched data set nts (in this . The matched data set consists of treated stude case students attending an online charter school) and demographically identical students in the control group . CREDO established two control groups for this analysis . The first was a traditional control group -mortar school operated by a traditional school district (brick of students who attend a brick- and - . These schools are those normally referred to in CREDO’s studies as TPS district) . Due to the dual nature between the comparisons of the treatment group, both online and charter, it was beneficial to make -and -mortar treated students and brick traditional schools and treated students and -and -mortar charter schools . This brick necessitated the creation of a second matched comparison group with students attending brick -charter schools as the contr ol group . This comparison group allowed CREDO to examine online -ness” of an online charter school as the “ Click for an infographic about here . compared to physical charter schools the Virtual Control Record method. t was hoped a third At the outset of the study, i focus on the comparison group would s by creating a dataset with students who attended online school “charterness” of the online charter operated by districts as the control group . Unfortunately, the number of students who attend online - trict matched dataset. -dis /online district schools is too small to allow for an acceptable online charter Selection of Comparison Observations A fair analysis of the impact of online charter schools requires a comparison group which matches the previous demographic and academic profile of online charter students to the fullest extent possible . As in CREDO employed the virtual control record (VCR) method of analysis developed by stud ies , this study charter student who is represented in . The VCR approach creates a “virtual twin” for each online CREDO ould differ from the online charter student only in that the student the data . In theory, this virtual twin w . The VCR matching protocol has been assessed against other possible charter school n online attended a 1 study designs and judged to be reliable and valuable by peer reviewers . CR approach, a “virtual twin” was constructed for each online charter student by drawing on Us ing the V very the available records of traditional public school (TPS) students with identical traits and identical or 1 -Savitz, N. et al. (2012). “Using an Experimental Evaluation of Charter Schools Forston, K. and Verbitsky t Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact to Tes -4019, U.S. Department of Education. Estimates,” NCEE 2012 6 credo.stanford.edu

17 2 prior test scores who were enrolled in TPS th at the charter students would have likely attended similar charter school . To better isolate the effect of attending an online charter if they were not in their online second da . For the ta set a school as opposed to just a charter school, a second VCR data set was created “virtual twin” charter student by drawing on the available records of was constructed for each online brick -and school students with identical traits and identical or very similar prior test -mortar charter scores who were enrolled in brick tar charter schools that the charter students would have likely -and -mor charter school . The second VCR data set using brick -and -mortar attended if they were not in their online charter school students to form the VCRs allowed CREDO to differentiate between the eff ects of online . If the effect sizes for online charter school attendance compared to just charter school attendance charter students compared to TPS VCRs was found to be similar to the effect sizes for online charter students compared to brick- -mortar charter VCRs, the effect sizes would be primarily attributable to and the online nature of the school. Factors included in the matching criteria were: Grade level • 3 Gender • Race/Ethnicity • • Free or Reduced -Price Lunch Eligibility • English Language Learner Status • Spe cial Education Status • Prior test score on state achievement tests Figure virtual twins linked to each online 1 shows the matching process used by CREDO to create the . In the first step, CREDO identifie s all TPS with students who tra charter school student nsferred to a given re referred to as “feeder schools” for that particular . These schools a charter school . charter school online attending a n online charter school Students eliminated from the match pool for each charter student are to ensure VCRs consist entirely of TPS students The feeder school method provides a strong . counterfactual as residential school assignment commonly used to place students in TPS has been shown to group demographically and socio -economically similar students into schools . This p ractice increase s the likelihood that students assigned to similar schools ha ve similar backgrounds, knowledge of school choice programs, and school choice options . Once a school i s identified as a feeder school for a online t particular charter, all the students in that TPS become potential matches for students in tha charter school . All particular the student records from all of a charter’s feeder schools were pooled – of 4 . this became the source of records for creating the virtual twin match 2 Achievement scores were considered similar if they were within 0.1 standard deviations of the online -online charter achievement. charter st udent’s pre 3 Gender is used as a match factor for all states except Florida due to lack of data availability. 4 Each charter school has its own independent feeder list, and thus a unique pool of potential VCR matches. 7 credo.stanford.edu

18 The VCR match s any of the TPS students from the match pool whose ing method then eliminate o not match exactly to the individual online . As part of the demographic characteristics d charter student in a lled charter match process, we also drop from the TPS match pool any students who enro n online school in subsequent comparison years . students at feeder schools in the year prior Using the records of TPS to the first year of growth, CREDO randomly select students with identical values on the matching varia bles in Figure 1 , s up to seven TPS identical or . Students with similar test scores were used only when similar prior test scores including very . The values for the selected TPS there were not enough TPS students with exact test score matches are then averaged to c reate values for the virtual twin . As all other observable characteristics students are identical, the only observable characteristic that differ s between the online charter student and the ir VCR is a n online charter school . The prior test score represents the impact on academic ttendance in a achievement of both the observable and unobservable student characteristics up to the time of the match, the year before the first growth measurement . Since we matched on observable characteristics concluded that any differences in the post -test scores are primarily and the prior test score, we charter school attendance attributable to online -and -mortar . The same process was used for the brick VCR match except feeder list was based on transfers from brick -and -mortar charter scho ols to online charter schools. Figure 1: CREDO VCR Methodolog y Brick - District VCR Matched Sample As stated above, this report uses two VCR groups . The first VCR data set created for these analyses matched online charter students with students from traditional brick- and -mortar district -run schools (TPS) . Due to the large number of feeder schools sending students to online schools, this data set had an credo.stanford.edu 8

19 5 . The online charter to brick -district match rate was 96 percent . As a result, exceptionally high match rate students the sample included in this analysis is highly reflective of the full population of online charter for the states included in the impact analysis. TPS feeder Table 2 shows the characteristics of the student bodies in the online charter schools, the schools, and all TPS schools for the states included in the impact analysis . The major difference between online charter students and the students attending feeder schools is the percentage of White the students enrolled in the schools (69%) is much higher than the feeder schools (45%). The online charter difference in the percentage of White students if offset by a decrease in the percentage of Hispanic schools to serve a much smaller percentage students . As would be expected, this also leads online charter (1%) than the feeder schools . Since written communications are the of English language learners (9%) settings, it should not be major form of interaction between students and teachers in many online surprising to find a lo online charter settings . We wer percentage of English language learners (ELL) in spanic student cannot determine whether lower ELL enrollment in online schools is the cause of lower Hi enrollment or an effect of lower Hispanic student enrollment. tudent Population Demographics by TPS Sector Table 2: S TPS Feeder Online C harter All TPS Schools Schools 6 11,574 166 Number of Schools 108,476 39% 51% 48 % Percent Students in Poverty Percent English Language Learner Students 9% 1% 8% Percent Special Education S 8% 11% 11% tudents Percent White 49% 45% 69% Percent Black 13% 13% 15% 32% 11% Percent Hispanic 27% 5% 6% 2% Percent Asian/Pacific Islander 1% 1% 1% Percent Native American - Percent Multi 3% 3% 4% Racial Average Total Enrollment per School 503 772 98 6 Total Enrollment 602,134 8,933,313 1 63,722 54, -district VCR population had a special education student rate identical to the feeder schools . The brick This rate is slightly higher than the rate of special education students enrolled in all TPS schools a cross the states included in the study . Online charter schools serve a slightly smaller percentage of students in poverty, those eligible for free or reduced lunches, than the feeder schools, but a higher p ercentage than all TPS schools . The average total enrollment for online charter schools is larger than all TPS feeder schools . 5 Match rate was the percentage of online charter students with at least a student in the comparison school who was an exact match on demographics and a close/exact match on prior achievement. 6 ration. Includes some multi -campus schools with separate IDs, but one administ 9 credo.stanford.edu

20 Some states have a large number of students who supplement their course work by taking one o r more . These students were not as the impact of classes via online methods included in the treatment group education could not be separated from their traditional class work. Additionally, students online their from schools which offer online re not included study in addition to other forms of distance education we unless the school had a separate school identifier for just the online students. Mobility Study Data Set One of the analyses included in this report focused on student mobility. The data set for the mobility portion of the report lable records from the 2007 -2008 consists of all of the online charter students’ avai -2013 as well as all of the records for all the TPS students included in the school year through the 2012 VCR for any online charter student . The data set was constructed by appending the data for each year of the study for each state included in the study . Within each state, all students who were either an online charter student or selected to be part of any online student’s VCR were flagged based on the records from the VCR match process . Once all the student records were p roperly marked, the files from each state were appended together to form a national panel data. As should be expected, the characteristics of the VCR students and the online charter students are similar (see Table 3 t identical as they are in a standard matched VCR ). The only reason the two samples are no In the traditional VCR matched data set is because the VCR students are not combined in a single value. This means for each data set, the TPS students who make up the VCR are combined into one value. ic student charter student, there is one Hispanic VCR . However, in the mobility data set, the VCR Hispan students are not combined. . There could be five Hispanic VCR students for one Hispanic charter student The differing number of VCR students assigned to each c harter student allows for some variance between the percentages of students by demographic categories . As part of the VCR match process, online students are matched multiple times based on the number of years the student appears in the data set . For analys . For this data set, the is, only the matches from the longest time period are included in the VCR students who make up the VCR from each match are included . This is why the ratio of VCR students to online charter students is higher than the 7:1 maximum rat io used for the standard VCR matched data set. credo.stanford.edu 10

21 Table 3: Student Record Demographics for Mobility Study Online Charter All TPS Students VCR Students 39% 56 % 53 % Percent Students in Poverty 8% 3 % Percent English Language Learner Students % 2 cial Education Students 8% 8 10% Percent Spe % 49% 73 % Percent White 69% Percent Black 15% 12 % 12 % 11 % 13 % Percent Hispanic 27% 5% 2 Percent Asian/Pacific Islander 2 % % Percent Native American 1% 0 % 1% Percent Multi - Racial 3% 1% 2% Total Enrollment 602,134 4,6 97,266 500,836 54, . Students may remain in The mobility data set includes a record for each year a student has a test score the data set for a different number of years based on their grade in a given year, the testing regimen of . The records for a single student are the state of residence, and t he students’ interstate mobility patterns labeled by period . The first period record for a student is the earliest record chronologically . In the mobility data set, it was possible for students to have up to six indivi dual year records . Table 4 includes the number of records in each period and what percentage of the data set is encompassed by each period. 4: Number and Percentage of Records per Period Table Percentage of Total N of Period Students Students 1 1,135,139 22% 2 1,134,562 22% 3 20% 1,044,064 17% 4 881,526 5 630,200 12% 6 294,949 6% Basic Analytic Model The primary question for this study is “ How did enrollment in an online charter school affect the academic growth of students? ” To answer this central question, we need to address multiple lines of inquiry around enrollment in an online charter school . For example, we explore, “How did the academic growth of online charter school students compare to students who are just like them but instead attended tr aditional public schools (TPS)?” As there has been little work in this research area, we believe credo.stanford.edu 11

22 our work will support the policy discussions about this rapidly expanding educational trend by extending school effecti veness. the pool of knowledge on online charter a more detailed with the demographic make -up of the tested includes Appendix A descriptive analysis . We include analyses of the demographics of students who were enrolled in the online charter sector des information on the percentage of students representing students in the data set. This discussion provi each race/ethnicity, eligibility for free or reduced priced lunches, English language learners, and special education students . The primary methodological challenge associated with any study of ch arter schools is selection bias . Even after controlling for student characteristics such as gender, poverty, race, and ethnicity, the fact that some students choose to enroll in charter schools and other students do not may indicate the existence of some unobserved difference between the two groups of students . The ideal solution to this problem is a randomized experiment that creates a control group that is identical to the treatment group before entering the online charter school. Several charter school studies have used admissions lotteries in oversubscribed charter schools to conduct randomized experiments. The approach is not applicable to most charter schools and especially not online charter schools as enrollments in online charter schools 7 are not co nstrained by physical space, thus they usually have no need to allocate seats by a lottery. In the absence of a randomized experiment, several recent studies have demonstrated that it is possible to successfully address selection bias by accounting for st udents’ prior academic achievement levels before entering charter schools (Gleason et al. forthcoming; Furgeson et al. 2012; Fortson et al. 201 5). The three previous studies of the achievement effects of online charter schools used variations on this appro ach. Unfortunately, however, it is not clear that the approach can succeed in eliminating all selection bias in the context of online schools. Because online schools differ radically from brick -and - mortar schools, the students who enroll might be quite dif ferent from those enrolling in conventional schools. For example, some students might enroll in online schools because they have had significant academic, behavioral, or social problems in conventional schools, which may, in turn, affect their later ement trajectories. If so, prior scores might not be predictive of future scores, regardless of achiev whether a student stays in a conventional school or moves to an online school. Given the uncertainties about whether online schools are subject to unique kinds of student selection, we used several different analytic approaches to test the sensitivity of findings to modeling approaches. virtual control records ( VCRs ) method developed by CREDO (Davis and Raymond The first approach uses con 2012), involving virtual s that closely mirror the matched charter school students on known trol , eligibility or participation in special support programs ( free or r educed -price demographic attributes lunch, English l anguage learners , or special education) , and prior academic achie vement . In order to determine the impact of attending an online charter school on student academic growth (the change in 7 online Although a small number of charter schools have enrollment constraints and hold admissions lotteries, it would be impossible to generalize from a study of the few online schools in such circumstances . 12 credo.stanford.edu

23 academic achievement), we employ statistical models which compare online charter students to their . The virtual twins rep resent the expected performance of charter students had they not virtual twins enrolled in online charter schools . Due to the dual nature of online charter schools, we include in this study findings for online charter students compared to brick charter students -district VCRs and online compared to brick -charter VCRs . The VCR method has been shown to produce results similar to those (Davis and Raymond obtained with randomized control trials and student fixed- effects approaches , such as those used in several published studies of charter -school impacts (for example, Bifulco 2012) and Ladd 2006; Booker et al. 2007; Zimmer et al. 2003, 2009). a study of The second approach uses a method that has been validated experimentally in charter management organizations (CMOs) (Furgeson at study demonstrated that an ordinary et al. 2012). Th least squares (OLS) regression that controls for demographic characteristics and prior academic achievement before entering a charter school produces results that are nearly identical to the results o f randomized experimental analyses using admissions lotteries. two parallel analytic approaches designed to address the student selection that is In addition, we use unique to online schools. Both of these approaches use comparison groups dents who consisting of stu enrolled in online schools at some point in their academic careers. These models recognize the key conclusion from the nonexperimental evaluation literature that the validity of a comparison group depends on its similarity on key characteristics (Cook et al. 2008). In the context of online schooling, an important characteristic is the student’s willingness to enroll in an online school. Selecting a comparison group of students who have enrolled in an online school at some point in time is one way to account for this characteristic. We describe these models as “chooser -matched” designs. -matched design employs a method that has previously been used to measure charter The first chooser - school effects on students’ academic attainment (Booker et al. 201 1). This approach identifies the effect of online schools by comparing the difference in achievement trajectories of two groups of students who are enrolled in online schools in the same grades and years. The difference occurs after one group switches to brick -and subsequently -mortar schools and the other does not. We identify the effect of online schools by comparing the achievement trajectories of students who switched to brick -and -mortar (the comparison group) and students who remain in online charter sc hools ( the treatment group), while controlling for any observable differences between the groups in the year before the switch. The second “chooser -matched” designs use s a comparison group of students who are enrolled in brick - and e period of treatment, but who are known to enroll in online schools later -mortar schools during th in the data set. This method, in essence, compares the achievement trajectories of current online students (the treatment group) with those of future online students (the comparison group), again controlling statistically for any observable differences between the groups. This method has been used in the past in an evaluation of after -school programs conducted for the U.S. Department of Education (Zimmer et al. 2007). As with the fir st chooser -matched method, this approach has the virtue of credo.stanford.edu 13

24 identifying a comparison group of students who have also chosen to enroll in online schools, only at a different point in time. CR analysis . Results for each set of The main body of the report contains results for the brick -district V in a separate subsection of Appendix B explained additional analyses are . Mixed Methods Analysis For this portion of the study, we merged information obtained from the online charter school survey atica Policy Research ( -level test data and school- administered by Mathem Mathematica) with student level effect sizes . These processes allow for the analysis of the relationship between school for the schools which have both s tudent da ta and survey characteristics and student academic growth . The models used for this section are not causal models, so we are describing a relationship responses The Mathematica survey covers a between two factors rather than claiming one factor causes the other. . The se practices , described in detail in Volume 1, include pedagogical wide variety of school practices concerns such as the method of curricula delivery, family issues such as expected parental participation, and school practices such as providing equipment or internet connectivity to students’ homes. This report includes two levels of mixed . The first correlates school -level average -methods analyses effect sizes with data from the survey conducted by Mathematica. The second mixes student growth data 8 The survey includes data on school level characteristics gleaned survey. with school- from the characteristics such as size, location, operational practices, expectations for parents and students, and expectations for teachers . Some of the questionnaire items are restricted to students of a cer tain grade . Other items are general and applied to all schools regardless of grades served . Because a particular educational practice might have differentiated impacts for younger students compared to older students, the survey includes a set th th grade students, 7 . These grade levels were grade students, and high school students of responses for 4 . Using this picked to be representative of elementary school, middle school, and high school respectively g relationships of a particular school -wide system enables the researchers to tease out the differin procedure on students of different ages . It also allows for schools which have differing procedures for students based on the students’ ages . The survey question with the smallest number of students contains resp onses from schools which collectively serve over 13,000 individual students. data responses was small The number of schools with average effect sizes and . Only 60 schools had both -level effects and data responses . Further, some questions were not ap school plicable to some schools because of the grade range of the students in that school . This greatly limits the generalizability of the findings . 8 -level analy By including the student sis, we were able to increase the analytic power of the statistical -level analyses allowed us to control for the differing characteristics models. Additionally, using student of the students within each school. credo.stanford.edu 14

25 Presentation of Results , we present the impacts of attending charter schools in terms of standard deviations . The In this report . A z base measures for these outcomes are referred to in statistics as z -score of 0 indicates the -scores student’s achievement is average for his or her grade. Positive values represent higher performance while negative values represent low er . Likewise, a positive effect size value means a student or group of . This remains true students has improved relative to the students in the state taking the same exam . As with the z -sco res, a negative effect regardless of the absolute level of achievement for those students size means the students have on average lost ground compared to their peers. (students It is important to remember that a school can have a positive effect size for its students are . Students with consistently positive effect sizes will improving) but still have below average achievement eventually close the achievement gap if given enough time; however, such growth might take longer to close a particular gap than students spend in school. .e. 0.08 is twice 0 .04), this must be done with care While it is fair to compare two effect sizes relationally (i It would be misleading to state one group grew twice as much as another as to the size of the lower value. if the values were extremely small such as 0 .0001 and 0.0002. Finally, it is impo rtant to consider if an effect size is significant or not . In statistical models, values which are not statistically significant should be considered as no different from zero . Two effects sizes, one equal to .001 and the other equal to .01, would both be treated as nil if neither were statistically significant. To assist the reader in interpreting the meaning of effect sizes , we include an estimate of the average number of days of learning required to achieve a particular effect size . This estimate is based on findings by Hanushek, Perterson, and Woessman that “student growth is typically about 1 full standard (2012) th th and 8 grade, or about 25 percent of a standard deviation deviation on standardized tests between 4 9 This tra nsformation is approximate and dependent on estimates of average from one grade to the next.” annual academic growth. Another study on the topic (Hill, Bloom, Black, and Lipsey, 2008) derived differentiated rates of growth by grade which would result in a lower number of days of lear ning for our estimates. While we evaluate the use of a more sensitive measure for computing days of learning, we continue to use the values from Hanushek et al. to maintain consistency with previous CREDO reports. 9 Using a standard 180 day school year, each 0.01 sd change in effect size is equivalent to 7.2 days of learning. credo.stanford.edu 15

26 3. Student Mobility enerally do not start school in an online setting, it is clear that students attending Because students g online charter schools may have a higher mobility rate than students in a traditional public school. The mobility rates of students matter because high mobility can be c orrelated with lower academic growth as well as higher likelihood of dropping out of school (South, Haynie, (Hanushek, Kain, & Rivkin, 2004) and Bose, 2007) . Mobility can be a tricky variable to follow because many states report a student’s pecific times of the year such as beginning of school and testing day, but do not report enrollment at s changes which occur between those times. To estimate mobility, CREDO linked student records longitudinally across the years of this study. Students were identified as being mobile if they experienced structural school change from one testing year to the next. A non -structural school change is one a non- which does not occur because the student aged out of their previous school. This method likely underestimates the number of students who voluntarily changed schools because it does not capture students who wait until a structural change to move to a new district or a school other than the one they would have attended. However, those students were going to experience a scho ol change no matter the choices they made, so the impact of the voluntary school change may not be greater than the forced school change the student was going to have to make anyway. As part of the discussion on mobility, CREDO also examined the characteristics of new online students in charter schools. Our view is constrained by the testing patterns of the various states which typically exclude the early elementary grades and are sporadic in the high school years . In addition to straightforward compariso ns of mobility rates between online charter students and brick - and -mortar students, we were also able to investigate questions such as: 1. How many years do online students remain in online charter schools? 2. to brick -and -mortar schools after What is the percentage of online students who return attending an online school? What grades are students in when entering an online school? 3. 4. What grade are students in when they leave an online school? These questions further the understanding of the experience of online charter students . Characteristics of Online Charter Mobility Some online charter school operators state that their students come to them with additional academic deficits beyond the typical student. Often they cite the students’ history of mobility as a c ause for these deficits. If it were true that students arrive at online schools with academic deficits created by high mobility, we would expect to find online students experienced higher mobility before switching to the online school than the comparison s tudents. In fact, students who switched to online schools have a pre- online school mobility rate of nine percent compared to eight percent of the comparison students. These findings place doubt on the argument that higher pre -online mobility creates widesp read , systematic academic deficits for students who eventually switch to online charter schools. credo.stanford.edu 16

27 The data in Table . 5 shows the entry grade of students who transitioned to an online charter school ts in their academic careers . Since all the states Students enroll in online charter schools at different poin 8, these grades are comparable and show included in the analysis require students to test from grades 3 – . There is a steady increas the relational pattern between student age and online charter enrollment e in the number of students enrolling in online charter schools as students age into middle school. The large th grade is likely an under estimate due to state testing patterns. drop off in enrollments in 9 Table 5: Grade at Initial Enrollment in Online Cha rter School for New Entrants Percentage N Grade 3 13,815 11.3% K - 10.3% 4 12,587 13,380 10.9% 5 17,691 6 14.4% 7 21,943 17.9% 18,147 14.8% 8 4.3% 9 5,243 13,669 11.2% 10 Nearly one- ne charter school for one year. half of students in our study (47 percent) are enrolled in an onli This number must be tempered with the fact 19 percent of the individuals in the study enrolled in an online charter school for the first time in 2012 . Students whose first entry into an online school is -2013 2013 can on ly have one year in an online charter school. On average, online charter students in our 2012- . Table 6 includes information on the percentage of students who study spend two years in online schools remained in an online charter school categorized by the student s’ entry year into an online charter school. Table 6: Duration of Student Enrollment in Online Charter Schools by Entry Year Entry Year 2 Years 3 Years 4 Years 5 Years 1 Year - 100 % 65 % 43 % 2008 29 % 16% 2009 - 100 % 63 % 2010 39 % 23% 2009 2010 - 2012 100 % 58 % 34% 56% 2011 100 % 2012 - 2012 - 2013 100% Obviously, the students first entering an online charter school in 2012 -2013 school year cannot be included in a discussion of persistence trends as many of those students may be shown to continue on past one year once more data is available. An examination of the first four years shows a decreasing percentage of students are remaining in online charter schools for a second year. This decrease has coincided with an increase in the number of students enrolli ng in online charter schools . credo.stanford.edu 17

28 Table 7 includes the percentage of individual students in each state who remained enrolled in online . In some states, online schools have not existed long enough charter schools for a given number of years . for students to hav e accumulated more than a few years in an online school Table 7: Percentage of Online Students Remaining in Online Charter Schools by State 5 Years 3 Years 4 Years 2 Years 1 Year State AR 64% 32% 16% 6% 100% 100% 37% 16% AZ 7% 3% 57% 29% 16% 8 % CA 100% 9% 100% 48% 21% 4% CO DC 72% 28% 11% 3% 100% FL 100% 19% 1% 1% 0% 11% 60% 23% 4% GA 100% 100% 83% 42% 20% 7% IL LA 39% - - - 100% MI 54% 14% 100% - - MN 100% 51% 23% 13% 5% NM 50% 15% 9% - 100% 9% 50% 22% 4% NV 100% 100% 57% 32% 17% 8% OH OR 46% 19% 10% 4% 100% PA 60% 32% 100% 19% 10% UT 100% 43% 15% 4% 1% WI 35% 14% - - 100% Total 53% 25% 13% 100% 6% - Duration not possible in given state Twenty -three percent of online charter student test scores in the data set were from a year in which the student experienced a non -structural school change . For TPS students, the rate was only eight percent . Of course, one of those moves for online charter students would be for the student to enter the online charter school e inflates the mobility rate for online students . Even after . This mandatory additional mov we remove the initial move to the online school from the estimate, students who attend an online charter . As we did not school still have a mobility rate of 15 percent, almost twice the rate of the VCR students find higher mobility for online charter students before transferring to an online charter, the conclusion is that online charter students have more mobility after transferring to an online charter school . Table 8 shows the mobility rates fo r online and traditional students by state . The full online results include the switch to the online school. The limited online percentages include all school switches for online students except the initial switch to an online school. The full traditional values are the rates for the comparison students. In most states even after removing the triggering school switch to an online charter school, students attending online charter schools still have mobility rates of at least 1.5 times the rates of the comparison students in that state (column 5 of Table 8 ). credo.stanford.edu 18

29 Table 8: Mobility Rates for Students by Sector and State Full Limited Full Comparison 10 Traditional Ratio Online State Online 14% 8% 5% 1.60 AR 28% 21% 10% 2.10 AZ 23% 16% 8% 2.00 CA 20% 8% 2.50 CO 29% 23% DC 18% 11% 1.64 FL 29% 17% 10% 1.70 12% 9% 1.33 GA 20% 16% 10% 6% 1.67 IL LA 11% 8% 1.38 23% MI 22% 16% 10% 1.60 MN 24% 15% 5% 3.00 NM 19% 7% 2.71 23% 24% 15% 8% 1.88 NV 12% 7% 1.71 OH 18% 26% 16% OR 2.29 7% PA 22% 13% 6% 2.17 26% 17% 7% 2.43 UT WI 21% 10% 2% 5.00 Total 15% 8% 1.88 23% A portion of the difference in mobility stems from the return of many online students to traditional . Using testing data, online students were flagged as returning to the schools after a period of time -online school after they complete a test in an online charter traditional sector if they have a test in a non . The rate of return for unique students from the online charter setting to a traditional setting is 22 school . One percent online education eventually return to a traditional setting within -in-five students who use the data window . Table 9 shows the percentage of online students who return to traditional settings remains steady as the number of students enrolling in online charter schools increases . Please note the rates in Table 9 of students returning from an online charter to a traditional setting is lower than the 22 . This is because the 22 percent figure is for unique students; whereas the annual percent figure given figures include multiple reco rds for students with multiple years in an online school. 2010, the Since 2009- annual percentage of students returning to the traditional setting has remained steady . 10 Comparison ratio=limited online mobility rate/full traditional mobility rate credo.stanford.edu 19

30 Table 9: Annual Rates of Return from Online School to Traditional Schools Online Charter Percent Returning to Traditional Setting Year Enrollment 2009 16,102 ‡ - 2008 2010 32,620 2009 - 16% - 2011 35,984 16% 2010 - 2012 43,471 2011 16% 2012 2013 52,843 17% - ‡Prior online charter status not available for all students. Table includes data f or online charter students who leave an online charter school and return to TPS . 10 As would be expected, grades in which students return to TPS has a similar but slightly lagged pattern as . Online charter students who return to TPS are the grades when students enter online charter schools th 11 grade year. most likely to do so in their 8 Table 10: Grade on Return to TPS from Online Charter School Percentage of Total Grade N Returns 4 2,889 11.3% 5 13.7% 3,490 4,568 17.9% 6 21.7% 7 5,524 6,240 8 24.5% The m online school years are extremely high. Even after eliminating the obility rate for students’ post- switch from the online school to the traditional setting, former online students have a mobility rate of 36% . This suggest s students who leave online schools have a more chaotic school experience post online. Mobility and Student Characteristics Another question related to mobility is whether student demographic characteristics are related to mobility ts separated by race -ethnicity, . To examine this, CREDO compares mobility rates for studen . poverty status, ELL status, and special education status - Ethnicity Mobility by Race Mobility varies greatly by the race Minority students, black students especially, -ethnicity of the student. have a history of high mobilit y between schools . High levels of mobility, or the life issues causing high levels of mobility, are likely related to lower academic performance. Among the VCR students in the mobility data, this same pattern holds true n average mobility rate . White and Asian VCR students have a 11 It should be noted the drop off in students returning to TPS in the upper grades could be due to fewer th te sts being given in those grades. Students who return to TPS after 8 grade may not be included since the lack of upper grade tests would mean those students would not be in the data set. 20 credo.stanford.edu

31 of just 6 percent racial students have mobility of 10 percent . Black . Hispanic, Native American, and Multi- The Black VCR mobility students have the highest mobility rate among the VCR students at 13 percent. rate is twice that of th e White and Asian students. . In addition to being higher overall, 23 The patterns are quite different for the students in online charters percent for online charter students vs. 8 percent for VCR students, the disparity between white students and minority students is much smaller for online charter students . This shift in the differences between groups is being driven primarily by the higher mobility rates for white students enrolled in online charter s shown in Table 11. The comparison ratio is the schools . The mobility rates for each group of students i relative difference between online charter student rates and VCR student rates . The results indicate that White students, and to a lesser extent Asian students, in online charter schools have much less stable educational histories as compared to their VCR counterparts. Table 11: Mobility Rates by Race -Ethnicity and Sector Online Charter VCR Students Group Comparison Ratio Students White 6% 22% 3.4 19% Asian 3.2 6% Black 13% 25% 2.0 Hispanic 10% 26% 2.7 N ative American 10% 25% 2.6 - 10% 25% Multi 2.5 Racial - populations Mobility by Student Sub Another set of student characteristics which have been shown to have an impact on educational attainment are students with exceptional needs . These are students who live in poverty, students who . Being a member of one of these sub - are English language learners, and special education students populations often comes with additional educational deficits . These deficits may be impacted by higher rates of mobility . Additio nally, disaggregating mobility rates by membership in these sub -populations can provide additional insight to the unique characteristics of the online charter population. little impact on mobility . There are Being an ELL student or special education student should have direct few direct factors with those characteristics which motivate a student’s family to more frequently While migrant families do tend to have a higher rate of ELL students, relocate to a different school zone. most ELL students are not fro m migrant families . Poverty, however, has been shown to be highly correlated with high student mobility . Families of students in poverty often live in rental properties rather than owning their homes hin a community, so we . This results in a lower transaction cost for moving wit tend to see many more moves for students in poverty . Students not in poverty generally have more stable home lives with less relocation . Figure 2 includes data for mobility rates of students from the various subpopulations. credo.stanford.edu 21

32 2: Mobility Rates by Subpopulation Figure 30% 24% 25% 23% 23% 22% 22% 21% 20% 15% 10% 9% 10% 8% 8% 8% 6% 5% 0% ELL ELL SPED SPED Non-ELL Non-ELL In Poverty In Poverty Non-SPED Non-SPED Non-Poverty Non-Poverty VCR VCR Online Online Online VCR Charter Charter Charter For online charter students, the mobility rates for ELL and special education students are approximately two -a-half times the rate of mobility for the same groups of students in TPS , the mobility -and . In fact rates are sl ightly lower for both online charter ELL students and special education students compared to special education students in online charters . But the difference between students -ELL and non- the non in poverty and non- poverty students who attend online charte r schools is only two percentage points compared to a four percent difference in the VCR comparison group. Overall, students who enroll in an online school demonstrate higher overall levels of mobility than VCR . However, the mobility of online cha rter school students before they transfer to the online students charter is similar to the rate of VCR students . Twenty -two percent of online charter school students eventually return to TPS schools. credo.stanford.edu 22

33 4. Impact Analysis wth of students in online charter schools to that of their For the impact analyses, we compare the gro -to VCRs -year change in achievement relative . This type of analysis provides information about the year to that of the rest of the students in the sample . On average, the effect sizes for students a ttending online are negative charter schools . A negative effect size does not mean the student did not increase in . A negative effect size means the student did not advance as much as expected academic achievement based on the student’s characteristics . Online Charter Students Compared to Brick- District Students of analyses examines the academic The first set students compared to the growth of online charter matched VCRs made up of students who attended brick- -mortar district -run schools . These schools and typically referred to as traditional public schools (TPS) are . Compared to their VCRs in the TPS, online students have much weaker growth overall. Across all tested students in online charters, the charter 180 math typical academic gains for deviation s (equivalent to are fewer days of learning ) -0.25 standard and -0.10 (equivalent to 72 fewer days ) for reading (see Figure 3 ). This means that compared to their twin attending TPS, t he sizes of the coeff an online charter sc hool leads to icients leave little doubt attending for the average student lessened academic growth . 3: Impa ct of Online C harter Attendance on Average Student Academic Growth , Reading and Math Figure 36 0.05 0.00 0 -0.05 -36 -0.10 -72 0.10** - -108 -0.15 Effect Size Days of Learning -0.20 -144 -180 -0.25 0.25** - -216 -0.30 Math Reading poverty, non SPED VCR The 0.00 line for this graph represents the average TPS non- -ELL, , White, non- student. ** Denotes significant at the .01 level. credo.stanford.edu 23

34 These results cover all students with a growth measure (i.e., at least two years of tested performance) in all the states in all the periods . Accordingly these average measures of academic growth reveal that the in figures, however mask the story charter students is not a positive one general case for online . The all- . Around the average, some online charters will perform better and some of the underlying distribution erage will perform worse than the av . While overall results establish a baseline for discussion, these results are not subtle enough to provide insight for policy implications. A clearer picture of the more rs that are associated granular distribution around the averages along with the student or school facto . with the distribution will add to a general understanding of the situation of online charters Figure 4 shows the results of this analysis . There is no consistent trend either upward or downward in the results . Instead the overall e ffect size in math stays fairly consistent over time . The overall effect size in 13. reading shows a gradual dip, but recovers part of that loss in 20 12- Online C Figure by Year, Reading and Math 4: Impa ct of harter Attendance on Academic Growth 36 0.05 0 0.00 -36 -0.05 0.09** - 0.10** - 0.10** - - 0.11** 0.12** - -72 -0.10 -108 -0.15 Effect Size Days of Learning -144 -0.20 - 0.23** 0.25** - 0.25** - 0.25** - - 0.26** -180 -0.25 -216 -0.30 2012-13 2010-11 2009-10 2008-09 2011-12 Reading Math , White, non- VCR ine for this graph represents the average TPS The 0.00 l poverty, non -ELL, non -SPED student. ** Denotes significant at the .01 level. In the 2009 CREDO charter school study, charter schools had on average weaker growth than their hool counterparts (Raymond, 2009) . The 2013 update to that study showed stronger traditional public sc (Cremata, Dickey, Lawyer, Negassi, Raymond, and results for the charter sector compared to the TPS credo.stanford.edu 24

35 Woodworth, 2013) -and -mortar charter schools in the 2013 . An examination of growth trends for brick . Taking into study showed a pattern of slow but gradual improvement over the past several years consideration the newness of the online sector, it is possible such a pattern might appear here as well . given sufficient time Results b y State In the full- data general To delve deeper, we also included analyses of online charter attendance by state. analysis, we use statistical methods to control for differences between states. In the online charter case by state analyses, we examine the impact of online charter attendance by each state as compared to the average student academic growth of and Figure 6 the zero line is the average growth state’s . In Figure 5 . A positive effect size means the average online charter student had stronger a VCR student in the state growth than the average comparison. A negative effect size means growth for online charter students was weaker than the average VCR comparison student. While the majority of states have negative effect sizes for students attending on line charter schools, there are a few exception al states ce or even positive effect sizes between online charter with no differen . Figure 5 shows the impact for online charter students and TPS students . Thirteen students in reading states have negative effect sizes in reading, two states positive, and in two states the differences were 12 . As was indicated by the general results, the average reading effect size is negative; not significant however in Wisconsin and Georgia, online charter student s have growth w hich was significantly stronger than their VCRs . While the value for Michigan is positive and larger than that of Georgia, the Michigan value is not significant . This means we cannot be certain the result is not spurious or due to chance; thus it is described as “not different”. 12 DC was not included in these analyses due to insufficient number o f schools. credo.stanford.edu 25

36 Figure Charter Effect Size by State, Reading 5: Online Days of Learning -216 -360 -288 -144 -72 0 72 AR 0.09** - AZ - 0.12** CA 0.09** - CO - 0.07** FL - 0.19** GA 0.01* IL - 0.044 LA 0.28** - MI 0.044 MN 0.08* - NV 0.17** - OH 0.11** - OR - 0.11* PA 0.14** - TX 0.18** - UT 0.12** - WI 0.06** 0.00 -0.10 -0.20 -0.30 -0.40 -0.50 0.10 Effect Size poverty, non -SPED -ELL, non The 0.00 line for this graph represents the average TPS White, non- VCR, student. * Denotes significant at the .05 level. ** Denotes significant at the .01 level. . The effect sizes for math were both more negative The effect sizes by state in math are shown in Figure 6 an s on math growth of attending In 14 states, the impact and larger than those for reading. online charter school were significantly w eaker than the comparison group . Three states had effect sizes which were had a positive effect size in math on average not different from the comparison groups. No state . The math and reading results show there is a large amount of variation in the effect iveness of online charter schools in promoting academic growth in students attending those schools . The reasons behind this variation is a topic for future study. Practices in t hose states who are producing positive results may hold useful lessons for the remaining states. credo.stanford.edu 26

37 Figure Charter Effect Size by State, Math 6: Online Days of Learning -396 -324 -252 -180 -108 -36 36 108 AR - 0.12* AZ 0.25** - CA - 0.33** CO - 0.19** FL - 0.46** GA - 0.20** IL - 0.032 LA 0.34** - MI - 0.008 MN - 0.24** NV 0.25** - OH 0.20** - OR 0.23** - PA 0.23** - TX - 0.39** UT 0.31** - WI 0.021 - -0.35 0.05 -0.05 -0.55 -0.45 -0.25 0.15 -0.15 Effect Size , White, non- poverty, non -ELL, non -SPED The 0.00 line for this graph represents the average TPS VCR student. vel. * Denotes significant at the .05 level. ** Denotes significant at the .01 le -populations Sub ethnicity - Race Exploring deeper into the performance question of schools requires us to examine the various sub - populations served by schools of charter schools (Cremata, et al ., 2013) , CREDO has found . In past studies e that -ethnic backgrounds have different impacts on academic growth evidenc students of different racial attending charter schools . It has become standard practice to report academic growth by racial - from . In the past, part of the motivation for the separate look at each student subgroup stems ethnic groups from the explicit mission of some charter school operators to serve communities whose students have historically fared poorly in school. to have greater online The student populations that charter operators serve was shown in Table 2 . proportions of White students and smaller shares of Hispanic and English Language Learner students credo.stanford.edu 27

38 13 While there is variation in the effect sizes -ethnic groups, they are still all consistently negative. of racial 12 has the effect s Table izes in math and reading equal to the difference in performance between TPS . Results were consistently less students and online charter students for each of the racial -ethnic groups negative for reading than for math across all groups . Additionally, reading effect sizes are much more to (56 days) (86 days) . White students in online 0.08 consistent between groups ranging from - -0.12 charters have larger differences in growth relative to their TPS peers than all other groups except Native -populations except Black students in math , but better than all sub Americans in reading . ial-Ethnic Group, Reading and Math 12: Effect Size of Attending Online Charter School by Rac Table - Ethnic Days of Days of Racial Learning Math Subpopulation Learning Reading - 0.11* * - 79 - 0.25** - 180 White 0.08** - 58 Black - 0.22** - 158 - - Hispanic 0.11** - 79 - 0.29** - 209 Asian/Pacific Islander - 0.09** - 65 - 0.26** - 187 Native American 0.12** - 86 - 0.30** - 216 - - - - 0.09** Multi - 65 - 0.26** Racial 187 The effects in this table represent the difference between a student of a specific race in TPS and a student of the same race in an online charter school. ** Denotes significant at the .01 level. Students in Poverty Race -ethnicity is not the only student characteristic which commonly has an imp act on students’ academic growth . Students in poverty, those who are English language learners, and special education . students also often have academic growth which differs from the typical comparison student generally lower than that for students who are not in The average growth for students in poverty is poverty. In this analysis, the baseline comparison is TPS students who are not in poverty . We isolate the relationship between poverty and growth. This leaves a picture of the difference in the impact of online charter attendance on students in poverty compared to similar students who are not in poverty. The bars for online charter schools in Figure 7 consist of two different colors. The blue portion of the bar represents the average impact of attending an online charter school which effects all online charter students. The remainder of the bar represents the average difference between being an online charter student in poverty and an online charter student not in poverty. The total length of the bar is the average expected impact on growth of being an online charter student in poverty compared to being a TPS student who is not in poverty. Figure 7 confirms that being a student in poverty results in lower academic growth in both math and reading for all s tudent groups regardless of the type of school attended with the online charter 13 The survey of online charter providers also showed that they did not target any particular student demographically, but rather sought students with particular academic profiles . Thus a breakout of environment as elsewhere. performance by the ordinary categories may not be as pertinent in an online 28 credo.stanford.edu

39 student having the more negative overall effect. Figures 8 and 9 are read in the same manner with the students face. blue portion of the bar representing the negative effect which all charter Compared to Students N ot in Poverty , Reading Figure 7: Overall Academic Growth for Students in Poverty and Math 108 0.15 72 0.10 0.05 36 0.00 0 -0.05 -36 - 0.12** -72 -0.10 0.10** - - 0.10** - 0.26** -0.15 -108 Effect Size 0.10** - -0.20 -144 Days of Learning 0.22** - -0.25 -180 -0.30 -216 - 0.11** -0.35 -252 - 0.37** -0.40 -288 Online Charter TPS Online Charter TPS Reading Math The 0.00 line for this graph represents the average non -poverty TPS student. ** Denotes significant at the .01 level. language learners English academically As with students in poverty, students who are English language learners tend to progress more slowly than students whose primary language is English . This is potentially even more of an issue online in ts typically rely more heavily on reading as the primary method of setting where studen an . Again, curriculum delivery the data show in the data set -language learners that English have weaker -English language learners growth as a group than non ish language 8 shows the growth for Engl . Figure native English speakers. learners as compared TPS credo.stanford.edu 29

40 -English Language Figure 8: Overall Academic Growth for English Language Learners Compared to Non Learners , Reading and Math 0.15 108 72 0.10 36 0.05 0 0.00 -36 -0.05 - 0.12** -72 -0.10 Effect Size -108 -0.15 0.26** - Days of Learning 0.15** - - 0.16** -0.20 -144 0.20** - -180 -0.25 - 0.06** -0.30 -216 - 0.29** 0.32** - -252 -0.35 Online Charter TPS Online Charter TPS Math Reading -ELL stu dent. TPS non The 0.00 line for this graph represents the average ** Denotes significant at the .01 level. Special education students Another sub -population with significant impacts from online charter attendance is special education growth than their students . Again, special education students as a whole demonstrate weaker academic comparison of overall academic growth of special the special education classmates as non- seen by -special education education students regardless of race/ethnicity or school type attended to non ). student VCRs (see Figure 9 credo.stanford.edu 30

41 9: Ove Compared to Non -Special Education Figure rall Academic Growth for Special Education Students Students , Reading and Math 0.05 36 0 0.00 -36 -0.05 - 0.12** -72 -0.10 -108 -0.15 - 0.26** - 0.20** -144 -0.20 - 0.21** Effect Size -180 -0.25 0.22** - Days of Learning -0.30 -216 - 0.14** 0.33** - -0.35 -252 -288 -0.40 - 0.40** -324 -0.45 Online Charter TPS Online Charter TPS Reading Math The 0.00 line for this graph represents the average TPS non -SPED student. ** Denotes significant at the .01 level. student Online charter schools again demonstrate an ability to reduce the impacts of being a SPED -SPED students . Math academic growth for students in online charters is significantly compared to non tion of the online compared to their non , represented by the orange por -SPED schoolmates less negative and their classmates VCRs , the red bar . However, the full effect of being charter bar, than that of the SPED a special education student in an online charter school is still more negative overall than being a special education student in a TPS. Interpretation of Subpopulation Effects included in the sub To help the reader to better understand the marginal differences in effect sizes - population analyses, we have included the two figures below. Figures 10 and 11 show the expected value 14 for student profiles with certain combinations of characteristics. The column on the he effect size of t for each student profile in a traditional public school left shows the expected value of the effect size setting. The column on the right shows the expected values for the same student profiles if the student 14 and Effect sizes in Figures 10 -poverty -ELL non represent growth of each profile relative to White non 11 SPED students. non- credo.stanford.edu 31

42 attended an online charter school. The higher a profile is positioned up the vertical axis, the stronger the effect size expected growth for a student with that profile. The number after the profile is the expected for that profile. who are not ELL, not SPED, ethnic group students The student profiles include a profile for each racial- and not in poverty. There are a dditional profiles are for each racial -ethnic group with one of the three additional fact ors (ELL, SPED, in poverty) included. Student profiles which do not specifically state they include ELL or SPED or in poverty do not have those features. We did not produce profiles for every -ethnicity and the three factors as doing so would have made the figures possible combination of race unreadable. However, as the effect sizes for ELL, SPED, and being in poverty are additive, any profile which includes a combination of ELL, SPED, and/or poverty would appear lower on the vertical axis than the profiles shown with only one factor. 10 and 11 demonstrate how the findings from the subpopulation analyses impact expected Figures ethnicity or other factors have weaker growth in student growth. All student profiles regardless of race- online charter schools than in TPS. This is due to the overwhelming negative impact on student growth from attending an online charter school. ELL students and SPED student s of a given race -ethnicity have weaker expected growth than students of the same race -ethnicity who are not EL ; however , as shown in Figures 8 and 9, online charter L or SPED se negative schools are more successful in minimizing the relative to their sector average in math. impacts This is most apparent in Figure 11 when comparing the performance differences between Asian non- ELL non- poverty non -SPED students with Asian ELL students between the two sectors. The distance between the dots representing the Asian non non- poverty non -SPED and Asian- ELL students on the TPS line -ELL is much larger than the same distance on the online charter line. credo.stanford.edu 32

43 Figure 10: Expected Values of Effect Sizes by Student Profile, Reading Reading Growth by Race- Ethnicity and Status Charter Online TPS 0.10 Asian non -Ell non -Pov non- SPED , 0.08 0.05 -Ell non , 0.00 SPED -Pov non- White non 0.00 Multi non -Ell non SPED , - 0.01 -Pov non- , - 0.02 SPED -Pov non- -Ell non Asian non Asian Pov , - 0.02 Hispanic non - -Ell non -0.05 , - 0.05 SPED Pov non- Native A. non- Ell non , - 0.08 -Pov non- SPED Asian ELL, -0.09 White Pov , - 0.10 -Ell non , - 0.10 Multi non -Pov non- SPED -0.10 Multi Pov , - 0.10 White non -Ell non -Pov non- SPED , - 0.11 , - 0.11 SPED -Pov non- -Ell non Black non , - 0.12 Asian Pov Hispanic Pov , - 0.15 Asian ELL, -0.15 -0.15 Asian SPED , - 0.15 0.16 White ELL, - -Pov non- , - 0.16 SPED -Ell non Hispanic non Multi ELL, - 0.17 Pov non - SPED , - Ell non 0.18 Black non - - Native A. Pov , - 0.18 , - 0.20 Multi Pov -0.20 , - 0.20 Native A. non- Ell non -Pov non- SPED , - 0.20 Black Pov White Pov , - 0.21 Hispanic ELL, - 0.21 , - 0.22 Asian SPED , - 0.22 White SPED Multi ELL, - 0.23 , - 0.23 Multi SPED 0.24 White ELL, - 0.25 Native A. ELL, - -0.25 Hispanic Pov , - 0.26 , - 0.27 Black ELL Hispanic SPED , - 0.27 , - 0.28 Black Pov Hispanic ELL, - 0.29 Multi SPED , - 0.30 -0.30 , - 0.30 Native A. Pov Native A. SPED , - 0.31 , - 0.31 Black ELL White SPED , - 0.31 Black SPED, - 0.33 0.33 Native A. ELL, - -0.35 Hispanic SPED , - 0.37 Black SPED, - 0.38 -0.40 Native A. SPED , - 0.40 -0.45 credo.stanford.edu 33

44 11: Expected Values of Effect Sizes by Student Profile, Reading Figure Ethnicity and Status Math Growth by Race- TPS Online Charter 0.15 , 0.14 SPED -Pov non- -Ell non Asian non 0.10 0.05 Asian Pov , 0.05 - - Pov non White non - , SPED Ell non 0.00 0.00 - Multi non 0.01 - Ell non - Pov non - SPED , -0.01 Asian ELL, Hispanic non 0.02 - , SPED - Pov non Ell non - - -0.05 Native A. non- Ell non -Pov non- SPED , - 0.06 , - 0.07 Asian SPED White Pov , - 0.10 Multi Pov , - 0.10 -0.10 SPED - - - 0.11 - Ell non , Black non Pov non -Pov non- -Ell non Asian non SPED , - 0.12 Hispanic Pov , - 0.12 Native A. Pov , - 0.15 -0.15 White ELL, - 0.15 0.16 Multi ELL, - Hispanic ELL , -0.18 -0.18 Asian ELL, Native A. ELL , - 0.21 -0.20 Black Pov , - 0.21 White SPED , - 0.21 Multi SPED , - 0.21 Asian Pov , - 0.23 Hispanic SPED , - 0.23 White non -Ell non -Pov non- SPED , - 0.25 -0.25 , - 0.26 Black ELL Asian SPED , - 0.26 0.26 - , Native A. SPED Multi non , - 0.27 SPED -Pov non- -Ell non 0.30 White ELL, - -0.30 Hispanic non -Pov non- -Ell non SPED , - 0.31 Black SPED, - 0.32 Multi ELL, - 0.33 Ell non Black non - SPED , - - Pov non 0.33 - SPED , - 0.35 Ell non -Pov non- Native A. non- -0.35 , - 0.36 White Pov 0.37 , - Hispanic ELL Multi Pov , - 0.38 White SPED , - 0.39 , - 0.39 Black ELL -0.40 , - 0.41 Multi SPED Native A. ELL, - 0.41 Hispanic Pov , - 0.42 Black Pov , - 0.44 -0.45 , Hispanic SPED 0.45 - Native A. Pov , - 0.46 0.47 Black SPED, - , - 0.49 Native A. SPED -0.50 credo.stanford.edu 34

45 Online -District Schools Charter Schools Compared to Brick ol study, CREDO introduced the idea of the school quality curve. The quality curve In its 2009 charter scho uses a statistical model to compare each charter school to a virtual school consisting of the VCRs for . This is a strong comparison as it allows the reader to see how students from each charter school online charter schools compare to a school of their peers . These measures use a smaller individual growth period data window made of the last two growth periods as opposed to the four growth period 15 To minimize the statistical inconsistencies which may arise from lyses. data window of the student ana including schools with only a few students, we limit this analysis to only schools with at least 30 tested students per year. hools with average growth statistically The quality curve consists of three categories, those sc significantly lower than that of their feeders, those with average growth which is not statistically different from their feeders, and those schools with average growth statistically significantly stronger than their . These three categories are distinct feeders . The placing of a school into each category has different meaning as to the performance of the school . As such, readers should resist the urge to combine categories from this analysis . Specifically, it is impro per and can be misleading to state “x% of schools performed stronger or no different than their local market” just as it is improper to combine the weaker . These numbers should always be reported as three separate categories. and no different schools Compa red to their comparison schools, online charter schools generally have significantly weaker academic growth . Figure 12 shows the quality curve in reading . As there are 101 schools in the quality curve, the numbers represent both the number and percentage o f schools in each category . Only two percent of the online charter schools outperform their comparison schools, 32 percent perform no comparison . In math, a full 88 percent differently, and 67 percent have weaker growth than their schools of online charter schools had significantly weaker growth than their comparison . These numbers are extremely weak compared to charter school performance found in previous CREDO studies . While these numbers clearly show students attending online charter schools are not per forming at the level of their comparisons , it is important to note the incredibly large size of the individual school feeder pools may have consequences on the strength of the aggregated VCR matches . With the elimination of the restraints of physical locat ion, online schools pull students from a much broader portion of the state than do standard schools of choice . This increases the number of schools in the comparison group and weakens the comparability between each online charter school and its feeders com pared to CREDO’s other studies . Online charter schools tend to serve a much higher percentage of white students than TPS . Previous studies have consistently shown white students have smaller effect sizes from charter attendance than minority students . Also , online charter students have higher mobility rates than the 15 The shorter period is necessary as the online charter sector in some sta tes, as well as many individual online schools, are expanding at an exponential rate and comparisons from the earlier years may not reflect the current state of performance for the smaller samples which make up individual schools. credo.stanford.edu 35

46 students who make up their VCRs . School instability has long been demonstrated to have a negative impact on student growth (South, Haynie, and Bose, 2007). Due to the large number of feeder schools from which online charter schools attract their students, the TPS comparison groups for the quality curve consists of a much larger proportion of the schools in the state than the typical charter school. As a consequence, the bar for online schools in this comparison was online charter schools outperformed their comparison high . In reading, even though only two schools 18 . Eleven of of the online charter schools had achievement higher than their state’s average achievement the 32 schools with growth no t significantly different from their comparison schools had achievement above their state’s average achievement, and six schools with weaker growth than their comparison had achievement above the state mean . In math, none of the online charter schoo ls had average school achievement scores higher than their state average. harter School Quality Curve: Reading a nd Math Figure 12: Online C Reading 2 32 67 Math 0 13 88 Significantly Weaker No Different Significantly Stronger Significantly Weaker No Different Significantly Stronger Even with these caveats firmly in mind, the percent of online charter schools whose students have weaker online school growth than their comparison is concerning . The qualifying argument of some providers is . Thus any educational gains many of their students would have otherwise dropped out of school entirely no matter how small are of benefit to the students and society . Thi s argument may be justified when applied to high schools students, of which online charter schools have a higher percentage, but does not s. take into account the outcomes for elementary and middle schools students enrolling in online school credo.stanford.edu 36

47 Network Affil iation schools to take advantage of Being part of a larger network of schools may allow online charter -scale in purchasing supplies and equipment . But more importantly online charter schools -of economies in a network may be able to leverage human capital gains across multiple schools. The overall results for online charter schools in a network do not show a significant difference in effect schools . sizes for schools which are part of a larger network as compared to independent online charter w no statistically significant difference for academic progress in either subject The results sho . This is not to say, however, that all networks of charter schools perform the same. Charter schools in the same network often share resources such as curricula, operational practices, and personnel training programs . If the schools within a network consistently produce common outcomes online charter s and other schools in other which are significantly above or below those of independent networks , it is reasonable to presume the schools in that network are doing something different from the other schools . The statistical models used already account for differences in the starting academic endowments of students . Further, due to the wide geographic range of online charter schoo ls, the results This points to network resources are likely not due to locale. such as work processes, teacher recruiting/training/retention, or other shared resources as the source of the network’s higher or lower performance . To investigate this, CREDO a pplies a statistical model which isolates the impact on student growth of affiliation with each network . Table 13 shows that even the students who attended the highest performing online network schools had academic growth which was weaker or not significantly different when compared to VCRs attending school in TPS settings. A value of 0.00 in Table 14 would be equal to the performance of the average brick - and -mortar TPS. credo.stanford.edu 37

48 Table 13: Effect Sizes by Network Compared to Average VCR, Reading and Math Days of Days of Reading Math Learning Learning 0.07 Network 1 - 0.17** - 124 48 - - 12 0.03 21 Network 2 0.02 - 0.02 - 15 - 0.19** Network 3 134 - 0.05** - 39 - 0.16** - 114 Network 4 - 0.05** - - 48 - 0.07** - 32 Network 5 - 0.07** Network 6 48 - 0.21** - 152 - Network 7 - 0. 09* - 66 - 0.16** - 116 0.27** - 83 - 0.12** - 191 Network 8 - - 0.12* - 84 - 0.21** - 150 Network 9 Network 10 0.14** - 98 - 0.28** - 202 - - Network 11 0.15** - 105 - 0.28* - 199 Network 12 - 0.15** - 107 - 0.20** - 147 Network 13 0.15** - 109 - 0.27** - 193 - - .16** - 114 0 0.25** - 177 Network 14 - - 0.17** - 121 - 0.30** - 218 Network 15 Network 16 0.18** - 126 - 0.18** - 130 - - Network 17 0.18** - 130 - 0.33** - 235 Network 18 - 0.22** - 156 - 0.36** - 260 Network 19 0.26** - 188 - 0.49** - 353 - - Network 20 0.28** - 204 - 0.50** - 360 Net work 21 - 0.35** - 250 - 0.38** - 274 The 0.00 value for this table represents the average TPS, White, non -poverty, non SPED student. -ELL, non- ** Denotes significant at the .01 level. Table 13 shows the impact of attending an online charter school as compared to TPS schools, but it is also interesting to see how networks perform within the online charter sector. Table 14 provides the results of this analysis using the same data as Table 13 re -centered on the average non -network online charter student. Table 14 show s a marked variation in the average performance of online charter schools by network as compared to the average independent online charter . A value of 0.00 in Table 1 4 school would be equal to the performance of the average independent online charter school. credo.stanford.edu 38

49 Table Compared to Independent Online Charter Schools , Reading and 14: Effect Sizes by Network Math Days of Days of Learning Math Learning Reading 0.16** 0.06 43 Network 1 115 0.08** 58 0.26** 187 Network 2 0.08** 58 0.05 36 Network 3 4 0.04** 29 0.08** 58 Network 0.03 0.19** 22 137 Network 5 22 0.02 14 Network 6 0.03 0.00 0 0.07 Network 7 50 Network 8 - 0.02 - 14 - 0.03 - 22 - - 14 0.02 14 Network 9 0.02 0.05** - - 29 - - 36 Network 10 0.04 - 0.05* - 36 Network 11 0.04 - 29 - Network 12 - 0.05** - 36 0.03 22 - Network 13 0.06 - 43 - 0.04 - 29 Network 14 - 0.06** - 43 - 0.01 - 7 Network 15 0.07** - 50 - 0.07** - 50 - 0.05** - 58 0.08** 36 Network 16 - - 0.09** - 65 - 0.10** - 72 Network 17 - 0.12** - 86 - 0.13** - 94 Network 18 - Network 19 0.17** - 122 - 0.26** - 187 N etwork 20 - 0.19** - 1 3 7 - 0.27** - 194 Network 21 0.25** - 180 - 0.15** - 108 - -poverty, non - The 0.00 value for this table represents the average Online Charter, White, non -ELL, non SPED student. ** Denotes significant at * Denotes significant at the .05 level. the .01 level. Students Compared to Brick- Charter Students Online Charter students and brick -district students It is possible the differences in performance between online charter is due to the charter nature of the online charter schools rather than the online nature . To address this concern, we created an additional matched data set in which we matched online charter students to brick -charter students using the same algorithm we typically use to match charter students to TPS (i.e. and matched online charter students to demographically identical students in brick- -mortar schools from which the online charter students transferred). We then repeated all the analyses using this brick -charter (see Appendix B for as VCR set results). The summary in Table 15 shows t he results between the two full samples were highly similar . There were no major differences between the two sets of analyses . These results that the findings presented above are a result of the online aspect of the schools as confirm opposed to the cha rter aspect. credo.stanford.edu 39

50 Table Summary of Significant Online Charter Impacts by VCR Group 15: Reading Math Brick - Brick - TPS VCR TPS VCR Charter VCR Charter VCR Negative Negative Negative Negative Overall Negative Negative Negative White Negative Black Negative Negative Negative Negative Negative Hispanic Negative Negative Negative Asian Negative Negative Negative Negative Native American Negative Similar Negative Negative Mixed -Methods Analyses vides insight into how growth differs from a The quantitative analysis of online charter impact results pro However, that information is the TPS student for those students who attend an online charter school. . starting point to the larger question of why does attending an online school impact the students’ growth To d elve deeper into the mechanisms behind the answer to the question of why, we can combine data on student achievement with information about the schools which students attend . We do this by estimating correlations between the presence (and in some cases dos age) of practices included in the 16 and student achievement for students who attended online charter schools. survey While these models may provide some insight into the relationships between school practices and student achievement, they are not causal, that is to say we cannot prove the presence of a particular school policy creates the impact seen in the quantitative analysis. Such correlational examinations are interesting in that they point towards areas for additional research using causal models as we ll as provide . It should also be noted the sample size of information for future policy trials by online charter providers 60 ) which limits the generalizability of these schools with both survey data and impact data was small (n= results. Student Testing Da ta and School Survey Data For the student -level comparisons, we were able to use statistical models which controlled for -ethnicity, gender, SPED, ELL, and poverty status of students differences in race to estimate effect sizes for several factors. Factors in the survey group naturally into clusters: curriculum, instructional practices , parent/ expectations, communications, student supports, etc . Results for the different clusters of student questions are presented below . Again, while these results provide i nformation about the relationship between online charter school characteristics/practices and student academic growth, they should not be considered causal. 16 dministered to TPS school leaders, these correlations relate to online charter As the survey was not a schools only. This means a positive or negative correlation represents growth which is stronger or weaker than the online charter average growth. 40 credo.stanford.edu

51 Self Paced Delivery - to be c onsumed self -paced A major characteristic of online education is the ability for curricula in a -and model by using a While some brick -mortar schools have broken away from the standard manner. lesson structure in which students work through self -paced lessons, usually via technological delivery, use the typical most e class lessons . still singl ask ed online charter schools if they offered courses that are The survey administered by Mathematica -paced entirely self -sever percent of schools state they offer some entirely self -paced courses . . Seventy CREDO’s analysis of student acad -paced emic growth finds students attending schools offering self - courses have academic growth in math which is not significantly different from schools not offering self courses , but stronger growth in reading. However, it is reasonable to propose the ability to work paced -paced course is a function of age . Younger students likely require more academic independently in a self -paced courses may differ by school support than older students, thus the impact of participating in self level. Figure 13 shows the effect size of attending a n online charter school which permits some level of self -paced courses by school level. Figure Attending an Online Charter School with Self -Paced Cla sses 13: Relationship between Growth and 108 0.15 0.09* 72 0.10 0.08* 0.06* 0.05* 36 0.05 0.03 0.01 0 0.00 - 0.01 Effect Size Days of Learning -36 -0.05 -72 -0.10 - 0.11 -108 -0.15 High Schools Middle School Elementary Schools All Levels Math Read nline the average o SPED -poverty, non -ELL, non- The 0.00 line for this graph represents charter, White, non student. * Denotes significant at the .05 level. credo.stanford.edu 41

52 Attending a school which allows self courses has a significant positive relationship in reading for -paced levels combined com -paced courses . schools of all pared to online charter schools which do not allow self is positive and significant for middle school Breaking the effect out by school level shows the relationship and high school students, but not significantly different for elementary students relationship in . The for students in math was not significantly math, however, was very different . The overall relationship -paced math classes a positive different from zero, and only for middle school students was access to self benefit on academic growth . While the effect size in math for high school students was large, it was not significant . This means the effect could be due to chance even with its large size. -paced course work . The question The survey results also contain information about the dosage of self asks what percentage of a school’s coursework is entirely self . The responses ranged from five -paced -paced with the most common answer being 100 percent percent to 100 percent of coursework being self . The statistical models show increasing th e percentage of self -paced work has a negative relationship on academic growth in both reading and math. At first, this may not seem logical, especially in reading where -paced courses has a significant positive effect size . But, the a having access to self y pparent inconsistenc can be explained by the concept that just because a proper dose of something is good, it doesn’t mean a larger dose is better. Synchronous vs. Asynchronous Another element of curricula delivery is whether students complete work at the same time as a group or on their own schedule. Synchronous delivery is typically described as all students receiving instruction . Synchronous instruction is exemplified by the model in which the teacher at the same time historical in Online schools can adopt various levels of synchrony teaches a lesson to the entire class all at once. . Some schools may function exactly like the traditional brick -and -mortar school . their curricula deliveries They may require all students to log in at specific times to recei ve instruction with the only difference from a traditional brick -mortar school being that the students are in different physical locations . -and Some online schools fully embrace the asynchronous model by allowing students to complete educational requirements whenever they wish. In fully asynchronous schools, students can meet their educational commitments at odd hours which better fit around the students’ other activities, such as work or training. Even the number of days or number of hours a student must devote to educational experiences can be flexible in a fully asynchronous setting . The Mathematica survey also addresse d the hours of instruction which was synchronous by school level. The statistical models d in either reading or math at any schoo l o not show any significant relationships level based on the hours of instruction which was synchronous . As with self -paced instruction, schools varied greatly on the amount of time they spent in synchronous instruction. Figure 14 contains the number of online ch udents in each school spend arter schools from the survey and the number of hours st in synchronous instruction. credo.stanford.edu 42

53 14: Count of Schools by Number of S Figure ynchronous Hours of Instruction 18 16 14 12 10 8 Number of Schools 6 4 2 0 0.75 0.5 0 5 4 5.5 6 7 8 10 12 15 18 20 25 30 40 3 2 1 Hours per Week of Synchronous Instruction Class Size The Mathematica survey includes information at each scho ol level, elementary, middle, and high, on the . The class size for ranged from one average course size in the online school in both reading and math students per class student per class to 180 . Table 16 has the average class size and maximum class size The impact of class size was significant and positive for middle school and high school -level. school by . While the effect size was very small, only .001, this is the impact per students in both reading and math additional student . Table and Maximum Class Size by School Level 16: Reported Average Maximum Class Size Average Class Size Elementary School 39 70 150 Middle School 60 71 180 High School School and Family Interactions nquiry into the relationship One of the more interesting sets of questions included in the survey was an i is of interest because ing question . This line of between the school and the family receiving services school the online student and their family may be located some s employ a wide variety of policies . As credo.stanford.edu 43

54 l’s center of operations, it is possible no one in the family has ever had an in distance from the schoo - Even if the family has visited the school person interaction with the teachers or a school administrator. operations center, it is still possible the teacher works out of a third location remote to the student and . These remote practices are different from the standard education model whereby the operations center teachers interact with students on a daily basis and provide parents with regular conference res by some online schools from the traditional educational model also include a . Departu opportunities shifting of the responsibility for supervising educational progress and participation from the teacher to the parents. Mathematica The nitors the interactions between the online survey includes a question about who mo teachers and students /parents . The options were: contact is not formally monitored, principal, other school administrator, lead mentor/teacher, other staff not listed . School leaders completing the survey were allowed to choose all answers which applied to their school. Results from the statistical models are 15 . Figure very revealing about the need for someone to monitor these interactions includes the relationships between student scores and attending a school wh ich uses each policy . Relationship between Monitoring Tea cher/Fa mily Interactions and Student Academic Growth Figure 15: 108 0.15 72 0.10 0.06 0.03 0.02 36 0.05 0.00 0 0.00 - 0.03 -36 -0.05 0.04 - 0.06* - 0.07* - -72 -0.10 Effect Size Days of Learning 0.13 - -108 -0.15 -144 -0.20 -180 -0.25 - 0.25** -216 -0.30 Not Monitored Other Other Staff Principal Lead Teacher Administrator Math Read s graph represents the average o SPED non- White, charter, nline -ELL, non- The 0.00 line for thi poverty, non student. ** Denotes significant at the .01 level. * Denotes significant at the .05 level. credo.stanford.edu 44

55 Not formally monitoring the interactions between teachers and families of online charter student s is a large significant negative . The results were also correlated with impact on math academic growth In negative in math when the supervision is delegated to a school administrator other than the principal. reading the only significant result occurs for schools where the interaction is monitored by a non - administrator, non -teacher staff member. Wh at is clear from these results is communication between the school and the family matters for online students, and the existence of that relationship needs to be monitored by someone other than just the assigned teacher to ensure the communication occurs. Part of the reason communication between the schools and the families of online charter students is important may lie in the roles the online school expects parents to fill in their child’s educational experience. Expectations for the role of the parent di ffer across online schools . In the survey, the principals are asked to select from a list of roles the school expects the parent to fill. It is worth noting the affirmative principal’s fulfilling these roles, only that response does not mean the parents are adequately the school has an expectation the parents will provide these supports when . This is a useful distinction how student outcomes interpreting vary with these expectations. role of the parent is likely to change with The ; accordingly, this survey item is asked the age of the student in relation to specific school -levels: elementary, middle school, and high school. Building principals were asked to select all the roles which apply to their school. Some schools selected all possible responses while o thers reported only some or none . Some replies are ubiquitous across all the schools of a level which means a relationship between that reply and student academic growth cannot be estimated. For exa school principals replied that they expect parents to monitor mple, all elementary and middle completion of assignments, this means we cannot estimate how strongly parental review of assignment completion . matters for student performance The strength of the relationships between the online charter school reportin g expected parental roles and and academic growth in reading and math are given in Figures 16 respectively . The only parental 17 roles which ha ve a consistent positive relationship to student academic growth are the expectation of parents verify ing seat ti me . In high school math , most of the parental roles were significant; however, this was primarily due to the fact that high schools which expected parents to actively participate in instruction and attend parent training sessions all also expected parents to monitor assignment . This means those two factors get a boost from the effect of monitoring assignment completion completion . Parents actively participating in instruction and filling other roles both have a consistently ic growth for all groups except high school math . While the statistical negative relationship with academ models in these analyses are not causal, the strong patterns we are seeing suggest the issue may be that schools are holding expectation s for parents which the parents do not meet . It would be hard to explain otherwise why a school expecting parents to actively participate in instruction would have a negative relationship with growth if parents were adequately meeting the expectation. credo.stanford.edu 45

56 16: Relationship between Expected Parental Ro les and Academic Growth, Reading Figure 144 0.20 0.16** 0.09** 0.05** 72 0.10 0.03 0 0.00 0.01 - 0.03 - -72 -0.10 0.09** - Effect Size - 0.14* 0.16** - 0.15* - Days of Learning -144 -0.20 0.17** - 0.20** - -216 -0.30 0.32** - -288 -0.40 Other Role Verify Seat Time Parent Training Actively Monitor Assignment Participate in Sessions Instruction Completion High Middle Elementary White, non- poverty, non nline -ELL, non- SPED The 0.00 line for this graph represents the average o charter, student. ** Denotes significant at the .01 level. * Denotes significant at the .05 level. credo.stanford.edu 46

57 Figure 17: Relationship between Expected Parental Roles and Academic Growth, Math 288 0.40 0.29** 216 0.30 0.15** 144 0.20 0.11** 0.06 0.10 72 0.04** 0.00** 0 0.00 - 0.03 -0.10 -72 0.09 - 0.11** - Effect Size -144 -0.20 Days of Learning - 0.23** -216 -0.30 - 0.26** -288 -0.40 -360 -0.50 0.48** - 0.49** - -0.60 -432 Verify Seat Time Other Role Monitor Actively Parent Training Participate in Sessions Assignment Instruction Completion Elementary High Middle The 0.00 line for this graph represents the average o charter, White, non- nline -ELL, non- SPED poverty, non student. * Denotes significant at the .05 level. ** Denotes significant at the .01 level. Methods of Class Communication Other items on the survey sought information on methods used by online schools to interact with students during instruction . There are questions which focus ed separately on asynchronous instruction and synchronous instructio n. For asynchronous instruction , the respondents are asked to identify all the methods used in their school . Possible survey responses were to engage students in asynchronous learning • Email • access to physical textbooks • interactive online exercises • using other websites with instructional focus or content • recordings of lectures • discussion forums or threaded discussion groups • social media other tools not listed above • credo.stanford.edu 47

58 Of all the options, only two have significant relationships with academic growth. g, having In readin 0 standard deviation (7 effect size in relation to access to recordings of lectures has a positive 0.1 2 days) . In math, having access to physical (paper) textbooks in schools which do not have access reading growth tionship with math growth compared to schools which do not . has 0.09 (65 days) positive rela Practices for synchronous instruction are also included in the survey . Because of the “real time” nature communications metho ds . Table of synchronous instruction, these practices are more centered on live 17 lists the various communication methods and their relationship with academic growth in reading and 17 In both reading and math, using audio conferencing in synchronous instruction has a positive math. . Providin relationship with academic growth g instruction through online chat forums has a strong negative relationship with math growth. Likewise, instant messaging does not appear to be an effective means of communicating “real time” reading instruction to students. 17: Tools Used to Suppor t Synchronous Instruction, Reading and Math Table Reading Math 0.01 0.07 Video Conferencing - 0.19 0.18 Screen Sharing Audio Conferencing 0.29** 0.13* Online Chat Forum 0.19 - 0.54** - Instant Messaging - 0.13** 0.00 Phone Calls 0.03 - 0.17 - 10 0.05 Text Messaging 0. 0.00 0.21** Other * Denotes significant at the .05 level. ** Denotes significant at the .01 level. Another question asks principals if their school provides technological support to students . Options include schools providing an internet connect ion, a computer, computer accessories, or assistive technology for students with disabilities . Of these options, none has a significant relationship with academic growth in either reading or math with the exception of assistive technology in math has a str -0.1 0) significant relationship with academic growth . This relationship emphasizes why we do not ong ( make causal claims in this portion of the study . It is difficult to imagine a situation where providing disabled students with assistive technology would cause the student to experience weaker academic growth . What is more likely is that students with disabilities so severe they require special adaptive equipment may not be fully compensated for in the statistical models which control for the average effect of students being in special education . 17 It is noteworthy that very li ttle impact was identified for these tools in general. Many methods show moderate to strong relationships which are not significant. This may be due to the small number of els cannot replies in that category. Having a small number of replies means the statistical mod differentiate between truly strong relationships and those which falsely appear strong by chance. 48 credo.stanford.edu

59 School -Level Data and School Survey Data level data from the impact analysis to produce school -level fixed effects CREDO used the student- above . Combining rincipal survey measures of academic progress which were then merged with the school p -level data with the survey data provides a slightly different lens through which to view the school -level comparisons provide a wider view of the relationship between the various outcomes . The student rowth, but the results can be heavily influenced by the largest schools which survey topics and academic g will have many more individual student records . By looking at how the survey factors relate to school- ly between the larger and level effect sizes, the weight of the relationships is distributed more even smaller schools . or does not use that Because many of the survey questions ask if a school uses a particular practice , it is possible to use a t -test to estimate the average relationship of that practice to the school’s practice effect size on student growth . This provides additional information beyond that derived from just using a correlation as it provides the reader with additional information on the relative size of the impacts of different educational practices . Many of the survey questions were grouped around related concepts such as parental roles, factors relating to principal experience and compensation, and factors related to school operational policies . The correlations between these questions and the school- level effect sizes have been grouped by general category below. It is worth noting that the survey data was collected from across the nation and values -response. The data included in the correlations below represents a subset of the were weighted for non ta as the below data was limited to only those responses which also had school -level coefficient survey da estimates. The use of a restricted survey data set in this section means the aggregated numbers presented here will likely be different from those presented i n the descriptive volume of the report. For purposes of any national discussion, the reader should refer to the values from the descriptive volume. The complete set of correlations between school- level effect sizes and survey responses is provided in Appe ndix C of this report. The reader should keep in mind that by chance, 5 percent of the correlations will be significant in each subject. To this end, the table in Appendix C includes all of the correlations and their p -values to allow for better interpretation of the significance of the relationship between each condition and the school -level effects. School Wide Policies - Students enrolled in online charter schools, especially asynchronous schools, may experience a variety of expectations on their individ ual participation. The presence or absence of clear- cut policies for student participation could be expected to have a strong relationship with academic growth. The Mathematica survey includes three items relating to student participation. Principals were asked if their school has a school -wide policy spelling out expectations for students in the completion of assignments, class participation, and attendance in synchronous portions of instruction. Only one school in the correlational data did not have schoo l-wide requirements for completion of assignments. Having clearly defined rules for class participation has a positive relationship with academic growth in reading, but the effect was not credo.stanford.edu 49

60 significant in math. However, there was a negative correlation in math between school effect sizes and schools reporting they monitored student participation by the pace at which students completed course assignments. -Wide Policies, Reading and Math Table 18: School Math Reading Effect Size Eff ect Size Correlation Correlation .37* Class Participation 0.14* .25 0.13 Attendance in Synchronous .24 0.06 0.01 Instruction .02 Monitors Pace of Student - .27 Completion of Assignments 0.13 - .38* - 0.29* - * Denotes significant at the .05 level. nstruction in online charter schools, the awarding of course credits Due to the use of asynchronous i based on seat time may not be an appropriate metric. Another means of awarding course credits to ts could earn students is through the assessment of course content mastery. Schools were asked if studen course credits through demonstration of mastery in none, some, or all courses. shows a negative 19 Table correlation exists between holding the policy of allowing mastery based credits in some subjects and school -level effect size in both rea ding and math. 19 also Table level effects and t he frequency with which includes results for correlations between school- schools assessed students. There was no significant correlation between the frequency of assessment of students and student academic growth in math. There was a moderate correlation in reading between more frequent assessments and academic growth for elementary and middle school students. -level effects for high school students. Frequency of assessment was not correlated with school 19: Course Credits and Assessment Frequency , Reading and Math Table Reading Math Correlation Effect Size Correlation Effect Size Seat Time Credits Only 0.03 - .03 - 0.01 .12 Mastery Based Credits in - .35* - 0.09* Some Courses - .33* - 0.12* Mastery Based C redits in All .04 Courses .04 0.02 0.01 School Participates in Title I - .36* - 0.08 .03 0.01 Frequency of Assessment Elementary Grades .49* n/a n/a .16 Frequency of Assessment Middle School Grades .42* n/a .11 n/a Frequency of Assessment .05 n/a - .10 High School Grades n/a * Denotes significant at the .05 level. credo.stanford.edu 50

61 Another set of school- wide policies included in the survey revolved around school funding. The principals where asked if the school received funding based on course completions as opposed to course enrollments, if schools received target funds for providing special education services, and if the school participated in the federal Title I program. Of these factors, only participation in T itle I had a significant relationship and only in reading. One major set of policy decisions which are usually set at the school- wide level is curriculum and instructional practices. The survey included a variety of questions related to the development of curriculum and methods for delivering the curriculum. In reading, receiving curriculum from the - management company was associated with positive school effect sizes. Correspondingly, reporting in nsible for developing curriculum was house developed curriculum and teachers of record being respo None of these policies had significant negatively correlated with school effect sizes in reading. correlations in math. Method of delivery for the school’s curriculum is another important factor which can impact student academic growth. Among the various delivery methods included in the survey, only one the frequent use th grade reading was significantly correlated with school - -guided synchronous instruction in 7 of teacher level effect sizes. ficant in math. A more specific breakout of synchronous The correlation was not signi instruction looked at the number of hours spent in synchronous instruction for each school -level. Most of these correlations were not significant except more hours of synchronous instruction in math w as th grade students. significant and positive for 4 credo.stanford.edu 51

62 Table -Wide Policies Relating to Curriculum and Instruction , Reading and Math 20: School Reading Math Correlation Effect Size Correlation Effect Size Some Curriculum Provided by .5 3* Management Company 0.17* .22 0.10 Majority of Curriculum Developed In -House by - .27* - 0.12* - .04 - 0.02 Individual Course Instructors Teacher of Record Responsible for Developing Curriculum .55* - 0.13* - .24 - - 0.08 Increased Frequency of Teacher -Guided Synchronous th .25 Discus Grade .33 n/a n/a sion 4 Increased Frequency of Teacher -Guided Synchronous th Grade .41* n/a .10 n/a Discussion 7 Increased Frequency of Teacher -Guided Synchronous n/a .07 n/a Discussion High School .16 Time in Synchronous th Grade .10 n/a Instruction 4 n/a .37* Time in Synchronous th Grade - .02 Instruction 7 .10 n/a n/a Time in Synchronous Instruction High School - .25 n/a .01 n/a * D enotes significant at the .05 level. Activities Student Support -and -mortar One issue in which online charter schools may differ substantially from the typical brick . The survey included several question s about various school is in student support activities of the school student support activities taken on by the school. These included activities common to all schools such as one -on -one interventions, providing guidance counselors, assessing student needs. Online charter schools also have some unique supp ort activities such as tech support for students or provision of internet services. The first step to providing services to students is assessing what services each child needs. The survey included a question about actions taken by online charter schools to assess student needs when a new student enrolls in the school. Table 21 shows the relationship between many possible types of entry assessments and school effect sizes. Of the steps listed, only assessments of parental or other home supports and the students’ learning disabilities have a significant relationship with the school effect size. credo.stanford.edu 52

63 Table 21: Entry Assessment for New Enrollees Reading Math Correlation Effect Size Correlation Effect Size .05 0.02 .01 0.01 Academic Skills Language Skills .27 0.07 .30 0.11 - English Potential Barriers for Online Learning 0.05 .12 0.04 .20 Parental or Other Home 0.06 .33* 0.10* Supports .27 .11 0.02 Student Learning Disabilities 0.10* .34* Other Disabilities .11 0.02 0.03 .12 Pull Records from Previous Sc hool - - - - Phone Call to Household 0.07 - .02 - 0.01 .17 .04 .23 - 0.06 - Home Visit - 0.01 - * Denotes significant at the .05 level. -on r, administrator, or parent has One -one interventions are practices taken by a school when a teache concerns that a student requires additional services to achieve academic success . When teachers and students are not physically present in the same location, intervention may look different from the Some online schools have tutors whose sole job is to provide interventions . Other standard classroom. schools expect the teacher to work directly with the students outside of the regular class time. Of course, even online schools are still required to provide special education sup ports required by the student ’s individual education plan (IEP). A series of questions about who provides the one- -one support show on some significant relationships between the provider and student academic growth. 22 shows the relationship between v arious providers and school -level fixed -effects estimates of Table academic growth for elementary students. Providing proper special education support for elementary of non - students in online charter schools is correlated with positive academic growth. Further, the use teacher tutors does not seem to provide the same level of academic growth as receiving one -on -one support from the class teacher in reading. The relationship in math is not significant. The results for middle school and high school students were s imilar to those for elementary students with regards to the use of tutors and coaches. Unfortunately, the number of schools in the upper grade levels -provided and special education faculty -provided one -on -one support was too who do not have teacher small t o compute a value for these relationships at the middle school or high school levels. Additionally, the amount of time a student spend in one -on -one instruction was not significantly correlated with student achievement for students at any level. credo.stanford.edu 53

64 Table -on -One Support to Students, Reading and Math 22: Providers of One Reading Math Correlation Provider Correlation Effect Size Effect Size ELEMENTARY .25 0.08 Teacher 0.01 .03 .45* - 0.09* - 0.03 Tutor/Coach .11 Special Education .53 * Faculty .41 * 0.10* 0.10* MIDDLE SCHOOL .20 - 0.09* .45* 0.05 Tutor/Coach - HIGH SCHOOL Tutor/Coach - .52* - 0.11* 0.001 .003 * Denotes significant at the .05 level. The survey also contained a variety of questions about other support services provided to students . Most of these programs did not have a significant relationship with t he school effect size. Table 23 includes the other support programs which did have a significant relationship with school- level estimates of student . The presence in a school of a program for talented and gifted students being associated with growth stronger growth seems logical. The negative relationships between academic growth and programs to support students who are parents may seem counterintuitive as we would expect those programs to help those students rather than hinder them. However, the fact that students in some schools are dealing with while students in other schools may not face that challenge, thus the school being a parent at a young age does not provide such a program, may explain the negative correlation. Likewise, it is hard to imagine -person tech support harms a student’s academic growth. Rather, students families which that in from have such a low level of computer literacy that they require outside support to set up their computer likely have other challenges which are the actual cause of the negative correlations. Finally, an increase in the number of guidance counselors serving an online school was correlated with significant correlation in math. significant positive growth in reading and a non- Table 23: School -Provided Supports Math Reading Effect Size Correlation Effect Size Correlation .27 Talented and Gifted Program 0.09* .41* 0.08 Programs for Students Who Have Children - .09 - 0.02 - .31* - 0.10* 0.09* In Up of Computer - .39* - Person Set - .23 - 0.08 - Guidance Counselors .39 * n/a - .01 n/a * Denotes significant at the .05 level. credo.stanford.edu 54

65 School and Family Interactions -level analysis, several elements of school and family interactions had significant In the student level analysis although the . This still holds true in the school- relationships with student achievement The differences are related to the weighting of the student values relationships are not all the same. which result from looking at the relationships using average school effect sizes instead of individual student values. For the student -level data, schools in which parents were expected to be actively involved in their child’s instruction have a negative relationship with growth . In the scho ol-level analysis, we again see a negative will actively participate in the student’s ’ expectation that parents schools relationship between instruction Table 24 ). For the remainder of parental roles, the results were and academic growth (see either not significantly related or could not be measured due to the small sample size and a lack of variation in responses. and Math 24: Relationship between Expected Parental Roles and Academic Growth, Reading Table Reading Math Correlation Effect Size Cor relation Effect Size Provider ELEMENTARY Actively Participate .29 .42* - 0.08* - 0.07 in Instruction Parent Training .06 0.02 .03 Session 0.01 Verify Seat Time .02 0.00 .21 0.05 MIDDLE SCHOOL Actively Participate in Instruction - .27 - 0.06 .24 0.07 Pare nt Training .22 - 0.01 - .03 - 0.08 Session - - Verify Seat Time - 0.02 .14 0.04 .10 HIGH SCHOOL Actively Participate in Instruction - .21 - 0.05 .24 0.08 Parent Training Session .01 0.00 .10 - 0.03 - Verify Seat Time - .05 - 0.01 .08 0.02 * Denotes signific ant at the .05 level. Professional Development and Compensation One of the processes by which schools support teacher improvement is through professional development opportunities. The survey inquired about the frequency and delivery format of professiona l development within online charter schools. The only format of professional development which had a significant correlation with student academic growth in ei ther math or reading was online -deliver ed credo.stanford.edu 55

66 profession development. There was a negative correlatio -.38) between the increasing frequency of n ( -delivered teacher professional development and student growth in math. The relationship in online reading was not significant. The correlations between the frequency of in- person teacher professional Schools which report having nt and student growth was not significant in math nor reading. developme teachers observed by master teachers or teaching coaches had significantly lower effect sizes in math than those who did not. One practice which did have a signific ant positive relationship with school effect sizes was providing teachers with diagnostic test results at the individual student level for purposes of planning instruction. This correlation was .34 in reading, but not significant in math. Professional Development Activities, Reading and Math Table 25 : Teacher Reading Math Correlation Correlation Frequency of Online - .03 - .38 * Professional Development Frequency Observed by and Received Feedback from - - .37* Master Teacher .10 Frequency Provided with Dia gnostic Test Results for Individual Students .34* .04 * Denotes significant at the .05 level. Schools also have a variety of professional development for school leaders. Among those included on the ant correlation with school survey, only site visits to other schools had a signific -level effect size. In schools where school leaders reported visiting another school for the purpose of improving their own work as a school leader in the past 12 months, the correlation with school effect size was .35 in reading . : School Leader Table 26 Professional Development Activities, Reading and Math Reading Math Correlation Effect Size Correlation Effect Size University Coursework - .04 - 0.01 - .01 - 0.00 .20 0.08* Visits to Other Schools 0.06 .35* Coaching by Leader of .16 0.04 Another School .01 - 0.00 - Participating in School Leader Network .20 0.05 .28 0.11 Workshop Presenter .17 0.04 .07 0.02 Workshop Participant - .31 - 0.10 - .11 - 0.05 * Denotes significant at the .05 level. credo.stanford.edu 56

67 Teacher incentives are another policy area which varies from school to school . Charter schools have more in the methods used to compensate teachers than the traditional public schools . The flexibility he online d a series of questions about factors that impact teacher salaries for t questionnaire include if a teacher would be paid more as a result of the factors listed in charter school . The questions ask ed . The two . Most of the options do not have a significant relationship with growth Table 27 below pay based on student growth a nd on the teacher holding an advanced degree . These except ions were Interestingly, while course were significantly related to student growth in reading. two factors completion as an influencing factor on teacher compensation was not significantly correlated with school .45) with school- -level effects, including student course completion was negatively correlated (- level effect size in math. While not direct compensation per se, tenure can also be an important means of rewarding teachers. We found a significant positive corr elation between teachers’ ability to earn tenure and school effect sizes in reading but not in math. Table 27: Influencing Factors for Teacher Compensation, Reading and Math Reading Math Correlation Correlation Effect Size Effect Size .08 Teacher Evaluati 0.06 0.02 on .29 Student Growth .41* 0.09* .30 0.09 Student Proficiency .08 0.02 .21 0.08 Course Completion Rates .17 0.06 .26 0.12 Holds Advanced Degree 0.08* .05 0.01 .39* 0.03 .19 0.06 Years Experience .12 .11 0.03 .25 Multiple Certifications 0.10 Hard to - Staff Position .09 - 0.02 .13 0.04 Number of Students - .20 - 0.06 .03 0.01 Mentor to Other Teachers .10 - 0.02 - .08 0.03 Teachers Can Earn Tenure .31* 0.11* .13 0.06 * Denotes significant at the .05 level. A similar question relating to comp ensation for school leaders was also included in the survey (see Table 28). The only compensation factor which had a significant relationship with student achievement level was student proficiency level. The correlation between basing school leader salary on student achievement level was .45. This significant relationship was not present in reading. credo.stanford.edu 57

68 Table 28: Influencing Factors for School Leader Compensation, Reading and Math Reading Math Correlation Effect Size Correlation Effect Size - .07 - 0.02 Number of En .06 - 0.02 rolled Students - 0.05 .24 0.07 Student Achievement Growth .22 .28 0.06 .45* 0.14* Student Proficiency Level Course Completion Rates .05 0.01 .25 0.09 0.04 .03 0.01 Reenrollment Rates .18 - .31 - 0.10 - Retention of Teachers - 0.05 .11 School Profit .23 0.05 .17 - 0.05 - * Denotes significant at the .05 level. Throughout the various related concepts, we did not find factors which impacted both reading and math. Likewise, we did not find consistent groups of factors within a concept which had significant relationships with school effect sizes. The absence of clear sets of factors which have a relationship with school effect sizes was in itself an interesting finding. The school -level survey did not reveal clear group of mechanisms level effect sizes. by whi ch to influence school- School leaders have a wide variety of responsibilities in any school. While the school leader of an online charter school has many responsibilities in common with the leader of a brick -and -mortar school, the online school may demand a different balance of responsibilities and that rebalancing may result in different outcomes. While we do not have comparative data, school leaders were asked to report what percentage of their time they spent on a variety of activitie s. We computed the correlation between the -level effect size on student academic growth. percent of time spent on several activities and the school School leaders spending higher percentages of their time with students, including discipline and academic gu idance, was correlated with higher school -level effects in reading. None of the other school -level effect size. leader activities was significantly correlated with school Table 29: Percent of School Leader Time by Task, Reading and Math Reading Math Cor relation Correlation Internal Administrative Tasks - .21 - .30 Observing Teachers .09 .07 Working with Teacher Coaches or Other .13 Instructional Leaders .04 Developing or Leading PD .21 - .01 - Reviewing Student Achievement Data .38* .30 Student Interac - .12 .05 tions Parent Interactions - .06 .28 * Denotes significant at the .05 level. credo.stanford.edu 58

69 Non Significant Findings - level analysis, we evaluated the relationships between the survey responses and As part of the school- each individual school’s estimated effect size in both math and reading. The majority of the relationships - were not significant. Table 30 contains a partial list of survey response items found to have non The full set of correlations is pro vided in significant correlations with student academic growth. Appendix C of this report. Table -Significant Correlations with Math and Reading Effect 30: Survey Items of Interest with Non Sizes Survey Item School monitors synchronous seat time Percentage of coursework which is self - paced ze Average class si School size Student Testing Data and Policy Changes . In our US Constitution, education is one policy domain that is relegated to state authority and control ution. As such, The individual’s right to a free public education is guaranteed in each state’s state constit every state has the duty to set the policies which govern the operation of schools within their state. This means education practices permissible in one state, may be banned in another. In fact, several states allow neither online schools n or charter schools at all . In the second volume of this report, the Center on Reinventing Public Education (CRPE) conducted an ols (Pazhouh, Lake, and Miller analysis of state education policies as they relate to online charter scho . They found that in those states which do allow online schools, policies governing online charter 2015) school s vary . Further, individual states can and do change their policies independent ly. This leads to a pattern of occasional policy shifts as es but others do not ; the overall some states change their polici pattern of policy shifts across all the states can be exploited for research purposes . We can use statistical models which allow us to examine the differences in student academic growth which correspond to the existence and changes in an individual state’s online school polices . In their analysis, CRPE identified education policies which may have a relationship with the academic level growth of online charter school students. CREDO then computed correlations between school- effects and the presence of three of these policies. The three policies included were: authorizer oversight fees, the existence of for -profit online charter schools with state- wide enrollment policies in a state, and if a state had specialized oversight p rovisions specifically for online charter schools. Authorizer oversight fees are fees charged to the charter schools by the organizations who authorize and have oversight authority over the charter schools. These fees are usually computed as a percentage of the per -pupil funding received by the charter school. As Pazhouh, Lake, and Miller state in their policy credo.stanford.edu 59

70 revi “Fees from large online schools can come to represent a large proportion of agency operating ew, ate and close consistently low -performing online charter revenues and may create a disincentive to regul -profit and state schools. ” The second factor, for -wide enrollment documents the presence of policies withi n the state which allow for BOTH the operation of for -profit charter schools and the ability of online charter schools to enroll students from any location within the state. Finally, s ome state laws include unique oversight and accountability provisions specific to online charter schools. Most of these provisions are partial measures, addressing authorizing entities and processes, special application requirements (i.e., technology plans), or accountability provisions regarding the frequency and manner of reporting. Table 31 below shows a significant negative relationship between authorizers collect ing oversight fees and student academic growth in math. Having online charter specific oversight policies and stronger charter laws in general have a significant and positive relationship with math academic growth. In reading, only the strength of the sta te’s charter law had a significant relationship with academic growth. These correlations fit the narrative provided by CRPE in the second volume of this report. 31: Correlations between Education Policies and School -Level Effects Table Math Reading orizer Oversight Fees - 0.21* - 0.12 Auth For - Profit and State - Wide Enrollment 0.19 - 0.11 Specialized Oversight Policies - 0.19 0.20* 0.25* Strength of Charter Law 0.33* 0.32* Strength of Charter Law Ranking 0.06 * Denotes significant at the .05 level. During the data window of this study, there were four policy changes which were likely to impact online charter schools . As these changes occurred over time within a state, we used student -level data to estimate a yearly school effect and then compared those school effects before and after the specific policy change. 32 shows the average change in academic growth associated with the Table implementation of each policy. Table 32 also contains a list of topics included in the regulation change. Details on the policy changes are available in the Pazhouh, Lake, and Miller volume of this report. Due to the existence of multiple simultaneous policy changes, it is not possible to disentangle which aspect of each law holds the causal mechanism hievement. in relation to student ac credo.stanford.edu 60

71 Table -Level Policy Change Description 32: State CO MN OH OH State HB HB SF- HB -3660 -2301 1528 1277 -11- - 0.07 - 0.27** Effect Size 0.16** 0.17* Policy Topics Contained in Law Accountability X X X X Oversight/Governance X X X Authorizing Communication X X X X Quality Review Funding X Enrollment Processes/Caps X X X Teacher Licensure X Assessment Equipment/Internet Access X * Denotes significant at the .05 level. ** Denotes significant at the .01 level. po licy change in Colorado was not correlated with a significant change in academic growth The . The policy change in Minnesota was associated with a large significant negative effect size . Both of the policy changes in OH were associated with stronger academic growth . The difficulty in making an analysis such as this is accounting for multiple policy changes in each law . For example, the list of changes associate d with Ohio SB 2301 cover several policies could be due to more student ac cess . The positive relationship brought about by the elimination of the enrollment cap and the requirement for districts to release up to three percent of students to attend online charter , or t he positive impacts could also be the result schools certification . The current data did not allow us to tease of the requirements for teachers to have state- out these possibilities. Over more time, comparing multiple changes in multiple states could allow more refinement of which policies are having what impact . Unfortunately, we are limited by the number of changes which took place within the data window of our study. Summary and Implications The purpose of this report was to present to online education stakeholders data -based information on the academic impact of attending online charter sch . The report combined student -level data, ools school -leader survey responses, and state policy data . Using academic data, we compared the growth of students attending online charter schools to that of students in TPS and students in brick- and -mortar charte r schools. We also combined student -level data with information from a survey conducted by Mathematica Policy Research . This mixed methods analysis permitted us to examine the relationship credo.stanford.edu 61

72 between a variety of online charter school policies and student aca . We also included demic growth Reinventing Public Education’s review of state policies . As online charter information from the Center on -depth examinations of the -studied area, this report represents one of the most in schools are a seldom topic. harter students had weaker growth than their VCRs. While results vary for each student, the Online c data showed the majority of online charter student records had academic growth in both math weaker and reading compared to their VCRs . The pattern of weaker growth remained consistent across racial - . Online charter schools were found to reduce the negative ethnic subpopulations and students in poverty impacts on growth in math for students who were English language learners and special education students relative to t -ELL and non- SPED peers compared to the size o f the negative impacts for heir non ELL and SPED VCRs to the the ELL and non- SPED VCRs . non- Pre The study of student -online mobility is the same for online charter students and their VCRs. nts who eventually enroll in online schools have pre- online mobility rates mobility showed stude charter similar to those of their VCR comparisons. However, after enrolling in online charter schools, these mes higher than their peers. students tend to become more mobile changing schools at a rate 2 to 3 ti Twenty -two percent of online charter students eventually return to TPS sector with the average time in an online charter school being two years. Positive growth acro ss a sector . Some online charter schools which w ere part of multi - is possible school networks had average impacts on academic growth which were stronger than the typical online charter. Online charter schools in Wisconsin and Georgia had academic growth in reading which on . Thes e findings show average was stronger than their VCRs it is possible for online charter schools to . outcome produce stronger growth, but it is not the common Few school practices had a strong relationship with academic growth . A review of the -level as rep orted in the Mathematica survey relationship between school practices and student academic growth found insignificant correlations between school practices and growth . Of practices in the mostly -paced clas ses survey which had strong positive correlations, attending schools which offered some self was the most wide -spread and was found to be consistent across all school levels . The findings on the expected parental roles was also revealing in that placing more instructional responsibilities on parents was strongly correlated with weaker growth across most settings. Teasing out the impact of state The role of state- level policies matter s in -level policies is difficult. . The s tate online charter education policy changes included in the study did have significant -level relationships with the academ ic growth of online charter students . With the data included in this analysis, it was not possible to tease out which aspects of the particular policy changes led to the changes in academic growth . This is a critical area for future study. Being an o school matters more than being a charter school. Finally, the major impacts of nline attending an online charter school appear to be primarily driven by the online aspect of the schools . credo.stanford.edu 62

73 Analyses comparing online charter school students to brick -mortar charter students produced -and results which were nearly identical to the results derived from comparisons of online charter students and TPS students . If the charter aspect of online charter schools or an interaction between the charter and online aspects were the driving factors of online charter school growth, we would have expected to find different results between the brick -mortar charter analysis and the TPS analysis . We did not. -and Implications iety to think beyond the bounds of Finding the best means to educate every student will require soc traditional schools . Online schools are a relatively new and rapidly expanding method of providing an alternative to traditional schools. The findings presented in this report establish a starting point for future implications of attending online charter schools . discussing the 1. Current online charter schools may be a good fit for some students, but the evidence suggests that online charters don’t serve very well the relatively atypical set of students that currently at tend these . Academic benefits from online charter schools are currently schools, much less the general population Online charter schools provide a maximum of flexibility for students the exception rather than the rule. PS setting . This can be a benefit or a liability as f lexibility requires with schedules which do not fit the T . Not all families may be equipped to provide the discipline and maturity to maintain high standards direction needed for online schooling . Online charter schools should ensure their p rograms are a good fit for their potential students’ particular needs. 2. Current oversight policies in place may not be sufficient for online charter schools . There is evidence that some online charter schools have been able to produce consistent academic benefits for students, . The charter bargain has been “Flexibility for Accountability” but most online charter schools have not s must be held to that concept . Authorizers must step up to their responsibilities and all charter school and demand online chart er providers improve outcomes for students . Authorizers should hold a firm line hose schools which cannot meet their end of the charter bargain. with t 3. States should examine the current progress of existing online programs before allowing expansion . Online schools have the potential to serve large numbers of students with practically no physical restraints on their expansion . As such, mechanisms which have typically played a role in regulating the growth of brick -and -mortar schools such as facility constr uction and limited potential student pools do not exert pressure on online schools . Without these natural constraints, online schools have the potential to expand more rapidly than traditional schools . This makes it critical for authorizers to ensure onlin e charter schools demonstrate positive outcomes for students before being allowed to grow and that online charter schools grow at a pace which continues to lead to improved outcome s for their students. credo.stanford.edu 63

74 Appendix A: DESCRIPTIVE PROFILE OF ONLINE TUDENTS CHARTER S 33 shows the number of students from each state by year included in the study sample. Table This count represents tested students with at least two years of data who were enrolled fulltime in the identified, . As can be , there was a wide variation in online charter seen in Table wholly online charter schools 33 enrollment across the states . Additionally, some states have stable enrollment patterns while others . In some states, the online charter enrollment rate increase have rapidly increasing enrollment numbers d ten -fold over the course of three years . The rate at which online charter enrollment is increasing in some . states provided emphasis on the need and timeliness of this study 33: Number of Matched Online C harter Students by State and Ye ar, Math Table 2009 - 10 2010 - 11 2011 - 12 State 2012 - 13 Total 236 235 228 1,166 AR 232 4,303 3,201 3,240 4,166 17,118 AZ 6,260 7,769 9,519 CA 38,400 9,845 CO 1,456 2,935 3,961 4,043 14,920 14 33 29 27 117 DC 68 6 25 107 FL 6 2,299 2,975 4,676 GA 4,012 15,436 IL 337 389 439 493 1,658 191 1,067 50 1,941 3,269 IN LA 0 0 467 927 1,394 MI 253 466 605 1,552 119 395 455 477 292 1,905 MN 3,334 2,912 2,743 11,655 NV 1,840 5,309 6,245 6,012 OH 6,582 27,772 OR 1,515 1,600 1,857 1,997 7,887 7,704 9,011 6,784 9,935 39,540 PA TX 364 802 3,492 5,603 10,269 UT 903 1,108 967 3,596 488 WI 439 682 336 ‡ 1,466 Total 31,005 39,087 50,432 55,202 199,227 ‡ 2012- 13 data was not available for Wisconsin. n in Table 2 . Figure The demographics of the matched sample are similar to the rates show shows the 18 race -ethnicity of the students in the brick -district VCR matched sample . The matched sample was made up predominantly by White students . One -in-four students in the matched sample were Black or Hispanic with Asian, Native American, Multi -Racial students making up the remainder of the sample. While the online charter demographics differ from those of both brick -and -mortar district and charter schools, they are similar to the demographics of online school s operated by distric ts. credo.stanford.edu 64

75 18: Race -District VCR Matched Sa mple Data Set , Math -Ethnicity of Brick Figure 2% 2% 3% 3% 3% 4% 100% 1% 1% 1% 1% 1% 1% 3% 3% 2% 3% 3% 2% 90% 11% 13% 11% 13% 14% 15% 80% 11% 11% 12% 12% 13% 12% 70% 60% 50% 40% 72% 70% 69% 69% 67% 67% 30% 20% 10% 0% 2008-09 2011-12 Total 2009-10 2010-11 2012-13 Native American Multi-Racial White Hispanic Black Asian , the percentage of students in poverty attending online As shown in Table 2 in the main body of the report her than the entire brick charter schools is lower than that of the feeder schools, but hig -district sector in the studied states 19, the percentage of students in poverty enrolled in online charter . Based on Figure schools has increased over the time of the study . The percentages of ELL students and special education students are steady across the years . As noted previously, the percentage of ELL students enrolled in online school s is much lower than in brick -and -mortar schools . This was true regardless of whether the online school was a district -run or a charter run school. credo.stanford.edu 65

76 Figure -District VCR Matched Sa mple Sub -Popula tions by Year, Math 19: Brick 100% 90% 80% 70% 60% 49% 47% 46% 50% 45% 44% 41% 40% 30% 20% 13% 13% 13% 13% 13% 12% 10% 1% 1% 1% 1% 1% 1% 0% 2010-11 Total 2008-09 2011-12 2012-13 2009-10 Students In Poverty SPED ELL The students in online charter schools were more likely to come from the lower deciles of academic the beehive graph represents . In Figure 20, the width of the block in achievement than the TPS students the percentage of students from each decile of achievement on their state’s proficiency exam in the year An equal distribution of students across all before the student enrolled in an online charter school. deciles would produce a cylinder shape in which every band is the same width. The difference in the width of the top and bottom bands indicates higher enrollment of lower achieving students in online charter schools . Fourteen percent of online charter students were in the fi rst (lowest) decile; whereas, only five percent of online charter students were in the highest decile. credo.stanford.edu 66

77 -Online Achievement Decile of Online C harter Students, Math Figure 20: Pre 10 5% 9 8% 8 9% 7 9% 6 9% 5 10% 4 11% State Achievement Decile 3 12% 2 13% 1 14% While there are some differences in the populations attending TPS, the TPS which lost students to online charter schools, and online charter schools, the sample used in our analysis uses pairs of students who are matched on observable characteristics which are known to have an impact on educational growth ched groups are identical or near- and achievement identical on all the match criteria shown in . The mat . Due to the high match rate (96%), we can be confident that the sample of matched students is Figure 1 the study states . By using test highly representative of the full population of online charter students in scores from before enrolling in online schools for our online students in addition to the other demographic factors, our matching process has included a proxy for the sum impact of the all the factors, . The prior test score ervable, which impact the students’ educational outcomes observable and unobs thus represents the sum educational progress of the student before entering an online charter school; have are identical in observable characteristics who students and have the same prior test score likely unobservable student characteristics with the same total impact on achievement for the student and . This holds true even if those unobservable at the time the students were matched their twin . The identical prior test identical between the student and their twin characteristics are not necessarily . This supports the score then functions as a proxy for the unobservable characteristics of the student dents. matched data set as a strong and proper counterfactual for the online charter stu credo.stanford.edu 67

78 Appendix B: TECHNICAL APPENDIX we After constructing a VCR for each charter student, then set out to develop a model capable of . The National Charter School Research Project provide measure of charter impact providing a fair d a very 18 useful to consider student growth rather than . First, it was begin the process useful guide to d a strong method to control for each student’s educational achievement . A growth measure provide t their academic ffec history as well as the many observable differences between students that a . The baseline model include d controls for each student’s grade, race, gender, free or achievement reduced price lunch status, special education status, English language learner status , and whether they 19 found that the . The were held back the previous year literature on measuring educational interventions best estimation techniques must also include controls for baseline test scores . Each student’s prior year Additional controls are also in , and cluded for state, year test score is controlled for in our baseline model. st nd year in charter, etc.) . The study’s baseline model is presented below. year in charter, 2 period (1 where the dependent variable is And A z-score for student i in -test is the state -by -test z-score for student i in period t; A -by is the sta te it-1 it 1; period t – is a set of control variables for student characteristics and period, Y is a year fixed effect, X i,t t is a state fixed effect; C is an indicator variable for whether student i attended an online charter i n S ε period t; and is the error term. Errors are clustered around charters schools and their feeder patterns as well. In addition to the baseline model above, we explored additional interactions beyond a simple binary to and “triple” interactions between the online charter enrollment . These included both “double” indicate schools on impact of charter . For example, to identify the charter variable and student characteristics the charter variable into “ online different racial groups, we estimate models that break online charter_black,” “ charter_hispanic,” etc. To further break down the impact of online charters by online race and poverty, the variables above were split again . For example, black students in charter schools are 18 Betts, J. and Hill, P. et al. (2006). “Key Issues in Studying Charter Schools and Achievement: A Review and Suggestions for National Guidelines.” National Charter School Research Project White Paper Series, No. 2. 19 Betts, J. and Tang, Y. (2011) “The Effect of Charter Schools on Student Achievement: A Meta -Analysis of the Literature.“ National Charter School Research Project. 68 credo.stanford.edu

79 alify for free and reduced price lunches (“charter_black_poverty”) and split further into students that qu those that do not (“charter_black_nonpoverty”). As part of the study, we conducted additional analyses using alternative model specifications . The purpose of using additional specifica tions is to ensure the robustness of the results, i.e. ensure the . The alternative specifications for this study findings were not an artifact of the analytic model chosen included completing the analyses using a data set made with VCRs from brick -mort ar charter school -and s on achievement rather than students, conducting two different ordinary least squares (OLS) model -year panel of student data for all students with test scores in the states included in growth using a multi OLS comp arison s intended to explore how choice related bias might impact the the study, and a set of report findings . The model for the OLS comparison (see model 3 below) was similar to model 1 with the exception that the dependent variable was growth. The results of these analyses are i ncluded later in this appendix. + = θ A ε + β X + + ρ Y C A σ S + γ t i,t -1 i,t i,t i,t i,t (3) We also examined the relationship between student records and responses to the survey administered to school leaders the records of students who . We assigned the schools’ responses from the survey to attended those online schools . We then dropped all students who attended schools which did not have a survey response . This analysis used a model which was a slight variation on model 1 above . A Δ θ A + β X + ρ Y + σ S + η Q + ε = i,t i,t i,t -1 t i,t s (4) Where Q represents the array of responses on the survey for a given online charter school. The other s variables were identical to those in model 1 above. The errors were still clustered around charter schools. Empirical Bayesian Shrin kage Tables 13 and 14 in the main body of the report include marginal and full estimates of growth by network for students who attended an online charter school which was part of a charter network . One of the reviewers suggested we might need to conduct em pirical Bayesian shrinkage to adjust the estimates due to the differences in the number of students included in each group . We computed the estimated coefficients applying empirical Bayesian shrin kage and found the adjusted estimates were similar to the un adjusted estimates . None of the estimates changed the level of significance or change d by a noticeable amount 34 includes the results for both original estimates and the adjusted estimates of network . Table marginal growth relative to non -network online charter schools . The values in Table 34 are comparable to those in Table 13. credo.stanford.edu 69

80 Table Compared to Independent Online 34: Empirical Bayesian Shrinkage of Effect Sizes by Network , Reading and Math Charter Schools Math Reading with EB with EB Reading Shrinkage Shrinkage Math 0.13** 0.06 0.06 0.16** Network 1 0.08** 0.08** 0.26** 0.26** Network 2 0.08** Network 3 0.07** 0.05 0.04 0.04** 0.04** 0.08** 0.07** Network 4 0.03 0.19** 0.18** Network 5 0.03 0.03 Network 6 0.02 0.02 0.03 Network 7 0.00 0.00 0.07 0.06 0.03 - 0.02 - 0.02 - 0.03 Network 8 - - 0.02 - 0.02 0.02 0.02 Network 9 - 0.04 - 0.04 - 0.05** - 0.05** Network 10 - Network 11 0.05* - 0.05* - 0.04 - 0.03 Network 12 - 0.05** - 0.05** 0.03 0.03 Network 13 0.06 - 0.05 - 0.04 - 0.03 - 0.06** - 0.06** - 0 .01 - 0.01 Network 14 - 0.07** 0.07** - Network 15 - 0.07** - 0.07** - 0.08** - 0.08** - 0.05** 0.05** Network 16 Network 17 - 0.09** - 0.08** - 0.10** - 0.09** Network 18 0.12** - 0.12** - 0.13** - 0.12** - - Network 19 0.17** - 0.16** - 0.26** - 0.25** Network 20 - 0.19** - 0.18** - 0.27** - 0.26** 0.15** Network 21 - 0.25** - 0.24** - 0.15** - * Denotes significant at the .05 level. ** Denotes significant at the .01 level. credo.stanford.edu 70

81 Alternative Specifications - and Mortar Charter School VCR Brick - stical models which compared the matched VCRs made contains information from the stati This section . Table includes the demographic -and 35 up of brick -mortar charter schools to online charter students . descriptive output for the brick -and -mortar charter school VCR data set emographics by Charter Sector Student Population D 35: Table Online C Charter Feeder harter Schools All Charters Schools 5,534 906 166 Number of Schools 51% 49% 48 % Percent Students in Poverty Percent English Language Learner 9% 7% 1% Students Percent Special Education Students 9 % 11% 9% 33% 42% 69% Percent White 21% 13% Percent Black 30% 29% 27% 11% Percent Hispanic 4% 4% 2% Percent Asian/Pacific Islander 1% 1% Percent Native American 1% - Percent Multi 3% 3% 4% Racial Average Total Enrollment per School 344 525 986 Total Enro llment 1,901,109 476,044 1 63,722 Table . The 36 includes the effect sizes for attending online charter schools for various subpopulations results while slightly different were similar enough to those found in the comparisons between TPS VCRs arter students to not merit repeating in the main body of the report. and online ch The marginal results are provided here for those with an interest in the results from this second control group. 36: Effect Size by Subpopulations for Online Charter vs. Brick- Char ter , Reading and Math Table Standard Standard Reading Error Math Error - Overall 0.12** 0.01 - 0.25** 0.01 White - 0.12** 0.01 - 0.23** 0.01 Black 0.08** 0.03 - 0.23** 0.02 - - Hispanic 0.13** 0.01 - 0.28** 0.02 Asian - 0.08** 0.01 - 0.23** 0.02 Native American - 0. 07 0.05 - 0.32** 0.04 The effects in this table represent the difference between a student of a specific race in TPS and a student of the same race in an online charter. * Denotes significant at the .05 level. ** Denotes significant at the .01 level. credo.stanford.edu 71

82 Figur 22 contain the effect sizes by state from attending an online charter school for reading es 21 and 21 with Figure 22 for math 5 and and math respectively for reading and Figure 6 . Comparing Figures shows t here is some variation in state effect sizes betwee n the two VCR groups, but in general the effect - sizes by state for the TPS VCR comparison in the main body and the effect sizes by state for the brick -and mortar VCR analysis are of the same direction and a similar magnitude . The similarity in results indi cate s the online nature of the online charter schools is a much stronger driver of their effectiveness than the charter nature . If the charter aspect had a stronger influence, the effect sizes between online charters -mortar charter VCRs would -and and brick differ more from the effect sizes between online charters and TPS VCRs. 21: Online Charter Effect Size by State for Online Charter vs. -Charter , Reading Figure Brick Days of Learning -360 -288 -216 -144 -72 0 72 AR - 0.11** AZ - 0.12** CA - 0.12** CO - 0.12** FL 0.16** - GA 0.01* IL 0.01** - LA - 0.32** MI 0.03* - MN - 0.16** NV 0.19** - OH 0.09** - OR 0.10 - PA - 0.15** TX 0.20** - UT - 0.15** WI 0.01 - -0.20 -0.10 0.00 -0.30 -0.40 -0.50 0.10 Effect Size for this - -poverty, non non White, -Mortar Charter, -and represents the average Brick graph The 0.00 line -SPED student. ELL, non * Denotes significant at the .05 level. ** Denotes significant at the .01 level. credo.stanford.edu 72

83 22: Online Charter Effect Size by State Brick -Charter , Math for Online Charter vs. Figure Days of Learning -360 -504 -288 -216 -144 -72 0 72 -432 AR 0.10** - AZ - 0.25** CA 0.30** - CO 0.21** - FL - 0.59** GA - 0.24** IL - 0.07** LA 0.48** - MI - 0.07* MN 0.21** - NV 0.28** - OH 0.18** - OR 0.21** - PA 0.20** - TX 0.41** - UT 0.35** - WI 0.03 -0.30 -0.40 -0.20 -0.70 -0.10 -0.60 0.10 0.00 -0.50 Effect Size non The 0.00 line for this graph represents the a verage Brick -and -poverty, non - -Mortar Charter, White, -SPED student. ELL, non * Denotes significant at the .05 level. ** Denotes significant at the .01 level. OLS Model on Multi - Year Panel Data Generalized For the panel data OLS analysis, we used achievement as the dependent variable. This was done to 20 s from the OLS Partial output ensure the findings were not directly related using a growth measure. The values for state- -level controls are not included for regressions are shown in Table 37. level and grade the sake of space. In reading, attending an online charter school had a significant negative effect size of . In math, the effect size for online charter attendance was -0.135, equivalent to 97 days less learning ing . Growth for students attending an online charter school were -0.347, equivalent to 250 days less learn significantly weaker than that of brick -and -mortar charter students . These findings support those presented in the main body of this report. 20 , where z_subj was the student’s achievement in a The growth measure used was z_subj – z_subj t0 t1 given year. 73 credo.stanford.edu

84 Table OLS Regression Output , Reading and Math 37: Panel Data Unrestricted Standard Standard Reading Math Error Error 0.610** 0.000 0.620** 0.000 z_orig_subj 0.105** 0.000 0.089** 0.000 z_orig_other_subj brick 0.000 - 0.011** 0.000 charter_ 0.009** - charter 0.135** 0.002 - 0.347** 0.002 online 0. 067** 0.000 - 0.024** 0.000 female - 0.132** 0.000 - 0.128** 0.000 lunch - 0.000 - 0.149** 0.000 ELL 0.330** 0.488** 0.001 - 0.517** 0.001 SPED - 0.053** 0.000 retained 0.104** 0.000 - re_black - 0.150** 0.000 - 0.177** 0.000 - 0.057** 0.000 - 0.052** 0.000 re_hisp re _asianpi 0.069** 0.000 0.152** 0.000 re_nativam 0.100** 0.001 - 0.097** 0.001 - - 0.001 - 0.026** 0.001 re_multi 0.015** 0.012** 0.000 0.016** 0.000 year_2009 0.008** 0.000 0.009** year_2010 0.000 year_2011 0.017** 0.000 0.022** 0.000 0.001 _cons 0.001 0.152** 0.129** Obs 55281185 54030479 R - 0.608 0.589 Sqr * Denotes significant at the .05 level. ** Denotes significant at the .01 level. - Year Panel Data Restricted OLS Model on Multi ta set . The restrictions to the data set We also analyzed an additional OLS model with a restricted da online school observation (they were in an online charter removed all students who did not have a pre- . during their first year in the data set) and limited the analysis to the first year in an online charter school These restrictions allowed us to isolate the specific impact of going to an online charter school, ensuring that estimated effects were not biased by treatment occurring in prior years. This method has been 21 Table 38 includes the shown to successfully replicate “gold -standard” e . xperimental impact estimates regression results for this analysis . The results of the restricted analysis showed a stronger negative trend than did the unrestricted OLS analysis . Students who attended an online charter analysis had significantly weaker growth in both reading with an effect size of - 0.239, equivalent to 172 days less learning and in math with an effect size of - 0.445, equivalent to 320 days less learning . 21 ., Furgeson, Chiang, H., Teh, B., Haimson, Gill, B and Verbitsky -Savitz , N. “Replicating Experimental J., J., pliance.” -Group Noncom Impact Estimates in the Context of Control Statistics and Public Policy, forthcoming. credo.stanford.edu 74

85 To examine if the declining achievement for online students w as in fact just a continuation of previously e computed the pre- declining achievement, w online charter growth trend for the students who would In the year before they entered an online charter school, the future eventually change to an online school. online students had negative academic growth . The change in reading achievement for this group in the year before they entered an online charter school was - 0.06 in reading, equivalent to 43 less days of 0.08 in math, equivalent to 58 days of lea . The conclusion of these analyses was that rning learning, and - while it was true students who eventually transferred to online charter schools had negative growth in l TSP before transferring, the steep decline in their growth after transferring to an online charter schoo online charter trajectory were not likely to be the indicated that the circumstances which lead to pre- source of the students’ lowered academic achievement found while attending an online charter school. 22 OLS Regression Out Table Panel Data Restricted 38: put, Reading and Math Standard Standard Error Reading Math Error 0.599** 0.000 0.610** 0.000 z_orig_subj 0.000 0.172** 0.000 z_orig_other_subj 0.188** brick 0.028** 0.000 0 .000 charter_ 0.000 onlinecharter - 0.239** 0.003 - 0.445** 0.003 0.081** 0.000 - 0.035** 0.000 female 0.096** 0.000 - 0.114** 0.000 lunch - - 0.273** 0.000 ell - 0.080** 0.000 sped - 0.265** 0.000 - 0.235** 0.000 retained 0.123** 0.002 0.137** 0.002 re_black - 0.116** 0.000 - 0.135** 0.000 re_hisp 0.059** 0.000 - 0.046** 0.000 - pi 0.000 0.132** 0.000 re_asian 0.048** 0.095** 0.001 - 0.101** 0.001 re_nativam - - 0.012** 0.001 re_multi - 0.022** 0.001 year_2009 0.001** 0.000 0.012** 0.000 year_2010 0.006** 0.000 - 0.001* 0.000 year_2011 0.009** 0.000 0.014** 0.000 _cons 0.001 0.080 ** 0.001 0.082** Obs 38278136 39526810 R - Sqr 0.649 0.605 -SPED The 0.00 value for this table represents the average TPS non- poverty, non -ELL, non , White, VCR student. * Denotes significant at the .05 level. ** Denotes significant at the .01 level. 22 Due to data access limitations, Table 30 did not include data for TX or IL; whereas, Table 29 did. We verified the unrestricted panel data coefficients were the same in models with and without TX and I L included. credo.stanford.edu 75

86 Onli ne Charter School Choice Analysis next We -matched” model s which explored “chooser deeper the impact of enrollment selection using two included only those students who attended online charter schools . Both of these models included nt variable and included controls for the student demographic achievement as the depende . characteristics as well as state specific dummy variables to control for mean differences between states model, we kept the records for only the students’ first year in an online c harter school and the e first In th year after the first year in an online charter school regardless if the second year was in an online charter 23 Students who would shows that or not . . Figure 23 online charter school students had negative growth eventually end up stayi had weaker first -year growth in ng in an online charter for only one year, leavers, . online charters than those students who would stay at least two years in an online charter school, stayers Both leavers and stayers had stronger growth in their second year than in their first year in an online school ; however, the growth in the second year was significantly smaller for those students who charter spent their second year in an online school, stayers, TPS compared to those students who returned to a in their second year , leavers . Figure 23: Average Growth for First Year in Online Charter and Subsequent Year by Stayer/Leaver Status 0.40 0.32 0.30 0.18 0.20 0.10 score - 0.00 0.00 - 0.09 0.06 - -0.10 Growth in Z 0.19 - -0.20 0.19 - -0.30 - 0.36 -0.40 First Year First Year Second Year Second Year Reading Math Stayers Leavers -Attending student. The 0.00 line for this figure represents the average Online Charter Ever 23 – A Growth=A -1 i,t i,t 76 credo.stanford.edu

87 We conducted regressions for both rea ding and math using the same data set as used for the above . We included a variable which indicated if the student remained in an online charter school in the graph The students who stayed in an online charter schoo l for the second year or returned to a TPS school. TPS were represented by second year were represented by the coefficient stayer . Those who returned to the coefficient leaver . The stayer coefficient is the marginal difference between the students who school, remained in online charter schools for the second year and the students who returned to a TPS leavers . The average change in achievement for the leavers is represented by the coefficient leaver . In reading, the students who left online charter schools after one year had second year growth of 0.33 equivalent to 238 days of additional learning) . The average growth of students standard deviations ( the 0.16 standard deviations who remained in online charter schools lagged behind that of those who left by - (the equivalent of 115 days less learning). Continuing Online Charter Enrollees Compared to One Year Enrollees , 39: Table – Marginal Results Reading se Coefficient 0.64** z_orig_read 0.003 z_orig_math 0.18** 0.003 stayer marginal to - 0.006 leaver 0.16** 0.33** 0.006 leaver 0.09** female 0.0 04 lunch - 0.08** 0.004 - 0.15** 0.018 ell sped - 0.18** 0.008 retained 0.012 0.19** - 0.006 re_black 0.06** 0.05** 0.006 re_hisp - 0.05** 0.010 re_asianpi re_nativam 0.07** 0.020 - 0.011 re_multi 0.00 year_2009 - 0.09** 0.006 year_2010 - 0.13** 0.006 year_2011 0.13** 0.008 - 0.011 _cons 0.01 Obs 107106 R - Sqr 0.654 The 0.00 value for this table represents the average Online Charter, White, poverty, non -ELL, non - non- SPED student. * Denotes significant at the .05 level. ** Denotes significant at the .01 level. credo.stanford.edu 77

88 In math, the differences between stayers and leavers was even larger . The students who left online TPS school had second year growth charter schools to return to a of 0.55 standard deviations (equivalent . Those who r to 396 days of lea average emaine rning) d in online charter schools had growth which was on -0.39 standard deviations (equivalent to 281 days of learning) less than the students who left online consistent with . The direction and magnitude of the coefficients from this analysis were charter schools those of the other analyses conducted. Table , 40: Continuing Online Charter Enrollees Compared to One Year Enrollees – Marginal Results Math se Coefficient 0.003 z_orig_math 0.60** 0.20** 0.003 z_orig_read stayer marginal to - leaver 39** 0.006 0. leaver 0.55** 0.007 - 0.004 Female 0.06** 0.09** 0.004 lunch - 0.018 ell 0.017 sped - 0.12** 0.007 0.13** 0.011 retained re_black - 0.09** 0.006 re_hisp 0.05** 0.006 - 0.12** 0.012 re_asianpi 0.08** 0.019 re_nativam - - 0.013 0.012 re_multi year_2009 0.08** 0.006 - 0.15** year_2010 0.006 - year_2011 - 0.11** 0.008 _cons - 0.15** 0.011 Obs 103136 - Sqr R 0.631 The 0.00 value for this table represents the average Online Charter, White, non- poverty, non -ELL, non - SPED student. * Denotes si gnificant at the .05 level. ** Denotes significant at the .01 level. The last analysis we conducted was the future online charter choosers analysis . For this analysis, we kept only students who would eventually attend an online charter school but who atten ded a TPS during their first year in the data set . We then kept their first year in the data set and their first year in an online charter school . We created a variable to indicate their enrollment in an online charter school . The model included student ac hievement as the dependent variable and student demographic characteristics as credo.stanford.edu 78

89 24 The regression results in Table 41 show attending an online charter school had independent variables. 0.17 standard devi ations . Likewise the a significant negative impact on reading achievement, - (122 days) -0.34 standard impact on math achievement of attending an online charter school (see Table 42) was (245 days) compared to the students’ first year in the data set. deviations 41: Future Online Charter Choosers, Readin g Table se Coefficient 0.6 5 ** z_orig_read 0.003 7 ** 0.003 z_orig_math 0.1 - 0.22** 0.052 read_missing - 0.17** 0.009 onlinecharter female 0.0 9 ** 0.004 0.07** 0.004 lunch - - 0.1 4 ** 0.016 ell ** 0.20 sped 0.008 - ** retained 0.008 0.07 re_black - 0.06** 0.006 re_hisp - 0.0 3 ** 0.005 re_asianpi 0.009 0.06** - 0.017 re_nativam 0.03 3 * 0.011 re_multi 0.0 0.02 ** year_2009 0.008 year_2010 0.0 3 ** 0.009 0.0 8 ** 0.010 year_2011 _cons - 0.0 2 0.010 Obs 120376 - R 0.622 Sqr The 0.00 value for this table represents the average TPS VCR , White, non -poverty, non -ELL, non -SPED student. * Denotes significant at the .05 level. ** Denotes significant at the .01 level. 24 -level dummy variables were included in the model, but are not State dummy variables and grade shown in the results table to conserve space. credo.stanford.edu 79

90 Table Future Online Charter Choosers, Math 42: Coefficient se 0.6 z_orig_math 0.003 1 ** 17** 0.003 z_orig_read 0. math_missing 2.1 2 ** 0.176 - - 0.34** 0.009 onlinecharter - 0.04** 0.004 female lunch 0.0 7 ** 0.004 - 0.0 3 0.016 ell - - 0.1 7 ** 0.008 sped retained 0.007 0.02** re_black 0.0 9 ** 0.006 - re_hisp - 0.0 4 ** 0.005 re_asianpi 0.011 0.13** - 0.017 re_nativam 0.06** 1 0.011 re_multi 0.0 0.0 1 year_2009 0.008 year_2010 - 0.00 0.009 0.07** 0.010 year_2011 _cons - 0.0 6 ** 0.010 Obs 118157 - R 0.612 Sqr The 0.00 value for this table represents the average TPS VCR , White, non -poverty, non -ELL, non -SPED student. * Denotes significant at the .05 level. ** Denotes significant at the .01 level. credo.stanford.edu 80

91 Appendi C: CORRELA TES OF SCHOOL -LEVEL EFFECTS WITH SURVEY x REPSONSES vey of Appendix C contains correlations between school -level effect sizes and the responses to the sur online charter school practices conducted by Mathematica. Correlations could not be computed for survey items with inadequate variation of responses. For example, if all the responses to a binary question (yes/no) were the same, a correlation cann ot be computed. Items for which a correlation could -“. not be computed are marked with a dash “ Table 43 includes the correlations and p -values for each item with sufficient variation. Those values “*”. Due to the high number of correlations which are significant at the .05 level are marked with a computed, it is likely at least some (5%) will be significant by chance. Based on the statistical principles used in this study, we expect 12 of the significant results in each subject to be the result of chan ce. To aid -value for each correlation. A lay explanation the reader in interpreting the results, we have included the p -value is that the p -value represents the likelihood a correlation is the result of chance. The lower of the p the p -value; the lower the likelihood that the result is due to chance. The traditional threshold for -value of .05 or less. Correlations with large p determining significance is a p -values should be considered to be due to chance regardless of the strength of the correlation. The column Response Type in Table 4 3 provides information on the type of response possible on the survey. The description ‘ binary ’ means the value of “1” was entered in the field if the practice in the survey question existed at the school and “0” if it did not. This means a positive correlation indicates that the presence of the practice described was related to stronger growth than the average online charter while a negative correlation indicates the presence of the practice school was related to weaker growth. The description ‘ ascending ’ means the value was dosage -based and coded with a higher number if the is related to condition occurred more frequently. Thus a positive correlation means more of the practice on ‘descending ’ means the value was do sage -based and coded stronger growth. Finally, the descripti with a lower number if the practice occurred more frequently. For a descending item, a positive correlation would indicate having less of the practice present in the school is associated with stro nger growth. Readers are advised to pay attention to the Response Type as it will have an impact on the interpretation of the results. For dosage based variables, the correlations were produced using standard Pearson correlations. Correlations between b inary variables and school -level effects were computed using a point bi -serial model which produces correlations between a binary and continuous variable equ ivalent to Pearson correlation. credo.stanford.edu 81

92 Table Correlations of School -Level Effects with Survey Resp onses, Math and Reading 43: Coeffi p - Coeffi p - Response cient cient value value Sig Sig Type Does your school’s program enable students to earn course credits by demonstrating mastery , regardless of “seat time”? No, students in all courses must meet seat requirements 0.12 0.3811 - 0.03 0.8143 Binary time any Yes, students in course can earn course credit by demonstrating 0.04 0.7478 0.04 0.7504 Binary mastery selected Students in courses, subjects, or grades can earn course credit through demonstration of mastery - 0.35 0.0056 * - 0.33 0.0098 * Binary Does your school’s program include courses that are entirely self - paced ? 0.00 0.9830 - 0.12 0.4232 Binary What percent of your courses are entirely 0.10 - - 0.05 0.7699 - 0.5871 Ascending self paced? total , how much time In is spent in synchronous instruction, , each week for an average student 0.10 0.5759 in the fourth grade? 0.37 0.0308 * Ascending How many students are involved in a typical fourth -grade math 0.7554 0.1122 section? 0.37 Ascending 0.07 H ow many students are involved in a typical fourth -grade English / Language Arts section? 0.37 0.1122 0.07 0.7554 Ascending credo.stanford.edu 82

93 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type In an average week, does a typical student in fourth grade spend -one any time in one -on with a interaction teacher or tutor (via chat, phone, tutoring, - 0.19 0.3034 - 0.30 etc.)? Binary 0.1061 How much time, on average, does a typical student in fourth grade -on -one spend in one interaction with a teacher or tutor (via chat, phone, tutoring, - 0.07 0.7214 - 0.17 0.4 157 Ascending etc.) per week? one on - one instructional support to students in fourth grade? Who provides - Teacher of record for the 0.2392 0.03 0.25 0.8863 Binary course - 0.45 0.0255 * - 0.11 0.6152 Binary Tutor/Coach [Removed] Binary f, Other instructional staf not listed above - - - Specify: - - Binary Other teacher - - - - Binary Special education faculty 0.0076 * 0.41 0.0478 * Binary 0.53 - - - Parent Binary - instructional method How frequently are the following (s) used in fourth grade? - 0.12 0.5693 Lecture 0.16 0.4331 Descending Teacher - guided synchronous discussion - 0.33 0.1128 - 0.25 0.2294 Descending Collaborative learning involving two or more students working 0.1895 0.07 0.7302 - 0.27 together Descending Individualized, student - dri ven independent study - 0.29 0.1611 - 0.33 0.1060 Descending What role, if any, is a expected to play to support the educational parent or guardian program of a student in the fourth grade ? Make sure the student keeps up with assignments - - - - Binary credo.stanford.edu 83

94 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type Actively participate in the student’s instruction 0.42 0.0376 * 0.29 0.1593 Binary - Participate in parent 0.7698 0.03 0.06 0.9054 Binary training sessions - 0.02 0.9197 0.21 0.3047 Binary Verify seat time Other role, not listed above Specify : - - - - Binary - total In , how much time is spent in synchronous each week , instruction, for an average student - 0.02 0.8937 in the seventh grade? 0.10 0.5566 Ascending How many students are involved in a typical seventh -grade math section? 0.26 0.2591 0.03 0.8993 Ascending How many students are involved in a typical -grade English / seventh Ascending section? 0.27 0.2393 0.03 0.8936 Language Arts In an average week, does a typical student in seventh grade spend any time in one -on -one interaction with a (via teacher or tutor chat, phone, tutoring, etc.)? - 0.23 0.1697 - 0.28 0.0834 Binary In an average week, how much time does a typical student in seventh grade spend in one -on -one interaction with a teacher or tutor (via chat, phone, tutoring, et c.) per week? - - - - Ascending Who provides one on - one instructional support to students in seventh grade? - Teacher of record for the - - course - - Binary Tutor/Coach - 0.45 0.0177 * 0.20 0.2870 Binary [Removed] Binary credo.stanford.edu 84

95 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type staff, Other instructional not listed above - - - - - Binary Specify: - - - Other teacher Binary - - - - - Binary Special education faculty - - - - Binary Parent instructional method (s) used in seventh grade? How frequently are the following 0.5929 0.316 2 - 0.10 0.19 Descending Lecture - - guided Teacher - 0.41 0.0209 * - synchronous discussion 0.5600 Descending 0.10 Collaborative learning involving two or more students working 0.21 0.2550 0.01 together Descending 0.9564 Individualized, student - driven independ ent study - 0.19 0.3093 - 0.19 0.2996 Descending What role, if any, is a parent or guardian expected to play to support the educational seventh grade ? program of a student in the Make sure the student keeps up with - - assignments Binary - - participate in Actively - 0.27 0.1392 0.24 0.1779 the student’s instruction Binary Participate in a parent 0.8642 - 0.22 0.03 0.2145 Binary training sessions - Verify seat time - 0.10 0.5810 0.14 0.4393 Binary Other role, not listed above - Specify : - - - - Binary In total , how much time is spent in synchronous instruction, , each week for an average student - 0.25 0.1350 in high school? 0.9715 Ascending 0.01 How many students are involved in a typical high school math section? 0.14 0.5604 0.01 0.9552 Ascending How many students are involved in a typical high school English section? 0.14 0.5592 0.02 0.9470 Ascending credo.stanford.edu 85

96 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type In an average week, does a typical high school student spend -one any time in one -on interaction with a teacher or tutor (via chat, pho ne, tutoring, - - - - etc.)? Binary In an average week, how much time does a typical student in high -on - school spend in one one interaction with a teacher or tutor (via chat, phone, tutoring, - - - - Ascending etc.)? - - one instruc tional support to students in high school? Who provides one on Teacher of record for the - - - - Binary course 0.9830 0.52 0.0030 * 0.00 Tutor/Coach Binary - removed Binary Other instructional staff, not listed above - - - Specify: - - Binary Other teacher - - - - Binary Special education faculty - - - Binary - Binary - - - Parent - instructional method How frequently are the following (s) used in high school? - 0.25 0.1729 Lecture 0.08 0.6406 Descending - Teacher - guided synchronous discussion - 0.16 0.3884 - 0.07 0.7145 Descending Collaborative learning involving two or more students working 0.25 0.1733 - together 0.8594 Descending 0.03 Individualized, student - driven independent study - 0.34 0.0593 0.01 0.9396 Descending What role, if any, is a expected to play to support the educational parent or guardian program of a student in high school ? Make sure the student keeps up with assignments - - - - Binary credo.stanford.edu 86

97 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type Actively participate in the student’s instruction 0.21 0.2477 0.24 0.1681 Binary - ticipate in parent Par training sessions 0.9637 - 0.11 0.5422 Binary 0.01 0.6621 - 0.7852 0.08 0.05 Binary Verify seat time Other role, not listed - Specify : - - - - Binary above sponse below Where does the school’s curriculum content come from? Please select the re majority of your school’s curriculum. that best applies to the Purchased from outside - 0.15 0.2514 - 0.13 0.3277 Binary provider(s) Provided by a school management organization that 0.16 0.2347 - oversees our school 0.6897 Binary 0.05 eloped in - house and Dev used by all instructors of the relevant courses - 0.09 0.5100 - 0.01 0.9579 Binary - Developed in house by individual course - 0.12 0.0362 instructors - 0.04 0.7762 Binary * Who monitors teachers’ contact with students and parents? Con tact is not formally - - - - Binary monitored - 0.2482 - 0.07 0.5890 Binary Principal 0.16 Other school - 0.01 0.9178 - - administrator Binary Lead mentor/ teacher 0.01 0.9482 - 0.03 0.8233 Binary Other staff, not listed 0 - above 0.5708 - 0.01 .9672 Binary 0.08 Do you have school - wide policies spelling out expectations for students in terms of ... Completion of assignments? - - - - Binary Class participation? 0.0303 * 0.25 0.1384 Binary 0.37 Attendance in 0.24 synchronous instruction? 0.1577 - 0.02 0.9211 Binary Does your school monitor attendance or student participation in any of the following ways? Pace of student’s completion of course assignments - - - - Binary Activity in the online system - 0.27 0.0858 - 0.38 0.0097 * Binary credo.stanford.edu 87

98 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type me involved in Seat ti synchronous work with a 0.20 0.22 0.1423 0.2004 Binary teacher Other measure of completion of course Specify: - - work - - Binary - programs and supports schools can offer to students. For each, please Below is a list of er your school offers this program or support. indicate wheth - on - one tutoring for One - - - - Binary struggling learners Supplemental group instruction for struggling - 0.12 0.4924 - 0.29 0.0903 Binary learners Dropout prevention or dropout recovery 0.10 0. 5664 0.9369 Binary program 0.01 - skills classes 0.08 0.6694 - Study 0.4671 Binary 0.13 Clubs or activities (e.g., literary magazine, cultural activity groups, 0.14 0.4254 pep club) 0.5030 Binary 0.12 Mental/behavioral health services 0.07 0.6747 0.03 0.852 6 Binary Music instruction 0.2931 - - Binary 0.19 0.8439 0.3574 - 0.03 Binary Fine arts instruction 0.16 Specialized instruction -language for English 0.3123 learners 0.11 0.5316 Binary 0.18 Speech and language therapy or services - - - Binary - Talented/gifted program 0.41 0.0156 * 0.27 0.1150 Binary Other services for students with IEPs - - - - Binary Please indicate whether your school offers any of the following programs or supports to high school students. Advanced Placement 0 .10 0.5568 Courses - 0.17 0.2763 Binary International Baccalaureate program - - - - Binary Supports for students who have children of their own - 0.09 0.5986 - 0.31 0.0462 * Binary credo.stanford.edu 88

99 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type How many students at this school participate in the dropout drop or out prevention 0.3620 0.8885 - recovery 0.03 Ascending program? 0.19 How many students at this school participate in Advanced Placement 0.1375 - 0.08 0.6758 0.29 Ascending ? Courses How many students at this school participate in the International Baccalaureate - - - - Ascending program? On average, approximately how often do teachers conduct assessments of students in a typical ... th 4 grade math section - 0.49 0.0197 * - 0.16 0.4728 Descending th grade English / 4 - 0.49 0.0197 * - 0.16 0 .4728 Descending Language Arts section On average, approximately how often do teachers conduct assessments of students in a typical ... h grade math section - 0.37 0.0547 * 7 - 0.11 0.5809 Descending th 7 grade English / Language Arts section - 0.42 0.0294 * - 0.14 0.4644 Descen ding On average, approximately how often do teachers conduct assessments of students in a typical ... 0.03 0.8825 0.10 0.5685 Descending High school math section High school English 0.7400 section 0.7862 0.04 - Descending 0.05 tically conduct an entry assessment for students who have just Does the school systema enrolled using any of the following measures or methods? Academic skills 0.05 0.7860 0.01 0.9332 Binary 0.30 - 0.27 0.1419 English 0.0823 Binary language skills Potential barriers for onli ne learning 0.20 0.2769 0.12 0.5132 Binary Level of parent or other home supports for online 0.0539 0.1290 0.33 * Binary learning 0.27 Learning Disabilities 0.11 0.5338 0.34 0.0495 * Binary Any disabilities other than learning disabilities 0.11 0.55 64 0.12 0.4990 Binary Pull student’s records - - - - from previous school(s) Binary credo.stanford.edu 89

100 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type 0.17 0.3412 - 0.02 0.9053 Binary Phone call to household - - 0.04 0.8268 0.23 Binary Home visit 0.2044 ments in any of the following Does this school promote student performance on state assess ways? Test preparation embedded in regular 0.09 0.6025 - 0.09 0.5979 Binary courses Separate test preparation course required in relevant 0.25 0.1557 0.17 0.3340 Binary grades/ subjects Intensive, targeted support for students who may have difficulty achieving proficiency standards on state 0.33 0.0583 0.17 0.3268 Binary assessments How frequently does your school actively send parents information on their child’s progress via email, phone, or postal 0.11 0.4843 - 0.05 mail? 0.7277 Ascending Does this progress report for parents include a measure of student engagement or participation? - - - - Binary How does your school respond when students are identified as disengaged? - - - - Binary Email parent - - - all to parent(s) Binary Personal c - Automated call to parent(s) 0.16 0.3054 - 0.07 0.6454 Binary Visit home - 0.03 0.8655 0.18 0.2433 Binary - Enlist social services 0.10 0.5260 0.09 0.5592 Binary Offer student incentive to participate 0.03 0.8628 0. 00 0.9868 Binary Other response, not * specify: 0.33 0.0341 listed above 0.20 0.1887 Binary – Letter mailed to home 0.21 0.1824 0.21 0.1734 Binary Are any of the following tools used to support ? asynchronous instruction Email - - - - Binary credo.stanford.edu 90

101 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type ysical (paper) Ph 0.17 0.3425 0.14 0.4357 Binary textbooks 0.04 - 0.10 0.5846 0.8443 Binary Online textbooks Interactive online - - - exercises Binary - Other websites with instructional focus or - 0.8888 content 0.03 0.2815 Binary 0.19 gs of lectures 0.06 0.7528 - 0.14 0.4373 Recordin Binary Discussion forums or threaded discussion groups 0.23 0.1970 - 0.01 0.9478 Binary - 0.6497 - 0.01 0.08 0.9743 Binary Social media (blogs, wiki) - Other tool not listed above (specify) 0.24 0.1790 0.02 0.9180 Binary - Are any of the following tools used to support ? synchronous instruction Video conferencing 0.03 - 0.14 0.4460 (Skype, FaceTime, etc.) - 0.8802 Binary Screen sharing/web conferencing 0.01 0.9636 - 0.24 0.1785 Binary Audio conferencin g 0.34 0.0551 0.10 0.5689 Binary 0.4793 0.9243 - 0.13 Binary Online chat forum 0.02 Instant messaging (IM) or - on - one chats - 0.30 0.0983 - 0.14 0.4381 Binary other one Phone calls - 0.03 0.8525 - 0.03 0.8523 Binary 0.07 0.28 0.1687 - - 0.7366 Binary Text messaging Other tool not listed above (specify) 0.0346 * 0.33 0.1431 Binary 0.47 What types of technology, if any, does this school provide, without charge, to students? Internet connection (e.g. internet service or subsidy for internet rvice, modem, router, se 0.13 0.4555 and/or hotspot) - 0.08 - 0.6500 Descending Computer (e.g. laptop or desktop computer, or tablet computer such as iPad) 0.03 0.8607 - 0.08 0.6493 Descending Computer Accessories (e.g. webcam, microphone, head set, cd/dvd drive, printer, or 0.8928 scanner) 0.08 0.6556 - 0.02 Descending credo.stanford.edu 91

102 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type Assistive Technology for students with disabilities 0.9780 - 0.32 0.0627 Descending 0.01 Does the school provide tech support to teachers in any of the following ways? No tech support is provided to teachers at - - - - Binary this school Live, personal support 0.11 0.4172 - 0.19 - 0.1542 Binary (via phone or chat) Manuals, written guides, - 0.02 0.8683 - 0.17 0.2035 Binary or FAQ documents Other support, not listed above (specify) - - - - Binary When is live, personal tech support available to teachers? Weekdays during - - - business hours Binary - 0.34 0.0557 0.20 0.2507 Binary Weekday evenings 0.12 Weekends 0.4952 0.25 0.1443 Binary - students How is tech support provided to ? No tech support is provided to students at - - this school - - Binary Manuals, technical guides, FAQ documents - 0.10 0.4596 - 0.23 0.0898 Binary Live phone or chat support - 0.06 0.6775 - 0.10 0.4799 Binary Troubleshooting via remote control of - 0.10 0.4830 - 0.1433 Binary computer 0.20 - 0.19 0.1620 0.23 Online ticketing system Binary 0.0830 In person set up of - 0.23 0.39 0.0033 * computer - - 0.0795 Binary Other support, not listed above (specify) - - - - Binary When is live, person al tech support ? available to students Weekdays during business hours - - - - Binary 0.0580 0.21 0.30 0.1827 Binary Weekday evenings Weekends 0.10 0.5213 0.11 0.4786 Binary In , how many total teachers are currently employed at this school? (ful l - time) 0.26 0.1020 - 0.06 0.7234 Ascending credo.stanford.edu 92

103 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type total , how many In teachers are currently employed at this - 0.05 0.7765 0.05 0.7410 time) Ascending - school? (part total What is the number of full -time equivalent (FTE) teachers employed by 0.1654 - 0.07 0.6750 0.23 Ascending the school? other instructional and support staff (including those contracted How many of the following for services) work in this school (in FTE units)? Teacher aides/instructional 0.7362 0.13 0.4602 assistants Ascending 0.06 0.22 ors 0.06 0.7673 - - 0.2590 Ascending Tut 0.39 0.0266 * - 0.01 0.9696 Guidance counselors Ascending Other instructional Ascending 0.3043 - 0.07 0.7021 support staff 0.20 From the list below, please rank the three most important factors when deciding wh ich candidates to offer jobs. Commitment to this school’s mission / willingness to work hard - 0.01 0.9152 - 0.16 0.2334 Binary Certification status (holds a valid teaching 0.2465 - 0.15 0.15 0.2493 Binary certificate) - College grade point ) - - - - Binary average (GPA College major in content area to be taught 0.08 0.5689 0.04 0.7600 Binary Score on a test (e.g. Praxis) - - - - Binary Experience teaching courses online - 0.17 0.1823 - 0.19 0.1377 Binary General experience as a teacher 0.04 0.7 652 - 0.04 0.7379 Binary - - - - Binary Master’s degree Performance in teaching sample class 0.16 0.2099 0.06 0.6566 Binary Quality of candidate’s -service teacher pre training program - - - - Binary credo.stanford.edu 93

104 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type Other factor(s), not listed - - - - - Binary specify: Are your school’s teachers covered by a collective bargaining 0.09 0.6093 0.22 0.2049 agreement? Binary Do teachers come to a central location to do most of their online teaching, or do they do most of their teaching homes ? - 0.18 0.2527 - 0.09 0.5722 Binary from their Is the teacher of record for a particular class responsible for ... - 0.27 0.1216 - 0.05 0.7929 Binary Lesson planning? Binary 0.55 0.0009 * - 0.24 0.1634 Developing curriculum? - - 0.4679 Lecturing? 0.26 0.1352 Binary 0.08 - - - - Binary Grading student work? One - on - one tutoring? 0.21 0.2353 0.03 0.8589 Binary Identifying struggling learners? - - - - Binary Communicating with - - - - Binary parents? Managing online learning environments (e.g. online forums or cussion boards)? 0.00 0.9845 dis 0.15 0.1409 Binary - Troubleshooting 0.05 0.10 0.5890 technical issues? - 0.7620 Binary - Other - Specify: 0.05 0.7998 0.14 0.4218 Binary Which of the following statements best describes the expectation for most 4th grade teachers of core academic subjects (reading, math, science, or social studies)? th - grade core Most 4 academic teachers - - specialize in a subject - Binary - Most 4th - grade core academic teachers are generalists, responsible for multiple subjects 0.03 0.8129 - 0.10 0.4421 Binary Approximately how many students, in total, is a full -time 4th grade teacher typically expected to teach? 0.05 0.7952 0.22 0.2648 Ascending credo.stanford.edu 94

105 p - Coeffi p - Response Coeffi cient value value cient Sig Type Sig Approximately how many students, in total, th is a full -time 7 grade teacher typically - 0.05 0.7558 - 0.04 0.7964 Ascending expected to teach? Approximately how many students, in total, high -time is a full school teacher typically - 0.02 0.8930 expected to teach? 0.6477 Ascending 0.08 Approximately how many students, in total, t-time 4th grade is a par teacher typically - - - - Ascending expected to teach? Approximately how many students, in total, th -time 7 is a part grade teacher typically expected to teach? - - - - Ascending Approximately how many students, in total, -ti me high is a part school teacher typically - - - expected to teach? Ascending - Does a typical fourth - grade math class include instructional staff in addition to the teacher (e.g. aides, tutors)? 0.39 0.0779 0.28 0.2145 Binary Does a typical - fourth grade Eng lish / Language Arts class include instructional staff in addition to the teacher (e.g. aides, tutors)? 0.36 0.1029 0.23 0.3035 Binary credo.stanford.edu 95

106 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type seventh Does a typical - grade math class include instructional staff in addition to the teacher (e.g. aides, 0.31 0.1102 0.17 0.3674 Binary tutors)? seventh Does a typical - grade English / class Language Arts include instructional staff in addition to the teacher (e.g. aides, 0.23 0.2564 0.05 0.7846 tutors)? Binary Does a typical high - school math class include i nstructional staff in addition to the teacher (e.g. aides, tutors)? 0.15 0.4445 0.18 0.3192 Binary Does a typical high class school English include instructional staff in addition to the teacher (e.g. aides, 0.14 0.4663 tutors)? 0.8709 Binary 0.03 Does this school provide teachers with paid time for professional development? - - - - Binary During the 2013 - 2014 school year, how frequently did a typical teacher participate with other teachers from this school in synchronous, online professional development? - 0.03 0.8807 - 0.38 0.0263 * Ascending credo.stanford.edu 96

107 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type - 2014 During the 2013 school year, how frequently did a typical teacher participate with other teachers from this school in in - person professional development at a 0.02 0.9206 0.11 0.5 423 Ascending central location? - During the 2013 2014 school year, how frequently did a typical teacher participate with other teachers in regular faculty meetings (online or in - 0.18 0.3214 - 0.03 person) for this school? 0.8671 Ascending How many times during the 2013 - 2014 school year did teachers experience the following at your school? Observed by and received feedback from a peer - 0.02 0.9206 - 0.01 0.9375 Ascending Observed by and received feedback from a master teacher or someone else who - 0.1 0 0.5730 - 0.37 0.0279 * Ascending coaches teachers Observed by and received feedback from a principal , administrator, or someone else who 0.19 0.2804 monitors performance 0.13 0.4711 Ascending Provided with diagnostic test results for individual students to help them determine which 0.34 0.0531 * topics/skills to focus on 0.8230 Ascending 0.04 Asked to submit lesson plans to master teacher, department chair, principal, or other 0.1069 administrator for review - 0.04 0.8407 0.28 Ascending credo.stanford.edu 97

108 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type Attended workshops, confer ences, or other based kinds of group- 0.0784 0.10 0.5682 0.05 training Ascending most important factors considered when evaluating three Please rank, in order, the teachers at this school. Observations of - 0.9328 0.12 0.377 6 0.01 Binary teacher’s instruction Teacher’s accessibility to students (e.g. logs of student- teacher communication, response time to student inquiries, time to grade - 0.14 0.2755 0.01 0.9224 Binary and return assignments) Feedback from other teachers or instructional 0 .25 0.0559 0.6356 Binary coaches 0.06 Feedback from students 0.5290 0.17 or parents 0.1866 Binary 0.08 Student course completion rate 0.3215 - 0.45 0.0003 * Binary 0.13 Student achievement growth 0.21 0.1101 - 0.09 0.4710 Binary Portfolio of examples of student work (e.g., student essays, lab - - - - Binary reports) Meeting expectations for 0.00 0.9755 - 0.17 0.1869 student engagement Binary Other factor(s), not listed - - - - specify: Binary - Are teachers in your school paid more based on any of the following: Teacher evaluation results 0.29 0.1061 0.08 0.6693 Binary Student achievement growth 0.41 0.0202 * 0.0901 Binary 0.30 Student proficiency levels 0.08 0.6577 0.21 0.2333 Binary Course completion rates 0. 0.17 0.3588 0.26 of students 1327 Binary credo.stanford.edu 98

109 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type Advanced degrees, such as master’s degrees or 0.39 0.0266 * 0.05 0.7928 Binary doctoral degrees 0.12 0.19 0.5041 0.2763 Binary Teaching experience 0.1487 0.5412 0.25 Binary Additional certifications 0.11 - to - staff Filling a hard on 0.4603 0.09 0.6099 0.13 positi Binary Number of students 0.20 0.2615 0.03 - 0.8731 Binary taught Serving as a mentor or - 0.10 0.6005 0.08 coach to other teachers Binary 0.6361 Can teachers at this 0.0501 * 0.13 school earn tenure? 0.31 Binary 0.4097 What opportunities do instructional staff in your school have to take on additional responsibilities to advance their careers? Supervise junior teachers (as a department chair or lead teacher) 0.25 0.1668 0.00 0.9964 Binary Become an instructional coa 0.28 0.1185 0.03 0.8650 Binary ch or master teacher Teach more and/or larger classes - 0.19 0.3057 - 0.07 0.7014 Binary Lead professional development for groups 0.25 0.1652 - 0.01 of staff 0.9591 Binary Approximately how long do teachers stay with th e school on average (months)? - - - - Ascending Approximately how long do teachers stay with the school on - 0.13 0.4500 0.27 0.1087 average (years)? Ascending Throughout the school year, what percentage of your work week, on average, do you spend o n the following tasks in this school? Internal administrative tasks, including human resource/ personnel issues, regulations, reports, school budget - 0.21 0.2480 - 0.30 0.0772 Ascending 0.6988 Observing teachers 0.09 0.6113 0.07 Ascending credo.stanford.edu 99

110 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type h Working wit instructional coaches, grade leaders, departmental leaders, or other instructional 0.13 0.4558 0.8397 Ascending leaders in your school 0.04 Developing or leading professional development activities - 0.21 0.2522 - 0.01 0.9429 Ascending for staff ent interactions, Stud including discipline and 0.38 0.0298 * 0.30 0.0799 academic guidance Ascending Student interactions, including discipline and 0.12 0.5016 0.05 - 0.7578 Ascending academic guidance - 0.06 0.7322 0.28 Parent interactions 0.1083 Ascendi ng Other task not listed - above, specify - - Ascending - Other task not listed above, specify - - - - Ascending Other task not listed above, specify - - - - Ascending Other task not listed - - - - Ascending above, specify re growth or student test - score levels Are student test sco included as a criterion in the evaluation of your performance? - Student test score growth is included in my 0.21 0.2504 performance evaluation 0.05 0.7813 Binary Student test - score levels are included in my perfor 0.10 0.5961 0.21 mance evaluation Binary 0.2281 Is your compensation as leader of this school, including salary and bonuses, affected by any of the following... Number of enrolled students - 0.07 0.7068 - 0.06 0.7168 Binary Students’ achievement on standardized growth assessments (or the school’s value added) 0.24 0.1752 0.22 0.2017 Binary credo.stanford.edu 100

111 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type - score Students’ test on state levels 0.1194 0.45 0.0073 0.28 * Binary assessments Student course completion rates 0.7751 0.25 0.1403 Binary 0.05 of current Reenrollment students across school 0.18 0.3269 0.03 0.8699 Binary years Retention of teaching staff - 0.31 0.0748 - 0.11 0.5334 Binary School’s operating profit or loss 0.23 0.1962 - 0.17 0.3168 Binary Binary Other (specify) - - - - In the past 12 months , have you participated in the following kinds of professional development activities as the leader of this school? University course(s) related to your role as leader of this school 0.04 0.8437 - 0.01 0.9463 Binary - Visits to other schools desig ned to improve your own work as leader 0.35 0.0504 * 0.20 0.2577 Binary of this school Mentoring, peer observation, or coaching by or for a leader of 0.16 0.3757 - 0.01 another school Binary 0.9470 Participating in a school leader network (e.g., a grou p of school leaders organized by an outside agency or through the internet) 0.20 0.2765 0.28 0.1054 Binary Workshops, conferences, or training in which you 0.6936 0.3559 0.07 were a presenter 0.17 Binary Other workshops or conferences in which you we re not a presenter - 0.31 0.0748 - 0.11 0.5334 Binary Have you (the leader of this school) participated in a principal training program? 0.12 0.5046 0.21 0.2385 Binary credo.stanford.edu 101

112 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type to the 2013 2014 Prior - school year, how many years did you serve as 0.08 school 0.08 the leader of th 0.6060 Ascending is 0.6011 Do you have prior experience as principal 0.19 0.2275 - 0.06 0.7222 Binary at another school? - and - mortar school, a virtual/online Was your previous experience at a conventional brick school or a combination of both? Conventional brick - and - 0.00 0.9824 0.07 0.5910 Binary mortar Virtual/on line school - 0.22 0.0905 - - 0.20 0.1187 Binary Both - 0.13 0.3209 - 0.01 0.9522 Binary How long were you a principal at the 0.5453 0.07 0.7 665 0.15 Ascending previous school(s)? - Before you became a school leader, how many years of elementary or secondary teaching experience did you - 0.31 0.0808 - have, if any? 0.4954 Ascending 0.12 How many years of teaching experience have you (the leader of a this school) had in ? - 0.18 0.3198 virtual/online school - 0.11 0.5180 Ascending Is this school its own LEA (Local Education Agency)? - 0.08 0.6279 0.02 0.8913 Binary A school’s funding can be impacted by a number of factors. Is the school’s funding impacted by the to tal number of courses completed ? - 0.13 0.4149 0.02 0.9204 Binary Does your school participate in the federal Title I program? - 0.36 0.0394 * 0.03 0.8667 Binary credo.stanford.edu 102

113 Coeffi Response p - - p Coeffi cient cient value value Sig Sig Type Does your school receive designated funding for special 0.2464 - 0.8817 - education services? 0.03 Binary 0.21 Does your school’s authorizer monitor any of the following student outcomes in your school - - - - Binary State test results 0.23 0.1910 - 0.05 Attendance rates 0.7573 Binary - - 0.4044 0.27 0.1272 0.15 Re Binary enrollment rates Course completion rates 0.8825 0.25 0.1515 - Binary 0.03 Is your school affiliated with a school management organization that provides curriculum or instructional support services? 0.30 0.0569 0.04 0.7785 Binary Does the management organization’s ce ntral office provide your school with any of the following? Curriculum and 0.2909 0.53 0.0098 * Instructional Materials Binary 0.22 Access to instructional - 0.10 0.6521 - 0.08 coaches? 0.7098 Binary Professional development for teachers, such as shops and in -service work - - training programs? - - Binary A system of diagnostic or formative student assessments and results? 0.15 0.4919 - 0.17 0.4215 Binary Technical assistance, support, or resources in areas in which student 0.9218 0. 15 0.4888 - 0.02 test scores are weak? Binary In your opinion, do state or local laws or policies impose constraints on your school’s growth? - 0.03 0.8695 - 0.27 0.1156 Binary credo.stanford.edu 103

114 References Betts, J. and Hill, P. et al. (2006). “Key Issues in Studying Charter Schools and Achievement: A Review and Suggestions for National Guidelines.” National Charter School Research Project White Paper Series, No. 2. Betts, J. and Tang, Y. (2011) “The Effect of Charter Schools on Student Achievement: A Meta -Analysis of the Literature.“ National Charter School Research Project. Cremata, E., Davis, D., Dickey, K., Lawyer, K., Negassi, Y., Raymond, M., and Woodworth, J. (2013). (CREDO) Report . Retrieved National Charter School Study . Center for Research on Education Outcomes 10 July, 2015 from: http://credo.stanford.edu/documents/NCSS%202013%20Final%20Draft.pdf Fortson, K., Gleason, P., Kopa, E., and Verbitsky -Savitz, N. (2015). “Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased.” Economics of 44: 100- 113. Education Review Gill, B J., Chiang, H., Teh, B., Haimson, J., and Verbitsky ., Furgeson, , N. “Replicating Experimental -Savitz Impact Estimates in the Context of Control -Group Noncompliance.” Statistics and Public Policy, forthcomi ng. Hanushek, E. A., Kain, J. F., & Rivkin, S. G. (2004). Disruption versus Tiebout improvement: The costs and benefits of switching schools. Journal of Public Economics, 88(9), 1721- 1746 . Retrieved 10 July, 2015 from: http://www.sciencedirect.com/science/article/pii/S004727270300063X . Is the US Catching Up? International and Hanushek, E.A., Peterson, P.E., and Woessmann, L. (2012) . Retrieved 10 July, 2015 State Trends in Student Achievement. Education Next, Vol. 12, No. 4. Fall 2012 http://educationn ext.org/is -the- from: -catching- up/ us Pazhouh, R., Lake, R., and Miller, L. (2015). “The Policy Framework for Online Charter Schools.” Center on Reinventing Public Education, University of Washington Bothell, 2015. Raymond, M. (2009). Multiple choice: Charter sch ool performance in 16 states. Center for Research on Education Outcomes (CREDO) Report . Retrieved 10 July, 2015 from: http://credo.stanford.edu/reports/MULTIPLE_CHOICE_CREDO.pdf South, S. J., Haynie, D. L., & Bose, S. (2007). Student mobility and school dr opout. Social Science Research , 36 (1), 68- 94. Retrieved 10 July, 2015 from: http://www.sciencedirect.com/science/article/pii/S0049089X05000700 credo.stanford.edu 104

Related documents

7340.2H  Bsc dtd 3 29 18

7340.2H Bsc dtd 3 29 18

ORDER JO 7340.2H Air Traffic Organization Policy Effective Date: March 29, 2018 Contractions SUBJ: contractions used by ed word and phrase This handbook contains the approv personnel of the Federal Av...

More info »
12.09   America's Best and Worst Cities for School Choice

12.09 America's Best and Worst Cities for School Choice

Amer ica’s Amer ica’s Best (And Worst) Cities for December 2015 School Choice PRISCILLA WOHLSTETTER & DARA ZEEHANDELAAR WITH DAVID GRIFFITH Nor m ber M. J. Petrilli thern a nd Mi chael d Forewor by A

More info »
Microsoft Word   cust subm Book 251 Cover.doc

Microsoft Word cust subm Book 251 Cover.doc

Summary of Reciprocity Agreements Between Texas and Other Jurisdictions Vehicle Titles and Registration Division Austin, Texas Issued November 2002

More info »
Microsoft Word   1. Final City Charter Sept 2012.doc

Microsoft Word 1. Final City Charter Sept 2012.doc

CITY CHARTER AMENDED AS OF NOVEMBER 2, 2010 CITY OF LONG BEACH CITY HALL 333 W. OCEAN BOULEVARD LONG BEACH, CA 90802 CITY CLERK DEPARTMENT 562-570-6101

More info »
Eli Hertz Reply 11 6 05c

Eli Hertz Reply 11 6 05c

R Even the most sacred precepts of international law can be manipulated to e pervert the truth. In its Advisory Opinion of 9 July 2004, the UN's Interna - p l tional Court of Justice ruled that Israel...

More info »
ES128202 PREMS 094415GBR2650 Have you say texte web.pdf

ES128202 PREMS 094415GBR2650 Have you say texte web.pdf

“Have Your Say!” Manual on the Revised European Charter on the Participation of Young People in Local and Regional Life

More info »
What Can We Learn from Charter School Lotteries?

What Can We Learn from Charter School Lotteries?

NBER WORKING PAPER SERIES WHAT CAN WE LEARN FROM CHARTER SCHOOL LOTTERIES? Julia Chabrier Sarah Cohodes Philip Oreopoulos Working Paper 22390 http://www.nber.org/papers/w22390 NATIONAL BUREAU OF ECONO...

More info »
Department of Defense   Law of War Manual (June 2015)

Department of Defense Law of War Manual (June 2015)

D E A R T M E N T O F D E F E N S E P N A L O F W A R M A W U A L J U N E 2 0 1 5 O F F I C E O F G E N ER A L C O U N S E L D P A R T M E N T E O F D E F E N S E

More info »
Inside Online Charter Schools

Inside Online Charter Schools

A REPORT OF THE NATI ONAL STUDY OF ONLINE CHARTER SCHOOLS Online Charter Inside Schools October 2015 Brian Gill Lucas Walsh Claire Smither Wulsin Holly Matulewicz Veronica Severn Eric Grau Amanda Lee ...

More info »
CUSTOMARY INTERNATIONAL HUMANITARIAN LAW   VOLUME I RULES

CUSTOMARY INTERNATIONAL HUMANITARIAN LAW VOLUME I RULES

Customary International Humanitarian Law Henckaerts Volume I: Rules and In 1996, the International Committee of the Red Cross, alongside a range of Doswald-Beck Customary International renowned expert...

More info »
Charter School Funding: (More) Inequity in the City

Charter School Funding: (More) Inequity in the City

Charter School 27% AVERAGE 27% AVERAGE FUNDING GAP FUNDING GAP Funding : LOCAL LOCAL FUNDING FUNDING (More) TO BLAME TO BLAME METRO METRO AREAS GET AREAS GET Inequity “C” or WORSE “C” or WORSE HOUSTON...

More info »
Building Excellent Schools Today (BEST) Annual Report

Building Excellent Schools Today (BEST) Annual Report

Building Excellent Schools Today (BEST) Annual Report Submitted to: Senate Education Committee Senate Finance Committee House Education Committee House Finance Committee Capital Development Committee ...

More info »
C:\Users\cb47108\AppData\Local\Temp\S1177enr

C:\Users\cb47108\AppData\Local\Temp\S1177enr

S. 1177 One Hundred Fourteenth Congress of the United States of America AT THE FIRST SESSION Begun and held at the City of Washington on Tuesday, the sixth day of January, two thousand and fifteen An ...

More info »
Michigan School Finance at the Crossroads

Michigan School Finance at the Crossroads

MICHIGAN SCHOOL FINANCE AT THE CROSSROADS: A QUARTER CENTURY OF STATE CONTROL Michigan State University Education Policy Report January 2019 David Arsen, anner Delpier, T and Jesse Nagel Copyright © 2...

More info »
Urban Charter School Study Report on 41 Regions

Urban Charter School Study Report on 41 Regions

Urban Charter School Study Report on 41 Regions 2015

More info »
CHARTER OF THE CITY OF SYRACUSE  1960   With Watermark

CHARTER OF THE CITY OF SYRACUSE 1960 With Watermark

Part C -- 1960* CHARTER OF THE CITY OF SYRACUSE __________ * Editors Note: Part C of this publication is the Charter of the city of Syracuse -- 1960, being Local Law No. 13 of 19 60, as amended. The C...

More info »
mcl act 279 of 1909

mcl act 279 of 1909

THE HOME RULE CITY ACT Act 279 of 1909 AN ACT to provide for the incorporation of cities and for revising and amending their charters; to provide for certain powers and duties; to provide for the levy...

More info »
BergmanMcFarlin school_choice

BergmanMcFarlin school_choice

∗ Education for All? A Nationwide Audit Study of Schools of Choice Peter Jr. Bergman Isaac McFarlin y Universit Teachers College, Columbia University of Florida and NBER December , 2018 Abstract: Scho...

More info »
crpe policy framework online charter schools final 0

crpe policy framework online charter schools final 0

The Policy Framework for Online Charter Schools October 2015 Robin Lake, and Larry Miller Rosa Pazhouh,

More info »