By Carlos Xabel Lastra-Anadón

The effects of voucher programs on educational achievement can be divided into two groups:

■        Effect on the achievement levels of students who participate in the programs.

■        Effects on the achievement level of children in public schools “threatened” by voucher programs (systemic effects).

On the first group, the evidence of the effect of voucher programs in the United States is inconclusive and lies in a spectrum ranging from neutral to positive. Some programs show improvements in certain subjects across the board (Washington DC), some programs show no improvement (Cleveland, Florida), some studies conclude that vouchers have positive effects on certain groups (African Americans in Dayton, DC and New York City), while for others it is too early to provide a definite answer (Milwaukee).

On the second group, the evidence is so far much more limited. Many of the voucher programs that have been developed are small in scope and so have limited systemic effects. However, wherever the systemic impact can be meaningfully evaluated, it has been positive (Milwaukee, Florida).

EFFECTS ON STUDENT ACHIEVEMENT

“Positive studies”- Washington DC Program, Studies on privately funded programs (DC, Dayton and New York City)

The most rigorous analysis of a large scale program is the one performed by Wolf et al. (2009) on Washington DC’s program, designed originally to provide new schooling opportunities for children in need for improvement or corrective action under No Child Left Behind with “expanded opportunities to attend higher performing schools in the District of Columbia”. It is authoritative because of three main characteristics which are necessary to establish robust conclusions but are not necessarily present in other studies:

■        It is based upon the comparison between a test group of participating students and a comparable control group that only differs from it because its members participate in the program. In this case, participation was decided through a lottery.

■        The sample is sizeable. This was made possible by the scale of the DC program (1,387 students in treatment and 921 assigned to control group).

■        Follows treatment and control group over a significant period, namely three years.

Wolf’s analysis of the program found statistically significant evidence of improvement in reading although not maths for students overall. The improvement amounted to an increase equivalent to 15% of a standard deviation for those students who participated in the program. This is roughly equivalent to 3.7 months of additional learning.

A different set of studies on the impact of student achievement that has benefitted from a randomized sampling methodology is the one conducted by Howell and Peterson (Howell and Peterson [2002] and Howell, Peterson and Campbell [2002]) through the private voucher programs of DC, Dayton (Ohio) and New York City. These programs are small in size and target specific communities. The effects on students are not positive across the board but are very significant for African Americans in both reading and mathematics, already after the first year and increasing after the second year. They are equivalent to an increase of 6.6 percentile points per student after three years in standardized tests. No statistically significant explanation has been found for this effect in African Americans and the difference persists even when controlling for other demographic and family characteristics.

“Negative studies”- Florida, Cleveland and Milwaukee

At odds with the results of these studies are the findings that other researchers have made in Florida, Cleveland and Milwaukee. All of them seemingly cast doubt on the effectiveness of vouchers. All of them are, however, based on data that is either interim or does not provide a sound basis for establishing conclusions. In addition, all of these programs are different in design, differing in everything from the amount of money that is allocated (on average, $870 in Cleveland to $6,500 in Milwaukee) to the demographics targeted (e.g., Cleveland targeting those under 200% of the poverty line, Florida targeting primarily those below 130%).

In Florida, Figlio (2009) found no significant gains in achievement for students in voucher programs compared to the control group. This result however is doubly undermined by the fact that it only takes into account results after one year of study and by concerns about the comparability of the control group, as the voucher program is geared towards the very poorest and lower performing students and the control group does not fully reflect this.

Similar concerns arise out of Belfield’s (2006) analysis of the Cleveland program, which again finds no significant positive effects of participation in the program. In addition, the Cleveland program is relatively ungenerous, with an average size of $870 per pupil.

Finally, Milwaukee has currently the largest single city voucher program in the country, with 18,000 participating students and so provides a very good ground for experimentation. Wolf and his team plan to perform over the next few years a comprehensive study on this program. The first year results do not show any significant positive effects (Wolf 2009), although it should be noted that neither did those of the DC program after 1 year, while now success its evident. This research group is committed to studying the effects of the program over the course of 5 years, finishing its evaluation in the academic year 2010-11.

SYSTEMIC EFFECTS

As noted above, voucher programs have often had limited scope when they have been tried. However, the limited evidence from systems where voucher programs are large enough to generate impact public school system dynamics points to significant positive effects in the public schools that have a credible threat of children leaving through a voucher program.

The most compelling and sustained evidence comes from the Florida public school system. It has over the last 10 years been more aggressive in developing accountability and choice mechanisms for its schools (under its A+ plan). All of Florida’s schools are graded yearly from A to F and one of its legislative provisions determines that two Fs in a period of four years would result in students being offered vouchers to go to private schools. All four studies performed on this program (Chakrabarti [2007], Greene and Winters [2004], Rouse et al. [2007], West and Peterson [2005]) have found that schools made very large improvements in student achievement when faced with the prospect of receiving a second F grade. Although it has not been fully settled to what extent it was the threat of students leaving the school or simply the stigma of receiving a failing grade, a narrowing of the gap between F-graded schools and the rest in reading, maths and writing is clearly observed.

Although Florida is the better studied program, a similar effect has been found in the Milwaukee program (Greene and Marsh [2008]), specially since it was broadened in 1990 to include a large pool of religious schools. It has concluded that, for public schools, the addition of one voucher-accepting school anywhere in Milwaukee modestly improves student achievement on both reading and maths.

conclusion

Both on the effect on the achievement of participants and the effect on achievement in public school systems, the results in the United States are so far not conclusive. In the studies that have been conducted the effects on students are only clearly positive in some cases (e.g., African American population or certain subjects), while they are neutral in others. Evidence of effects on improving the public schools system is so far limited only to the cases of Florida and Milwaukee and, although it is in both cases positive, more research is needed to ensure the transferability of the results to other environments and enable the isolation of the impact of specific design principles.

REFERENCES

Angrist, J., Bettinger, E., Kremer, M. (2005): Long-Term Educational Consequences of Secondary School Vouchers: Evidence from Administrative Records in Colombia. (MIT Working Paper). Cambridge: MIT.

Bellfield (2006): The Evidence on Education Vouchers: An Application to the Cleveland Scholarship and Tutoring Program. (Report from the National Center for the Study of Privatization in Education). New York: NCSPE.

Chakrabarti, R.  (2007): Vouchers, Public School Response and the Role of Incentives: Evidence from Florida (Federal Reserve Bank of New York Staff Report, 306). New York City: FRBNY.

Figlio, D. (2009) Evaluation of Florida’s Corporate Tax Credit Scholarship Program

First Follow-Up Report – Participation, Compliance and Test Scores in 2007-08. (University of Florida, Working paper). Gainesville: University of Florida.

Greene, J. and Winters, M. (2004): “Competition Passes the Test,” Education Next, 4:3

Rouse, C.E., Hannaway, J., Goldhaber, D., Figlio, D. (2007): Feeling the Heat: How Low Performing Schools Respond to Voucher and Accountability Pressure (CALDER Working Paper 13). Washington DC: Urban Institute.

Greene, J. and Marsh, R. (2008): “The Effect of Milwaukee’s Parental Choice Program on Student Achievement in Milwaukee Public Schools”, The Comprehensive Longitudinal Evaluation of the Milwaukee Parental Choice Program. Report from the School Choice Demonstration Project. Fayeteville: University of Arkansas

Howell, W. and Peterson, P. (2002): The Education Gap. Vouchers and Urban Schools. Washington DC: Brookings Institution Press.

Howell, W. and Peterson, P. and Campbell, D. (2002):

 “School Vouchers and Academic Performance: Results from Three Randomized Field Trials”,

 Journal of Policy Analysis and Management, 21:2.

Wolf, P. (2008): “Summary Report”. The Comprehensive Longitudinal Evaluation of the Milwaukee Parental Choice Program. Report from the School Choice Demonstration Project. Fayeteville: University of Arkansas.

Wolf, P. et al. (2009): Evaluation of the DC Opportunity Scholarship Program. Impacts After Three Years. Washington DC: US Department of Education.

West, M., Peterson, P. (2005): The Efficacy of Choice Threats Within School Accountability Systems. (Harvard PEPG Working Paper 05-01). Cambridge: Harvard University