Executive Summary

The goal of the study was to examine differences in mean National Assessment of Educational Progress (NAEP) reading and mathematics scores between public and private schools when selected characteristics of students and/or schools were taken into account.

Among the student characteristics considered were gender, race/ethnicity, disability status, and identification as an English language learner. Among the school characteristics considered were school size and location, and composition of the student body and of the teaching staff. In particular, if the student populations enrolled in the two types of schools differed systematically with respect to background characteristics related to achievement, then those differences would be confounded with straightforward comparisons between school types.

The present report examined results from the 2003 NAEP assessments in reading and mathematics for grades 4 and 8. NAEP draws nationally representative samples of schools and students. In 2003, over 6,900 public schools and over 530 private schools participated in the grade 4 assessments. Over 5,500 public schools and over 550 private schools participated in the grade 8 assessments.

Hierarchical linear models (HLMs) were employed to carry out the desired adjustments. HLMs were a natural choice because they accommodate the nested structure of the data (i.e., students clustered within schools) and facilitate the inclusion of variables derived from student and school characteristics. In this study, the focal parameter was the mean difference between mean NAEP scores for two populations of schools. (This difference was not identical to the difference in mean scores between the two student populations, though the discrepancy was typically small.) HLMs were used to compare all private schools to all public schools, as well as to compare, separately, certain categories of private schools (i.e., those for which sample sizes were sufficient to report reliable estimates) to all public schools.

Statistical significance was determined at the .05 level using t tests on model results.

Results From Grade 4
Reading

In the first set of analyses, all private schools were compared to all public schools. The average private school mean reading score was 14.7 points higher than the average public school mean reading score, corresponding to an effect size of .41 (the ratio of the absolute value of the estimated difference to the standard deviation of the NAEP fourth-grade reading score distribution). After adjusting for selected student characteristics, the difference in means was near zero and not significant. In the second set of analyses, Catholic schools and Lutheran schools were each compared to all public schools. The results, both with and without adjustments, were similar to the corresponding results for all private schools.

Mathematics
In the first set of analyses, all private schools were again compared to all public schools. The average private school mean mathematics score was 7.8 points higher than the average public school mean mathematics score, corresponding to an effect size of .29. After adjusting for selected student characteristics, the difference in means was -4.5 and significantly different from zero.

(Note that a negative difference implies that the average school mean was higher for public schools.) In the second set, Catholic schools and Lutheran schools were each compared to all public schools. The results, both with and without adjustments, were similar to the corresponding results for all private schools.

Results From Grade 8
Reading
In the first set of analyses, all private schools were compared to all public schools. The average private school mean reading score was 18.1 points higher than the average public school mean reading score, corresponding to an effect size of .58. After adjusting for selected student characteristics, the difference in means was 7.3 points and significantly different from zero. In the second set, Catholic, Lutheran, and Conservative Christian schools were each compared to all public schools. The results, both with and without adjustments, were generally similar to the corresponding results for all private schools. The only exception was that the average difference in adjusted school mean scores between Conservative Christian schools and all public schools was not significantly different from zero.

Mathematics
In the first set of analyses, all private schools were again compared to all public schools. The average private school mean mathematics score was 12.3 points higher than the average public school mean mathematics score, corresponding to an effect size of .38. After adjusting for selected student characteristics, the difference in means was nearly zero and not significant. In the second set, Catholic, Lutheran, and Conservative Christian schools were each compared to all public schools. While the results for Catholic schools, both with and without adjustments, were very similar to the corresponding results for all private schools, the results for the other two types differed.

The initial difference between Lutheran schools and all public schools was substantially larger (19.5 points) than was the case for all private schools. The average difference in adjusted mean mathematics scores between the two types of schools was 4.9 points and significantly different from zero. On the other hand, the initial difference between Conservative Christian schools and all public schools was substantially smaller (5.1 points) and not significant. The average difference in adjusted school means between Conservative Christian schools and all public schools was -7.6 points (i.e., a higher average school mean for public schools) and was significantly different from zero.

Comparison of Results for Grade 4 and Grade 8
Overall, there were many similarities in the results for the two grades. In both reading and mathematics, analyses employing unadjusted NAEP scores indicated that the average private school mean score was higher than the average public school mean score, and the difference was statistically significant. Including selected student characteristics in the model, however, resulted in a substantial reduction in the difference in all four analyses.

The reduction varied from 11 to 15 score points. For grade 4 reading and grade 8 mathematics, the average difference in adjusted school mean scores was no longer significant. For grade 4 mathematics, the difference was significant, and the adjusted school mean was higher for public schools. Only for grade 8 reading was the difference still significant with a higher school mean for private schools. For all four analyses, with student characteristics such as gender and race/ ethnicity incorporated in the model, the inclusion of school characteristics (e.g., teacher experience, type of school location, school size) had little impact on the estimate of the average difference between the two types of schools.

Variance decompositions yielded similar results for the four grade-subject combinations. Most of the total variance was due to heterogeneity among students within schools rather than heterogeneity among school mean scores. The combination of selected student and school characteristics accounted for about one-third of the total variance for grade 4 and about two-fifths of the total variance for grade 8.