Teachers vs Students: Do School Ratings Really Matter?
Takeaway: Widely published school performance ratings are heavily correlated with the makeup of their student bodies, but this relationship is driven by selection bias rather than true school effectiveness.
Key Points
Ratings Conflate Background with Quality: Conventional school ratings reward schools that enroll higher-income and White students, mistaking advantaged family backgrounds for effective teaching.
True Quality Is Race-Neutral: Leveraging admissions lotteries in New York City and Denver, researchers found that a school’s causal "value added"—how much it actually boosts student learning—is essentially unrelated to its racial composition.
A Policy "Free Lunch": Statistically removing racial imbalance from school growth ratings eliminates demographic bias without sacrificing predictive accuracy, offering a fairer and more accurate gauge of school quality.
Anyone who has ever searched for a home on Zillow or Redfin has encountered GreatSchools ratings. These 1-to-10 badges do more than summarize test scores—they shape neighborhood property values, guide where families choose to live, and influence which public schools districts decide to close or replicate. Yet parents and journalists have long noted an uncomfortable pattern: schools with more White and Asian students almost always score at the top, while schools serving Black and Hispanic students are frequently rated near the bottom.
This dynamic raises a fundamental question for parents and policymakers alike: Are predominantly White schools genuinely better at educating children, or do standard ratings simply reflect the socioeconomic advantages students bring from home?
To answer this, economists Joshua Angrist, Peter Hull, Parag Pathak, and Christopher Walters analyzed middle school admissions data from New York City and Denver. Both cities allocate middle school seats using centralized assignment systems that rely partly on randomized lottery tiebreakers.
This setup provided a rare natural experiment: by comparing students with identical preferences who were randomly assigned to different schools, the researchers could isolate the true causal impact of attending a specific school—known as its "value added"—separate from the background of its enrolled students.
Findings
The researchers compared conventional ratings with causal value added and uncovered three major findings.
First, standard "levels" ratings—which simply measure the share of students passing state exams—are terrible indicators of school quality. In New York, a one standard deviation increase in a school's pass-rate rating corresponded to just a 0.21 standard deviation increase in actual learning growth. These raw scores largely mirror family income and race rather than classroom effectiveness. Growth-based "progress" ratings, which track how much students improve from year to year, perform far better, but they still exhibit a substantial bias toward schools with higher White enrollment.
Second, when school quality is measured correctly using admissions lotteries, the link between school effectiveness and racial composition disappears. In both Denver and New York City, middle schools with higher White enrollment shares generated no more learning growth, on average, than schools serving predominantly minority students. Highly sought-after screened schools and schools with larger White student bodies were, on average, overrated by public metrics.
Third, the authors demonstrated that adjusting growth ratings to remove their correlation with student race does not hurt their predictive power. In fact, "race-balanced" progress ratings were slightly more accurate at predicting causal value added than raw ratings. In economic terms, this represents a rare "free lunch": a straightforward statistical adjustment eliminates racial bias while making ratings a better guide to school performance.
Limitations and Context
The findings directly apply to middle schools within two large urban districts featuring centralized choice systems. While national data show similar correlations between race and public ratings across the United States, further research is needed to determine whether true value added is equally race-neutral in suburban districts or across different grade levels.
Additionally, how parents might alter their residential or enrollment decisions when presented with race-balanced ratings remains an open question.
Why This Research Matters
When public ratings mismeasure school quality, they reinforce residential and racial segregation by nudging families toward wealthier, whiter schools under the false impression that they offer better instruction. Simultaneously, they penalize high-performing educators serving disadvantaged populations. Adopting race-balanced ratings would provide families with clearer signals of instructional quality and ensure school accountability policies reward the schools that help students learn the most.
Learn More
Title: Race and the Mismeasure of School Quality
Authors: Joshua Angrist, Peter Hull, Parag A. Pathak, and Christopher R. Walters
Journal: American Economic Review: Insights
Publication Year: 2024
URL: https://www.aeaweb.org/articles?id=10.1257/aeri.20220292
Econ Today Explains
Economic Concept: Selection Bias
Selection bias occurs when the individuals in a group being studied differ systematically from those outside the group in ways that affect the outcome. When evaluating schools, students do not end up in classrooms by pure chance; families with higher incomes and more educational resources often select into certain neighborhoods and schools.
If an analyst simply compares average test scores across schools, they risk attributing the advantages of wealthier students to the quality of the school itself. To overcome selection bias, economists use natural experiments—such as randomized admissions lotteries—to ensure they are making true "apples-to-apples" comparisons.
This article was written by AI but reviewed by a real human.
Comments