Also read "What Matters Most in Measuring," for Rotherham's take on the new U.S. News & World Report high school rankings.

Americans love rankings, and one has to look no further than the local newsstand for evidence. Every week, it seems, a magazine or newspaper runs a headline about the number one way to lose weight, the five hottest new soul singers or the top ten most powerful dishwashers. Education has not been spared from this national obsession, and U.S. News and World Report's annual ranking of colleges and universities shows just how influential such ratings can be: Some colleges have altered their admissions practices in order to boost their score in the newsmagazine's ranking.

While there is no national ranking of the country's elementary and secondary schools, Newsweek publishes a list of the top 100 American high schools. The annual list has become increasingly influential since its inception in 1998, and now some newspapers, such as the Journal News in suburban New York City, have begun to cover the release as if it were a horserace, detailing the ups and downs of individual schools and publishing tables on which schools did and did not make the grade.[i]

The highly regarded Washington Post education writer Jay Mathews developed the Newsweek formula. (The Washington Post Company publishes Newsweek; read Mathews' response to this paper).[ii] He calls it the Challenge Index, and it is a simple measure: Divide the number of Advanced Placement (AP) and International Baccalaureate (IB) tests taken by students at a high school by the number of graduating seniors.

Mathews focuses on AP and IB because the courses are supposed to give students an opportunity to experience challenging, college-level work while still in high school. And, according to Mathews, the percent of students taking the exams is better than other measures at gauging how effectively a high school prepares its students for post-secondary success.[iii] Further, because the same AP and IB tests are given to students nationally, the Challenge Index allows Mathews to compare schools from different states.   

While the number of students taking AP and IB tests is one key indicator of a good high school, we believe that the Challenge Index is a seriously flawed measure of overall quality. A successful high school should show high levels of student achievement, graduate almost all of its students and not let any demographic subgroup suffer at the expense of others. Most national and local experts and policymakers share these values. To be sure, graduation rates and student achievement are hardly the only indicators of a school's quality. At a minimum, however, America's best high schools should be expected to meet these basic criteria.

Yet our analysis shows that many schools on Newsweek's list do not meet these minimum standards. Using publicly available student performance data, we found that many schools on Newsweek's 2005 ranking have glaring achievement gaps and high dropout rates. By presenting them as America's best, Newsweek is misleading readers and slighting other schools that may in fact be better than those on Mathews' list. For example, the magazine ranks Eastside High School in Gainesville, Fla., as the third best high school in the nation, but only 12 percent of Eastside's black students were reading at grade-level in 2004. And Newsweek ranks Hillsborough High School in Tampa, Fla., as America's 10th best high school, but only 17 percent of black students and 26 percent of Hispanic students met the state's modest grade-level standards in 2004. While some students at Eastside or Hillsborough may be receiving a challenging education, it's clear that many are not. And Eastside and Hillsborough are not outliers. In fact, schools with substantial inequities in student achievement make up a significant proportion of Newsweek's list of best high schools.

Our research shows that Newsweek's methodology is far too focused on one discrete indicator of school quality and that many schools that fail to make the Newsweek list may be doing a better job educating all of their students.

Certainly, ensuring that more students take rigorous courses is important, and Newsweek's emphasis on the practice is commendable.  However, the inclusion of schools with significant achievements gaps and very low performance for some student groups raises two important questions:

• Is Mathews' methodology the best way to find out which high schools are America's best, in light of the increasing public availability of school data?

• Is this list fair to other non-selective public high schools?

We think the answer to both questions is no.

Method

Our analysis is straightforward. We collected student performance data for the 100 schools in Newsweek's 2005 “America's 100 Best High Schools” issue.[iv] The data was available on Standard and Poor's Web site, www.schoolmatters.com, and came from state assessments and accountability systems. When information was not available through this site, we went to the Web sites of state departments of education. Most of the data we obtained reflects student performance on state tests administered in spring 2004.  In a minority of cases, the available data was from 2003 or 2005.[v]  

To try to get a better understanding of how schools on Newsweek's list are serving different populations of students, we looked at disaggregated student performance data on state assessments and calculated the achievement gaps for three student populations that are often underserved: black students, Hispanic students and students from economically disadvantaged backgrounds. For each group, we examined the proficiency gaps: the difference between the percent of proficient students from one subgroup, according to the standards defined by their state, and the percent of those proficient from another group. We compared the percent of black and Hispanic students proficient in reading/language arts and math with the percent of white students, and the percent of economically disadvantaged students with the school-wide average.[vi]  We also collected the graduation rates of schools on Newsweek's list and, when available, graduation rate data disaggregated by race, ethnicity and family income. 

Of the 95 schools on Newsweek's list for which we were able to find data, 73 reported data on at least one of these three subgroups: students who are black, Hispanic or economically disadvantaged.[vii] Of the other 22 schools, six reported no disaggregated data, and 16 had disaggregated data for white students only.

Findings

Black-White Test Score Gaps
Sixty-one schools on Newsweek's list reported data for both black and white students in reading, and the average difference between the percentages of white and black students who were proficient was 26 points. The achievement gaps ranged from 67 percentage points at Florida's Eastside High School to a negative 5.1 percentage points (meaning more black than white students were proficient) at Brighton High School in Rochester, N.Y., with a median gap of 25.5 points. Forty schools on Newsweek's list had black-white reading achievement gaps of greater than 20 percentage points.

In math, 56 schools reported data for both black and white student subgroups, and the average gap between the percentages of white and black students who scored proficient was 31 points. Differences ranged from 61 points, again at Eastside, to negative 3.8 points at Great Neck South in Great Neck, N.Y., with a median gap of 32.5 points. Thirty-nine schools on Newsweek's list had a black-white math achievement gap of 20 percentage points or greater.

Hispanic-White Test Score Gaps
In the 58 schools with disaggregated data for white and Hispanic students in reading, the average gap between the percentages of white and Hispanic students who scored proficient was 19 points. Gaps ranged from 57 points at Chapel Hill High School, in Chapel Hill, N. C., to negative 10 at Clarke County High School in Berryville, Va., with a median gap of 19 points.  Twenty-seven schools on Newsweek's list had a Hispanic-white reading achievement gap of 20 percentage points or greater.

In math, 54 schools reported data for both white and Hispanic students, and the average difference between the percentages of white and Hispanic students proficient in math was 18 points. Differences ranged from 53 points at Gunn High School in Palo Alto, Calif., to negative 4 points at Rochester's Brighton, with a median gap of 17 points. Twenty-six schools on Newsweek's list had Hispanic-white math achievement gaps of 20 percentage points or greater.

Gaps By Income
In the 60 schools with disaggregated data for economically disadvantaged students in reading, the average difference between the percentages of economically disadvantaged students proficient and the school-wide average was 19 points. Differences ranged from 56 points at Palo Alto's Gunn to negative 2 points at Pittsford Mendon High School in Pittsford, N.Y., with a median gap of 13 points. Twenty-seven schools on Newsweek's list had achievement gaps in reading of greater than 20 points between their economically disadvantaged students and school-wide averages.

In math, 55 schools reported data for economically disadvantaged students, and the average difference between the percentages of economically disadvantaged students proficient and school-wide averages was 20 points. Differences ranged from 68 points at East Chapel Hill in Chapel Hill, N.C., to negative 2 points at the Science Academy in Mercedes, Texas, with a median gap of 19 points. Twenty-five schools on Newsweek's list had achievement gaps in math greater than 20 points between their economically disadvantaged students and school-wide averages.

Graduation Rate Gaps

We examined the graduation rates of Newsweek's top high schools for two reasons. First, graduation is an important indicator of school performance. Second, graduation rates are particularly important in considering the accuracy of the Challenge Index, because the denominator in the Challenge Index is the number of graduates, rather than the number of students who should have graduated in a given year.

We found graduation rates for 84 schools, and the average graduation rate for these schools was 91 percent.[vii] Although this average rate seems quite high, researchers have extensively documented methodological flaws in the graduation rate data reported by many schools and states.[ix] The tendency is for schools to underreport dropouts, so these graduations rates should be approached cautiously. Still, some schools on Newsweek's list reported graduation rates that are disturbingly low. While 77 schools report graduation rates of 90 percent or higher, nine schools on Newsweek's list had graduation rates of less than 75 percent.

Nevertheless, it is possible to compare gaps in graduation rates within each high school because the same method is used to calculate the graduation rate for each group in the school. The results were disappointing. Only 26 schools reported graduation rates for black students as a separate subgroup, and the average rate was only 71 percent. The graduation rates for blacks ranged widely from 43 percent at St. Petersburg High School in St. Petersburg, Fla., to 100 percent at both Science/Engineering Magnet in Dallas and Grapevine High School  in Grapevine, Texas.  The median black graduation rate was 65 percent. All but one of these schools also reported graduation rates for both black and white students, and the average gap between the graduation rates was 15 percentage points, with a median gap of 18 points.

Nineteen schools reported graduation rates for Hispanic students, and the average rate was 81 percent. These graduation rates also ranged widely, from 50 percent at Florida's Hillsborough High School to 100 percent at Newport High School in Newport, Wash., and Bellevue High School in Bellevue, Wash.  The median Hispanic graduation rate was 85 percent. Seventeen schools reported graduation rates for both white and Hispanic student subgroups, and the average graduation gap between the two groups was 10 percentage points, with a median gap of 9 points.

Twenty-one schools reported data for economically disadvantaged students, and the average graduation rate was 63 percent. Again, these numbers were all over the map. Interlake High School in Bellevue, Wash., reported that none of its economically disadvantaged students graduated, while Science/Engineering Magnet in Dallas graduated all of its low-income students. The median graduation rate for economically disadvantaged students was 61 percent. The average gap in graduation rates between economically disadvantaged students and school-wide averages was 17 percentage points, with a median gap of 19 points.

Implications

While some schools on Newsweek's list may be among the best in the nation, a closer look at the data reveals that many do not meet a reasonable definition of a good high school. Indeed, some of the schools on the list have such significant achievement gaps that they should be on a list of schools needing improvement rather than on one for best schools.[x] And it is not merely the case of a few outliers. In fact, so many of the schools on the list have such significant gaps in achievement among their student subgroups that it calls into question the entire Newsweek enterprise.

First, we should explain why it's possible that so many schools that score well on the Challenge Index could still have substantial achievement gaps:

• The Challenge Index does not reflect how AP and IB tests are distributed. If a school has more students taking AP or IB tests than the number of graduates, it does not mean that every student in the school is taking an AP or IB test. In fact, a school with a small number of students taking many tests will receive a high Challenge Index score even if it is providing a lousy education to the rest of its students.

• Similarly, some of the schools on the Newsweek list may have selective school-within-a-school-type programs in which a few students take a lot of tests, and, again, make the whole school look good. While the Newsweek index only lists schools where at least half the students are admitted on an open-enrollment basis, this filter is not sufficient.

• The design of the Challenge Index does not account for schools that have high dropout rates. Non-completion is an enormous problem among minority students. Based on the most commonly accepted methodology for calculating completion rates, only 56 percent of black students and 52 percent of Hispanic students finish high school with a degree.[xi] Because the Challenge Index uses the number of students who graduated in a given year as the denominator, rather than the number of those who should have done so, it fails to account for the large number of students who drop out.

• The achievement gaps detailed in these schools are not that different from those in other schools. According to the National Assessment of Educational Progress, minority students lag behind their white peers by about four grade levels by the end of high school.[xii] However, if these Newsweek schools are going to be held up as the gold standard, they should not have such large achievement gaps.

In Newsweek's defense, it is very difficult to measure the quality of a high school. For one, state assessment practices and procedures are far from uniform. Some states test students at the end of courses, while others use graduation exams. State accountability systems are also often not sophisticated enough to rank schools. Moreover, by the time students reach high school they have often suffered from years of educational neglect, and it's extremely difficult to measure how much a high school has helped each student improve.

We hope our research produces two results:

First, Newsweek should characterize its rankings more precisely, and thus more accurately. Certainly, there is value in AP and IB course-taking, and schools that encourage more students to take such advanced classes should be lauded. However, the current Newsweek approach gives readers the wrong impression about the overall achievement of these schools. These schools score highly on one very narrow measure that ignores many other important aspects of school quality. At the very least, Newsweek should change the language surrounding the release of the list to make it clear what these 100 high schools are actually best in: having students take AP and IB exams. The Newsweek list also slights the work of other excellent open-enrollment public high schools that are striving to provide an excellent and equitable education to their students but may have fewer AP and IB test-takers.

Second, Newsweek should use more sophisticated measures.  Despite the shortcomings of the adequate yearly progress (AYP) requirements in No Child Left Behind, AYP is a more comprehensive measure of school performance than the Newsweek ranking, because it looks at how well different student subgroups are performing. To be sure, AYP could not be used to measure the best schools in the country; while the approach helps policymakers root out low-performing schools, it is too limited to serve as a ranking mechanism for the country's best schools.

Newsweek could, however, take some simple steps to improve their ranking. One option is to tighten the definition of what constitutes an open-enrollment school. They could also include disaggregated data and graduation rates in their rankings and reward schools that do well with students from diverse backgrounds. As state data systems become more developed, information about course-taking patterns and other measures of college preparation—as well as longitudinal data about performance—could also be included.

A second option is to develop a more elaborate metric for determining school quality and couple it with professional judgment. The Broad Foundation's selection process for the Broad Prize for Urban Education is one model.  Each year, the Broad Foundation convenes a review board to examine the achievement of the nation's largest school districts.[xiii] As part of the process, the review board examines copious amounts of data, but the members base their final decision on both subjective and objective observations.[xiv] This method is obviously more laborious than simply using the Challenge Index. But adding more indicators to the Challenge Index would produce a much more accurate ranking.

Jay Mathews claims that he would broaden the ranking to include other school characteristics such as teacher quality if those factors could be accurately measured.[xv] In the past, this was a reasonable point.[xvi]  Today, however, there is a wealth of publicly available data on schools, including data about teacher quality. All the data in this report, for instance, came from publicly available sources, and much more data than what appears here could be used to rate the nation's schools.

Conclusion

Newsweek's Challenge Index measures exactly what it purports to: the number of AP and IB tests taken by students in a school. In doing so, the newsmagazine provides a comparative measure of one aspect of high school quality. Unfortunately, the magazine also uses the formula to compile a list that it hails as the nation's best schools. But a closer examination of these schools shows that some grossly shortchange disadvantaged groups and have a significant number of drop outs.