by Stuart Singer, The Teacher Leader
Jay Mathews is one of the most important educational writers in this country. He has brilliantly chronicled the strengths and weaknesses of school policies for decades. He is a “hands-on” journalist who spends hours observing, interviewing and reporting on education. Mr. Mathews’ contributions to academics cannot be underestimated. But because of this influence it is necessary to address some flaws in his statistical analysis of schools in the United States.
Sometimes silence is not golden
When the 2013 Challenge Index was announced I was determined to keep quiet. After all I had already shared my mathematical concerns with this tool on a number of occasions. But then two incidents made it impossible to keep that vow.
The first occurred during a morning news segment on the NPR radio station in San Francisco. It concerned what was referred to as “a sad irony”. The reporter intoned that on the very day the American Indian Public Charter High School of Oakland, California was named by the Washington Post as the top school in the nation, the Board of Education in that city was demanding that it be closed down for a variety of issues including the misuse of funds. The reporter went on to say “…this designation (number one in the country) by the Washington Post was based on an analysis of the school’s high percentage of college-level exams, the success of the students on these difficult tests and the poverty-level of the student body.”
Unfortunately, the broadcaster misstated the criteria utilized in this ranking system. Here is the actual methodology according to the Post:
“The index score is the number of college-level tests given at a school in 2012 divided by the number of graduates that year. Also noted are the percentage of students who come from families that qualify for lunch subsidies (Subs. lunch) and the percentage of graduates who passed at least one college-level test during their high school career, called equity and excellence, (E&E).”
The voice on the radio misinterpreted the “also noted” to mean “included”. There is a significant difference.
My second problem with this year’s list is located in a follow up article written by Mr. Mathews. In addressing the issues surrounding the problems facing The American Indian Public Charter School, his headline read “Nation’s Best School May Be Closed”. I have no knowledge about this school. It may well be among the finest in the country. But to label it the nation’s best as a result of one data point is hard to accept.
Good data, bad data and misused data
The Challenge Index has access to a significant group of important educational statistics. In addition to the number of tests administered and graduates, it has been given the percentage of students on subsidized lunch and the specifics of how every individual fared on the various exams. It is the use or non-use of this information that is troubling.
In the Challenge Index the category of “E&E” delineates how many students have passed at least one college-level exam in their high school career. Thus, if a student takes ten exams and passes one, they would count exactly the same as someone who passed nine. When studied carefully this statistic could raise some eyebrows. The aforementioned American Indian School has a Challenge Index of more than 23. That translates into the average student at the school taking more than 23 college-level exams. But its E&E rating of 80% means that one in five members of that student body failed every such test they took. While no precise numbers are given, it could be possible that there are individuals at this school who are failing dozens of these exams during the course of their academic careers. Merely passing one of these 23 would place someone on the “good” portion of the percentage. A far more revealing statistic would be the actual pass rate on all of the testing in the school.
A data analysis role model
There is no human activity that utilizes statistics to the extent of Major League Baseball. If something occurs on the field of play, there is a weighted-number assigned to it. And in more cases than not, the resulting conclusions accurately reflect reality. The Challenge Index would be wise to incorporate some of the techniques found in MLB.
In baseball the “E&E” would equate one batter who in five appearances has a home run, two doubles and a single with another hitter who in his five at-bats recorded four outs and a single. One success in 23 would equate to a batting average of .043 which would never earn a place on the positive side of a baseball ledger.
Not all student bodies are equal
No one can question the adverse effects of high-poverty in a student body. This fact is clearly evident in the 2013 Index. It is not surprising that of the 20 top ranked DC area schools only two have a poverty rate above 23%. Meanwhile of the 26 schools at the bottom only four are below 23%. To ignore the wide range of subsidized lunch populations is inevitably going to lead to a statistical bias.
Baseball understands the impact of financial resources. Money talks in baseball. There is a reason why the Pittsburgh Pirates have not had a winning season in two decades and the New York Yankees have been to the World Series seven times during that same period of time. Every year the team payroll of the Pirates is among the lowest in the league while the Yankees are firmly at the top. Thus, if a franchise can overcome a severe economic disparity, the sport acknowledges such success. Despite facing the same money issues as Pittsburgh, the Oakland A’s are consistently successful. In 2012 they won the American League West Division and are currently in first place once again. Overcoming adverse circumstances is a major accomplishment. There is a reason why the movie “Money Ball” featured the Athletics not the Pirates or Yankees. The Challenge Index should incorporate a similar economic assessment of a school’s student body.
Recognizing excellence in education is critical to improving academic success. But these measures need to accurately reflect the relative performance of the institutions based on a variety of measurements. Such precision requires more than one single piece of data.