Truth is, after all, a moving target
Hairs to split and pieces that don’t fit
How can anybody be enlightened?
Truth is, after all, so poorly lit
– Neil Peart
There’s a world of difference between truth and facts. Facts can obscure the truth.
– Maya Angelou
You gotta hand it to ‘em. The Kansas Association of School Boards (KASB) knows how to take a set of facts, manipulate them to draw an erroneous (pre-determined) conclusion, and present it as if Winston Churchill himself was declaring that this “truth is incontrovertible.”
I am speaking of KASB’s recently published State Education Report Card 2016 Update.
The Report Card claims that Kansas ranks 10th in overall student achievement, a point of pride for KASB and the Kansas education community. However, just as quickly as that declaration came and went, the focus turned to the nine states that rank higher. Any guesses what they all have in common? That’s correct, they all spend more money per student than Kansas. (They’re also all to the north and/or east of Kansas. Maybe we should do as the late Senator Pat Moynihan suggested when presented with similar data: move closer to Canada.) The unspoken conclusion from KASB is that we should be proud of our ranking, but if we want to get better we need more money for education.
KPI’s Dave Trabert, in this article, took the report and subsequent ranking to task by exposing that the report is nothing more than a ruse to justify more money to the schools.
The purpose here is not to echo those remarks. I will show that it is impossible to give Kansas (or any other state) any comparative ranking using the data chosen by KASB. Without getting too wonky, the ranking relied heavily on graduation rates, included post-secondary attainment (sort of), and included NAEP, ACT and SAT scores.
A close look at this data supports the notion that such an overall ranking undertaking is a quixotic exercise.
Graduation rates. A look at the percentage of students entering post-secondary education who have to take remedial college courses is an indicator that graduating from high school isn’t much of an accomplishment anymore. That aside, Kansas has an overall graduation rate of 85.7%. Is that good or bad? Does it mean we’re worse than Alabama’s 86.3%? Are we better than Virginia, which graduates their students at an 85.3% clip? What about New Mexico? Their graduation rate is 68.5%, by far the lowest in the nation. Is that a reflection of the schools, their graduation requirements, or a function of the student population? The point is, you can’t compare graduation rates across states, there are too many variables that contribute to the number.
Post-secondary. This factor, too, cannot be meaningfully compared state-to-state. According to the KASB analysis, North Dakota is ranked second in the rate of those pursuing post-secondary education, while a disproportionate share of the bottom ten and are from the least affluent states, most in the south. We all understand that many factors go into an individual attending and/or completing a post-secondary degree. How much of that is a direct reflection of a state’s K-12 system? (As a technical point, KASB included high school graduation in this category, essentially double counting that data.)
Assessments. This is the one category KASB used that does allow state-to-state comparisons – with a caveat. NAEP scores were included, but not effectively utilized. KASB chose to include in the rankings those who scored in the Basic category, therefore considering scoring at that level an accomplishment. It’s curious they would include this category in an outcomes-focused analysis, since those scoring in Basic are definitely not “college ready” and arguably not even at grade level. So why include that data? (KASB also committed a research faux pas by double counting all students in the NAEP analysis). A more objective view of NAEP results and trends comes from the annual report published by Education Week. The 20th edition of Quality Counts for 2016 ranks Kansas 40th among the 50 states and D.C. with a letter grade of D. A full review of that report is available here.
But the most curious data used in the Report Card is SAT scores. Despite the fact that only 5% of Kansas high school students take the SAT, KASB chose not only to use it in their analysis, but gave it a weighting 6 times greater than the NAEP indicator of All Students at Proficient Level. It is a generally accepted premise that those who take the SAT are among the most highly motivated students – certainly not a cross-section of all graduates and presumably those hoping to attend some university out of Kansas. How can you compare the scores of Kansas with states that have much higher rates of SAT participation, even as high as 100%? Well, it is possible to adjust statistically, as KASB and others have done. But that still doesn’t change the fact that KASB considers a very small, non-representative indicator like SAT 6 times as important as a NAEP indicator in calculating student outcomes. Subjecting a variable to statistical manipulation doesn’t make it meaningful.
Please understand that it isn’t that KASB used the wrong data to show Kansas as 10th in the nation. It’s that there is no set of data that can possibly provide meaningfully comparisons and rankings of the states. It just can’t accurately be done. This is a classic example of using facts to mask the truth.
The Report Card avoids the incontrovertible truth that Kansas performs about average in a nation that doesn’t perform very well. It avoids the incontrovertible truth that achievement gaps between the haves and have-nots are at persistent, unacceptable levels – regardless of the outcome measure. Perhaps the Kansas Report Card would have some level of credibility if it focused on how to deal with those achievement gaps. As such, it should be considered nothing more than an avoidance instrument.
It should be no surprise that the first statement on the first Power Point slide of the report is “Serving Educational Leaders,” since they are the only ones served by this kind of report, not the rest of Kansans.
