The Kansas Association of School Boards (KASB) continues to take data manipulation to new levels in their just-released report Comparing Kansas 2022. KASB claims that “the most recent national data show Kansas ranks 10th in the nation on a weighted average of…measures that are part of the State Board of Education’s Kansans Can program and new accreditation system.” KASB chose 16 measures to make the 10th place claim.
KASB puts a colorful map on the front page then fills the rest of the report with definitions and numbers set to overwhelm and confuse readers, hoping their conclusion that Kansas is indeed 10th in the nation will go unquestioned by those who do not examine the data itself.
The most recent previous “Comparing Kansas” report was released in 2019. In that year, KASB had Kansas ranked 9th in the nation. KPI responded with this blog which highlighted KASB’s poor methodological structure. That methodology hasn’t improved since the last report.
Indeed, a close look inside the statistics employed by KASB shows a shoddy methodology that required more manipulation than Obi-wan Kenobi ever dreamed of wielding on the Empire’s storm troopers.
The following table, taken directly from the KASB report, is a summary of the way KASB got Kansas in the top 10. Notice that only 4 of the 16 measures have Kansas in the top 10, but KASB has massaged those numbers to give Kansas such a lofty national ranking.
KASB acknowledges that “(m)ost of Kansas’ rankings are in the 20’s, the middle band of states, with a few areas ranking in the top 10 or ‘teens,’ and its lowest ranking is 30th”. But they employ some verbal sleight-of-hand by stating that some states “have much wider variation. Florida, for example, ranks in the top 10 in some areas and in the 40’s in others.” That statement somehow justifies a top-10 ranking despite 75% of all KASB’s indicators rank Kansas outside the top 10.
Just look at the “Kansas Rank” column. How could Kansas rank 10th in the nation given those standings? Easy, it’s all in how KASB weighted those rankings. It must have taken quite a bit of trial and error to weight these variables to come up with an overall ranking of 10th. And with a nod to Tom Petty, the weighting is indeed the hardest part. Somehow, some way, KASB engineered those rankings to come up with an overall ranking of 10th which is highlighted at the bottom of the table.
How did they do it? In short, they disproportionately weighted the indicators in which Kansas ranked high. In fact, those indicators are the performances of three very small subgroups of students who do not represent demographically the state’s student body and do not reflect the overall performance of Kansas students.
Graduation rates of limited English proficient (LEP) students and Students with Disabilities – national ranks: 3rd and 6th
First of all, this is an excellent forum to serve as a reminder that high school graduation is more of an attendance achievement than an academic achievement. Also, it is important to understand that comparing graduation rates from state to state is not a reliable measure. Although the feds require uniform methods of calculating graduation rates, graduation requirements vary from state to state. States are free to change their requirements as seen fit. In fact, the Kansas State Board of Education is now in the process of changing graduation requirements.
Those two issues aside, the KASB report gleaned data from the National Center for Education Statistics (NCES), which is a reliable source, since KSDE no longer reports certain demographic data. According to the most recent NCES data, 9% of all Kansas students are LEP (NCES refers to them as English Learners -EL). That does not mean that 9% of Kansas graduates are LEP students.
Given the nature of LEP services, in which a disproportion are elementary students, the actual percentage of LEP students in that graduation cohort is likely to be much lower. Regardless, KASB has weighted the graduation rate of that relatively small subgroup (6.7%), the same as the universe of all of the approximately 35,000 graduates across the state. (Note: KSDE does not publish the total number of students who graduate in a particular year, nor subgroups).
Regarding students with disabilities, which is an umbrella term for special education, again there is no data on the number (or percentage) of Kansas graduates who have disabilities, only the rate is reported. According to KSDE, roughly 15% of all Kansas students are in special education.
Armed with two relatively high national rankings, KASB took the results of those two small subgroups of graduates, and gave those groups twice the weighting of the graduation rate of all students. To be specific, KASB assigned the total number of graduates a weighting of 6.7% and the weighting of LEP and disabled students a weighting of 13.4%. If that makes sense to you…
SAT scores – national rank: 9th
For the class of 2021, only 2% of Kansas students took the SAT college entrance exam. That number is in stark contrast to the 79% of students who took the ACT. (The state now pays for any student who wants to take the ACT.) Yet KASB chose to weight those two the same. The huge difference in the rate of students taking those two exams is not unusual. In most states, a large majority of students take either the ACT or the SAT, but not both. Predictably, the lower the percentage of students taking one of the two tests, the higher they rank across the country.
But the adjoining table shows how Kansas ranks among those states in which 2% or fewer of the class of ’21 took the SAT. Of the 12 states
with 2% or less participation, Kansas ranked tied for 7th with Missouri. It is important to understand that all 12 of these states ranked higher than the state with the highest SAT and at least 10% participation, Massachusetts, which has a mean of 1184.
So, who is that two percent of the Kansas class of ’21 who took the SAT? According to the SAT Suite of Assessments Annual Report for Kansas, 721 students took the SAT. Compare that to 29,284 (79% participation rate) who took the ACT. PrepScholar’s description of those Kansas students says “likely only the most prepared, ambitious high schoolers” take the SAT. The 2% is not only a low number, but also definitely NOT a representative sample of high school students whose results can be extrapolated to the universe of high school graduates. Even those within that 2% are not representative of the student population. Of those 721 students, 171 (24%) are Asian. According to the latest Digest of Education Statistics, Asian students comprise only 2.8% of the entire Kansas school population. It is also important to note that the Asian students had a mean SAT score of 1293, far above the total mean for Kansas. And one more note, it is impossible to determine whether these students were in traditional public schools, private schools or home schools, since SAT does not report that information. And just for good measure, nearly half of the parents of these students hold graduate college degrees. It’s safe to say that the SAT data says little about the 450,000+ students in Kansas public schools.
So, I ask you this: Does it make sense that 9th place SAT ranking is given the same weighting as the 27th place ranking Kansas is given for ACT performance? It does to KASB. Also, see if this passes the smell test: Since both are college entrance exams, does it seem logical that there would be such a range in the two rankings within the same class of students if all else were equal?
Here’s a summation of the data was indeed manipulated regarding SAT results: KASB employed SAT scores that came from a non-representative sample of 2% of the class of ’21, a group made up of a disproportionate share of 171 high-performing students, some of whom might not have even been enrolled in public schools, and given those scores an overall weighting of 8.3% in their calculation of Kansas’s nationwide ranking, a weighting that is higher than NAEP proficient achievement (a combined 7.8%).
Speaking of assessment achievement, here’s how the report not only downplays Kansas’s middle-of-the-pack NAEP rankings, but fallaciously conflates standards with achievement. “A national report found that these Kansas standards for ‘cut scores’ are the highest in the nation compared to NAEP results.” That claim has become a go-to line by the educational establishment as a way to justify low assessment scores. As KPI details in this article, standards and achievement are mutually exclusive. And just for good measure, KASB throws in this statement to minimize assessment results, “(t)est scores are a measure of how well students are mastering academic skills and preparing for post-secondary studies but are of limited use if students don’t finish high school.” Everybody got that?
As a sidebar to assessment scores, here’s a point the State Board of Education should heed: Under the heading of “Academic Preparation for Postsecondary Success,” KASB doesn’t use Kansas state assessment scores. Why? Because there is no way to rank Kansas among the other 49 states because those comparisons are not possible. It is not possible due to the fact that each state has its own unique test and reporting requirements. This is further evidence that it is disingenuous, in the least, to declare as state board members have that “Kansas has the highest test scores in the nation.” Believe me, if there is data to back up that outrageous claim, KASB would be leading with that ranking in their methodology.
In my first research methods class in grad school, long before spending 17 years in Kansas public school classrooms, the professor taught us a very important lesson when reviewing/critiquing social science research. He told us not to focus on the math, but instead question the assumptions that lead to the math. KASB’s math is fine, it’s their poorly constructed assumptions, primarily through the use weighting trickery, that lead to such an invalid conclusion.
Once again, much work has been done to avoid the reality of student achievement in Kansas. The need to refocus on students via choice, transparency, and accountability is overwhelming.
The takeaway from this report should be obvious, data can be manipulated to one’s content, but, pardon the cliché, garbage in, garbage out.
