In this previous article, KPI showed the State Board of Education (SBOE) members’ level of confusion, frustration, and ignorance during the July 12 meeting regarding the entirety of the state assessment scoring process. (The hour-long report and discussion can be seen here).
The purpose of this piece is to show that despite all the efforts described by KU’s Dr. Neal Kingston – the lead individual hired by the SBOE to run the state assessments – and KSDE’s Beth Fultz to make the process seem highly ‘scientific,’ each step in the process is actually quite subjective. And all that attention to process detracts from these two realities that garnered no attention at the meeting:
- State assessment scores are going down, and
- Income-based achievement gaps are embarrassingly high regardless of what process is being utilized.
The process
The first step in the entire process, the standards adopted by the SBOE – in this case, 10th-grade math – are subjective. There is no ‘right’ or ‘wrong’ set of standards. Setting standards is a political process, “policy,” as Dr. Kingston referred to it. Look no further than to Common Core as an example. Furthermore, Kansas school districts are not required to teach curriculum that supports state-adopted standards. That’s another subjective process determined by local school boards and best discussed in another piece.
The testing system, which follows the adoption of those standards, is nothing more than a series of subjective decisions, starting with the test itself. The questions that are put on the test, the number of questions, and the testing process/procedure are all vital but still subjective decisions.
As a parallel, the process of categorizing the results is extremely subjective. Despite the attempt of Dr. Kingston to refer to the methods as “tried and true” or “research-based,” decisions are made that are not ‘right’ or ‘wrong.’ The SBOE determined how many levels of performance there were to be – in this case, four levels. As Beth Fultz detailed – as described in the previous article – that was a very political process. In other words, very subjective. And perhaps the most subjective decision of all is the process of defining those four levels. What does it mean to be a Level 1,2,3 or 4? In Kansas, the definitions of the four levels are the same verbatim – save for one word. In terms of math ability, the Level 1 operative word is limited. In Level 2, the word is basic. The word for Level 3 is effective, and for Level 4, the word is excellent.
A scale score range had to be employed to give each student a raw score. In the Kansas state assessments, students are not scored on a percentage correct basis, students are scored from 220 to 380 points. There is no magic to that. Then the cut scores had to be developed. As Dr. Kingston pointed out, this is a very involved process that utilizes a number of math educators across the state who ultimately must agree on the numbers.
Finally, those scores are presented to the SBOE for adoption. And as board members Mah and Arnold alluded, they would be interested in seeing the distribution of scores across those four levels before making a final decision on accepting those cut scores. Obviously, another subjective decision here.
To be clear, subjectivity doesn’t make the process any less crucial. On the contrary, these decisions are extremely important, it’s just that attempts to dress them up in overly-scientific jargon misses the forest for the trees.
The results
What’s missing in this entire subjective process is the objective, measurable results that were not part of the board discussion at the meeting. We’re missing the view of the entire forest as staggering achievement gaps between kids at different income levels are a constant, never mind stagnating achievement overall.
First, since the inception of the process that is now utilized – pursuant to the federal ESSA requirements and has generated test scores since 2016 – the scores in both math and ELA are trending downward. And as pointed out in the last article, these lower scores are a function of the lack of emphasis and importance put on state assessment results. The graph below is very telling. It shows the percentage of all students who scored in Level 1, the lowest of the four levels. Regardless of all the discussion and debate over what the four levels mean, there is consensus that Level 1 is clearly below grade level. The percentage of students scoring below grade level is trending up, COVID notwithstanding.
Second, income-based achievement gaps have existed for years and years, and those gaps in performance escape the concern, even the attention of the SBOE and KSDE. Regardless of the way the process is sliced and diced, one that has changed significantly over the years ranging from standards setting to the distribution of results, achievement gaps between low-income and not-low-income students are significant, persistent, and are not getting better. The three figures below are evidence of that.
The first is a graph taken directly from KPI’s research paper At-Risk Funding: Increased Money Fails to Increase Achievement. The figure shows the gap between low-income and not-low-income – all grade levels – during the No Child Left Behind testing years. The second figure is a table that shows 10th grade math results in the post-NCLB years, 2016-2021 (2022 data has not yet been released by KSDE).
Two different time periods, two different sets of standards, tests, definitions, grading, etc., etc., yet extremely similar achievement gap results. The percentage point difference between the two income groups is consistently in the 20s.
And for good measure, the final table takes this phenomenon out of the state assessment realm and into NAEP results, a test KSDE recognizes as valid and is considered the “Gold Standard.” Once again, a different population (NAEP tests a representative sample of students across the state), different time periods, different questions…Same results, with gaps consistently in the 20+ percentage point range.
Kansas isn’t alone in experiencing these income-based differences, but there is no excuse for willfully ignoring these results. The state legislature funds the at-risk program to the tune of nearly one-half billion dollars each year. This money is supposed to be specifically targeted toward reducing achievement gaps. It clearly isn’t working, as KPI has revealed over and over, and Legislative Post Audit found in this report. As long as the SBOE and Education Commissioner Randy Watson continue to downplay the importance of state assessments and continue pursuing fluff initiatives like Kansans Can and the trip to the moon school redesign project, these numbers will not get better. Furthermore, local districts are complicit by neglecting students’ needs in determining how this money is to be spent. Hopefully, recent legislation that beefs up the state requirement of local districts to address student achievement during the budget process will begin a reversal of this systemic disregard.
We can only hope that at the August SBOE meeting, in which the board is slated to approve the new 10th-grade state assessment math cut scores, there is at least some attention given to the fact that no matter what set of cut scores they adopt, achievement gaps will remain and results will continue to get worse. Given the SBOE’s and Commissioner Watson’s ‘see no evil’ viewpoint, that is likely nothing more than wishful thinking.
The real answer lies in the Legislature giving Kansas students “a fighting chance” by opening avenues of educational choice. Here is a laundry list of choice alternatives that would not only improve students’ chances but would also move the public school system from institution-centered to student-centered.
- Universal education savings accounts (ESA) – patterned after the new Arizona Universal ESA law that is available to all parents in the state to spend what they believe appropriate for their children’s education.
- Revise the yet-to-begin open enrollment program to mirror the South Dakota open enrollment program that puts no restrictions on students moving to a school district of their choice.
- Expand the existing tax credit scholarship program to serve more students.
- In light of the recent U.S. Supreme Court decisions that have struck down the prohibition of public funds to religious schools, begin the process of amending the language of the state constitution to reflect that change.
- Amend the charter school law to allow authorizers other than just school districts. This would remove charter schools from being under the thumb of local school boards, which is now the case and why there are fewer than a dozen charter schools in the state.
- A version of a Parents’ Rights Bill that requires a level of transparency in the way their children are taught.
