Back to Conclusions and Policy OptionsBy Stephen Clements
From Kentucky's Teachers: Charting a Course for KERA's Second Decade
pp. 55-64, published 1999
However, the connections between teacher knowledge and skills and student academic achievement levels are not necessarily straightforward, nor are they easy to measure. Commonwealth physicians—to continue the analogy—may treat their patients, but they cannot easily control the population from among which their patients are drawn, or force their patients to take the treatments or medications prescribed. Similarly, teachers must provide instruction to all students, regardless of their socioeconomic background, family support, or commitment to learning. Nevertheless, as noted a few paragraphs earlier, research has shown, and intuition confirms, that teacher skills, knowledge, and capabilities significantly affect student achievement. Shifting achievement levels should therefore be acknowledged as part of any discussion of the Kentucky education system.
Several kinds of data exist regarding academic achievement levels of Kentucky’s school children. Not surprisingly, different types of measures reveal different things about student skill levels. Moreover, the measures used to gauge Kentucky students’ academic progress have shifted over the years, as periodic changes have been made in the statewide student assessment system. Most of these changes have been made because state officials have been dissatisfied with tests that do not provide school officials, parents, and policymakers with the best information available about academic achievement. Education policymakers in other states have faced the same dilemma regarding student achievement tests.
In the late 1970s, for example, Kentucky officials, along with those in numerous other states, especially in the southeast, participated in the "back to basics" movement. This movement was itself a response to mounting national evidence that standardized test scores were sagging (e.g., the SAT), as well as the perception among growing numbers of Americans that schools had become lax and strayed from their primary mission of promoting basic skills and competencies. In 1978, Kentucky’s General Assembly passed the Education Improvement Act, which, among many other things, mandated statewide achievement testing. Hence, efforts to gauge elementary and secondary student progress came to Kentucky.
Norm-Referenced Tests: The CTBS SeriesBy 1982, the state began administering to students in grades 3, 5, 7, and 10, the Comprehensive Test of Basic Skills, Third Edition (CTBS/3), a test commercially available at the time, produced and marketed by CTB/McGraw-Hill.(1) This particular test was administered thereafter biannually through 1988. Its successor, the CTBS/4 test, was administered during 1989 and 1990, and the CTBS/5 test was administered in 1997 (as well as in the spring of 1998). The 1989-90 and 1997-98 CTBS assessments were given because of growing dissatisfaction with internally produced tests and to provide better comparisons of Kentucky students to those in other states.
The CTBS assessments are so-called norm-referenced tests. This means the scores of Kentucky students who took the test are not reported based on the number of correct answers but based on how scores compared to those of students in other states. An important factor in such assessments is the year in which the scoring norms are set. Frequently a norm will be set for such a test and then not updated for several years. Consequently, scores may rise farther above the norm as teachers and students become familiar with the test.
Figure 1 shows scores on these assessments for Commonwealth students. These data indicate that Kentucky school children at various levels have scored close to the national average on the CTBS tests over the years, with elementary students slightly outscoring those at the higher grade levels. Scores on the CTBS/3 drifted upward after 1982, in part because the same version of the test was given and old norms employed—this typically happens under such conditions, and does not necessarily mean students are improving as rapidly as the results suggest. The slightly-below-the-national average scores for 1997 might be due to the administration of the newly normed CTBS/5 test. They might also be accounted for by the long lag time between 1990 and 1997, years when no national standardized norm-referenced tests were given to large samples of Kentucky students. The 1998 CTBS/5 results should be available soon, so a comparison with 1997 scores can be made.
Figure 1: Trends in CTBS Mean Scores, Kentucky Students, Various Years
Criterion-Referenced Tests: KIRIS and NAEPExactly how valuable to policymakers, school officials, and parents is the information provided by CTBS-type scores? Such results may assure Kentuckians that their children perform as a group at levels similar to those of students in other states. But the education reform movement has been propelled by the twin suspicions that student performance nationally was much too low and that the norming process employed in standardized, multiple choice tests yielded far too many "above average" scoring states. For those who delved into problems in Kentucky’s public school system during the latter 1980s, for example, when that system was being reviewed by state courts, it seemed counter-intuitive that Commonwealth students were scoring near national averages, given documented school problems and funding variations.
Indeed, concerns about the questionable usefulness of CTBS scores for judging students’ academic achievement levels were much on the minds of Kentucky education policymakers during this time. During 1989 and 1990, the joint executive-legislative Task Force on Education Reform was conducting hearings and working toward the school system overhaul known as KERA. Task Force members spent many hours struggling over the problems associated with assessing student progress and consulting with testing experts from around the country.
In the end, the subcommittee on curriculum, and later the entire Task Force, rejected reliance upon norm-referenced standardized tests such as the CTBS, and instead opted to create a performance-based testing system that would be linked to established performance standards and a new core curriculum, use the scoring categories from the National Assessment of Education Progress (NAEP), and drive the state’s reward and sanction system. This approach was included in HB 940, the 1990 reform law. The new tests would involve authentic academic tasks—writing and thinking skills—and, inasmuch as they were linked to a core curriculum, would be worth "teaching to."
The assessment system that was created to embody these principles was known as KIRIS, for Kentucky Instructional Results Information System. Enormous amounts of time, effort, and financial resources were invested in developing KIRIS during KERA’s first several years. Much work had to be done to create this new testing system and its related components, given that no off-the-shelf assessment tool was available for purchase and implementation in Kentucky.
As noted, the General Assembly stipulated that KIRIS tests be similar to the NAEP tests. NAEP assessments have been given to national samples of 4th, 8th, and 12th grade students since the early 1970s, in subject areas such as reading, history, geography, math, and science. These tests have two primary characteristics. First, they ask students to engage in a range of activities that employ a variety of different types of skills. The test, in other words, is not simply a multiple-choice device. And second, the assessments are scored based primarily on how well examinees fare relative to predetermined performance criteria, not simply in terms of how well all other test takers fared.
NAEP staff use an extensive national consensus process in developing content frameworks for the tests, such that teachers, parents, curriculum experts, and others are involved in determining what students should know and be able to do at different points in their school experiences. NAEP reports score results in several ways, but its most popular method involves classification of the percentage of students who score at one of several different levels: Below Basic, Basic, Proficient, and Advanced.
The creators of the KIRIS tests borrowed heavily from the NAEP approach. As with NAEP, the KIRIS tests employed multiple choice questions, but also open response questions and other types of assessment tools. KIRIS also employed a scoring system similar to NAEP’s: Scores were reported at the Novice, Apprentice, Proficient, and Distinguished levels, which roughly paralleled the score categories of NAEP. It is noteworthy, however, that there is not a complete correlation between KIRIS and NAEP results. KIRIS tests were developed to be an integral component of KERA’s school accountability system and were designed to provide useful school-level data as opposed to student-level data. Therefore, KIRIS results primarily provide information about the progress of students at certain grade levels in each school toward the outcomes defined in the Kentucky curriculum framework. NAEP scores, on the other hand, provide information about aggregate student performance vis-à-vis NAEP-defined skills and knowledge.
In addition to creating the KIRIS tests, KERA assessment administrators had to pursue several additional tasks simultaneously for the accountability system to be put into place. One was to turn the learning goals for Kentucky students—identified in the fall of 1989 by the Council on School Performance Standards—into specific learning outcomes.(2) Panels of Kentuckians originally identified and approved a list of 75 "valued outcomes." This list was later streamlined into 6 learner goals and 57 academic expectations.(3) The bulk of the expectations that were measured as of 1998 are those which are most subject-oriented in content. Those goals which involve student attitudes and behaviors are not well covered by the KIRIS assessments. (4)
Another task was to develop curricular materials linked to the specified learning goals. This led eventually to the creation of two documents: the Core Content for Assessment, which represented the specific material that would be used in developing assessment items, and Transformations: A Curriculum Guide, which would cover the core content material but also provide broader curricular guidance regarding the knowledge and skills Kentucky students should have to meet the academic expectations.(5) When these materials were completed, they were circulated to all Kentucky schools.
The creation of the KIRIS tests themselves was undertaken by New Hampshire-based Advanced Systems in Measurement and Evaluation, Inc. (ASME), under contract.(6) These tests have most recently (1997-98) been administered in schools at grades 4, 5, 7, 8, 11, and 12, although in earlier years—1991-92, 1992-93, 1993-94, and 1995-96—they were given only to students in the 4th, 8th, and 12th grades. In the first several rounds of testing, the exams focused on Kentucky’s core curriculum and academic expectations in reading, mathematics, science, social studies, and writing. More recently, KDE has added assessments of arts and humanities, practical living, and vocational studies, although too few longitudinal data have been gathered in these areas to provide useful information. The tests originally included both multiple-choice and short-answer sections. Students also prepared writing portfolios, which were graded and included in the calculations for school accountability scores.(7)
According to KIRIS results, Kentucky students made fairly significant progress during the first few years of KERA implementation, both collectively and at many schools. This is especially true with regard to moving large percentages of students from the "novice" to "apprentice" categories, as is shown in Tables 1-3. Schools have had relatively less success, however, in moving students into the "proficient" and "distinguished" categories.
In terms of aggregate KIRIS score averages, Kentucky students made progress between 1992-93 and 1994-95 in most subject/grade areas, but lost ground in 10 of 15 subject/grade areas between 1994-95 and 1995-96. The 1996-97 data showed index score improvement in most areas. But 1997-98 data showed slippage again in some areas (see Figures 2-4).
Table 1: Percentage of Grade 4/5 Students by KIRIS Performance
Table 2: Percentage of Grade 7/8 Students by KIRIS Performance
Table 3: Percentage of Grade 11/12 Students by KIRIS Performance
Figure 2: Grade 4/5 KIRIS Averages, 1992-93 through 1997-98
Figure 3: Grade 7/8 KIRIS Averages, 1992-93 through 1997-98
Figure 4: Grade 11/12 KIRIS Averages, 1992-93 through 1997-98
Other tests only partially corroborate the large KIRIS achievement gains reported by KDE during the assessment’s first years. Specifically, data from NAEP show Kentucky students making modest progress, but not nearly as much as shown by KIRIS. Beginning in the early 1990s, NAEP assessments began to include large enough samples of students that state-by-state comparisons could be made. To date, assessments with state-level data from NAEP have been conducted at various grade levels in reading, math, and science.
According to the reading and math results—for example see Table 4—Kentucky students have been making modest progress in those subjects, gradually moving into higher achievement categories. But students elsewhere in the nation are improving as well, with the result that Kentucky students still lag behind national average achievement levels.
Table 4: KY and U.S. NAEP Results in Reading and Math
NAEP has reported science achievement for its state-by-state results differently than for reading and math. According to the 1996 science report, Kentucky eighth-graders scored very close to the national average. This score placed Kentucky in a cluster of states that included upper tier southeastern states such as Maryland, North Carolina, Virginia, and West Virginia, but also states from other regions, such as New York, Arizona, Rhode Island, Texas, and Washington.
Table 5: NAEP 1996 Science Assessment Summary for All States
Aggregate ACT scores of college-bound Kentucky high school seniors have been flat for several years, although math and science scores from subsets of Kentucky high schoolers show these students faring better than their peers in other southeastern states.(8)
Figure 5: ACT Exam, KY and U.S. Scores
The ACT scores, reported in Figure 5, suggest that KIRIS results, at least those reported at the 11th or 12th grade levels, could be mildly inflated. They also indicate that the KERA reforms are having significantly less impact at the high school level. The ACT is not, of course, taken by a random sample of Kentucky high school seniors, but rather those interested in going to college. However, students from other states are self-selected from the same "pool," and hence the results are more or less comparable.
Drawing conclusions about achievement levels of Kentucky students based on this collection of test results can be hazardous, especially given the discrepancies between KIRIS scores and those of the other tests. As noted, the national standardized test results provide information of limited usefulness, inasmuch as neither the nature of the tests nor their content corresponds to Kentucky’s KERA-driven core curriculum and assessment systems. Such tests, or at least those recently normed, have nevertheless found Kentucky students scoring near the national average at most grade levels. This does not tell Kentucky policymakers how well students should be scoring, however.
The KIRIS results, on the other hand, do not correspond directly to results on other tests given nationally or in different states. It is therefore difficult to determine how to size up the state’s own internal test scores. According to KIRIS reports, with minor exceptions Kentucky students at various grade levels have been making slow but steady progress, at least in terms of moving students from the "novice" to "apprentice" achievement categories. The schools have made relatively little progress, however, in moving significant percentages of students into the "proficient" and "distinguished" categories. Moreover, given problems with the KIRIS system and the inability to match KIRIS results with those from other tests, it has not been possible to verify KIRIS results as well as policymakers within the state would like.
Indeed, in late 1998 yet another critical evaluation of the KIRIS system was published, this one by RAND Corporation researchers.(9) The authors documented in much more detail the disjunction between KIRIS scores and those on selected nationally administered tests. They concluded that features of the KIRIS test administration process and the high stakes nature of the assessment combined to inflate KIRIS scores substantially. They also recommended a series of measures designed to reduce the likelihood of scoring problems in the future.
Even assuming some KIRIS score inflation, however, the full array of test scores provide evidence of some academic progress in the state. This is especially so given the large percentages of impoverished students in the state. As Figure 6 shows, Kentucky has struggled for years with higher-than-average child poverty rates, a fact which certainly has significant educational consequences. In light of these percentages, it should be good news that test results show Kentucky students scoring close to national averages.
Figure 6: KY and U.S. Poverty Rates Among Children (under Age 18)
On the other hand, the results also show that Kentucky students have a long way to go to fulfill the promises of KERA. According to both KIRIS and NAEP, for example, relatively few Kentucky students perform at the "proficient" level, which is the performance category recognized by many experts as the skill level a student must reach if he or she is to function effectively in school, society, and the economy. Kentucky is to be commended for attempting to create a NAEP-like scoring system, which provides more useful results than does the CTBS. But Kentucky policymakers should also acknowledge that much remains to be done in terms of moving large percentages of students into higher performance categories.
CATS: The Post-KIRIS Assessment SystemDissatisfaction with various aspects of the KIRIS system led the 1998 Kentucky General Assembly to pass legislation requiring the State Board of Education to revamp the KERA assessment system. The process of creating the new system—to be called the Commonwealth Assessment and Testing System, or CATS—is currently underway, but there is still uncertainty about how different it will be from its predecessor. It is likely that CATS will be quite similar to KIRIS in numerous respects. The new assessments, however, will incorporate additional multiple-choice items from nationally standardized tests, so that Kentucky students can more easily be compared to their counterparts in other states. The tests will also consume less classroom time than KIRIS. New baseline scores for individual schools will also be set, so that it will be several more years before bona fide accountability rewards and sanctions can begin. Other components of the assessment system—for example, the 4th-grade writing portfolios—will also likely be altered. The intention is to create a more reliable and valid assessment, one that is less burdensome on teachers and students and that yields more useful information.
Regardless of the details of the new plan that should emerge soon, the switch to CATS from KIRIS will have two results. First, Kentucky will continue to pursue a largely uncharted and risky path toward useful, performance-based assessments, which will drive the accountability system and pressure schools to ensure that students continue to make academic progress. This will help keep Kentucky on the cutting edge of achievement testing and systemic school reform.
On the other hand, the CATS system, like KIRIS before it, may not be designed to gather data in such a manner that the scores from individual students in individual classrooms can be compared. Instead, CATS data will likely provide school-level information. This type of information should indeed be useful for making reward and sanction decisions within the strategy enumerated in KERA. But it will not help us measure the academic progress of particular students. Nor will it provide information that will help officials determine which types of teachers are particularly effective, inasmuch as this is not currently a mandate for the new system.
Back to Conclusions and Policy Options
Ahead to Appendix B: Kentucky's Teacher Training System
This and additional information about statewide testing in Kentucky can be found in Anthony J. Nitko, A Guide to Tests in Kentucky (Frankfort: Kentucky Institute for Education Research, 1997). The KIER executive director prepared a short section found near the beginning of this publication entitled, "A Recent History of Statewide Achievement Testing in Kentucky." Return to text.
These goals were as follows: (1) Students are able to use basic communication and mathematics skills for purposes and situations they will encounter throughout their lives; (2) Students shall develop their abilities to apply core concepts and principles from mathematics, the sciences, the arts, the humanities, social studies, practical living studies, and vocational studies to what they will encounter throughout their lives; (3) Students shall develop their abilities to become self-sufficient individuals; (4) Students shall develop their abilities to become responsible members of a family, work group, or community, including demonstrating effectiveness in community service; (5) Students shall develop their abilities to think and solve problems in school situations and in a variety of situations they will encounter in life; (6) Students shall develop their abilities to connect and integrate experiences and new knowledge from all subject matter fields with what they have previously learned and build on past learning experiences to acquire new information through various media sources. Return to text.
The original list of 75 outcomes was revised because of objections raised to a number of the outcomes, which were said to involve affective issues and which would be difficult to teach and assess. Other outcomes were criticized for being too vague or laden with jargon. Moreover, the term "outcomes" was becoming a political liability, and this prompted the renaming of the list as "academic expectations." Return to text.
Discussed in Joseph M. Petrosko, "Assessment and Accountability," 1996 Review of Research on the Kentucky Education Reform Act (KERA) (Frankfort: Kentucky Institute for Education Reform, 1997). Return to text.
The processes involved in accomplishing these tasks—or at least KDE’s role in those processes—have been described by former KDE official Betty Steffy in her book Kentucky School Reform: Lessons for America (Lancaster, PA: Technomic Publishing Company, Inc., 1993). Return to text.
Several recent documents have appeared that aptly summarize development of Kentucky’s assessment system, the research and analyses that have been conducted, and the student achievement outcomes of these and other tests administered to Kentucky school children in the years since KERA’s passage. These materials have been summarized in Petrosko, "Assessment and Accountability." See also Anthony J. Nitko, "A Guide to Tests in Kentucky: A Description and Comparison of the Comprehensive Tests of Basic Skills, the California Achievement Tests, the TerraNova (CTBS/5), and the Kentucky Instructional Results Information System Assessment" (Frankfort: Kentucky Institute for Education Reform, 1997). Return to text.
Math portfolios were also originally part of the KIRIS plan, but these have been removed because of various scoring problems. Return to text.
This is the finding of an SREB analysis of data collected on students from Kentucky secondary schools that participate in the SREB’s High Schools That Work (HSTW) initiative. In reading, Kentucky HSTW students had an average score of 280, while those in other states averaged 273. In math, Kentucky scores were 292 compared with a regional average of 285, and in science the Kentucky average was 287, compared with 283 elsewhere. KDE press release No. 97-007, February 3, 1997. Return to text.
Daniel M. Koretz and Sheila I. Barron, The Validity of Gains in Scores on the Kentucky Instructional Results Information System (KIRIS). (Santa Monica, CA: RAND, 1998). Return to text.