A Lesson on Norm-Referenced Testing
from Ames, Iowa

The summary paragraph of that section begins with the sentence, The real problem in the continued use of norm-referenced tests is the perception on the part of many that they are useful and meaningful. I admit that I am one of the many. I also admit that they would be less open to criticism - and even more useful and meaningful - if they were not norm-referenced. Far preferable would be the AP Calculus scoring (although I prefer starting from 0, when earned, rather than 1) where everybody can fail (it's running at about 40% failure, I hear) or everybody can get a 5, the top score possible, and life-changing credit depends on that score. That said, however, assessment experts such as Professor Romberg throw the baby out with the bath. The tests themselves are fine. Even the alternative reporting of the scores is fine (little Johnny is reading at a grade-level 3.2 but his math is at 4.1 and he's only second semester ...) This reporting is somewhat arbitrary but it really is very informative to school boards, to teacher evaluators, to teachers, and to parents - even to students themselves as they become more educationally mature.

Despite the obvious theoretical drawback (i.e., some must be first and some must be last even if all are competent or all are incompetent by some more sophisticated assessment), the traditional percentile scores are useful and meaningful. There are enough items, carefully enough chosen, that the argument is red herring. To find that little Johnny, though outwardly normal, is performing at the 25th percentile is an important warning to involved teachers and parents that intervention may be necessary. The 90th is information that probably it is not. Similar scores at the high school level provide information that some course choices and some career choices may be more viable than others, at least at this point in their educational history.

The problem with the new assessment is that it is not assessment at all. It is refusal to assess when refusing to assess forthrightly would draw well deserved public outrage. California's CLAS, Kentucky's KIRIS, etc., all share the same problem in mathematics. They are so unreliable that they can have no consequences or people will (and rightly will) sue if they are used in making individual student decisions. The ITBS, CTBS, SAT-7, ERB, etc. do not have that problem. Abandoning their annual use becomes excellent cover for incompetence. If weak curricula, schools, or instructors are judged as students are judged, expect to hear the time-honored excuse of weak students, The test is not fair!

This would appear to be one reason Professor Romberg does not like norm-referenced tests. He is also the head of a math curriculum project called Mathematics-in-Context (MiC) now commercially available from Encyclopedia Britannica. The following are partial results of a MiC pilot program in Ames, Iowa, a district that once had a particularly strong mathematics program. These words are those of Marianne Jennings for the Arizona Republic.

In 1985, the Ames [Iowa - no connection with the ITBS which is actually published in Itasca, Illinois] Community School Board introduced a standards-based "math in context" program. "math in context" means no rote memorization, heavy dependence on calculators, and use of "real-world problems" [how many insects will it take to eat 300 acorns?]. The following chart demonstrates the impact on the Ames school children, who are, for the most part, children of folks who work at Iowa State University [i.e., they were a group of high performers before they got math in context].

     GRADE    1985-86 ITBS    1994-95 ITBS
       3            96             47		
       4            98             31
       5            98             26
       6            90             37
       7            99             21
       8            99             24

These numbers came from an article in the Ames Daily Tribune of February 21, 1995, that reported strong public concern regarding this record of student performance. Parents filled the Ames Community School Board meeting room Monday seeking solutions for a dramatic drop in basic math skills in some of the lower grades. Many parents blasted a "math-in-context" program, saying it is the culprit in their diminishing abilities to handle basic math skills on standardized tests.

My first assumption was that this was the same Mathematics-in-Context (MiC), out of the University of Wisconsin, that I knew was being piloted there, but apparently that is only partially the case. This article never uses the upper case form but does indicate pilots in the 1993-94 year with full implementation in grades five through eight in 1994-95. One associate professor at Iowa State was quoted as saying that she was horrified by the charts showing drops in math skills over the last year, that is, the 1994-95 year. How much of the drop prior to that - or even during that year - is due to the capital form (MiC) and how much is due to a continuing history of worsening performance for other reasons such as lower case mathematics-in-context, calculators, group activity, group evaluation, social promotion, deliberate absence of textbooks, etc., would be speculative. The before data year, 1985-86, was chosen because that was the year that regular use of calculators was introduced into the Ames school system.

It was reported by another credible source that the MiC emphasis is only in grades six through eight so we should look especially carefully at the grade n to grade n+1 data for those grades and that year. That kind of cohort information over several consecutive years can be very informative which is part of the reason that the effort by self-styled assessment experts to abandon these annual tests is so disturbing. From the attached chart of both the national and the state data that came from that Ames newspaper article, we have 59 to 36 and 77 to 43 at the national level and 51 to 21 and 63 to 24 at the state. Note that the least of these declines matches the greatest of that of the lower grades and the seventh to eighth grade drop represents more than half of the entire drop from the strong 1985-86 numbers in a single year. It would appear that the official University of Wisconsin materials were even more damaging than their lower case predecessor.

In fairness to the Ames program, it should be reported that, after concerted effort, the scores are now back up somewhat although I have no idea how much. The numbers have been requested but I have not yet received them. Somewhat disconcerting, however, is that part of this recovery appears to have been from teaching directly from past exams, a practice contrary to the recommendations of ITBS and its honor system. Apparently the data from an entire grade level was discarded because of an instance of using the current year's test, itself. There is more to be taught than just mathematics, of course. Students will remember an incident like that for the rest of their lives. Such abuses only add credence to those who argue for the elimination of these useful and meaningful exams - the very exams that brought to light the dramatic decline in student performance.

                            Ames, Iowa  ITBS Data

               National Percentiles         State of Iowa Percentiles

 Grade         85-86  93-94  94-95             85-86  93-94  94-95
   3            90     71     70                96     44     47
   4            94     60     56                98     49     31
   5            90     59     40                98     51     26
   6            96     59     52                90     51     37
   7            94     77     36                99     63     21
   8            93     51     43                99     28     24