The following statements have been submitted to the National Assessment Governing Board, which now has design authority over the potential Voluntary National Test in mathematics:
January 20, 1998
Attn: Munira Mwalimu
Aspen Systems Corporation
2277 Research Boulevard, Mail Stop 4-D
Rockville, Maryland 20850
To Whom ...
I write in regard to the plans for the Voluntary National Test in Mathematics. Let me preface my remarks with the comment that I am a firm believer in the necessity of testing and was delighted to hear President Clinton giving the idea such publicity. His "algebra is algebra" quote is now a favorite of mine, acknowledging credit, of course.
That said, I could hardly believe my eyes when I saw the list of people on the committee last summer. It had as anti-testing of a bias as I could have imagined. This goes all the way to the chairman of the committee, John Dossey. This gentleman was involved in development of the book that has gotten much amusing notoriety as "Rain Forest Algebra", being so branded by a writer who had spent time with her daughter's book. Instead of algebra being algebra it had become a PC, environmentally green, form of something quite different. If it worked, of course, that would be different but it was her bright daughter's lack of algebra competence that brought up the subject in the first place.
I was a strong supporter of the move to place the VNTM under the oversight of NAGB, exactly where it belongs, but problems still remain. Most overwhelmingly, this same John Dossey has not been replaced by someone who believes that algebra is algebra! That should be the first change in approach or the exam is probably doomed before it rises from its ashes (assuming, of course, that it ever does). The country is in the midst of renewed mathematics education awareness (e.g., the Loveless articles in Education Week documenting the "Second Great Mathematics Rebellion"). This should have precluded the use of Professor Dossey on the list of active participants. Even if the eventual exam turns out to be exactly the same, some of us will assume that he was instrumental. Dossey's presence is poor, perhaps fatal, politics.
Regarding the actual exam, much ado is made to insure the interested reader that "the tests will be individualized versions of NAEP" and that "they are fully consistent with NAEP's content and performance standards." As long as this is just education rhetoric, there is no problem. Say anything you like. Doing it would be quite another matter. NAEP does not get down to the individual student level so individual reliability is not a problem. In fact, "No student takes the entire assessment." The VNTM will get down to the individual student level, so it is a very real problem. NAEP does not get down to the individual student level so scoring the structured free response items called "Extended Open-Ended Items" (attachment) is not a problem. The VNTM will, so it is a huge problem. There cannot be any or it will be prohibitively expensive and too slow in getting the results back to parents and schools. If NAGB is clever enough (see my suggestion at the end of the attachment) to devise a test that can be entirely machine scored and yet "fully consistent" - so that it can have enough items of enough different types to be reliable at the individual student level and still have the results out in a couple of months, it will be successful and schools and districts will quickly be pressured to sign on. Parents, school boards, and math curmudgeons such as myself will mandate it. If it is not that clever, NAGB had better just lie and ignore full consistency.
Nothing will make a laughingstock of the exam more quickly than if a few schools' local math geniuses do badly or worse, a few village idiots score well. Both occurred in our 1993 and 1994 experiment with CLAS here in California but, since the results didn't get back to individual students, only those on the inside - or anyone who understands probability - knew it. Also, getting the hand-scored portion of the results back was a huge problem, far beyond what the test planners had anticipated. It's a tough realization to which the NAGB must come, given its longstanding reform-minded history, but it is the only way Clinton's dream will ever be realized. Lo and behold, ITBS, CTBS, ETS, etc. have been right all along. There is no need to have any free-response items on this kind of broad-screen exam, in fact the negatives far outweigh the positives. Enough items for reliablity, immediate scoring, and cost make it a no-brainer.
Respectfully submitted,
Wayne Bishop, Ph. D.
Mathematics and Comp. Sci.
California State University, LA
Los Angeles, C 90032
Attachment from the NAEP website:
Extended Open-Ended Items
Extended open-ended items require students to consider a situation that demands more than a numerical response or a short verbal communication. These items require the student to carefully consider a situation within, or across, the content strand areas, understand what is required to "solve" the situation, choose a plan of attack, carry out the attack, and interpret the solution derived in terms of the original situation. The response mode requires that students provide evidence of their work on some of these aspects of the solving process and communicate their decision-making steps in the context of the problem.
For example, consider the following problem developed for the 1992 NAEP mathematics assessment:
This question requires you to show your work and explain your reasoning. You may use drawings, words, and numbers in your explanation. Your answer should be clear enough so that another person could read it and understand your thinking. It is important that you show all your work.
Radio station KMAT in Math City is 200 miles from radio station KGEO in Geometry City. Highway 7, a straight road, connects the two cities.
KMAT broadcasts can be received up to 150 miles in all directions from the station and KGEO broadcasts can be received up to 125 miles in all directions. Radio waves travel from each radio station through the air, as represented below.
[Graphical Image of concentric circles around a radio tower.]
On the next page, draw a diagram that shows the following.
Be sure to label the distances along the highway and the length in miles of the part of the highway where both stations can be received.
In this example, students must use logic and diagrams to communicate the reasoning behind their solution to this task. Hence, several key elements of mathematical power are being measured here.
[Instead, ask some questions that can be machine scored that assume they did draw the picture but don't require it to be submitted (except maybe to the teacher for a later discussion about the item).
i. How far apart are the cities? (If they miss this one ...! But some will.) ii. How far toward Math City must someone drive in order to pick up station KMAT? iii. For how many miles can both stations be received?]
The Board's present mandate, in math as in reading, is to ensure that the content and item specifications for the proposed national tests are faithful to the NAEP frameworks and that the reporting standards for such tests are those of NAEP itself.
I will not second-guess the intent of Congress in making that assignment. I do, however, urge the Board to understand the changing context of math education in the United States, and to weigh its decisions accordingly.
I need to begin with a big mea culpa. Much of what I'm about to tell you is a problem can fairly be said to have begun on my watch, as a member of this Board for eight years.
In 1990, we began to steer NAEP toward the recommended standards of the National Council of Teachers of Mathematics (NCTM), which had been published in 1989. In revising the NAEP math framework for the 1992 assessment cycle, and revising it yet again for the 1996 cycle, the Board steered ever closer to NCTM's view of what constitutes desirable standards, curricula, instructional methods and assessment strategies. Steadily more emphasis was placed, for example, on "open response" test items, on calculator use, and on conceptual understanding. That evolution is plain in numerous NAEP documents, including the 1996 mathematics Report Card. The three "math abilities" and three forms of "mathematical power" depicted in that report, for example, are pure NCTM doctrine.
Why this evolution? In retrospect, I would say there were two main reasons:
Second, NCTM was riding high as arbiter of K-12 math for the U.S., perhaps even for the world. Its standards were held up as a model that other subjects might aspire to. Never mind that they never included actual student performance levels or benchmarks. Nor, prior to their universal dissemination, were NCTM's standards subjected to field tests or clinical trials to ascertain whether they actually boost student performance. (The Food and Drug Administration would never allow a medical device or procedure to reach the open market under such circumstances. Nor would responsible physicians use it without solid data arising from large experiments.)
Yet few responsible voices were heard even to challenge the NCTM view of math or to offer alternatives. (I wrote an essay in Education Week in January 1993 entitled "What if Those Math Standards Are Wrong?" but everyone just assumed I was a fuddy-duddy.)
Indeed, I think it's accurate to observe that the Board's math frameworks for NAEP would likely have embraced the NCTM approach even more fully had it not been for concerns about trend lines and about the assessment getting out too far in front of school practice. But the frameworks and assessments went a great distance toward NCTM, ever closer as the 1990's progressed. I would estimate that by 1996 the NAEP math framework had become at least 75% reflective of NCTM's recommendations.
That's a mighty quick and abrupt shift for the "Nation's Report Card", a hint, it's now clear, of the danger inherent in allowing so sensitive and consequential a measuring stick to be bent to the contours of curricular fads and educational "innovations" that carry plenty of big names and paper credentials but that have not, in fact, proven themselves in the real world.
Even 75 percent, however, did not satisfy the insatiable NCTM and CCSSO. They made plain (in the draft national test specifications that became public last summer) that they were bent on achieving perfect conformity between NCTM's view of math and the assumptions on which any new national test would rest. Such a result would, of course, strengthen NCTM's hegemony over how math is taught throughout the United States. Calculator use, for example, would be universal. Old-fashioned arithmetic would be minimized. And so forth. The national tests would become a weapon in NCTM's campaign to obliterate what it views as unenlightened math.
And that, I believe, is exactly what would be happening today if the original national test plan had continued unchecked. It's evident that the U.S. Department of Education was a co-conspirator with CCSSO and NCTM and, so long as it was in charge, had no intention of applying the brakes. (This is not surprising. Senior administration officials have not hidden the fact that they, too, are NCTM acolytes.) The resulting test probably would not be comparable to NAEP. And passing it would be set at the "basic" level, far below the "proficient" standard that NAGB has worked so hard to establish as the necessary level of academic attainment for young Americans. But the original plan did not prevail. In the muddled compromise that finally emerged from Congress, NAGB is more or less in charge and is supposed to ensure that any national tests be faithful to the curricular assumptions and performance standards of NAEP itself.
There's just one hitch. The math world is again changing. Since those NAEP frameworks were developed (with, as I said, perhaps 75% NCTM influence), earthquakes have begun to rattle the once-solid structure of NCTM math. It's too early to be sure, but there is reason to suppose that American education has started moving away from some key NCTM assumptions.
There is, in fact, something of a brushfire underway in this land, a kind of populist revolt against NCTM-style math and against the constructivist view of education more generally. Angry parents are being heard from, denouncing what they sometimes term "fuzzy" or "rain-forest" math and demanding to know why their fifth graders are still counting on their fingers. State legislatures and boards of education are rallying to defend the multiplication tables. (Consider the recent decisions taken in California.) The more we learn about math curricula and pedagogy in countries that score higher than we do on international assessments, the more doubts crop up about certain NCTM assumptions concerning the nature of mathematics and the optimal modes of math instruction. (Eighth graders in Korea and Japan, for example, do not use calculators on tests, but wipe the floor with us on international math assessments.) Perhaps most significantly, major math organizations (real professors of mathematics, not math educators) are beginning to point to fallacies and inadequacies in NCTM's assumptions when universally applied at the elementary-secondary level. Second graders grappling with subtraction are not, after all, the same as Ph.D. candidates in mathematics.
What does this mean for NAEP and NAGB? To me, it means that if a nationwide math consensus project were launched today and conducted honestly, i.e. with a full measure of consumers and real scholars and not again dominated again by CCSSO and NCTM, the result would almost certainly be less reflective of NCTM- style math than was the 1996 framework. The real situation in the United States, in other words, is different from that which CCSSO and NCTM took for granted last summer. They supposed that the NAEP framework was the fuddy-duddy document, the one that needed to be conformed to NCTM doctrine. But that's simply not true. In fact, it appears to me, in today's context, that the NAEP framework is itself worrisomely derriere garde, reflecting the math assumptions of 1989, assumptions now no longer universally shared and no longer viewed as the cutting edge but as the receding edge of a failed revolution.
Secretary Riley recently called for an end to the "math wars". That's a splendid sentiment, even if many in his own Department believe it must mean the surrender of long division to the "problem solving" forces of the NCTM. Perhaps the development of new specs for voluntary national tests is the place to seek a just and lasting peace. Maybe NAGB can become the peacemaker. But for that to happen, I respectfully suggest, the Board must itself go back to ground zero, jettisoning the specifications that it inherited, and starting afresh. That means enlisting new people, new assumptions, new item specs and, in time-let's be honest-developing a new NAEP math framework, too. Anything less will be perceived as obliviousness to the growing portion of the American population that today doubts that the direction pointed by NCTM is the way the country ought to go. Anything less is also apt to lead to a decision by Congress to block future movement toward a national test. For all its faults, in matters of curriculum the Congress is not a bad proxy for the consuming public. I don't believe most of the public wants its children to learn fuzzy math. I don't believe the Congress does, either.
One final thought, voiced with more than a little chagrin because this, too, began on my watch. I believe the Board should probably start afresh on reading, also, both for purposes of national tests and for NAEP itself. A new reading framework is indicated, because just about everything I've said about math today could also be said about the existing reading framework (and-with greater vehemence-about the draft national test specifications), substituting "whole language" for "NCTM". But that was this morning's hearing.
Thanks for listening.