Testimony on Item and Test Specifications for the
Voluntary National Test in 8th-Grade Math
January 27, 1998
Mary Damer
Instructor: Northern Illinois University
Classroom Behavior Consultant

As a college educator who trains special education teachers, a former principal, and a parent of children in elementary school, I initially welcomed the announcement of a new, rigorous national test which would assess high standards in math and reading. I believed that rigorous high-stakes testing would provide citizen consumers with quality information they have been seeking related to the effectiveness of their local schools. I hoped that improved testing in math and reading would provide objective information that might help stem the decline in math and reading literacy that I have witnessed these past twenty-five years.

After reading and analyzing the test specifications for the proposed Voluntary National Test in Mathematics, I was disappointed to see that the proposed math assessment reflects the NCTM-based math standards which are currently so controversial. Many of my concerns arise because the proposed math test is based on what skeptics are calling "fuzzy math" curriculum, a curriculum which does not yet have any longitudinally-based research support. My own community of St. Charles, Illinois implemented an NCTM-based math program, The University of Chicago, "Everyday Mathematics Program," four years ago. Since then, education consumers living in our community have become so concerned about a perceived "dumbing down" of the math curriculum, that we have tracked achievement test scores for the past three years. Our observation reveals math scores which have steadily decreased. Barrington, one of the first suburbs in Illinois to adopt the same NCTM program, recently returned to a more traditional math curriculum, because after several years their students’ math skills were sinking.

In my capacity as a behavior consultant, I visit several NCTM-based math classrooms each year because disruptive student behavior increases when the students do not have the necessary preschools for assigned math tasks. The students’ frustration leads to increased "acting out" behaviors. In one rural district I visited this year which has had the "Quest" NCTM-based math program for several years, half of the sixth graders did not yet know their multiplication and division facts. The teacher was forced to give all of the students calculators when they were learning how to multiply decimals, because they were unable to compute the numbers independently. I have never observed an NCTM-based math class in which more than one-half (more commonly one-third) of the students have the skills required to complete the assignments. When I read on page 2 of the math specifications booklet, "the VNTM will include items that measure goals that reflect the best thinking about the mathematics 8th-grade students should know and be able to do", and then look at the example selection reflecting NCTM- based curriculum, I question the authors’ rationale for making that claim.

I have conducted extensive library searches and asked other professionals for objective data spanning more than one year which supports this new NCTM-based approach to math learning. To date, I have found none.

I believe that the American public would eagerly embrace a rigorous national test in math - a test that is economically feasible and which maintains the highest standards in regards to internal and external validity and reliability. Unfortunately, this proposed Voluntary National Test in Math does not meet that criteria. Savvy consumer parents and educators will not voluntarily introduce this test into their local school’s testing programs for the following reasons:

1.Because the curriculum which this test is based on is so controversial, consumers will have questions about whether this math test measures what it claims to measure. I find myself unconvinced that this proposed math assessment assesses math achievement. Surveying, the sample items, I can’t help but wonder whether this test with its emphasis on problem solving, assesses math aptitude rather than achievement. The authors of the math test specifications’ booklet state on page 1, "The Voluntary National Test in Mathematics at Grade 8 (NVTM) is intended to provide information to students, their families, their teachers, and their schools about student achievement with respect to challenging National standards." Reading this statement, one would clearly define the proposed NVTM as an achievement test, a test describing what a student has learned to do.

However, a survey of the sample math questions which reflects many of the "gamey" type of problem-solving exercises in NCTM-based math programs, appears to show more problems similar to those on an aptitude test - a test providing information on a student’s cognitive abilities telling the examiner what the person can learn to do.

Some of the proposed test questions directly assess math achievement, such as the one on page 87 asking the student to identify equivalent fractions or the one on page 83 asking for an interpretation of a graph. Other questions such as those listed on pages 91, 98, 113, and 114 possibly reflect a student’s cognitive ability rather than achievement. Educational research has little or no data that links success with such problem-solving test items to math achievement. Years of research will be necessary to make an informed decision about the inclusion of such items. Until we have data linking these complex problem-solving items to math achievement, I would advise prudent educators to take the best math testing utilized by higher performing nations, or to utilize the TIMSS, even though the message it provides is painful.

2. I question the accuracy of many of the assumptions expressed by the authors of the math test specifications report. Some of the assumptions described on page 15 include: an assumption that students know how to compute using whole numbers and that they have developed reasonable fluency in using algorithms, both for multidigit pencil-and paper computations and in mental arithmetic; assumptions that students should have developed a conceptual understanding of fractions, decimals and the use of percent. My own experience sitting in hundreds of middle school classrooms and working with sophomore and junior year college students has revealed that the inability of many American students to perform simple computations and to work with decimals and fractions has rendered them mathematically illiterate and unable to progress into algebra-level math.

A typical college sophomore who is studying to become a teacher and who needs to learn to compute rate per minute and various types of percentages in my college classroom functions at a fifth or sixth grade level in mathematics. The typical college student I see does not automatically know how to round decimals, how to fluently figure out equivalent fractions, how to compute percentages in new situations, how to reduce fractions, how to multiply or divide fractions, how to solve ratio and proportion problems. The typical college student has not committed to memory formulae or measurement equivalencies. I have watched math teachers in middle school classrooms trying to teach NCTM-based math lessons make errors when they had to tell whether a fraction equivalent was correct or incorrect. I have seen them pause in the middle of a computation problem until a student shouted out the correct math fact that they had forgotten. I have observed entire fifth and sixth grade students adding and subtracting on their fingers when calculators weren’t present. The authors of this test specifications manual need to include their data support for making these assumptions about the current average math skills of eighth graders. They do not reflect current ability levels of students in math classrooms throughout Illinois. Because calculators will be allowed in this assessment, gross deficiencies in math calculation and basic operational skills may remain hidden.

3. I believe that savvy education consumers would quickly embrace a rigorous math testing program which maintained high standards, which demonstrated valid and reliable information, and which provided higher quality consumer information than is now available from standardized achievement tests. This test will not provide parents or educators with enough information for intervention if their child does not meet the established standards. Parents and educators not only want criterion- referenced interpretations telling what competencies in reading their child has developed, but also norm referenced interpretation comparing each student to the class, the school, the nation, and to similar communities in the nation. Education consumers want:

• data that will help them judge what degree of confidence is to be place on the information provided by the test and what limitations are inherent in that information.

• normative data that will help them evaluate the schools in which they or their children are involved.

• information about the amth achievement of representative groups of 4th graders in communities with similar demographics

• information about the math achievement of representative groups of 4th graders in private schools

• documentation that the Lake Wobegon effect has been avoided

• documented proof of the content validity of this math assessment

• documented proof of the reliability of this math assessment and information about how the reliability coefficients will be determined. Descriptive and statistical characteristics of each group on which reliability data are based should be provided.

• Standard error of measurement established at each of a number of score levels in order to know over what range of scores the test maintains its accuracy.

• Evidence on the relationships of the test to other factors. The test specifications that have been provided do not provide objective information about how the test will provide for any of those factors. No data is provided to support the rationale for the sample test items selected as examples. How will the testing be conducted to link individual student performance on this assessment to the TIMSS?

4. This math assessment will be classified as a "high-stakes" assessment, since students will only have one opportunity to take the test which covers a large domain of material. The test specifications booklet informs the reader that approximately 20% of the test will involve non-machine scored items which will take up 50 % of the students’ testing time on the reading assessment. This decision to include performance assessment has no valid research support. Although performance assessment has become an educational fad of the 1990’s, it has the following disadvantages :

• Performance Assessments include fewer questions and thus sample a smaller portion of the whole domain than a standardized test which takes the same amount of time to complete. This lack of adequate coverage can lead to a distortion of instruction and abnormally high scores. When fewer tasks are included on the performance assessment, the accuracy and fairness of the test is reduced. Because 50% of the students’ time will be spent answering the performance assessment questions, the number of questions asked will be substantially reduced.

• Performance Assessments are significantly more expensive than standardized tests. Estimates range from their being 5 times more expensive to 30 times more expensive. Performance Assessments require trained scorers who spend large amounts of time rating students’ work. In this high tech age of computerization, test designers should write questions with answers that can be computer scored. Well designed multiple choice items will measure students’ ability to analyze, synthesize information, make comparisons, draw inferences, and evaluate performance.

• Performance Assessments are not time-effective. A superintendent or principal who is deciding what assessments to administer as part of the school district testing program is dealing with increasingly reduced instructional time in typical classrooms.

  My student teachers are fortunate when they are able to teach reading for four days in any week. The average number of days in which they can present instruction in any subject area is closer to 3.5 days. School assemblies, teacher inservices, special unit activities and celebrations, state testing weeks, achievement testing weeks, field trips, holidays, special parties and activities, DARE programs, all interfere with regular instruction. Because of the time constraints, any testing program needs to deliver valuable information for the time allotted to the testing. Because 50% of the students’ time taking is spent answering performance assessment questions, the student is required to make fewer responses and decisions, thus providing less feedback

• Performance Assessments introduce more bias because they call for a greater degree of subjective judgment than traditional testing methods. Individuals grading the students’ work may differ greatly in their evaluation no matter how detailed the scoring rubric, leading to increased bias. In England when the assessments that make up the General Certificate of Secondary Education were changed to put more emphasis on performance tasks and less on written answers, the gaps between the average scores of ethnic groups increased rather than decreased. Studies have shown that when highly trained scorers assign numbers from a performance assessment rubric, their reliability decreases after a weekend break.

5. The specifications report for the Voluntary National Test in Math reveals the misleading results that can arise with performance assessments. A look at the scoring rubric on page 92 of the math specifications report, exemplifies a sample rubric directing the scorer to give points to a student for trying. The student receives one point simply for attempting to recognize the common factors. If the student provides the correct response, the student receives two points. In order to receive full credit (three points), the student must also provide an explanation for why the answer is correct. In addition to mathematically solving the problem, the student must also exhibit language skills (usually in writing) to receive full credit. The test is no longer assessing only achievement in math. Full credit for sample questions on pages 114 - 120 requires written descriptions accompanying the answer .

Any child who has poor fine motor skills, who is a slow writer, who is a slow or inadequate speller, or who has indecipherable/sloppy writing might receive a lower score on the math assessment. This lower score on the National reading test will reflect deficiencies in writing rather than in math. School officials and parents analyzing the student’s score will be unable to ascertain whether the lower performance is a result of writing ability or of math skills.

One might claim that a test assessing both math and writing would be a more rigorous assessment instrument. Because the test rubrics do not require the student to write responses using grammatical conventions or correct spelling, only lower level writing skills will effect a student’s total score.

6. Districts that have chosen not to implement NCTM-based math curriculum as well as districts that cannot afford to purchase the new NCTM-based math curriculum and retrain their teachers will probably choose not to participate in the Voluntary National Test in Mathematics. The mismatch in content between this test and other math curricula, whether Saxon math or the older traditional math programs, will limit the usefulness of this test’s data.

7. During a discussion with an educator who has developed achievement test questions for a popular achievement test , I was surprised to learn recently that today’s achievement test questions usually include the formula required to answer a related question. The educator was not sure when the practice of providing necessary formulae began, but indicated that test companies started doing this sometime during the past decade. I was disappointed to see this same practice, which represents curriculum "dumbing-down," reflected in the specifications for the Voluntary National Test in Mathematics.

8. Bias will be introduced into the Voluntary National Test in Mathematics by allowing optional calculator use. Students who have access to calculators because their school district will allow them or because they can afford them will conceivably show higher achievement on this assessment. How will standardization be established for this test when the conditions under which it is given are not the same? How will valid predictors for TIMSS scores be established when the test conditions have not been the same?

Research does not yet substantiate justification for using calculators in the early grades or with basic computational problems. Students lacking basic computational skills could conceivably attain the same achievement scores as students who have them, because calculators are allowed for the entire test. In Illinois, proposed new mathematics performance assessments ("Performance Assessment in Mathematics: Approaches to Open-Ended Problems" 1995, Illinois State Board of Education) show students in third grade who need to draw pictures in order to correctly add, earning the same or better scores than children who can add fluently.