QED Research Reviews

Mathematically Correct thanks QED and Robert C. Noël for permission to reproduce these reports.

The University of Wisconsin's Center on Organization and Restructuring of Schools promotes what it calls "authentic" education, by which it means the "... construction of knowledge, through disciplined inquiry to produce discourse, products and performances that have meaning beyond success in school."

In the Center's view, the "... conventional curriculum excessively emphasizes reproducing [prior] knowledge ... rarely helps students develop in-depth understanding," and emphasizes student tasks having little meaning beyond "simply showing the teacher, the parent, a college, or an employer that the student has mastered the requirements of schooling." (Newmann and Wehlage, pp. 8-11)

These researchers employ the term "authentic" in a number of specialized ways. They claim to have shown, among other things, that "authentic" achievement in mathematics is greater in schools and classes which employ "authentic pedagogy."

1. The study did not use established achievement tests to assess student performance in mathematics. Rather, these researchers attempted to measure what they call "authentic student performance" by designing their own unique assessment methods, which did not systematically assess computational and algorithmic skills.

• No data pertaining to the validity of their measurement scale was reported.
• Information about the reliability of their measure of "authentic student performance" was reported. Each student's work was graded independently by two researchers. Over all student work in the study, graders were in agreement only about half of the time (54%).

2. Students in the study were not all studying math according to the same curriculum. There was no way of knowing whether they studied the same math topics or whether they spent the same amount of time on any given topic. This left open the possibility that differences in math achievement were due more to differences in course content than to the claimed effectiveness of "authentic instructional practices."

3. The researchers graded work that was done by students on different problems assigned by different teachers at different schools instead of giving all students the same assessment tests. This left open the possibility that differences in math achievement were due more to differences in the achievement tests themselves than to the claimed effectiveness of "authentic instructional practices." This possibility was examined statistically. It was found that:

• "authentic instructional practices" had relatively little effect on "authentic student performance" in mathematics, and
• "authentic assessment tasks" alone had major effects on "authentic student performance." In the researchers' own words: "what you test is what you get." (Newmann, Marks and Gamoran, footnote 19).

Thus, the main relationship hypothesized in the study -- that "authentic pedagogy" significantly increased student achievement -- was obscured by student reactions to the methods used to measure math achievement.

I. The main claim of this study -- that so-called "authentic pedagogy" contributed to higher levels of so-called "authentic achievement" in mathematics -- is unwarranted on the basis of the evidence presented. Ostensibly higher levels of achievement were, in fact, explained by the nature of the achievement tests themselves. This not withstanding, the researchers ventured the prediction that:

• "regardless of social background, an average student would increase from about the 30th percentile to about the 60th percentile as a result of experiencing high versus low authentic pedagogy. " (Newmann and Wehlage, p. 22)

Even if the claimed effects of "authentic pedagogy" had been established, which it was not, this prediction about an hypothetical average student is not as it seems.

• Math achievement was measured using an esoteric "authentic achievement" scale of proven unreliability and unproven validity. Standardized tests were not used.
• In terms of actual scores, the putative increase was from 5.4 to 6.8 on an "authentic achievement" scale ranging from 3 to 12. In percentile terms it appears larger.
• Given the nature of the researchers' "authentic achievement" scale, the increase might well fall within the margin of error of the measure.

II. Even if the study's claims were scientifically warranted, which they are not, they may not be directly relevant to the use of Mathland in Santa Barbara schools. For the study to be applicable, local instructional practices with Mathland would have to match the practices these researchers defined as "authentic." However, the specific classroom practices that were deemed to be "authentic pedagogy" were not listed in the research reports.

This study was conducted in elementary schools serving high concentrations of children from low-income and ethnically diverse populations in Washington, D. C., Cincinnati and San Francisco.

Among other things, the researchers examined the relationship between different approaches to teaching mathematics at the elementary level and student learning. The researchers distinguished between two bundles of instructional practices: "meaning-oriented instruction" and "skills-oriented instruction." They used standardized testing to measure student achievement in math.

The researchers claim that students who receive "meaning-oriented instruction" have significantly higher mathematics achievement scores than do students receiving "skills-oriented instruction."

1. This study did not demonstrate that so-called "meaning-oriented" math instruction is more effective than instruction which emphasizes arithmetic computational skills, because, according to the researchers:

• "... most meaning-oriented teachers, in effect, hedged their bets by including a good deal of arithmetic computation drill. " "Orienting instruction toward meaning thus did not preclude an emphasis on discrete skills." (Knapp et all, pp. 143-144)

An unambiguous comparison of the two instructional approaches would have required classes which made exclusive use of one or the other.

2. The classroom practices labeled "meaning-oriented" instruction in mathematics are not identical to the classroom practices associated with Mathland.

• With both "meaning-oriented" instruction and Mathland, there is a strong emphasis on learning concepts and learning to think; and there is a wide variety of mathematical content, including statistics and probability, geometry, measurement and patterns and sequences. In both, teachers frame problems with more than one solution, discuss alternative solutions and make frequent use of manipulatives, calculators and cooperative learning groups.
• However, key elements of Mathland's constructivist approach were not included among listed "meaning-oriented" practices, especially Mathland's deemphasis of computational practice, its deemphasis of homework, the absence of a student textbook, and Mathland's emphasis on discovery learning in which teachers intentionally withhold problem closure.

Thus, it does not follow that findings from "meaning-oriented" classrooms apply to Mathland classrooms.

3. It is not clear if the study's findings apply across the full array of real world teachers, because, according to the researchers:

• they "... excluded teachers who were apparently less capable, experienced, or committed to their profession." (Knapp et al, pp. 143-144)

4. Nor is it clear that the study's findings apply outside of high-poverty schools. As its title suggests, the study was about high-poverty schools, most of which were located in inner cities.

• The researchers make no claim that their findings apply to student populations characterized by low student turn-over rates, supportive and tranquil home and community environments, well nourished and healthy children, two-parent families and fluency in English.

I. The following finding appears to be warranted by this study.

• In high-poverty schools under the tutelage of less capable, committed or experienced teachers, students in classes which combined an emphasis on conceptual understanding and a broad range of math topics with an emphasis on arithmetic computational skills scored higher on standardized math achievement tests than did students in classes which emphasized arithmetic computational skills alone.

II. Significant differences between the conditions and definitions of this study and conditions associated with the use of Mathland in Santa Barbara elementary schools mean that this study does not provide scientific warrant for the use of Mathland.

III. If this study is to be interpreted as suggestive, though inconclusive, it suggests that gains in mathematics learning may be possible through combinations of the new math approach and the more traditional approach. It is not a question of either/or -- it is both.

The main focus of Lee, Smith and Croninger's study was on the effects of changes in the way high schools are organized on levels of student achievement.

A small part of their study looked the effects of high school instructional practices on student achievement in mathematics. They claim that students in high schools which make more widespread use of specific instructional practices they call "authentic" score higher on standardized tests of mathematics achievement.

1. This study does not provide support for the claim that instructional practices called "authentic" produce higher student achievement scores in math. The effects of "authentic" practices were not isolated in the research; both traditional and "authentic" practices "... occur together in classes." These items themselves are not mutually exclusive. (Lee, Smith and Croninger, Appendix A, p. 49)

• So-called "authentic instruction" included such items as: use of calculators, students working in small groups, explaining math work orally, student-led group discussions, oral reports, use of books other than math textbook, use of manipulatives and use of computers (in descending order of usage).
• Traditional instruction included such items as: reviewing student work, responding to questions, copying teachers' notes from board, completing individual assignments and doing story problems or problem-solving activities (in descending order of usage). (Lee, Smith and Croninger, Appendix A, p. 49-50)
• In the schools studied, "authentic practices" were: a) less commonly used and than traditional practices, and then b) used in addition to not as a substitute for traditional practices. (Lee, Smith and Croninger, Appendix A, p. 49-50)

2. For similar reasons, neither does this study offer support for the efficacy of "constructivist learning," the educational theory underlying Mathland. Key elements of Mathland's constructivist approach were not included among the "authentic practices," especially Mathland's deemphasis of computational practice, its deemphasis of homework, the absence of a student textbook, and Mathland's emphasis on discovery learning with the teacher as facilitator not disseminator.

3. Nor does this study support the belief that math achievement scores for students taught with Mathland will be higher than for students taught with traditional instructional practices. Listed "authentic instruction" practices only partially match the Mathland approach. Therefore, their findings may not be extrapolated to Mathland classes or schools.

4. Any otherwise valid findings of this study are restricted to high schools, where math and science are taught by specialists. They may not be generalized to elementary schools where math and science are taught by generalists who must teach all subjects.

5. In speaking about learning gains for a hypothetical "average student," the researchers failed to report the margin of error surrounding any such predictions from their model. Neither the standard error nor the proportion of unexplained variance is included in the reports.

I. The following finding appears to be warranted by this study.

• Students in high schools where the mix of instructional practices in mathematics emphasized "authentic" practices (as defined) scored higher on standardized math achievement tests than did students in high schools where the mix of instructional practices emphasized "traditional" practices (as defined). [NB: Traditional instructional practices were used in some combination with "authentic" practices in all schools in the sample.]

However, with certain statistics missing from the report, it is not possible to evaluate the claim that:

• ... an average student who attended a 'high authentic instruction' school would learn about 78 percent more mathematics between grades 8 and 10 than a comparable student in a 'low authentic instruction' school..." (Newmann and Wehlage, p. 25)

On the surface, "78 percent more" is impressive. But it is also deceptive. Seventy- eight percent of what?

• In the authors words, "...it is evident that students don't gain very much..." either between grades 8 and 10 or between grades 11 and 12. (Lee, Smith and Croninger, p.20 , emphasis added). So, it is 78 percent of 'not very much.'
• Moreover, gains in mathematics learning fall off appreciably between grades 11 and 12. (Lee, Smith and Croninger, p.56 figure 1). So, it becomes 78 percent of even less than 'not very much'.
• Worse, more course taking apparently does not lead to more learning. (Lee, Smith and Croninger, p. 22) At schools where students averaged more math courses, achievement scores were no better between grades 11 and 12 than were scores at schools where students averaged fewer math courses.

This kind of information does nothing to allay the fears of college-bound families that their children are not learning enough in school. These predominantly middle class families can only view with alarm a major, if uncelebrated, finding of the study.

• The decline in math learning between grades 11 and 12 is greater for students of higher socioeconomic status, (Lee, Smith and Croninger, p.57, Table 2A) despite the fact that the math achievement tests were "tailored" to raise the scores of many students of lower socioeconomic status. (Lee, Smith and Croninger, Note 10, p. 32, the "tailoring" of tests was done for students whose math scores at grade 8 were relatively low. The relationship between lower scores and lower socioeconomic status is well known.)

II. The incomparables between the use of Mathland in elementary schools and what these researchers call "authentic" instructional practices in high schools deny scientific warrant for the use of Mathland.

III. If this study is to be interpreted as suggestive though inconclusive, it suggests that gains in mathematics learning may be possible through combinations of the new math approach and the more traditional approach. It is not a question of either/or -- it is both.