Note that this attitude is in direct contrast to that shown by a different math series (Saxon Math) that is in nearly every way the anti-CPM. Saxon Publishers actually offers free books to any school first establishing a Saxon program on the condition that the school conduct a carefully designed test of the success of Saxon students vs. students in school's current program, based on ability matched classes, using a pre-test/post-test method, with one of three well characterized, time tested exams as a measure of performance. Saxon explicitly asks for the results of such tests, no matter what the outcome. (This example is used for comparison only, it is not intended as an endorsement of Saxon math).
As another example, the November, 1995, issue of Consumer Reports in Education from Effective School Practices (PO Box 10252, Eugene, OR 97440), presents the results of four studies of the Connecting Mathematics Concepts program. These studies attend to pre-existing differences among students, and employ well-known outcome measures. With a large student base, one has to wonder why CPM has not reported studies of this nature. (This example is used for comparision only, it is not intended as an endorsement of Connecting Mathematics Concepts).
Multiple sections in the supplement to the CPM newsletter contain over-stated summaries of the conclusions from various CPM assessments. The nature of these studies, and their weaknesses are covered in some detail so you can be ready to counter them when they are presented. It is interesting to note that none of the data presented below or summarized in these packets have been published in reputable, refereed journals.
1) CSU/UC Math Diagnostic Testing Program (MDTP)
In 1993 or Spring 1994, eight schools in which some students were taking CPM Math 1 and some were taking traditional Algebra I had their students take MDTP exams for either algebra (3 schools) or geometry (5 schools). Average scores for the two groups from each school are compared. There are no pretest data and no information about how students were assigned to classes (i.e. we do not know from this presentation if "bright" students were unevenly assigned to one or the other group). This is a serious consideration.
In Standley Middle School in San Diego the better math students were all assigned to CPM 1 and only less adept math students, including those who flunked CPM 1 the year before, were assigned to traditional algebra. Lack of a pre-test/post-test design makes it impossible to judge how students progressed with CPM relative to traditional instruction.
Finally, there are no data about how the study was controlled for teacher motivation and commitment. This is especially important as CPM teachers, especially early in the program, are selected for their desire to be in this new program and undergo special training before and during the year, while the traditional teachers are likely to be those who are leftover and unencouraged.
The results of the eight comparisons show five with no significant differences between CPM and traditional. Three tests (1 algebra and 2 geometry) show significant (p<0.01) or highly significant (p<0.001) differences in favor of CPM. Interestingly, each of the highly significant scores come from schools with a small number of CPM students relative to traditional students (58:207 and 59:310), raising serious questions about student assignment, as discussed above.
Interestingly, between the Fall of 1994 and the Fall of 1995, no new data of this type were added to the pool. Since this is the only generally recognized test of math readiness or math performance used by CPM, lack of continued testing, or lack of publicity concerning such testing, may be highly suggestive.
It should also be noted that the means for both programs in all schools are substantially below the mastery (aka, minimally acceptable competency) level of 70% correct, with results ranging between 35% and 56% correct. Thus, the majority of students in both CPM and Traditional programs did very poorly on the exams.
2) SAT Exams
Students completing "Geometry" (either CPM Math 2 or traditional) were administered a PORTION of an SAT exam in 1992. Approximately 150 CPM students and approximately 75 non-CPM students from each of three Sacramento area schools took the test, for a total of 600. (These are the numbers in the newsletter and come from CPM. They don't add up). In each school, CPM teachers chosen for the study selected/recruited teachers of non-CPM classes for use of their students in the study. All of the weaknesses of experimental design discussed above continue to apply. The policy of having CPM teachers select the control group may lead to an even greater opportunity for bias. The final results from this study, as summarized by CPM without the presentation of data, suggest no difference between traditional and CPM instruction. Teachers have reported serious concerns regarding the biased selection techniques use in CPM assessments.
3) CPM End of the Year Assessment
Questions designed by CPM for CPM students were given to 2,500 students (65% CPM) at the end of the 1993 school year. Each student took only 3 questions out of a 19 question exam. All tests were graded CPM-style. Each CPM group was composed of the students of a teacher who VOLUNTEERED to be included in the test. Each CPM teacher then recruited a colleague who was teaching a non-CPM class to supply the non-CPM students. In addition to all of the problems of student assignment to classes, the sources of pro-CPM bias are so large and of such magnitude as to make it unsurprising that CPM students did better than non-CPM students. Added testing from 1992 and 1994 brings the number of students taking either two or three questions of the tests to 13,000. Teachers have reported serious concerns regarding the biased selection techniques use in CPM assessments. Some reports indicate that they may actually consider such studies as evidence to recommend against the use of CPM texts.
4) Golden State Exams
These exams are not designed for program or school assessment, do not have systematic reporting of results (each teacher or school must report scores independently to CPM authorities, leading to an inherent reporting bias), cover results from less than 30% of students and are based on an exam that was redesigned in 1993 to be more in line with the California Math Framework (e.g. to be more CPM-friendly). Furthermore, participation rates vary from school to school. It is reasonable to expect that ability levels vary as well. The data are too incomplete to judge relative performance of CPM and non-CPM students, but we have seen data from our own sources suggesting that CPM students do not substantially worse, relatively, on this test even though they do score substantially worse on more traditional tests of algebra competence.
Summary
Although the existence of these four sets of test data may sound impressive at first, the bottom line is that they do not provide evidence for the effectiveness of CPM. Given the importance of valid assessments to the CPM cause, one has to wonder why years have gone by without the appearance of such data. It is significant to note that at least one school district has opted against CPM after (and apparently on the basis of) their review of the "evidence" provided by the authors.