A new, unpublished study of CPM effectiveness relative to Traditional methods, using a CPM-biased exam, reveals
The study reports scores for Algebra I students from four high schools -- two of which were exclusively CPM and two of which were part CPM and part Traditional. The pre-test consisted of the MDTP Algebra Readiness test. The report indicates that within each of the two mixed schools there was no difference in pre-test scores between the CPM students and the Traditional students. The pre-test scores from the two schools that were exclusively CPM appear to have been excluded from these comparisons. With the inclusion of the students from the all CPM schools, there appears to be a significant difference between the pool of CPM students and the pool of Non-CPM students in their pre-test scores.
Significantly, the pretest data indicate that a large fraction of the students in the study were unprepared for Algebra. About 65% of students in the CPM group and about 85% of students in the Traditional group scored below the mastery level (70% correct) on the readiness pre-test. No student in the Traditional group scored above 90% correct on the pre-test. The mean for the CPM students was roughly 60% correct, and the mean for the Traditional students was roughly 50% correct. We expect 20% correct by chance alone. These scores are very low. As such, the study seems to best apply to the ability of teachers and programs to teach Algebra to students who should be taking Pre-Algebra. From the data, little can be said regarding the effectiveness of programs for students who were prepared for Algebra at the start of the course.
The post-test consisted of 20 of the available 50 MDTP Elementary Algebra items. Two word problems specifically designed for CPM evaluation were also included in the test. The utility of this 2-item subtest is unknown, and only the data from the MDTP items will be considered. The 20 items from the MDTP included in the criterion measure appear to have been selected in a manner that favors the CPM curriculum via elimination or vast under representation of whole content areas. The Elements which have been removed or de-emphasized are key components of a Traditional Algebra I course which are largely or completely absent from CPM I. The following tables indicate the breakdown of the original MDTP items into subscales as well as what appears to be the distribution of questions in the Modified MDTP used in this study:
MDTP Subscale # Items in original MDTP -------------------------------------------------------------------- Linear & Quadratic Equations (2 Subscales) 13 Arithmetic 6 Geometry 7 Graphs 6 Rational Expressions 5 Exponents and Square Roots 6 Polynomials 7 Modified MDTP Subscale # Items in Modified MDTP -------------------------------------------------------------------- Linear & Quadratic Equations 5 Arithmetic 5 Geopetry 5 Graphs 5
Very few of the students in either program achieved mastery (70% or 14 items correct) on the post-test, even though the selected items for the post-test may be among the easier items from the MDTP Elementary Algebra exam. In fact, the post-test mean appears to be between 35% and 40% correct (again, 20% correct would be expected by chance). Students in both programs appear to be doing even more poorly on the post-test than one would expect from pre-test scores. The data presented lead to the compelling conclusion that if you start with a group of students who are ill-prepared for Algebra, you are likely to end up with a group of students who do poorly on an Algebra exam -- irrespective of the method of instruction.
Even given the biased reformatting of the exam, the fact that most students in the study were unprepared for Algebra, the fact that most students were unsuccessful on the post-test, and questions about equivalence of teachers in the two programs, the difference between CPM and the Traditional program of instruction is not significant.
Interestingly, although this study involved a relatively small number of teachers, it clearly shows that the effect of teacher competence on student performance is very substantial. Because the quality of the teacher makes such a large contribution to outcome scores, it is essential to a well-designed study that teachers be randomly assigned to programs, and that a large sample of teachers be included. Neither of these appears to have been the case in this study. In addition, one doesn't know if there were very strong and very weak teachers in each school, nor does one know the relative distribution of strong and weak teachers between the CPM and Traditional classes. One wonders whether teacher differences are greater for CPM classes than for traditional classes. Unfortunately, the data in this study are not adequate to address this point.
The sources of variance evaluated include pretest, program, teacher and gender, together with their interactions. Certain non-significant interactions have been pooled into the error term. The appropriate analysis for the available data would be a partially nested and partially crossed analysis of covariance. It appears that this may be the form of analysis that was employed.
The teacher effect is very substantial, so which teachers taught which methods is very important. The author notes that this effect is larger than usual. One has to wonder whether teacher differences are greater for CPM classes than for traditional classes. Unfortunately, the data in this study are not adequate to address this point.
Because of the strong teacher differences, the F-ratio for program differences must use the Teacher mean square as the error term, not the within-cell mean square. The correct F-ratio appears to be the second of the two presented tests for program effects, where the difference between programs is not significant.
What the report refers to as the second analysis is a graphical display of estimated curves between pre-test and post-test data without significance tests attached. This provides no evidence for generating inferences.
What the report refers to as the third analysis presents correlations among various scores within groups. The pre-test to post-test correlation for Traditional students is .55, while the correlation for CPM students is .69. According to the table of F-ratios, the program differences in regression are not significant (p=.1029).
The most misleading portion of the report appears in the opening summary: "Three types of analyses were done. Each showed that CPM students outperformed their non-CPM counterparts." With respect to the subset of MDTP items, none of the analyses showed that the CPM program differed significantly from the Traditional program, and only one of the three analyses even addressed this question with inferrential statistics.