The Essential Elements are shown on the left. They are the standards and they came first. The TAAS test specifications are on the right and are supposed to be based on the Essential Elements.
There are two levels of abstraction. A global level applies to all grade levels and consequently doesn't say very much. Beneath that are grade-level specifics.
Translation between Standards and Specifications is not uncommon in designing tests, since more detail is typically needed. In the process, there can be emphasis changes that may be helpful or harmful - in this case, perhaps a little more helpful than harmful.
Test items are actually written to match the grade level details of the TAAS test specifications on the right.
Each white arrow represents a translation process. Thus, even if the global Essential Elements provide a good set of objectives, there are several opportunities for these goals to be lost in translation.
The blue double arrow represents a check point. The TEA has provided a document giving the correlation between the Essential Elements and the TAAS at this level. While this correlation cannot validate the whole process, it allows us to make some judgments based on the TEA's own documents.
The TEA documents on the correlation between Essential Elements and TAAS suggest that two important policy decisions have influenced the development process.
First, each TAAS grade level typically correlates roughly to three grade levels in the Essential Elements - the on-target grade level Elements and those for the prior two years. If we take an average of on-target, one year behind, and two years behind we see the TAAS targets average about a year behind the Essential Elements.
Second, there has evidently been a special treatment allowed for the Exit Exam since Texas students are not required to take algebra. Consequently, the specifications for the Exit Exam are just about the same as for the eighth grade test.
The gap between the bars in the graph and theoretical on-target tests indicated by the white line shows the discrepancy between the TAAS and the Essential Elements.
This is not to say that the Essential Elements set goals that are as high as they should be, but even so, the TAAS design doesn't keep up with them.
Rather than attempt to uncover the full history of the development process, we choose to simply evaluate the end products - the test items themselves.
We used the new California Math Standards as our guide. The new standards have been highly acclaimed as both being clear and specific and being on-track with more successful countries like Singapore and Japan.
Using these standards, we were able to judge the grade level of each TAAS High-School Exit Exam item. The distribution of item grade levels indicates that most fell in the 4th to 6th grade range. The mean grade-level rating was 5.3.
By checking each item for four years of TAAS tests against specific California Standards, it was also easy to tell what 7th grade topics are not covered well in the TAAS Exit Exam.
Here are some item counts found over the four years of tests. Keep in mind that if there aren't four items found, then the topic cannot be tested each year.
Several of the content areas found were never tested, or were only tested by one item in one of the four years.
Note, for example, that no items at all asked about prime numbers.
No items at all asked for absolute values.
Even area and volume were not regularly tested.
Note also that only two items in four years asked for equations to be solved, and they are very simple problems at that.
Here is one of the highest level items from the Exit Exams. We gave this item credit as a question related to the properties of real numbers.
Here is an item that asks for the solution of a pair of simultaneous linear equations.
Of course, the students don't really find the solution, they just read the point of intersection from the graph.
Here is a curious item. Students are supposed to find the area of the shaded region. However, the dimensions of the inner white rectangle are not given.
This item asks for the BEST predition of the next year's expenditures. There is no real way to define what is best here.
Of course, the problem expects students to look for a rough linear trend. However, some students may just as well think that the best prediction is the one with the lowest expenditures - our future business majors.
Here are some more items from the High School Exit Exam.
Remember that these are multiple choice but the response choices are not listed here. This means that the items are even easier than they look here.
Here students estimate how long a pencil is in metric units, and they find the perimeter of a rectangle.
These are useful basic skills, but not what expectations should be for high school students.
Here are the final two examples from the Exit Exam.
Note that the last question asks about change from $20. This is not high school level material.
What the TAAS asks for high school graduation is essentially what should be expected for elementary school graduation - 4th to 6th grade content.
This graph is really just a test of the validity of our methods. If our ratings are correct, than Texas students should get more of the items we see as 3rd or 4th grade items correct than those we see as 5th or 6th grade items. This is the result we find as the graph indicates.
Before, we looked at the discrepancy between on-target grade level testing indicated by the white line and the TAAS Instructional Targets shown by the yellow bars.
But, now we can estimate the grade levels of test items based on the California Standards as indicated in green. Remember that the California Standards are designed to be competitive with Singapore or Japan.
This is just another way to show how seriously low the Texas assessment expectations really are.
Now we move on to the situation in algebra. Texas has an end-of-course algebra examination as well as the grade-level TAAS tests.
We looked at each of the algebra test problems for four years and rated them as low, moderate, or high-difficulty level algebra problems. We added a category for items that were really pre-algebra problems, and even another category below that.
This graph shows the distribution of ratings for the algebra exam items.
Basically, the algebra exams are actually testing a combination of pre-algebra and the lowest difficulty level of algebra content.
Here are some of the limitations that were found by studying the algebra exam items.
Note especially the last of this group - factoring has long been one of the hallmarks of introductory algebra, but it just isn't there.
Here are more limitations to the algebra exam.
Another hallmark of algebra is the use of the Pythagorean Theorem and the related distance formula, but these two are also not needed.
This is the final list of limitations to the algebra exam.
The last bullet here refers to the type of machine-scored tests where students must indicate a numeric response rather than using a multiple-choice format. This is useful in mathematics since it means students can't work backwards from the answer choices. However, the algebra exam makes little use of this technique.
Here we illustrate some of the limitations.
This is part of a chart students are given when they take the exam. This way, they don't even have to know that the area of a rectangle is the length times the width.
This is one way to avoid actually having to factor an expression in algebra.
For those who haven't seen this method of drawing pictures of problems, the light blue annotations have been added to represent how student thinking is supposed to work here.
The item seems to ask students to factor 8x2+10x+3
However, the figure is printed right in the test, so students just count one side as 4x + 3 and the other side as being 2x + 1. That's the answer here. Thus, factoring is reduced to counting.
Here's an example of a word problem, but you can see that the level of difficulty isn't what we expect in algebra.
Multiple choice tests in math must be done carefully to avoid letting students just work backwards from the response choices. Here, for example, student's don't solve this quadratic equation, they simply plug in the answer choices until it works.
The relationship of linear equations to their graphs is an important topic in pre-algebra that gets repeated in algebra. In this case, the item asks for an equation with a slope of 2.
However, the answer choices all have a slope of 2 so students can ignore that part entirely. No knowledge of slope is required.
Worse than that, since x is 1 in the ordered pair and the coefficient for x is always 2 in the response choices, this item really taps the students knowledge of the fact that 2 + 2 = 4.
Here, students are being asked to find where the best fit line through the yellow dots would cross x=16, which is indicated by the light blue line. The solution can be found by a 4th grader.
Here, we review the weaknesses of the exam.
In summary, the expectations for the Texas algebra student are really quite minimal. Students who are competent at only the lowest difficulty levels should still be expected to pass the exam.
Even though the expectations in the algebra exam are at a very low level, we see a problem when we look at the algebra test in relation to the TAAS tests.
Basically, the problem is that the TAAS items are at such a very low level that there is a big jump between the 8th grade criteria and the algebra criteria.
This suggests that students who pass the TAAS exam each year could still be at risk of failing even this low-level algebra exam.
In short, the TAAS may be sending out the wrong message. It may be setting up such low expectations that it indirectly contributes to failure to learn algebra.
A hypothetical case is indicated here. The red line is intended to represent the achievement curve at baseline. The green line represents where a state might like to be after they have raised achievement.
Now, assume that the method used to raise achievement is to set some minimum requirement. This would have to be at a fairly low level, otherwise too many students will fail.
The yellow arrow illustrates where a minimum requirement might be set. Unfortunately, placing this hurdle at such a low level may have an overall negative impact by forcing the whole system to focus on these low achievement levels.
To continue with the hypothetical case, if the system puts too much energy into their low-level objectives, it could end up moving the achievement curve to look more like the light-blue curve here.
Notice that the actual number who fail to meet the low-level hurdle has declined somewhat from the red to the light blue curve. However, failure hasn't gone down as much as we wanted in the green curve. Meanwhile, the focus on low-achievement levels has brought the overall blue curve way down.
We often hear about algebra as a gateway subject since it opens up so many opportunities. We know that students who complete algebra in eighth grade have far greater success rates in higher education and better career chances than those who do not. But, with the low-achievement goal system in place, the system can actually make it more difficult to take advantage of this. At least in this hypothetical model, it becomes even more difficult to use education as a path to opportunity.
One last point about this idea. If the test is too easy it won't even measure the achievement levels on the right hand side of the graph. In this case, the test distributions will be negatively skewed and the decreased achievement in the light blue curve will be hidden.
Now here's the real data. We see over a three year period that the very lowest achievement levels are declining, while the upper end of achievement is obscured because the test is just too easy.
There are two important take-home messages here:
One is that what appears to be a benefit overall may be misleading since part of the story is hidden by using a test that is too easy. Thus, the system may not work to promote greater achievements in algebra.
The other is that while a reduction in the number of students failing an easy test is a good thing, the benefits can only go so far. Passing such an easy test doesn't mean that these students will now have real opportunities.
It may well be that any gains that the TAAS can bring have run their course, and that further achievement gains will require expectations that are more in line with those of our international competitors.
The American Federation of Teachers has recently pointed out that Texans are not the only group that has fallen into the trap of using tests that are too easy.
The AFT feels that this is a problem that plagues the whole country. In their report, the AFT rated test items as easy, medium, and hard for five of the big tests used in the U.S.
As you can see from the graph, there just weren't any hard questions and there were very few moderate questions on any of these tests. While Texas was the weakest of the lot, none of the tests were seen as adequate.
The AFT report supports our findings for the Texas exams. We believe that we need much greater mathematics achievement across America, and that we need tests that will help to drive up achievement. It simply won't do to have an assessment system that can't even measure high achievement levels. The Texas assessment system is essentially blind to anything above modest achievement.
The lesson for Texas is clear. It is time to give up the low expectations for the students of Texas. It is time to start expecting them to achieve more than sixth grade competence to graduate from high school, and it is time for Texas to replace their assessment system with one that can measure higher levels of achievement. Until this happens, the testing system simply cannot promote greater mathematics achievement. Until this happens, the system is failing to open up the algebra gateway. The students of Texas deserve this opportunity.