Revised Conceptual Understanding Weighting System (CUWS)
Check 1 (Skills): Does this task involve only computational/algebraic skills or memorized facts with no understanding of the concept required? Yes - assign a weight of 0 (Level 0) to the task and skip the remaining checks; No - continue to the next check.
Check 2 (Method): Could this task be answered completely using a method that the student or teacher might have developed in prior course work? Yes - assign a weight of 1 (Level 1) to the task and skip the remaining checks; No - continue to the next check.
Check 3 (Conceptual Understanding): Does the task involve any of the following? No - assign a weight of 2 (Level 2 - conceptual understanding) to the task; Yes - assign a weight of 3 (Level 3 - reasoning from conceptual understanding) to the task.
· interpreting the meaning of a mathematical characteristic in a novel setting
· making connections between different representations (numerical, graphical, symbolic or narrative) of a mathematical characteristic
· evaluating mathematical statements and providing relevant examples or counterexamples as appropriate
· analyzing sample work on a task and identifying flaws with accompanying reasoning
· proving or justifying
Background:
While considering how well the exam questions that we had written assessed our students' conceptual understanding, we realized that we were asking ourselves a series of questions about each problem. We decided to try to formalize this questioning process so that we would then have a tool for evaluating the extent to which conceptual understanding is being assessed by a given problem in our course. That tool could then be used to weight the scoring of exam problems, giving us scores for our students (separate from their exam scores) that indicate more specifically their level of conceptual understanding on that exam. In addition, this tool also gives us the ability to track our students' proficiency at specific types of mathematical understanding.
To begin with, we recognized that in every math course there are computational and/or algebraic skills needed for the work in that course. We want our students to become proficient with these skills, so we put some problems on exams that assess these skills. These skills problems, however, do not require any conceptual understanding so our weighting system should assign them a weight of zero. We then began to realize that on some of the problems that we had specifically written to test students' conceptual understanding, students were in fact more likely answering the problem by adapting methods that they had developed previously in class work. Being able to adapt a method in this way demonstrates some conceptual understanding but not as much as when a student must reason directly from the meaning of a concept. Our weighting system should assign these types of problems a weight of one. If no previously developed method can be applied to solve the problem, we assume that students must be reasoning from concepts in order to answer that question. Our weighting system should assign a weight of at least two to these types of problems. If in addition students were required to interpret the meaning of a mathematical characteristic in a novel applied setting, and/or the students were required to make connections between different representations (numerical, graphical, symbolic or narrative) of a mathematical characteristic then students are demonstrating an even deeper understanding of concepts when successfully answering that question. To reflect this, these types of problems should receive a weight of three. This Conceptual Understanding Weighting System (CUWS) weights each problem (weights ranging from 0 to 3) according to the extent to which the problem assesses students' conceptual understanding - depending upon both the characteristics of the problem itself and how a student has previously encountered (in this course) the concept/s being tested by the problem.
The original CUWS was revised in 2015 and the revision is detailed in the paper Conceptual Understanding Weighting System: A Targeted Assessment Tool. The revision is informed by the five task types that Swan (Swan, 2008) identifies as promoting conceptual understanding mentioned above as well as the eight task types described by Pointon and Sangwin (Pointon & Sangwin, 2003). However, all of the task types mentioned in Swan (Swan, 2008)and Pointon and Sangwin (Pointon & Sangwin, 2003) do not immediately translate to a bullet in Check 3 as some tasks fall under the Level 0, Level 1 or Level 2 categorization. For instance Pointon and Sangwin Type 1 corresponds to Level 0 and Type 2 corresponds to Level 1. The way in which Pointon and Sangwin’s Types 3-8 (Pointon & Sangwin, 2003)compare to the CUWS Levels depends on what the students have encountered in classwork before they approach that task. For instance, a task that asks students to prove could be a Level 0 (a proof that they have already done in class and have memorized), Level 1 (a type of proof, for example induction, that they have practiced many times and are being asked to complete in a slightly altered setting) or now Level 3 (a novel proof) due to the new revisions. One of the task types described by Swan (Swan, 2008), interpreting different representations, was already considered in Check 3 but we added bullets to represent two more: evaluating mathematical statements and analyzing sample work. A bullet was not added to Check 3 for classifying mathematical objects. Therefore this type of task will either be classified as Level 0 or 1 if it is a familiar situation and object for the students, or as Level 2 if it is not something they can complete from memory or by applying an existing method. It is possible that such a task could become a Level 3 task if it also requires students to engage in one of the checks in Level 3, for instance making connections between different representations. Differentiating two levels of conceptual understanding (Level 2 and Level 3) is consistent with the idea of knowledge quality discussed in Star and Stylianides (Star & Stylianides, 2013) where it is highlighted that students understanding of concepts could range from superficial to deep. The authors believe these are appropriate weights for such a task and this relative ranking is also reflected in Pointon and Sangwin 2003 which places “Classify some mathematical object” as a Type 3 task just above factual recall and routine calculations and algorithms. The other task type from Swan which is not included in Check 3 is “Creating Problems” as this seemed better suited as a developmental task for conceptual understanding. We therefore worked with a subset of the five tasks (interpreting multiple representations, evaluating mathematical statements, analyzing reasoning and solutions) that we felt would appropriately require students to demonstrate an ability to reason from conceptual understanding after that understanding had been developed.
More generally, Wiggins and McTighe (Wiggins & McTighe, 2001) “have developed a multifaceted view of what makes up a mature understanding, a six-sided view of the concept”. The six facets that they mention are: can explain, can interpret, can apply, have perspective, can empathize and have self-knowledge. These facets are seen to varying degrees in each of the characteristics for reasoning from conceptual understanding highlighted in Check 3. For instance, “interpreting the meaning of a mathematical characteristic in a novel setting” would involve can explain, can interpret and can apply and “analyzing sample work on a task and identifying flaws with accompanying reasoning” could conceivably involve all six facets.
Based on the literature on demonstrating conceptual understanding and ranking understanding discussed above the authors determined that the five points in Check 3 represent the main attributes of a task that requires deep conceptual understanding. This coincides with the experience of the authors in their classroom testing. However, future work is planned to expand the content to which the CUWS is applied and investigate any additional task types that would demonstrate deep understanding.
To see examples of how we have used the CUWS see Pages 10 - 16 of the paper attached below. This paper has been published in a slightly different form in the journal Transformative Dialogues and is available here.