Student Learning Retention Experiments:
A method for assessing the learning and skills that students "carry forward" into subsequent classes in the same discipline or "carry away" into other disciplines.

Background:
This method was developed to meet the needs of faculty in two NSF consortia (ChemLinks and ModularChem) for the reform of undergraduate chemistry (thirty institutions overall). These faculty have developed; which explore fundamental areas of chemistry by focusing on real-world questions, and use a variety of pedagogical styles to emphasize active learning. Module developers and adapters want to know about the efficacy of their work, including answers to the question, "What knowledge and skills do students carry forward from this class into subsequent (chemistry) classes; or carry away into other disciplines?; The method can be used in any discipline, but may be particularly relevant in the following situations:

1. where it is important for students to build their understanding and skills in a sequential way
2. where "client" disciplines are depending on members of your department to engender particular knowledge and skills in their future students
3. where science faculty wish to investigate the knowledge and skills that non-science majors carry away from their courses.

Overall objectives:
Ongoing development and field-testing of a naturalistic assessment method that achieves the following:

1. offers a way to measure the extent of vertical and lateral transfer and integration of student knowledge within and across disciplines
2. creates a dialogue about learning objectives among innovative faculty--whether in the same or in different departments--and thereby acts as a catalyst for educational change.
3. gives feedback on a dimension of learning of agreed importance to faculty innovators, their funders, and all interested colleagues
4. allows refinement and continued use as a matter of good professional practice beyond the ending of any funded initiative and its formal evaluation

Student Learning Retention:
Ensuring that students carry knowledge and skills forward with them from earlier classes into subsequent classes--whether in the same discipline or others--is a commonly-agreed-upon professional aim. However, it is rarely made explicit or evaluated. Indeed, on many campuses, the ongoing professional conversations necessary for its achievement are not a normal feature of the institutional structure and culture, whether within or among departments. From our ongoing interviews with faculty (as part of our evaluation process), it is clear that many of those who have developed and/or taught innovative classes are looking more critically at the fit between their learning objectives, their teaching, and their assessment practices than they did at the outset. Some are interested in collaborating in the development of campus-specific experiments in student learning retention, both for the purposes of evaluating their collective reform efforts, and for their own longer-term professional use. The last three years (1997-2000) of the ChemLinks and ModularChem initiatives offer a unique opportunity to design (and to refine in working practice) several different models of the experiment that can be adapted by others.

Experimental Structure:
During the 1997-98 academic year, we are setting up a number of experiments at participating institutions of different types. Their design will vary according to the particular configuration of classes in each department. However, the central features of the design will be constant. The following description assumes a single pair of experimenters. However, the method works equally well with small groups of "GIVER" and "RECEIVER" faculty. A GIVER teacher solicits the active collaboration of a colleague who will RECEIVE students from one of the GIVER'S classes. The GIVER and RECEIVER will begin to meet before the start of the GIVER class, and meet as many times as are necessary to negotiate each of the issues listed below. Meetings should continue on an as-needed basis in order to resolve any difficulties and make any adjustments to the design:

1. Agree upon the GIVER and RECEIVER classes or sections to be used in the experiment
2. Solicit from each other, discuss, and agree upon a small number of specific learning goals that the GIVER teacher wishes to see the students carry forward into the subsequent class, and that the RECEIVER teacher wishes to receive. These may include knowledge, understanding, skills, approaches to learning, attitudes, etc. NOTE: In initial trials of the method, it is recommended that the number of items (learning goals) to be jointly tested should be kept to around six. Also, field testers report that it is easier to develop tests that explore understanding rather than skills.
3. Agree upon the timing of the tests in the GIVER class. For each agreed-upon item, the assessments should be set within the normal class assessment pattern. NOTE: This will be easier in the GIVER class. As discussed below, the RECEIVER teacher may have to give the tests for learning retention outside of the normal testing pattern for the class.
4. Clarify each point in time in the RECEIVER class when the teacher will begin to draw upon each item of student learning. Testing should be undertaken just after the RECEIVER teacher begins to draw upon the students' presumed understanding of each item in class. By testing just after this point, faculty are more apt to measure understanding rather than mere recall. NOTE: Testing for all agreed items at the start of the RECEIVER class is less effective because this will advantage students who have recently completed the GIVER class, compared with those who took it some time earlier. Field-testers have suggested that the length of time between the giver and receiver classes may prove to be a critical variable. Because students begin to remember what they know about any matter once they begin to work with it again, it is better for the RECEIVER teacher not to test items before they begin to refer to them in class.
5. Develop student assessment instruments - tests - for both the GIVER and RECEIVER classes. Both parties must agree that each other's tests are fair and appropriate for students at these class levels. Some adjustment of questions may be needed until both/all faculty are satisfied. NOTE: Field-testers report that this is the most difficult aspect of the method. Both sets of assessments need to be agreed upon before students take the first test in the GIVER class. Some faculty who are accustomed to setting their tests, exams, etc., close to the time that the students take their tests find it difficult to agree upon the form of these assessments before their own class begins. Adjustments in the detail of both sets of assessments can be made to accommodate changes in actual classroom activities after classes have begun, so long as changes are discussed and agreed upon by the experimenters.
6. Develop criteria by which to grade student performance in both sets of assessments.
7. Agree on grading and on grade comparison and adjustment methods that will ensure consistency in application of the grading criteria in both sets of assessments. NOTE: In an optimal procedure, the experimenters would agree to grade all of a particular set of questions in BOTH the GIVER and RECEIVER class assessments. This would allow for the greatest consistency of a single problem across classes. Moreover, the total amount of grading required would be the same as if each graded only their own assessments. This system works for groups of GIVERS and RECEIVERS as well as pairs. An slightly less effective procedure would be for each person to grade their own student assignments, but pass a randomly-selected 10% of the papers to their partner for regrading. The GIVER and RECEIVER can then compare grade levels and the quality of answers after grading. On the regraded papers, the GIVER and RECEIVER should agree on the scores for at least 90% of the shared items. Where a group of faculty are GIVERS and/or RECEIVERS, a randomly-selected 10% of each person's set of papers could be passed to each group member for consistency checks. If the consistency is below 90% agreement, then the scoring procedure should be revised and all the papers should be rescored (at least for items that show large numbers of disagreement).
8. Agree upon a format for the comparison of the student performance data from the two classes. NOTE: A draft template for data entry is currently being field-tested and is available either on the three web sites, or direct from Joshua Gutwill (gutwill@socrates.berkeley.edu)
9. Decide whether the intention to test for learning retention will be explained and discussed at the outset with students in the GIVER and RECEIVER classes. NOTE: One group of experimenters have found it useful to do this, particularly in the RECEIVER class where the results of the tests were used as the means to review the commonest problem areas and motivate students to address them.
10. Decide whether the grades for tests in the RECEIVER class will be "counted" in the final class grade, or used only diagnostically, and as experimental data between participating faculty.
    

Variations in Format: The simplest, and perhaps most common, format for these experiments is likely to be GIVER-RECEIVER pairs (see Figure 1). However, there may be more than one GIVER teacher (e.g. where two or three faculty are teaching sections of the same class), and more than one RECEIVER (see Figure 2). RECEIVER teachers may be in the same discipline, a related or "client" discipline, or a totally different discipline.

Figure 1. Simple GIVER-RECEIVER design.

Figure 2. Multiple GIVER-RECEIVER design.

Both GIVERS and RECEIVERS may be innovative teachers, or RECEIVERS may be more traditional. In a group experiment, various mixtures of more and less innovative faculty can be included. It is just as important to know how well students retain what they learned in a sequence of traditional classes, as whether they retain learning from more innovative classes sufficiently well to satisfy the more doubtful members of the department. Chains of three faculty (or groups) may be formed, such that the first RECEIVER becomes a GIVER for a third level RECEIVER. What matters is that variations should arise solely from the need of the participating faculty to know how effectively students carry forward particular aspects of their learning into other classes.

Variations will also reflect the particular configuration and timing of classes in any department. By field-testing the method in departments with different class configurations and in institutions of different types, the ChemLinks and ModularChem evaluators expect to learn about the relative efficacy of different forms of the experiment.

This method is primarily for the use of participating faculty. However, Joshua Gutwill is collecting and compiling the results from different campus experiments, and we expect the results from the first groups of experimenters in summer 1998.

• Keith Kester
  Colorado College Chemistry Department
  E-mail: kkester@cc.colorado.edu ; phone: (719) 389-6440
  

• James Swartz, Dean, and Lee Sharpe, (Chair)
  Grinnell College Chemistry Department
  E-mail: swartz@ac.grin.edu ; phone: (515) 269-3006
  E-mail: sharpe@ac.grin.edu ;phone: (515) 269-4285
  

• ChemLinks --http://Chemlinks.beloit.edu
• MC2 --http://mc2.cchem.berkeley.edu