EUGENIA ETKINA
Graduate School of Education, Rutgers, The State University of New Jersey,
e-mail address: etkina@rci.rutgers.edu.
This paper describes how to use weekly reports written by the
students as a feedback tool in teaching science. The weekly reports
help students to summarize their knowledge about the new concepts,
to learn how to ask questions and to predict what questions their
teacher is likely to ask. The reports help a teacher to know immediately
about the difficulties her (his) students experience while learning
new material, to adjust her (his) teaching to the students' needs
and to match the levels of difficulty of learning and testing.
The experience described in the paper shows the existence of a
common mismatch between learning and assessment and offers a solution
to this problem.
While teaching physics in high school and college I passed through the usual stages. First, I thought that my students would be able to repeat what I did if I told them exactly what to do and how to do it. I assumed that if I explained the material well, they would be able to solve the problems (Mazur, 1997). I concentrated on the depth and logic of the explanations, tried to demonstrate as many experiments as possible, and looked for the best examples and various methods of assessment. I did not know that at that time I was following the path typical for physics instruction: "Not very long ago, the physics community considered research in physics education an activity in which instructors thought carefully about the subject matter and worked to produce clear and unambiguous lectures, textbooks and laboratory manuals. Improvement in instruction was often equated with an increase in clarity of presentation" (McDermott, 1997, p. 42). Still this strategy left me with students who in spite of all my efforts did not "fall in love with physics" and not only had no clue about what was going on in class but hated the subject.
The second stage began after 5 - 6 years of teaching, when I finally realized that there was only a slight correlation between the quality of my work as a teacher and the amount of knowledge my students possessed (Mazur, 1997). Then I tried different teaching techniques - classical and constructivist approaches, individual and group work. But my students still had typical misconceptions about the laws of physics and never did as well as I wanted them to do (Clement, 1982).
The third stage was reached after 14 years of teaching and was
even more frustrating. I realized that until I know what students
think, I cannot really teach them. I started to see that "what
the instructor says or implies and what the student interprets
or infers as having been said or implied are not the same…
There are often significant differences between what the instructor
thinks the students have learned in a physics course and what
students may have actually learned" (McDermott, 1991, p.
303). I could see it in preconceptions, misconceptions, and misinterpretations
that my students had, and in the need to explain the same material
over and over again with very little improvement in the students'
knowledge. This experience led me to the conclusion that if I
want my students to understand the basic concepts of physics I
need to know what is going on in their minds after each class.
I was ready to "focus on what students are actually learning"
(McDermott, 1997). From research in physics education one can
learn a great deal about the typically most difficult moments
in physics instruction, and the common misunderstandings (Clement,
1982, Throwbridge & McDermott, 1980). But I wanted to devise
a method that could help me to have a feeling for what my particular
students had troubles with and what they thought I was teaching
them. This was the fourth stage.
I wanted to know what my students thought they have been learning in class. Though it would not necessarily mean that they really understood the material, at least it would help me to immediately "catch" those who were completely lost. The immediate knowledge about my students' difficulties would help me to prevent deep gaps in their understanding before it was too late. Another important aspect is understanding of what concepts and skills are the most important in the material the students are learning. The ideal case is when a teacher can communicate this hierarchy naturally while teaching. But how will one know if this is the case? All this meant that I needed a tool that would help me to make my students' thinking visible.
What exactly did I want to know about my students' experience in class?
1) What did my students think they had learned during a certain class, or period of time?
2) What did they think remained unclear to them?
3) What did they expect me to ask them tomorrow concerning the material taught today?
But would this work be beneficial to the students? I thought that the first question would help them to summarize what they had learned and review the material concentrating on the most important concepts (in the process of teaching I always focus the attention of my students on what kind of knowledge they are learning: experimental facts, models, physical magnitudes, laws, limitations, etc.). It means that in answering the first question the students would try to sort the new knowledge, assigning pieces of it to certain structural elements of physics.
The second question would help my students formulate questions about the difficulties they encountered in the material. I have always thought that in teaching questions are more important than answers and assigned extra points for good questions in class. The Piagetian idea that "cognitive disequilibrium" is essential for learning (Victor & Kellough, 1997) emphasizes the importance of asking questions. There is some disagreement about how important for learning students' questions are. Some authors consider students' questions irrelevant for learning (Ennis, 1986); some think that a teacher should be able to predict possible questions and base the teaching process on them (Macmillan & Garrison, 1983). Others advise us to base instruction on students' questions asked without prior knowledge (Scandamalia & Bereiter, 1992); and there are those who think that it is better to start generating students' questions after the soil has been plowed (Miyake & Norman, 1979). But most educators and psychologists agree on the following: "To teach someone something is to answer that person's question about some subject matter" (Macmillan & Garrison, 1983). In other words, before teaching any topic, a teacher has to stimulate the students' interest by addressing the potential questions that they already have but have never formulated.
Work on the third question would help the students to concentrate
on what ideas and skills I might test later. Problems and questions
in tests are based on the most important concepts. If the students
can predict these test problems and questions it means they understand
which ideas are the most important in the material they learned.
Being better prepared for tests is an additional benefit.
The Idea at Work
In the summer of 1997 I taught an introductory physics course, "Extended General Physics" at Rutgers University. This course was intended for at-risk students, mostly female and minority students - bio and pre-med majors who need physics as a requirement. The eight-week course had the following format: 3 lectures, one lab, one mini-lab and three recitations per week. The first recitation in the week had the purpose of going over the major concepts of the lectures of the previous week, using simple exercise problems. The second recitation had the format of group problem solving of context-reach problems (Heller, Keith & Anderson, 1992). The mini-lab contained simple, qualitative, hands-on experiments, while the lab had quantitative experiments. The last recitation was supposed to be dedicated to the homework problems. The activities each week were based on theoretical material taught during lectures the previous week. At the end of each week, students were required to hand in the homework (regular questions and problems from the corresponding chapters of the book). To provide the regular feedback that I wished, I decided to ask the students to write weekly reports in which they were to answer the three questions, phrased as follows:
1. What did you learn during this week?
2. What questions remained unclear?
3. If you were the professor, what questions would you ask to find out whether the students understood the most important material of this week?
These reports were graded and they had the same weight in points as the homework itself. To encourage students to ask questions in class as well as in the reports, I gave them extra class points for interesting questions. (The grading system in the course was not based on a curve but was a point-accumulating system.)
I planned to use their report questions to correct my lecturing. But after the first week it turned out that my expectations of what questions the students might provide did not match what they actually listed in the reports. Their questions turned out to be so interesting that I decided to devote the entire last recitation each week to discussing them.
Every week, while grading their reports I made a list of the difficult questions that they provided and then discussed them using the group method in class (Mazur, 1997). The average number of different questions each week was 15 (I had 16 students in the class). It frequently turned out that a question asked by one student could not be answered by anyone in the class, yet none of the others had even been able to formulate the question. I therefore realized that if a question is asked by only one student I should not assume that everyone else knows the answer. I also realized that I had not imagined what difficulties my students would have even with the easiest concepts and logical connections. The analysis of the questions they asked showed that once a teacher makes certain logical connections for him or herself it can be difficult for the teacher to understand that these logical connections are not natural for the students.
As the course progressed I changed the requirements for the reports,
asking the students to mention the homework problems that corresponded
to what they had written in response to questions one and two.
It meant that instead of just mentioning a concept a student learned
during the week, he or she was required to find a homework problem
based on this concept. For example: "I learned the concept
of average velocity; problems 4 and 6 from the homework are based
on this concept." "I did not understand how to graph
speed vs. time; that is why I think I could not do problems 7
and 8." This helped me to distinguish those problems the
students could not solve because of technical details from those
for which they lacked conceptual understanding. Surprisingly,
by the end of the course, most of the students found it more interesting
and challenging to analyze their homework problems from a conceptual
point of view, rather than simply repeat what they had learned.
Part 1. What Did I Learn This Week?
It turned out that it was rather easy for the students to explain what they had learned during the week. Analysis of the reports indicated that what I wanted the students to learn and what they thought they had learned matched by approximately 90%. Different students approached this question differently. Some concentrated more on listing facts and laws they had learned ("I learned the difference between a vector and a scalar." "I learned how acceleration of an object depends on the net force acting on it"). Fewer students listed skills they learned ("I learned how to draw free-body diagrams."). And very few could analyze the material they learned conceptually ("Always derive a formula for the net force using a free-body diagram."). Analyzing this part of the reports I could see which of the students had read the textbook in addition to attending lectures. ("There are two different ways to derive the formula for apparent weight."). I could also identify those who tried to emphasize the concepts ("For example, it is silly to say a person sitting down is not moving. The reason for this is that the person is really moving in reference to perhaps an astronaut standing on the Moon. However, the person sitting down is not moving in reference to the ground (Earth). Therefore we should always specify the reference frame that we are talking about.").
It was very interesting to observe how different students were changing as the course progressed. Most could not put their thoughts into coherent sentences at the beginning. One student wrote, "I learned that vectors are magnitude with direction using arrows to show vector direction", and "Can't say something moves unless it is with respect to something." This happened because students are not usually required to give long explanations during class, and the tests they take are mostly multiple choice. Thus the language skills necessary to describe phenomena do not develop and the students cannot explain what they think even if they can solve problems. The requirement to write in a physics course was unusual for the students and at the beginning they did not think that it was important for physics. This skill improved significantly during the course. For example, the same student after 5 weeks wrote "For wave propagation we need a vibrating source, and a medium in which particles interact with each other. The difference between wave motion and the other kinds of motion we learned before is that in wave motion only energy propagates not substance." To construct this statement the student needed to analyze and synthesize the material from the whole chapter on wave motion.
Sometimes students listed what they learned in part one of the reports and then later, while summarizing what remained unclear, they would pose a question revealing that in fact what they thought they had learned was not understood at all. For example, after a week spent on the subject of wave motion one student wrote: "We understood that the velocity of a wave depends on the medium where the wave propagates, the frequency is determined by the source and the wavelength depends on both." Yet, in the second part of the report the same student wrote: "What determines the velocity of a wave?" This question meant that she could not identify how the medium affects the velocity, though the formula was derived in class and listed in the book. Mathematical symbols in the formula and the statement she made in the first part did not overlap to produce understanding.
Some students used this part of their report to communicate with me, explaining which concepts were difficult to grasp ("I think we did not spend enough time studying the difference between static and kinetic friction"); sharing how interesting some ideas were ("Now I know why astronauts orbiting the Earth experience apparent weightlessness!") or even telling me how to organize lectures ("Could you briefly go over the material of the previous lecture when starting a new lecture? It helps a lot to review and prepare myself for the new material."). I found it very rewarding, because this put me in touch with the feelings of the students.
Part 2. What Questions Remained Unclear?
It was more difficult for the students to answer the second question in the reports, in which they had to identify the concepts they did not understand. The fact that out of a total of 254 questions asked, 66 (26%) were based on the homework problems rather than concepts (the students would simply say, "I did not understand how to do problem 55 in the homework.") confirmed the idea that to ask a question one needs to know something (Miyake & Norman, 1979).
Sometimes the questions students asked could be used as a topic for a later class ("Is gravitational potential energy the only kind of potential energy or there are others? If yes, what is responsible for them?"). This is how elastic potential energy, interference, and standing waves were introduced in the course - as a consequence of questions asked in the weekly reports.
The level of questions varied greatly. They ranged from simple questions such as: "What is a physical pendulum?" to "How do you label forces on a free-body diagram?" to such sophisticated ones as, "In wave motion we can find three different velocities: the velocity that shows how fast displacement changes, the velocity that depends on the medium, and the velocity that is the product of wavelength and frequency. How do these different velocities relate to each other? Which one depends on which and why?" To ask this question the student had to put together pieces of knowledge from kinematics, dynamics and wave motion, integrate these concepts, and began synthesizing a new concept.
Again in this part of the reports some students wrote general comments such as, "homework this week gave me a lot of trouble," and "I did not feel confident doing my homework this week." These students did not represent the majority of the class but still I thought that it was very helpful for them to be able to complain about how difficult their work was. I think it made their learning and our communication more human and changed the whole atmosphere in the course.
Part 3. What I Would Ask if I Were the Professor?
The most difficult part of the weekly reports for the students was to predict what the professor would ask. They listed a total of 200 conceptual questions and qualitative and quantitative problems in this part. I analyzed these questions and problems from the point of view of the concepts they covered and compared these to the concepts that I was going to test later. Students' questions covered approximately 65% of the material on which I had planned to test them.
After the first week they were really confused by this part of the task and could not come up with questions. Some of them gave general advice. For example, "To test your students regarding their knowledge of the material in week 1, you could perhaps ask students to tell you in writing all they know about a certain topic in the material that we learned. Along with a question from each topic, you could also ask a problem related to that topic." Or "I would ask students what equations can be used in specific situations." After being told that these responses were too vague, the students started to list the numbers of the problems of the homework that they considered the most important. Only gradually did they start to learn how to predict the questions that I would ask. Some of their questions were at a very low level (for example, "What does Archimedes' principle state?"). Many of the questions indicated a desire to relate the phenomena they learned in class to their own lives: "What are everyday examples of the Doppler effect." A few reflected typical difficulties the students have in understanding certain concepts: "Is saying 'one wavelength' the same as saying 'one wave cycle'"? Two students out of 16 (both received an "A" in the course) were able to devise sets of questions that I could use as ready quizzes almost every week, starting with week three. For example, during the week in which we studied work, energy, momentum, and laws of conservation of mechanical energy and linear momentum, the questions asked by one of these students were the following: "(1) Why does the Earth do no work on the Moon? (2) A pilot travels all over the world and lands at the same place where he took off from. Will the work done by the net force be positive? Negative? Zero? Explain. (3) How do you find the work done by a force that is not constant? Explain. (4) A person walks to the back of the train. As she walks her momentum increases. What, if anything, happens to the train's momentum? (5) What is the momentum and kinetic energy of a 5000 kg truck moving at 10 m/s? If the speed is quadrupled, by what factor do momentum and kinetic energy change?"
In this set of questions and problems almost all of the main concepts taught during the week (work done by constant and changing force, conservation of mechanical energy and momentum) were covered, qualitative questions were mixed with quantitative problems, and the level of difficulty varied from a simple "plug-in problem" (5) to a very difficult conceptual question (4) involving many ideas such as vector properties of linear momentum, linear momentum of a system, an isolated system of objects, and the law of conservation of linear momentum.
However, despite some improvement in students' responses, I was not satisfied with this part of their reports as much as with the first and second. Why did it turn out to be the most difficult? I think the answer is that during all the years of their previous studies the students had been taught to do what the instructors wanted them to do without really understanding the reasons for this. It means that the logical connections, the differences between the most important and least important questions, were hidden from the students behind the walls of the numerous problems they had to solve. The problems and questions were not associated with the concepts because the concepts were not considered to be the most important part of learning. With our widespread multiple choice testing system, students concentrate on picking and choosing the right formulae for solving problems, not on identifying the concepts on which these problems are based (Van Heuvelen, 1991).
I assigned the questions the students asked in the second and third part of the reports to four levels of difficulty. Questions that asked for factual information were assigned to the "minimal level" (for example "What is a physical pendulum?"). Questions that asked for comparative information were described as "low level" (for example "What is the difference between a simple pendulum and a physical pendulum?"). Conceptual questions and questions about different procedures done in previous class were to be considered at a "moderate level" (for example, "How can we prove that the period of a simple pendulum does not depend on its amplitude?"). The "highest level" was assigned to questions that required explanations not given in class before and that usually start with "Why?" (for example, "Why is it that only a force that is linearly proportional to the displacement can provide a system with simple harmonic motion?").
This approach helped me to analyze what students thought they did not understand (part two) and what they expected me to test them on (part three) and compare these two different kinds of questions. I graphed the number of questions at each level as a percent of the total for both parts of the weekly reports (Fig. 1 below).
Figure 1.
The majority of the questions for both parts of the reports fell in the "low" and "moderate" categories. The students were asking questions that helped them to clarify and apply previously taught concepts. This means that they tend to assign the same value to the content as I do. But what interested me the most were the tails of the graph. The number of "minimal" level questions was much higher for part three (the questions that "I would ask if I were the professor") while the number of "highest" level questions was considerably higher for part 2 of the reports ( the questions that "remained unclear" for the students). In other words the students expected me to ask them much easier questions than they wanted to ask for themselves. Why? I think it meant that they expected me to test them at a rather low level. This guess was confirmed by the first exam, which the students described as too difficult even though it was no more difficult than their own questions listed in part two of the reports. Such complaints usually originate from a mismatch between a student's expectations of what should be included in the test and what is actually included in it. By the second exam, the level of difficulty of the questions in the third part of the reports increased and there were fewer complaints about the difficulty of the exam.
The problem of grading tests was addressed by Smith (1995, p. 2). Discussing the problems associated with grading he wrote, "When students are about to take a test, they need to know what you think is important in the material they are studying. The more clearly you communicate your expectations, the more they can take ownership for their grades." (1995, p.2). Using weekly reports a teacher can go a step further - by helping the students to realize themselves what is important in the material they learned and what will be included in the test. In this case all a teacher needs to do is to explain what level of difficulty corresponds to a certain grade. My own experience shows that the students are willing to take bigger challenges during tests if they can predict what will be included.
The experience of writing weekly reports turned out to be beneficial for me and for the students. During all eight weeks of the course I knew about the difficulties my students were having, I could "catch" their misunderstandings as soon as they appeared and, most important, I could see what concepts were the most difficult for them. It gave me the opportunity to correct my work, and involve the students in regular discussions of the important questions.
The questions asked in part three of the reports were very useful for me in writing the exams. I also realized how important the understanding of the individual student's difficulties was for me. I could improve my teaching because I knew more of what my students were thinking and this understanding was based not only on a "typical student" with "typical misconceptions" but also on real responses of real students.
The students gave high marks to the use of the reports in the course. In interviews conducted 3 months after the end of the course, when the same students were taking the second semester of physics (with no weekly reports) they said: "At the time of the first semester I did not really like the weekly reports. They took extra time. But now I realize that they helped me to review the material each week, and focus on the most important concepts." "The reports taught me to think about what I did not understand. They helped a lot." "The reports taught me how to formulate my questions." "Due to the reports I could understand why certain problems were included in the exams." "I never took physics before and I thought that I would not be able to pass it. But because I could ask as many questions as I wanted I became confident." Students felt very strongly that the friendly atmosphere in the course helped them to ask questions freely, without being afraid that the questions might be used against them later. For the weekly reports to serve their purpose, the students should trust their teacher.
The weekly reports gave me a feedback tool that was easy to use and did not cost anything except some extra time for grading. But this extra time was well spent because of improvement in my teaching and change in the students' learning that occurred. For me, the weekly reports became an essential part of teaching - as important as labs or lectures, and a wonderful way to create a positive student-teacher relationship for learning.
The constant support and helpful suggestions of Dr. Angela O'Donnel
are deeply appreciated. The critical reading of the paper by Dr.
David Ehrenfeld and his assistance in preparing the manuscript
is most gratefully acknowledged.
References.