Many educators are aware that a key to the assessment of mastery in many areas of science education may lie in the development of instruments which can assess the ability of students to retain the knowledge learned from one course to the next, as well as the related ability to transfer that knowledge to different contexts. Thus, the ability to track students progress throughout a science curriculum becomes essential to the development and testing of such assessment tools.
Development of longitudinal student records for the purposes of program or course assessment was one of the first projects undertaken by the New Traditions initiative. The entire story has been one of various aspects and epochs of computing technology struggling to catch-up with themselves. The task was fraught with numerous difficulties from the outset in the Spring of 1995. At that time the University of Wisconsin-Madison stored all its records on an IBM 3270, which meant that the file format was unreadable by modern desktop computers. Thus, the services of a COBOL programmer would be required to download the records.
On the plus side, desktop database management software was just coming into its own at that time with tools such as Microsoft's Access, and Corel's Paradox for Windows just becoming available. Moreover, seamless interfaces between modern servers and desktop computers were also being developed. As a result, "DataWarehouses" were being developed in which large databases were made available on servers with ODBC (Open DataBase Connectivity) protocols that allowed desktop computers to extract data from servers over the Internet.
Our first attempt consisted in trying to convince the UW-Madison to include longitudinal student records in their first modern server-based DataWarehouse which was scheduled to go on-line in January 1996. Unfortunately, this project is still in progress as it has been subsumed into the monumental task of converting the entire UW-Madison electronic record system into a modern server-based facility.
Interestingly, one of the largest problems involved with this larger project is the development of a "data-dictionary" explicating the arcane data fields that have been used over the years. For many of these fields no one is absolutely sure of precisely data is found in them, much less of the best way to define the field.
Undeterred, our second attempt consisted in hiring a University COBOL programmer to develop routines that would download most of the student records which were of interest based on a list, or definition of a set of student IDs. For a given cohort of students the established routines could be quickly modified (on the "front-end") to download the student records corresponding to the specified set of IDs (typically, we were interested in cohorts of students who enrolled in a particular course(s) during some given time period). The results presented below were derived from these sorts of mainframe database routines.
Once we had the data on our desktop, our first goal was to develop a general profile of the characteristics of those students who were successful (or not) in the traditional freshman chemistry curriculum and then track their progress through various gateway courses into majors and to graduation. These baseline data were then to be compared to students who took courses designed by the New Traditions faculty so as to check for differences in longitudinal outcomes. To this end we considered cohorts of entering Fall freshman students (with no advanced standing) who were enrolled in first-semester general chemistry and some math course in their first (Fall) semester (typically, this amounted to about 1,000 students per cohort). Enrollment in both math and chemistry in the first semester was considered a proxy for student interest in majoring in a (physical or related) science.
Of course, a student's incoming preparation levels are bound to be correlated with their first-semester performance, but we were somewhat surprised to find that only the students with "excellent" high school science preparation (cf. below) had better than a 50-50 chance of achieving a B or above in their first-semester chemistry and math courses. On the other hand, the bulk of the rest of the students had much lower than a 50% "success" rate.
We examined several measures of incoming preparation level including: standardized test scores; high school grades and number of high school units in science and math; as well as the UW-Madison placement scores. After careful examination of the predictive capabilities of the various measures we settled on the combination of a 3-valued measure and a 2-valued measure yielding a single 6-valued measure.
The 3-valued measure divided a cohort into 3 groups labeled: Excellent, Good, and Other (denoted E, G, O in the charts below); based on the four yearly summary grades a student received in the two math and two science courses of their sophomore and junior years in high school, (senior grades were not available from the database). The "Excellent" group consisted of those with 4 A's (out of the four yearly grades); the "Good" group was comprised of those with all B's or above (but not 4 A's); the "Other" group was the remainder of the cohort. This divided students into three reasonably sized groups . Although high school grades are by no means standardized, this measure was used because of the longitudinal history of the student's success with science courses that it reflected.
In order to enhance this 3-valued variable with a measure that reflected some degree of standardization we considered the two-valued variable which indicated whether or not a student enrolled in standard calculus for science majors course (or higher) in their first semester. This measure reflected both the student's UW-Math placement score (i.e., whether their placement was above the cut-off for calculus), as well as the confidence (and/or need) the student had to actually enroll in calculus given that they scored above the cut-off.
We then examined the first-semester "success" rates for each of the six groups defined by all combinations of these two incoming preparation measures (denoted by E/H, E/L, G/H, etc. in the charts below). We defined "success" by whether a student received a B or above in both their first-semester math and chemistry courses. Interestingly, those in the "Excellent" group who enrolled in calculus had a 75% first-semester success rate, while the other five groups each had less than a 50% success rate. In addition, there was little difference in first-semester success rates between male and female students.
Finally, we examined the graduation patterns of each of the twelve groups defined by the six possible values of the incoming preparation measures and the two possible values of the first-semester "success" variable. In particular, we were interested in each group's: graduation rate; SMET (Science, Math, Engineering, and Technology) major rate. The findings are presented below.
Below we present two of the charts that one will find upon downloading slide presentation available in the "Sample Results" section which immediately follows this section. The data represents a cohort of entering freshman students enrolled in general chemistry and math their first semester at the University of Wisconsin-Madison in the Fall of 1989.
Many science faculty and students alike have been well aware of the fact that the presentation of introductory science curricula is often designed primarily for majors (often associated with a "weed-out" effect for the majority of students who will not be majors in the given discipline). Even so, we were somewhat surprised to see the dramatic contrasts in "success" rates as depicted in Chart 1a. below. (See the overview section above for the definition of the E/H, E/L, G/H, … groups.)
These contrasts persist to a considerable extent through graduation. Chart 3. below shows the numbers of students in the cohort who graduated in SMET majors (or not) broken down by their incoming preparation levels (E/H, E/O, etc.) and their 1st-semester success status.
*success = B or above in Math and Chemistry
**others includes those who did not finish degrees by Jan. '96
We draw the reader's attention to the following observations regarding Chart 3:
One might hypothesize that a freshman science curricula more targeted to the G group of students would be more successful in meeting the NSF goals of science "literacy" for all students.
The longitudinal student record databases were developed and analyzed by Dr. Steve Kosciuk, (statistician at the LEAD Center), for the New Traditions Chemistry Curriculum Reform Initiative.
Dr. Steve Kosciuk -LEAD Center statistician
room 427 1402 University Ave.
University of Wisconsin-Madison
email: kosciuk@engr.wisc.edu -- phone: 608-265-5926
