New Traditions beginning-of-semester
survey
FLAG homepage
Return to top of page
- Analysis/interpretation of findings
Perhaps, the most basic point to realize regarding the analysis
of Likert scale type survey questions (i.e., questions of the
type "strongly disagree=1,…, strongly agree=6")
is that the scale (in this case, from 1 to 6) is ordinal
as opposed to metric.
In other words, a response of 5 can be assumed to be higher than
a response of 3, but not necessarily 5/3(=1.667) times higher
than 3. Moreover, there is absolutely no guarantee that the contrast,
say, between 5 and 3, for one respondent is the same as that for
another respondent. On the contrary, one can be virtually certain
that each respondent has their own "scale." Nevertheless,
one may observe interesting contrasts between different groups
of respondents. For example, one may find (as we did) that male
freshman chemistry students tend to report higher confidence levels
(higher percentages of 5s and 6s) than female students at the
beginning of the semester, but that this gap narrows by the end
of the semester. In general, questions framed in a scale "invariant"
way will be both more reliably answered, and the answer more easily
interpreted. Thus, although "statistically significant"
differences in average responses may be a useful method to flag
potentially interesting contrasts, the interpretation of such
differences will be of one sort in the case that a large proportion
of one group responds at the top of their scale (in comparison
to the other group), and of quite another sort in the case where
most of the responses of both groups are near the middle of the
scale. [Readers familiar with statistical methods will also note
that the implications for statistical analyses are also far reaching.
In particular, it means that the more appropriate statistical
methods (i.e., the methods with p-values that are more reliably
and easily interpreted) are those which are "invariant"
under arbitrary monotone changes of scale. Thus, rank tests, permutation
tests, chi-square test, and log-linear methods are more appropriate
than "normal theory" tests such as t-tests or Analysis
of Variance (ANOVA) methods.]
- As to the specifics of this survey, the first ten questions
query students about ten different aspects of their confidence
in their ability to engage in the task of learning chemistry.
We planned to use this information in a variety of ways. One was
to see how this correlated with "incoming characteristics"
such as high school grades, standardized test scores, gender,
and ethnicity. By the same token, we also wanted to see how confidence
levels correlated with how students characterized their learning
strategies in responses to the following two blocks of survey
questions. Finally, we wanted to see how all these factors correlated
both with student performance in the course and with their responses
to the same and additional questions on the end-of-semester survey.
In particular, we wondered whether increases in confidence levels
were associated with the self-reported relative importance/effectiveness
of particular learning strategies--such as group work or consultations
with the TA, or with particular course components--such as special
lab projects or workshops etc.
Return to top of page
Sample results
Return to top of page
- Who designed and tested the tool
This survey was designed by the LEAD Center in its role as evaluator
for the New Traditions initiative. It was developed on the basis
of interviews with 36 students enrolled at UW-Madison Chem 104,
and Chem 108, Madison Area Technical College, and San Jose State
University in the Spring semester of 1995. A preliminary version
was piloted in the Spring of 1995 at the above three institutions.
Subsequent versions with a variety of revisions were administered
at the same institutions during the Fall and Spring semesters
from Fall 1995 through Spring 1997.
- Name and location of original designer(s)and tester(s)
The survey was originally designed by Dr. Steve Kosciuk, LEAD Center
statistician, and implemented by many of UW-Madison Chemistry faculty,
who taught freshman chemistry courses from Fall 1995 through Spring 1997.
For further information contact:
-
Professor John Moore
Department of Chemistry, University of Wisconsin-Madison
1101 University Ave. Madison WI 53706
rm. 1321 Chemistry bldg.
email: jwmoore@chem.wisc.edu;
Phone: 608-262-5154
-
Dr. Steve Kosciuk,
LEAD Center, University of Wisconsin-Madison
1402 University Ave., Madison, WI 53706
email: kosciuk@engr.wisc.edu;phone:
608-265-5926;
LEAD Center administrative office: 608-265-5920
Return to top of page
-
Others who have used this tool
-
Professor Steve Branz
Department of Chemistry, San Jose State University
San Jose, CA 95192-0101
email: branz@leland.stanford.edu;phone:
408-924-4999
-
Professor David Shaw
Department of Chemistry, Madison Area Technical College
email: dbshaw@facstaff.wisc.edu ;
UW phone: 608-262-2940; MATC phone: 246-6656
-
Professor Kim Kostka
Department of Chemistry, University of Wisconsin-Rock County
email: kkostka@mail.uwc.edu;phone:
608-758-6532
Return to top of page
Return to top of page
