This tool was originally designed to assess the impact of a broad variety of factors related to the processes by which students learn chemistry including: confidence in their ability to master fundamental aspects of chemistry; as well as their perceptions of what were the most effective strategies by which they solved chemistry problems and grasped chemistry concepts. Additional questions on this survey were designed to assess outcomes related learning processes which also rely on student self-report (e.g., students' perceptions of science as a career; ability to work in groups; enjoyment of science). The tool is designed to be given at the end of the semester in conjunction with a beginning-of-semester survey administered in the first or second week of class.
Those who wish to administer only one survey at the end of the semester may obtain related "before-and-after" information by using a version of the NT end-of-semester survey (before & after version as MS-Word doc) in which students are asked (at the end of the semester) to compare, e.g., their confidence levels at the beginning of the semester with those at the end of the semester. However, the interpretation of this latter sort of before-and-after data will be somewhat different than that arising from comparison of responses on the beginning-of-semester and end-of-semester surveys.
Perhaps, the most basic point to realize regarding the analysis of Likert scale type survey questions (i.e., questions of the type strongly disagree=1,…, strongly agree=6) is that the scale (in this case from 1 to 6) is ordinal as opposed to metric. In other words, a response of 5 can be assumed to be higher than a response of 3, but not necessarily 5/3(=1.667) times higher than 3. Moreover, there is absolutely no guarantee that the contrast, say, between 5 and 3, for one respondent is the same as that for another respondent. On the contrary, one can be virtually certain that each respondent has their own "scale." (only some of which may coincide).
Nevertheless, one may observe interesting contrasts between different groups of respondents. For example, one may find (as we did) that male freshman chemistry students tend to report higher confidence levels (higher percentages of 5s and 6s) than female students at the beginning of the semester, but that this gap (not surprisingly) narrows by the end of the semester. In general, questions framed in a scale "invariant" way will be both more reliably answered, and the answer more easily interpreted. Thus, although "statistically significant" differences in average responses may be a useful method to "flag" potentially interesting contrasts the interpretation of such differences will be of one sort in the case that a large proportion of one group responds at the top of their scale(s) (in comparison to the other group), and of quite another sort in the case where most of the responses of both groups are near the middle of their scale(s). [The implications for statistical analyses are also far reaching. In particular, it means that the more appropriate statistical methods (i.e., the methods with p-values that are more reliably and easily interpreted) are those which are "invariant" under arbitrary monotone changes of scale. Thus, rank tests, permutation tests, chi-square tests, and log-linear methods are more appropriate than "normal theory" tests such as t-tests or Analysis of Variance (ANOVA) methods.]
As to the specifics of this survey, questions 3 to 42 query students about relative impact that people and course activities/materials had on both their learning of chemistry and the confidence they had in their ability to engage in the task of learning chemistry. We planned to use this information in a variety of ways. One was to see how these responses correlated with "incoming characteristics" such as high school grades, standardized test scores, gender, ethnicity, and the beginning-of-semester survey. By the same token, we also wanted to see how people and course activities/materials correlated with students' characterization of their learning strategies (queried in questions 65 to 105).
Finally, we wanted to see how all these factors correlated with student performance in the course. In particular, we wondered whether increases in confidence levels, and/or high course grades were associated with the self-reported relative importance/effectiveness of particular learning strategies such as group work or consultations with the TA, or with particular course components such as special lab projects, workshops, etc.
