| This report presents summary results of an evaluation of LSA’s Quantitative Reasoning requirement. According to The LSA Bulletin, “The goal of the Quantitative Reasoning requirement is to ensure that every graduate of the College achieves a certain level of proficiency in using and analyzing quantitative information.” Students may fulfill the requirement by passing one course designated for QR credit (QR1) or two courses designed for half credit (QR/2). Because a wide margin (82%) of students fulfill the requirement through the former option, this evaluation focuses specifically on the impact of QR1 courses. |
A. Evaluation Methods
B. Key questions and findings:
C. Key implications of these findings
The evaluation was based on a survey of LSA first- and second-year students about quantitative reasoning gains they reported making from their QR1 or Fall Term courses. Most of the survey was derived from a University of Wisconsin assessment study of its QR requirement, which found that student self-reports about learning gains on 14 different quantitative reasoning skills were as valuable a source of data as pre- and post-tests of authentic QR-related problems (Halaby, 2005). The instrument was developed by a study team that included UW’s Director of Testing & Evaluation and other quantitative researchers. In addition to the 14 UW gains items, the U-M survey asked students if they felt that the course met LSA’s goals for the QR requirement, if they could give an example of an application of the course, and what instructional methods helped them learn. The survey was pilot tested on U-M students in six QR1 courses, as well as a control group, in Spring/Summer 2009. When asked to give their feedback about the survey, students reacted very favorably. The 14 QR gain items had a Cronbach’s alpha of 0.95, indicating the high consistency of the items in measuring the same underlying concept.
In Fall 2009, all first- and second-year LSA students who were taking a QR1 course for the first time were surveyed at the end of the term. The survey was distributed to 2,624 QR takers and 1,419 responded, a 54% return rate. Additionally, a parallel survey was distributed to a control group of 900 first- and second-year students who had never taken a QR1 course. 384 of these students replied to the survey, a 43% response rate. Both groups of students were asked, “Compared to when I started [QR course or the Fall Term], I am now BETTER able to:” in regard to 14 different skills, which were aggregated into an overall mean QR score (1=Strongly Disagree to 5=Strongly Agree, N/A).
B. Key questions and findings were:
1) Do LSA students perceive that QR1 courses contribute to gains in their quantitative reasoning skills?
For first-time QR takers, the overall mean QR score was 3.52, indicating that these students tend to agree that they gained key quantitative reasoning skills in their Fall Term QR1 courses (1=strongly disagree, 5=strongly agree that gains were made in quantitative reasoning skills).. Strongest gains were reported for “solve problems using arithmetic, algebra or statistics” (M=3.83) and “use quantitative information to solve problems” (M=3.79). Weakest reported gains for all QR takers, hovering around a neutral response, were “using statistics to evaluate factual claims” (M=3.22), “understand randomness, uncertainty and risk” (M=3.18) and “understand the difference between correlation and causation” (M=3.16).
2) Do these reported learning outcomes differ by subgroups of students, e.g., by gender, race/ethnicity, incoming math proficiency, and class rank? Do they differ by discipline?
Looking at first-time QR takers only, there was no statistically significant difference in mean QR score between men and women, nor between underrepresented minorities and others.Dividing up QR takers’ ACT math scores into quartiles, there was no statistically significant difference between the four groups of students. However, the general trend is that students with the least math proficiency going into a QR1 course perceive that they gain the most. The mean QR score was significantly higher for sophomores, compared to first-year students, a 0.19 gap
In Fall 2009, eight departments offered QR1 courses. Looking by department, there was a range of 3.42 to 4.39 on the mean QR score (but an important caveat is that there was a small sample size in some departments due to the sampling procedures). Reported gains for the eight departments offering QR1 courses are generally similar, indicating that the variety of courses that fulfill the QR1 requirement function well to achieve relatively equivalent outcomes.
3) Are these gains comparable to those reported by other LSA students who have never taken a QR1 class, or are there statistically significant differences between students who take QR1 courses and those who do not (overall and by specific skill)?
Compared with the control group (first- and second-year LSA students who had never taken a QR1 course), QR1 takers had a slightly higher (+.10), but significantly different, overall mean QR score (p<.05). On individual items, QR1 students scored significantly higher than the control group on seven of the 14 measures of QR skills:
On three items, non-QR students rated their gains higher:
Interestingly, in the UW study, the non-QR group scored higher than the QR sample on all three of these items as well (Halaby, 2005). These seem to be the only items that could just as easily apply to general analytical skills, as opposed to quantitative reasoning specifically.
There was no statistically significant difference between the two groups on the remaining four items:
4) Controlling for student demographics, what is the impact of taking a QR1 course on students’ reported learning gains? Given that QR/2 courses also can be used to fulfill the requirement, does having taken at least one QR1 or QR/2 class have a greater impact on students’ reported gains?
Regression models indicate that enrollment in a QR1 course is a significant predictor of the overall mean QR score, even when controlling for gender, race/ethnicity, class rank, incoming math proficiency, and participation in a QR/2 course. We also can conclude that both QR1 and QR/2 courses have a positive impact on students’ perceived learning gains, but QR1 is a slightly stronger predictor of self-reported development of quantitative reasoning. Sophomore status is also significant, showing a small positive effect on students’ reported learning gains as well.
5) Do students who took a QR1 course perceive their course met the goals of the LSA Quantitative Reasoning requirement?
QR students were presented with information about the goals of the QR requirement and asked if they felt that the course met these objectives. Students tended to agree (3.9) that their course fulfilled LSA’s goals (1=strongly disagree, 5=strongly agree).
6) Can students who took a QR1 course give an example of how they are better able to use and analyze quantitative information?
When asked to describe an example of the way in which they left their QR course being better able to use and analyze quantitative information, fewer than half the respondents (42%) offered a response.
Although many themes were course-dependent, two themes that cut across the QR courses were students’ development of evaluative skills (e.g., evaluation of ways that the media presents statistics) and applications to everyday problem-solving tasks.
7) Is a requirement necessary? In other words, would students likely take a course that emphasizes the development of quantitative reasoning even if they were not required to do so?
Students who took a QR course were asked, “If LSA did not have a Quantitative Reasoning requirement, would you still plan to take a course at U-M that met these goals?” Two-thirds of students indicated that they would, although for social science/humanities, the proportion dropped to less than half (48%). Given that LSA’s goal is to develop quantitative reasoning in every graduate of the College, this response suggests that a requirement is necessary.
8) What instructional activities do students find useful for the development of quantitative reasoning skills?
When asked about instructional activities that are helpful to QR learning, only about a third (35%) of students selected “solving real-world problems,” which may help explain why the majority of students did not provide an example of the way in which they could apply their QR course concepts/skills.
By department, there were several distinctive trends in what students reported was useful for QR learning:
|
C. Key implications of these findings include:
Reference: Halaby, C. N. (2005, October). Two assessment studies of the general education quantitative reasoning “A” requirement at the University of Wisconsin-Madison. Report to the General Education Assessment Council Subcommittee on the Effectiveness of the Quantitative Reasoning “A” Requirement. Madison, Wisconsin: University of Wisconsin-Madison. Available: http://www.ls.wisc.edu/gened/Reports/commA_surveyreport.pdf [1]
Links:
[1] http://www.ls.wisc.edu/gened/Reports/commA_surveyreport.pdf
[2] http://www.crlt.umich.edu/category/tags/assessment
[3] http://www.crlt.umich.edu/category/tags/quantitative-reasoning