1997 Annual Report

University of Nebraska-Lincoln
April 1998

 

Appendix D
The Comprehensive Education Program: A Condensed Report

 

Assessment is an ongoing process aimed at understanding and improving student learning. It involves making our expectations explicit and public; setting appropriate criteria and high standards for learning quality; systematically gathering, analyzing, and interpreting evidence to determine how well performance matches those expectations and standards; and using the resulting information to document, explain, and improve performance.
        Angelo, T. A. (1995). Reassessing and Defining Assessment. AAHE Bulletin. November, pp. 7-9.

The General Education Planning Committee, which proposed the Comprehensive Education Program to the nine undergraduate colleges in September of 1993, assumed from the beginning that a program as far-reaching as CEP would have to be assessed on a continuing basis in order to provide the feedback needed to achieve its goals in an ever-changing university environment. The members of the Committee, however, possessed little expertise in educational assessment, which had not then taken much of a foothold at UNL. Subsequent events proved that the task of examining the outcomes of CEP has been more difficult though not less interesting than the Committee could have imagined.

By the time a practicable CEP assessment plan had been drafted and a team assembled, CEP was a year old. Some opportunities for baseline studies and early intervention had been lost, yet the program was still so new that its effects were diffused and hard to measure. Nonetheless, the Assessment Team was able to take some steps in the direction of the ideal suggested by Tom Angelo in the epigraph; and even some misdirected steps proved instructive. The CEP Assessment Team sought answers to four questions.

1. Are the two entering classes of students (1995-96 and 1996-97) governed by the Comprehensive Education Program moving at a satisfactory rate toward completion of their Essential Studies and Integrative Studies requirements?

We analyzed data compiled by Dr. Earl Hawkey, of Registration and Records, showing the number of students in each cohort and the number of ES and IS classes taken by each, broken down by college or General Studies. The following charts reveal a positive finding for both groups on both requirements.

Essential Studies requires the students to take nine courses in eight areas or "domains" (two in Human Behavior) Understandably, there are significant variations in the rate at which students in different colleges are taking courses in various ES domains. On average, however, the students in the more recent cohort had completed two of the requirements and are halfway or better to completing five others (the Humanities and Race, Ethnicity, and Gender being completed by substantially fewer than 50% of the students). The earlier cohort, after two years, had on average completed or exceeded the ES requirement in all but two domains. Moreover, enrollment figures supplied by Dr. Hawkey reveal that students chose a wide variety of ES courses from the sometimes extensive lists of options in each domain but that enrollments tended to cluster in a relatively few courses in most areas. It is reasonable to conclude that both cohorts are moving at a steady and fairly rapid pace to the fulfillment of the ES requirement, though these averages do not tell us much about individual students. There may be a few, especially in highly structured programs, who are struggling to take all nine ES courses.

The following two tables also indicate good progress toward fulfillment of the Integrative Studies requirement. The 1996-97 cohort averages 3.5 IS courses per student, and no college has an average lower than 3. The earlier cohort averages 6.2 IS courses per student, with no college average lower than 5.69. The data we have received do not reveal at what level the students are taking their IS classes, but we can assume that these first and second year students enrolled mostly in 100 and 200 level IS classes. We need in the future to monitor whether these two classes of students will have access to sufficient Integrative Studies courses in the upper division. In the first two years of the program, some transfer students had trouble finding enough 300 and 400 level courses to complete their IS requirement. Particularly in 1996-97, however, departments and colleges made a concerted effort to increase the number of upper division IS courses.

Table 1. IS/ES Courses Taken by Students Entering Fall Semester of 1996-97

Essential Studies Course Catgories
College # Students # Courses IS Courses Commun-
ication
Math & Stats Human Behavior Science & Tech Historical Studies Human-
ities
Arts Race, Ethn & Gender
CASNR 272 2781
(10.2)
1099
(4.0)
369
(1.36)
76
(0.28)
309
(1.14)
1059
(3.9)
96
(0.35)
35
(0.13)
47
(0.17)
12
(0.04)
ARCH 100 1046
(10.5)
37
(3.7)
183
(1.83)
58
(0.58)
114
(1.14)
108
(1.08)
18
(1.8)
33
(0.33)
234
(2.34)
10
(0.1)
A&S 927 9027
(9.7)
3234
(3.5)
1207
(1.3)
533
(0.57)
1448
(1.56)
1741
(1.88)
544
(0.59)
312
(0.34)
250
(0.27)
93
(0.10)
CBA 537 5326
(9.92)
1619
(3.0)
897
(1.67)
275
(0.51)
892
(1.66)
521
(0.97)
327
(0.61)
302
(0.57)
210
(0.39)
29
(0.05)
E&T 292 2589
(8.87)
1070
(3.67)
51
(0.17)
482
(1.65)
286
(0.98)
921
(3.15)
139
(0.48)
84
(0.29)
39
(0.13)
21
(0.07)
FPA 116 1614
(13.9)
355
(3.06)
148
(1.28)
19
(0.16)
75
(0.65)
36
(0.31)
43
(0.37)
25
(0.22)
268
(2.31)
3
(0.03)
Gen. Stud. 871 8090
(9.29)
3193
(3.67)
1216
(1.4)
225
(0.26)
1568
(1.8)
1089
(1.25)
569
(0.65)
333
(0.38)
376
(0.43)
38
(0.03)
HRFS 61 633
(10.4)
242
(3.97)
86
(1.41)
19
(0.31)
134
(2.2)
89
(1.46)
22
(0.36)
7
(0.11)
53
(.87)
9
(0.15)
JOURN 164 1589
(9.69)
504
(3.07)
164
(1.0)
43
(0.26)
277
(1.69)
175
(1.07)
225
(1.37)
68
(0.41)
129
(0.79)
13
(0.08)
TC 247 2403
(9.73)
984
(3.98)
514
(2.08)
39
(0.16)
476
(1.93)
274
(1.11)
281
(1.14)
136
(0.55)
78
(0.32)
8
(0.03)
Totals 3587 35098
(9.78)
12671
(3.53)
4835
(1.22)
1769
(0.49)
5579
(1.55)
6013
(1.68)
2264
(0.63)
1335
(0.37)
1684
(0.47)
236
(0.07)

 

Table 2. IS/ES Courses Taken by Students Entering Fall Semester of 1995-96

Essential Studies Course Catgories
College # Students # Courses IS Courses Commun-
ication
Math & Stats Human Behavior Science & Tech Historical Studies Human-
ities
Arts Race, Ethn & Gender
CASNR 231 4699
(20.3)
1615
(7.0)
583
(2.52)
195
(0.84)
562
(2.43)
1360
(5.89)
119
(0.52)
82
(0.36)
164
(0.71)
26
(0.11)
ARCH 55 1163
(21.1)
335
(6.1)
151
(2.75)
54
(0.98)
126
(2.29)
152
(2.76)
61
(1.11)
39
(0.71)
145
(2.64)
15
(0.27)
A&S 696 14148
(20.3)
4691
(6.7)
1293
(1.86)
880
(1.26)
2303
(3.31)
2881
(4.14)
822
(1.18)
615
(0.88)
652
(0.94)
302
(0.43)
CBA 475 9809
(20.7)
2702
(5.69)
1351
(2.84)
791
(1.66)
2074
(4.37)
999
(2.1)
442
(0.93)
444
(0.93)
596
(1.25)
131
(0.28)
E&T 226 4463
(19.7)
460
(6.46)
243
(1.08)
696
(3.08)
366
(1.62)
1493
(6.61)
147
(0.65)
88
(0.39)
115
(0.51)
15
(0.07)
FPA 99 2809
(28.4)
542
(5.47)
172
(1.74)
59
(0.6)
156
(1.58)
103
(1.04)
80
(0.81)
67
(0.68)
406
(4.1)
30
(0.3)
Gen. Stud. 267 5348
(20.0)
2009
(7.52)
619
(2.32)
234
(0.88)
1140
(4.27)
767
(2.87)
268
(1.0)
286
(1.07)
312
(1.17)
98
(0.37)
HRFS 86 1841
(21.4)
727
(8.45)
214
(2.49)
79
(0.92)
395
(4.59)
241
(2.8)
69
(0.8)
51
(0.59)
115
(1.34)
41
(0.48)
JOURN 144 2912
(20.2)
845
(5.87)
277
(1.92)
102
(0.71)
587
(4.08)
309
(2.15)
297
(2.06)
142
(0.97)
248
(1.72)
64
(0.44)
TC 261 5487
(21.0)
2137
(8.19)
673
(2.58)
248
(0.95)
899
(3.44)
735
(2.82)
383
(1.47)
249
(0.97)
264
(1.01)
128
(0.49)
Totals 2590 52679
(20.3)
16063
(6.2)
5576
(2.12)
3338
(1.29)
8608
(3.32)
9040
(3.49)
2688
(1.04)
2063
(0.80)
3017
(1.16)
850
(0.33)

2. Is there an observable difference between IS courses and non-IS courses in terms of the IS criteria?

Integrative Studies courses are expected to emphasize various types of critical thinking, writing on which the instructor comments, oral expression, and the consideration of human diversity. All tenured and tenure-line faculty at UNL were surveyed about their experience of teaching IS courses (if they had done so) and non-IS courses. The survey was intended to establish a baseline comparison of teaching practices in IS and non-IS courses just two years after the initiation of CEP, to gather information on faculty attitudes about Integrative Studies and CEP in general, and to discover faculty attitudes toward and their instructional use of co-curricular activities. We received 420 usable responses, including 152 from faculty who had taught both types of courses and 47 from those who answered questions only on an IS course.

Most of the questions inquired about activities included in the faculty's classes that pertain to the IS goals. A scaling of the responses yielded these comparisons between IS and non-IS classes.


Table 3: Critical Thinking Scale
  N Min. Max. Mean Std. Dev.
IS Course 195 0 1 .85 .14
Non-IS Course 344 0 1 .73 .23

Table 4: Writing Scale
  N Min. Max. Mean Std. Dev.
IS Course 198 0 1 .67 .36
Non-IS Course 356 0 1 .52 .38

Table 5: Oral Expression Scale
  N Min. Max. Mean Std. Dev.
IS Course 199 0 1 .70 .30
Non-IS Course 364 0 1 .50 .38

Table 6a: Diversity Discussed in IS Courses Table 6b: Diversity Discussed in Non-IS Courses
  N Percent   N Percent
No 31 15.6 No 103 28.1
Yes 148 74.4 Yes 191 52.2
Not Applicable 20 10.1 Not Applicable 72 19.7
Total 199 100 Total 366 100

These data reveal a real difference between IS and non-IS classes in the foregrounding of classroom activities and goals connected with Integrative Studies. However, they also indicate that these instructional methods are valued by faculty in many non-IS courses as well. Data on IS classes extracted from the survey (not reproduced here) find a statistically significant difference only on the oral expression scale between classes of less than thirty-one students and those higher than thirty. Non-IS classes show a statistically significant drop in all categories when the enrollment exceeds thirty. Finally, except for the diversity results, which were based on only one question, the data cannot be interpreted to say that IS goals are being met only by some instructors. The most that can be surmised is that teachers of IS classes are taking the criteria seriously and working toward a full integration of IS activities into their classrooms.

The final question on the faculty survey asked for comments about the Comprehensive Education Program, and 138 respondents took the opportunity to reply to that question or to write about other aspects of the survey. These responses cannot serve as a systematic, campus-wide poll on CEP, but the range of answers may well be representative of faculty opinion on the program. A number of faculty respondents seem to feel, in general, that CEP is a positive formulation of UNL's general education requirements, though some favor Essential Studies over Integrative Studies, or the reverse. Several note that CEP matches (or sometimes improves upon) the kind of education they received as undergraduates, while others say that it codifies their own priorities in the classroom. Even those in favor of the program are concerned that it be assessed rigorously. Some seem satisfied with the program itself but decry the complexities or uncertainty of the IS course approval process. Others worry that the aims of its in particular cannot be realized while faculty numbers decrease and class size increases.

At the other end of the spectrum, several respondents dismiss CEP out of hand as a waste of effort and resources. Some see it as an administrative showcase that is, in fact, a change without a difference. Others focus on the "diversity" aspects of the program and see all of CEP as an ideological, "non-academic" incursion into the curriculum or classroom. Several respondents complain of a deleterious effect of CEP requirements on the advising process. Finally, many respondents reveal in their comments or assert openly considerable ignorance of the goals, requirements, and structure of CEP . Taken as a whole, the faculty's comments raise many issues for further assessment and feedback. For example, it would be helpful to have greater insight into the obstacles to Integrative Studies teaching posed by large class sizes, and methods to overcome them. Likewise, ways to streamline CEP advising could be shared, allowing advisors to spend more time on mentoring and other substantive advising functions. Finally, knowledge of the actual nature and goals of the Comprehensive Education Program is not as widespread among faculty (and probably students) as it must be in the future.

3. What has been the experience of students taking Integrative Studies classes?

A survey was sent to 1300 students who had enrolled in an IS course in the Fall of 1996, yielding 646 responses. The survey was intended to gather information about the frequency of activities in the class related to the IS criteria, to elicit students' comments on their learning in and attitudes toward the IS class they took, and to discover information about students' co-curricular activities and their attitudes toward such experiences.

Most of the questions about classroom activities on this survey asked for frequencies, the choices being Never, Rarely, Sometimes, and Often ("Never" assigned a value of 1 and "Often" a value of 4). The results, scaled like those in the faculty survey, are these)

  N Min. Max. Mean Std. Dev
Critical Thinking Scale 641 1.0 3.9 2.39 .62
Writing Scale 643 1.0 4.0 2.70 1.04
Oral Expression Scale 645 1.0 4.0 2.63 .82
Human Diversity Scale 642 1.0 4.0 2.50 1.09

 

All four sets of results fall into the range between "Rarely" and "Sometimes," though the large standard deviations indicate a wide variety of responses. In addition, 67.4% of the students agreed that the course made them more interested in the subject; 77% that they could apply the course learning outside the classroom; and 45.3% that they were more actively engaged in this class than in others they had taken. Scaled comparisons of the student responses (with all answers except for "Never" collapsed into "Yes" answers) with faculty answers (remembering that the classes taken and taught were not identical) resulted in few statistically significant differences. 78.7% of the students rate their IS class as a positive experience, and 52.2% say that they learned more in this class than in others.

The portrait of the students' IS experience, while slightly less positive than the faculty reports, leads to similar conclusions. Integrative Studies is a new program, one with high expectations and rigorous standards. At this point it is not for all students and faculty the intensive educational experience the General Education Planning Committee hoped it would ultimately be, but it is headed in the right direction.

Both the faculty and the students report in their respective surveys that they value highly the potential learning and developmental power of co-curricular activities, but neither group was widely familiar with the Essential Experiences Program, the co-curricular component of CEP. Perhaps more important, neither group reported much experience with a tie between classroom learning and co-curricular activities.

4. What developmental level of the intellectual skills associated with Integrative Studies do present UNL students reveal in their written work?

The Assessment Team made its first direct attempt at outcomes assessment by collecting portfolios of all written work (plus syllabi and assignments) from a sample of students and analyzing them by means of a matrix that represented a developmental model of the Integrative Studies learning objectives that had been developed earlier in the year. Since only two academic years had elapsed since the inception of the Comprehensive Education Program, it was too early to assess the outcomes of the Integrative Studies requirement. Thus the sampling did not focus on students who had taken one or more IS class. Rather we attempted to measure a random selection of students at each class level by the IS learning goals in order to establish a baseline view of the intellectual and writing skills of UNL undergraduates before the full effect of CEP could be expected. The hope was to construct a standard against which future outcomes assessment of IS could be measured.

After a lengthy process of recruitment that including mailings and follow-up phone calls, 121 students agreed to collect their written work for the year. Despite modest incentives and an elaborate support system, however, only 66 portfolios were available for the reading team by semester's end. Moreover, many of those portfolios were incomplete, probably both because students forgot to include some items and because some of their written work was not returned to them. As a result, the materials were both too limited and too inconsistent for us to place serious weight on the outcomes analysis. That is unfortunate, not only because of the resources invested in the project but because the rather intriguing results cannot be used as a baseline for future assessment. Those results include the following.

  1. The very large range of scores and standard deviations for the whole sample in each category would seem to confirm what is already known that the UNL undergraduate student body is quite diverse in preparation, ability, and achievement.
  2. In general there is an observable if not steady progress on the scale used in this assessment from the first-year students to seniors.
  3. The firstyear students in this sample seem generally well prepared to do beginning university-level work. In only one category (intellectual bias) do they fall out of the second quartile into the first quartile, the latter representing a skill level that we think most faculty members would consider inadequate for any university course.
  4. The seniors in this sample have not reached a very high level of intellectual skill in the aggregate. In only three categories does the mean for seniors break into the third quartile (problem solving, research, conclusions), and not even very strongly in those three (though the standard deviations for the seniors are generally higher than for the other classes).

Conclusions and 1997-98 Assessment Plans

  • The definition with which this summary begins applies exclusively to outcomes assessment, but program assessment involves a wider variety of measures. Moreover, any assessment leads to knowledge about the process itself. Here is what we learned last year.
  • The Comprehensive Education Program is evolving into a stable and functional component of undergraduate education at UNL. Colleges and departments have made steady progress in identifying sufficient ES and IS courses to help students move toward graduation, and advisors and the students themselves have taken the requirements seriously.
  • The faculty needs time and continued development support to achieve the full educational potential of all Integrative Studies courses. Even so, both students and faculty seem satisfied overall with their IS classes.
  • The whole university community should engage in an ongoing conversation about the aims and nature of general education at UNL and how the Comprehensive Education Program serves those ends.
  • We do not have a clear enough sense of the learning objectives of the Essential Studies requirement to do a meaningful outcomes assessment. It is not sufficient to assume that taking a course in mathematics, historical studies, or the arts will yield a valuable educational result. Ultimately, college faculties will need to discuss in some depth the desired ends of our distribution requirement.
  • The Academic Affairs office is well positioned to carry out some kinds of assessment activities: large-scale surveys; tracking course offerings, enrollments, and patterns of student fulfillment of requirements; faculty development (in cooperation with the Teaching and Learning Center); and providing information about assessment techniques. Likewise,this level of the university can assemble aggregated data from departments and colleges to create a picture of a whole program such as CEP.
  • The Academic Affairs office, however, is not well positioned to do outcomes assessment of IS or ES courses. Every authority agrees that assessment of student learning is most successfully carried out in focused and planned ways at the department level by faculty actually teaching the students. Such assessment should not be an add-on task for busy faculty but rather a natural outgrowth of their continuing attempt to improve their students' educational experience. That, not the writing of reports, is the true aim of assessment. Thus the greatest impact of assessment should be at the department level, where the goals of CEP are actually achieved and where general education is blended with the specialized study of the major. Reports that go forward through the colleges to upper administration should focus on the interplay of assessment and improvement.

With these conclusions in mind, the Assessment Team has set the following priorities for 1997-98.

  1. The dissemination of information about the Comprehensive Education Program and CEP assessment to as many pertinent faculty as possible (this report being one component of that process).
  2. Conducting focus groups of Integrative Studies course instructors to explore the variety of experiences faculty have had teaching Integrative Studies courses; to discover what faculty do not understand about IS goals; to discover what obstacles exist to the achievement of those goals; and to explore what faculty development efforts are needed.
  3. Piloting student evaluation questions about their experience of critical thinking in the classroom to see whether such questions provide faculty with valuable information for instructional improvement. Disseminating the results of these trials to interested faculty.
  4. Piloting a classroom artifact study, an outcomes assessment of students' written work with an improved rating matrix, and again making the results of the pilot available to faculty.
  5. Seeking and disseminating information about assessment efforts already in practice that bear upon CEP and its goals. Cooperation with other institutions engaged in the assessment of general education.
  6. Conducting a survey of alumni to discover their perception of the UNL undergraduate experience as a preparation for life after graduation (to be used as a baseline for later CEP assessment); a survey of graduating seniors with the same focus; and a survey of first-year students to gauge their educational expectations.

[The full report from which this summary is drawn is available from the offices of the college deans across the campus, the members of the University-Wide Assessment Steering Committee, or the members of the UNL Curriculum Committee.]

University-wide Assessment Contact Information: