Research and Evaluation Priorities
for Distance Education in Nebraska:
A Delphi Study
White Paper for
Distance Education Action Team
Nebraska Network 21 (NN21)
S. Kay Rockwell, Principal Investigator
Final Revision: April 9, 1999
Appendix 1: Delphi Survey, Second Round
Research and Evaluation Needs for Distance Education
ID# _____
This is the second and final round for the Delphi study focusing
on research and evaluation priorities in Distance Education for Educational
Institutions in Nebraska.
Below are new items suggested by those who responded to the first
instrument. Please rank how important it is to conduct research or evaluation
on items A1 through E1 according to the following scale:
Scale:
VI = Very Important
I = Important
N = Nice to know but neither important nor unimportant
U = Unimportant
VU = Very Unimportant
A1. Assess how funding formulas can be changed to fairly reward all institutions
in a collaborative distance education project.
VI
I N U
VU
Comment:
B1. Identify training needs of distance education teachers.
VI
I N U
VU
Comment:
B2. Identify factors that contribute to effective teacher competencies
in distance education.
VI
I N U
VU
Comment:
B3. Identify the kinds of support/assistance that is necessary for individual
instructors to develop courses.
VI
I N U
VU
Comment:
B4. Assess "innovative" instructional processes to identify what best
helps distance students learn.
VI
I N U
VU
Comment:
C1. Identify what makes collaborative distance education offerings successful.
VI
I N U
VU
Comment:
C2. Identify how instructors teaching a collaborative offering interact
with learners from other institutions.
VI
I N U
VU
Comment:
D1. Identify effective and fair teacher evaluation processes.
VI
I N U
VU
Comment:
D2. Conduct a meta-analysis of the research on different types of distance
delivery modes.
VI
I N U
VU
Comment:
E1. Assess how to include training for faculty to learn about adult
education theory and practice so the distance education instructors become
more action oriented.
VI
I N U
VU
Comment:
Below are the items that were listed in the first Delphi round. Each
item is again listed with the mean noted. In some cases, stems of the original
items have been altered for clarification. On this round, please decide
if you agree or disagree with the mean. Please assess if the item's mean:
MI = should reflect More Importance
A = is an Accurate representation of importance
LI = should reflect Less Importance
Please circle the appropriate response to each item.
A. What types of assessments, are needed to serve planning
decisions for implementing distance education in Nebraska?
Note: Topics in this category will help (a) define objectives
for
distance education across the state and (b) judge whether
proposed objectives are sufficiently responsive to the
assessed needs.
How important is it to . . .
1. Assess how technology is organized for connectivity :
-Among the pods across the state
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Between NET and educational organizations
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Between K-12 and higher education institutions
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
-Among higher education institutions
Mean = VI x . . . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
2. Assess how the technology is coordinated:
-Among the pods across the state
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Between NET and educational organizations
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Between K-12 and higher education institutions
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Among higher education institutions
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
3. Assess factors that facilitate or inhibit coordination/cooperation
for programming (i.e. course development, mutual course use, credit
transfer, etc.) . . .
-Among higher education institutions
Mean = VI . x . . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Among secondary/elementary units
Mean = VI . . . . . . . . . xI . . . . . . . . . N . . . . . . . .
. U . . . . . . . . . VU
MI A
LI
-Between higher education institutions and secondary schools
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
4. Assess client expectations for access to various educational
opportunities for . . .
-obtaining credit courses and/or degrees
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-professional improvement
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
- personal enrichment
Mean = VI . . . . . . . . . xI . . . . . . . . . N . . . . . . . .
. U . . . . . . . . . VU
MI A
LI
Comment:
5. Ascertain the opportunities that different educational
institutions have to meet the client expectations for . . .
-obtaining credit courses and/or degrees in-state, regionally,
and nationally
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-professional improvement in-state, regionally, and nationally
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
- personal enrichment in-state, regionally, and nationally
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
6. Identify how the different educational institutions are relating
their distance education programming efforts to their institutional mission.
Mean = VI . . . . . . . . . I . . x . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
7. Diagnose problems learners have with various distance
delivery strategies (i.e. interactive TV, audio-conference, web-based,
multi-media, etc.) in regard to . . .
-equipment requirements
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-access to the learning opportunity at a convenient time
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-access to the learning opportunity at a convenient place
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-overall student costs
Mean = VI . . . . . . . . . I . x . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-motivation or desire to participate
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-using the technology required in the delivery process
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
8. Ascertain why potential learners fail to take advantage of
distance education offerings.
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
9. Identify characteristics of successful distance learners
(i.e. self-regulation, independent inquiry, collaborative tendency, familiarity
with computer tools, etc.)
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
B. What types of assessments are needed to serve structuring decisions
for distance education in Nebraska?
Note: Topics in this category would facilitate designing
distance
education
programs and procedures.
How important is it to . . .
10. Assess resources needed as well as those available in regard
to:
-Resource systems that support faculty development?
Mean = VI . . x . . . . . . I . . . . . . . . . N . .
. . . . . . . U . . . . . . . . . VU
MI A
LI
-Time faculty need to develop and teach distance courses?
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
- Financial costs to an institution?
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Cost benefit?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
11. Identify and define terms that relate to distance education
(i.e. system, method of delivery, pods, etc.)?
Mean = VI . . . . . . . . . I . x . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
12. Assess the pros and cons of different distance education models
or methods for addressing various learning objectives . . .
-POD’s
-Television-based
-Computer based
-Telephone based
-Combinations of the technologies
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
13. Identify which strategies are the most effective in making
a distance education learning experience successful.
Mean = VI . x . . . . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
14. Identify the pros and cons of distance education models that
are designed primarily for . . .
-in-state use?
Mean = VI . . . . . . . . . I . x . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-regional use?
Mean = VI . . . . . . . . . I . . x . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-national use?
Mean = VI . . . . . . . . . I . . . . x . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-international use?
Mean = VI . . . . . . . . . I . . . . . . . x . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
15. Identify expectations for instructor qualifications required
for distance education courses that are used . . .
-in-state?
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-regionally?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-nationally?
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-internationally?
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
16. Determine the factors that impede or enhance the development
of a structure that will make distance education work programmatically
(i.e. competing bureaucracies, etc.)?
Mean = VI . . . x . . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
17. Assess past courses, workshops, or conferences to identify
success and failures for . . .
-degree programs?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-continuing education?
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
18. Assess past marketing strategies to identify how to best create
a market?
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
19. Determine how competitive interests can cooperate or collaborate?
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
C. What types of assessments are needed to serve decisions relative
to how distance education is being implemented in Nebraska?
Note: Once questions are answered in this category, procedures
can be monitored, controlled, and refined.
How important is it to . . .
20. Compare traditional face-to-face delivery with various distance
delivery methods?
-face-to-face with computer-based (Lotus Notes, etc.)
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-face-to-face with television-based
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-face-to-face with telephone-based
Mean = VI . . . . . . . . . I . . . x . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-face-to-face with multi-media processes
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
21. Identify processes distance education instructors are using
to customize the educational experience for students along with the benefits
and drawbacks of these processes.
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
22. Assess the learners perceptions about the advantages and disadvantages
of using various technologies.
Mean = VI . . . x . . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
23. Compare learner reactions of distance delivery to traditional
classroom delivery in terms of . . .
- knowledge or skills acquired
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-interaction with the instructor
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . .
. . . U . . . . . . . . . VU
MI A
LI
-interaction with other learners in the course/workshop
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-feeling a part of a learning community
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-being able to apply and use the content
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-learning styles of the learners
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
24. Identify factors that influence how quality is judged from
the perspective of different audiences such as different institutions,
accreditation agencies, faculty, general public, and students.
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
25. Identify what is being done to control quality and maintain
rigor?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
26. Identify indicators that are useful for controlling quality
and maintaining rigor?
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
27. Identify factors that are encouraging or discouraging educators
to work together for. . .
-program development and delivery
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-creating new educational structures (i.e. models, systems, etc.)
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
28. Identify barriers and incentives for using distance delivery
by the learner?
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
29. Identify barriers and incentives for implementing delivery
by the . . .
- instructor?
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
- administration?
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
- technical support people?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
30. Identify structures that are in place that are blocking delivering
education via distance?
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
31. Identify incentives and barriers for incorporating multiple
technology use . . .
-within the infrastructure
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-by the instructor
Mean = VI . . . . x . . . . I . . . . . . . . . N . .
. . . . . . . U . . . . . . . . . VU
MI A
LI
- by the learners
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
32. Compare the benefits and drawbacks of various administrative
models such as . . .
-lead instructor with instructors-of-record at sites
Mean = VI . . . . . . . . . xI . . . . . . . . . N . . . . . . . .
. U . . . . . . . . . VU
MI A
LI
-cohort designs
Mean = VI . . . . . . . . . xI . . . . . . . . . N . . . . . . . .
. U . . . . . . . . . VU
MI A
LI
-facilitator licensed processes
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
33. Research how instructors are transferring skills from using
traditional campus-based instruction to using distance-based instruction.
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
D. What types of evaluations are needed to serve decisions
relative to the outcomes, or impacts that distance education is having
in Nebraska?
Note: Questions in this category are important in judging
program attainments.
How important is it to . . .
34. Document participation and completion rates.
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
35. Assess how well instructional outcomes are reached in . .
.
-K-12
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Formal higher education courses
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-Non-formal professional and personal growth workshops (i.e. seminars,
video-conferences, etc.)
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
Comment:
36. Study the maturation of distance instruction as instructors
design and deliver educational processes for distance delivery.
Mean = VI . . . . . . . . . I . . x . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
E. What types of research is needed about education in general?
Note: Topics in this category will help the general educational
context within which distance delivery is emerging.
How important is it to . . .
37. Create a long-term vision about educational systems that will
serve Nebraskans in 2020?
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
38. Assess how institutions are integrating distance education
into their strategic plans?
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
39. Find methodologies that will help determine if the education
programs meet felt needs.
Mean = VI . . . . . . . . . I . x . . . . . . . N . .
. . . . . . . U . . . . . . . . . VU
MI A
LI
Comment:
40. Identify if distance education creates changes in the learning
process? If so, how?
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
41. Identify how distance education can facilitate lifelong learning?
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . .
. . . . U . . . . . . . . . VU
MI A
LI
Comment:
42. Study how the change process is managed by . . .
-students
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-faculty
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . .
U . . . . . . . . . VU
MI A
LI
-administration
Mean = VI . . . . . . . . . I x . . . . . . . . N . .
. . . . . . . U . . . . . . . . . VU
MI A
LI
Comment:
|
