Research and Evaluation Priorities
for Distance Education in Nebraska:
A Delphi Study
White Paper for
Distance Education Action Team
Nebraska Network 21 (NN21)
S. Kay Rockwell, Principal Investigator
Final Revision: April 9, 1999
Findings
Two issues were of interest in the findings: the items included on the
instrument, as well as the rankings of the items. Of the 98 items
ultimately included on the instrument, 94 were ranked as important or very
important. The other four were ranked as being neither important
nor unimportant.
As the data were being analyzed, it became apparent the issues for research
and evaluation priorities were, in all likelihood, issues that distance
educators felt were important to address as distance education opportunities
are implemented in more and more situations. In other words, research
and evaluation issues become synonymous with general issues about distance
education that need to be addressed.
Types of research, needs or opportunity assessments
or evaluations needed for planning decisions
Topics in the category focusing on planning decisions help (a) define
objectives for distance education across the state and (b) judge whether
proposed objectives are sufficiently responsive to assessed needs.
Out of the 27 items suggested in the planning decisions category, 15 were
ranked as very important, nine were on the upper side of important, and
three were on the lower side of important (Table 1).
Major interests for research and evaluation focus on identifying how
to better cooperate and coordinate distance education programs, as well
as how to cooperate and coordinate in technology use. As these
things improve, it is important to identify the impact the improved strategy
has on learners.
Collaboration and Coordination
The highest interest is on collaboration among postsecondary institutions.
There is slightly less interest on collaboration between postsecondary
and secondary institutions and even less interest in collaboration between
secondary and elementary schools. It is important to identify funding
formulas that fairly reward all collaborative participants.
Technology coordination among higher education is important, as is connectivity
among these institutions. Technology coordination and connectivity
among the grouping of schools identified as "pods" is also of interest.
There is much less interest in Nebraska Educational Television’s (NET's)
role in this coordination. There appears to be more interest in research
on how the technology is used rather than on the processes needed to coordinate
and connect the technology.
Distance Learners
Several items relate to concern about student success. Understanding
the characteristics of successful distance learners ranks high, along with
potential problems they may have with required equipment. There is
less interest in exploring problems learners have using the technology.
Diagnosing problems learners have with access in terms of time and place
is very important, along with how the required use of technology affects
motivation. Less concern was noted for diagnosing problems with overall
student costs.
It is very important to determine learners expectations for credit courses
and professional improvement and less important to identify expectations
for personal enrichment. There is more interest in ascertaining opportunities
different institutions have to meet learner expectations for obtaining
credit courses than for professional improvement or personal enrichment
needs.
It is very important to determine why potential learners fail to take
advantage of distance learning opportunities. Although not stated
as a marketing issue, these items could be used in marketing research to
develop and refine effective marketing strategies.
Table 1. Types of assessments needed to serve planning decisions
for implementing distance education in Nebraska
| VERY IMPORTANT (x = 1.000 to 1.499a) |
| x |
How important is it to... |
| 1.029 |
...diagnose problems learners have with various distance delivery strategies
(i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in
regard to equipment requirements? |
| 1.130 |
...assess how technology is organized for connectivity among higher
education institutions? |
| 1.145 |
...assess factors that facilitate or inhibit coordination/cooperation
for programming (i.e. course development, mutual course use, credit transfer,
etc.) among higher education institutions? |
| 1.152 |
...identify characteristics of successful distance learners (i.e. self-regulation,
independent inquiry, collaborative tendency, familiarity with computer
tools, etc.)? |
| 1.154 |
...assess how technology is organized for connectivity between K-12
and higher education institutions? |
| 1.168 |
...diagnose problems learners have with various distance delivery strategies
(i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in
regard to access to the learning opportunity at a convenient time? |
| 1.229 |
...assess client expectations for access to various educational opportunities
for obtaining credit courses and/or degrees? |
| 1.245 |
...assess how the technology is coordinated between K-12 and higher
education institutions? |
| 1.277 |
...assess how the technology is coordinated among higher education
institutions? |
| 1.339 |
...assess client expectations for access to various educational opportunities
for professional improvement? |
| 1.340 |
...diagnose problems learners have with various distance delivery strategies
(i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in
regard to access to the learning opportunity at a convenient place? |
| 1.344 |
...diagnose problems learners have with various distance delivery strategies
(i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in
regard to motivation or desire to participate? |
| 1.373 |
...ascertain opportunities that different educational institutions
have to meet the client expectations for obtaining credit courses and/or
degrees in-state, regionally, and nationally? |
| 1.378 |
...assess how technology is organized for connectivity among the pods
across the state? |
| 1.391 |
...ascertain why potential learners fail to take advantage of distance
education offerings? |
| UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a) |
| 1.504 |
...ascertain opportunities that different educational institutions
have to meet the client expectations for professional improvement in-state,
regionally, and nationally? |
| 1.517 |
...assess factors that facilitate or inhibit coordination/cooperation
for programming (i.e. course development, mutual course use, credit transfer,
etc.) between higher education institutions and secondary schools? |
| 1.628 |
...diagnose problems learners have with various distance delivery strategies
(i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in
regard to using the technology required in the delivery process? |
| 1.666 |
...assess client expectations for access to various educational opportunities
for personal enrichment? |
| 1.735 |
...assess how the technology is coordinated among the pods across the
state? |
| 1.856 |
...assess how funding formulas can be changed to fairly reward all
institutions in a collaborative distance education project? |
| 1.888 |
...ascertain opportunities that different educational institutions
have to meet the client expectations for personal enrichment in-state,
regionally, and nationally? |
| 1.897 |
...identify how the different educational institutions are relating
their distance education programming efforts to their institutional mission? |
| 1.984 |
...diagnose problems learners have with various distance delivery strategies
(i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in
regard too overall student costs? |
| LOWER SIDE OF IMPORTANT (x = 2.000 to 2.499a) |
| 2.118 |
...assess factors that facilitate or inhibit coordination/cooperation
for programming (i.e. course development, mutual course use, credit transfer,
etc.) among secondary/elementary units? |
| 2.204 |
...assess how technology is organized for connectivity between NET
and educational organizations? |
| 2.305 |
...assess how the technology is coordinated between NET and educational
organizations? |
aScale:
1 = Very important
2 = Important
3 = Neither important nor unimportant
4 = Unimportant
5 = Very unimportant
|
Types of research or evaluations
needed to serve structuring decisions
Topics in the category focusing on structuring decisions facilitate
designing distance education programs and procedures. Of the 24 items
suggested in the structuring decisions category, nine were ranked as very
important, 10 were on the upper side of important, three on the lower side
of important, and two fell out as neither important nor unimportant (Table
2).
Major interests for research and evaluation appear to concentrate on
effective strategies for successful distance learning experiences, the
support needed from the educational institution and training needs for
distance education teachers.
It is most important to identify strategies that are the highly effective
for a successful distance learning experience B including assessing Innovative
instructional processes to identify what best helps distance students learn.
Determining factors impeding or enhancing the development of a structure
supporting distance education programming is also ranked high.
Also of concern is assessing the financial resources needed and available
for course development, along with the time faculty need to develop and
teach via distance. Determining a cost/benefit ratio ranks slightly
lower.
Training needs and competencies of distance learning teachers are a
great concern. High on the list is identifying the types of support/assistance
instructors need, training needs of teachers and effective teacher competencies.
Sufficient resources are also viewed as important, as well as resource
systems supporting faculty development.
Less important is an assessment of the pros and cons of different learning
models, as well as instructor qualifications for in-state distance learning
courses.
Preference is given to developing models designed primarily for in-state
use. Assessing models designed for regional, national or international
use is viewed as less important. Instructor qualifications for such
regional, national and international models are also viewed as less important.
There is less interest in assessing past courses to identify successes
and failures as part of a continuing education effort for teachers.
It is also viewed as less important to assess past marketing strategies
to identify how to create a market. This is apparently more of an
implementation concern than a structuring concern.
There is little interest in defining and identifying terms relating
to distance education.
Table 2. Types of assessments needed to serve structuring decisions
for distance education in Nebraska
| VERY IMPORTANT (x = 1.000 to 1.499a) |
| x |
How important is it to... |
| 0.971 |
...identify which strategies are the most effective in making a distance
education learning experience successful? |
| 0.987 |
...determine the factors that impede or enhance the development of
a structure that will make distance education work programmatically (i.e.
competing bureaucracies, etc.)? |
| 1.068 |
...identify the kinds of support/assistance that is necessary for individual
instructors to develop courses? |
| 1.324 |
...identify factors that contribute to effective teacher competencies
in distance education? |
| 1.328 |
...assess resources needed as well as those available in regard to
financial costs to an institution? |
| 1.350 |
...assess resources needed as well as those available in regard to
resource systems that support faculty development? |
| 1.351 |
...identify training needs of distance education teachers? |
| 1.372 |
...assess "innovative" instructional processes to identify what best
helps distance students learn? |
| 1.402 |
...assess resources needed as well as those available in regard to
time faculty need to develop and teach distance courses? |
| UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a) |
| 1.608 |
...determine how competitive interests can cooperate or collaborate? |
| 1.622 |
...identify expectations for instructor qualifications required for
distance education courses that are used in-state? |
| 1.626 |
...assess resources needed as well as those available in regard to
cost benefit? |
| 1.673 |
...identify the pros and cons of distance education models that are
designed primarily for national use? |
| 1.689 |
...assess pros and cons of different distant education models or methods
for addressing various learning objectives (pod's, television based, computer
based, telephone based, combinations of the technologies)? |
| 1.693 |
...assess past courses, workshops, or conferences to identify success
and failures for continuing education? |
| 1.697 |
...identify expectations for instructor qualifications required for
distance education courses that are used regionally? |
| 1.705 |
...assess past courses, workshops, or conferences to identify success
and failures for degree programs? |
| 1.711 |
...identify expectations for instructor qualifications required for
distance education courses that are used international use? |
| 1.725 |
...identify expectations for instructor qualifications required for
distance education courses that are used nationally? |
| LOWER SIDE OF IMPORTANT (x = 2.000 to 2.499a) |
| 2.067 |
...assess past marketing strategies to identify how to best create
a market? |
| 2.124 |
...identify pros and cons of distance education models that are designed
primarily for regional use? |
| 2.272 |
...identify the pros and cons of distance education models that are
designed primarily for international use? |
| UPPER SIDE OF NEITHER IMPORTANT NOR UNIMPORTANT (0 =
2.500 to 2.999a) |
| 2.791 |
...identify pros and cons of distance education models that are designed
primarily for in-state use? |
| LOWER SIDE OF NEITHER IMPORTANT NOR UNIMPORTANT (0 =
3.000 to 3.499a) |
| 3.137 |
...to identify and define terms that relate to distance education (i.e.
system, method of delivery, pods, etc.)? |
aScale:
1 = Very important
2 = Important
3 = Neither important nor unimportant
4 = Unimportant
5 = Very unimportant
|
Types of assessments needed to serve decisions relative
to how distance education is being implemented
Once questions about the topics in the category focusing on implementation
are answered, distance education procedures can be monitored, controlled
and refined. Of the 31 items suggested in the implementation category,
12 were ranked as very important, 11 were on the upper side of important,
six on the lower side of important, and two were clustered around neither
important nor unimportant (Table 3).
While there is some difference between what is more or less important
for research or evaluation in the implementation category, this section
seems to focus on a few main themes: learner issues, instructional delivery,
administration and quality control.
The most important issue deals with identifying learners' barriers and
incentives in using distance delivered education. In addition, considerable
emphasis is placed on comparing classroom-based instruction and distance
learning. The most interest is on application and use of content,
followed by knowledge or skills acquired, learning styles, interaction
with other learners and with the instructor and feeling a part of a learning
community.
There is a great deal of interest in identifying what makes collaborative
distance education offerings successful. Specific implementation
issues of concern relative to instructors and instructional processes include
the barriers and incentives for implementing distance delivery by the instructor,
the processes used to customize the educational experience, factors encouraging
educators to work together and the instructors' use of multiple technologies.
Another important issue deals with identifying structures blocking distance
delivery. This appears to be an administrative issue. Other
administrative issues of importance include factors encouraging or discouraging
educators from working together for program development and delivery.
The issue of quality and the maintenance of rigor is listed as an important
item, along with identifying factors influencing how quality is judged
from the perspective of different audiences
Multiple technologies are addressed in three different responses.
The most important relates to learners perceptions about their advantages
and disadvantages, followed closely by an identification of the incentives
and barriers for instructors to incorporate multiple technologies.
Such incorporation within the infrastructure was also listed as a valuable
item.
Of less importance is identifying of the benefits and drawbacks of various
administrative models such as cohort designs, lead instructor with instructors-of-record
and facilitator licensed processes. There is also less interest in
research into how instructors are transferring skills from the traditional
classroom to distance-based instruction.
There is also less interest in comparing traditional face-to-face delivery
with the various distance delivery methods that are television-based, computer-based
and telephone-based.
Table 3. Types of assessments needed to serve decisions relative
to how distance education is being implemented in Nebraska
| VERY IMPORTANT (x = 1.000 to 1.499a) |
| x |
How important is it to... |
| 1.128 |
...identify barriers and incentives for using distance delivery by
the learner? |
| 1.134 |
...identify what makes collaborative distance education offerings successful? |
| 1.142 |
...identify structures that are in place that are blocking delivering
education via distance? |
| 1.181 |
...compare learner reactions of distance delivery to traditional classroom
delivery in terms of being able to apply and use the content? |
| 1.220 |
...identify barriers and incentives for implementing delivery by the
instructor? |
| 1.224 |
...identify processes distance education instructors are using to customize
the educational experience for students along with the benefits and drawbacks
of these process? |
| 1.233 |
...assess learners perceptions about the advantages and disadvantages
of using various technologies? |
| 1.296 |
...compare learner reactions of distance delivery to traditional classroom
delivery in terms of knowledge or skills acquired? |
| 1.344 |
...identify incentives and barriers for incorporating multiple technology
use by the learners? |
| 1.364 |
...identify factors that are encouraging or discouraging educators
to work together for creating new educational structures (i.e. models,
systems, etc.)? |
| 1.389 |
...compare learner reactions of distance delivery to traditional classroom
delivery in terms of learning styles of the learners? |
| 1.441 |
...identify incentives and barriers for incorporating multiple technology
use by the instructor? |
| UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a) |
| 1.519 |
...identify factors that are encouraging or discouraging educators
to work together for program development and delivery? |
| 1.526 |
...compare learner reactions of distance delivery to traditional classroom
delivery in terms of interaction with other learners in the course/workshop? |
| 1.529 |
...compare learner reactions of distance delivery to traditional classroom
delivery in terms of interaction with the instructor? |
| 1.611 |
...identify factors that influence how quality is judged from the perspective
of different audiences such as different institutions, accreditation agencies,
faculty, general public, and students? |
| 1.616 |
...compare learner reactions of distance delivery to traditional classroom
delivery in terms of feeling a part of a learning community? |
| 1.617 |
...identify barriers and incentives for implementing delivery by the
technical support people? |
| 1.637 |
...identify indicators that are useful for controlling quality and
maintaining rigor? |
| 1.655 |
...identify what is being done to control quality and maintain rigor? |
| 1.677 |
...identify incentives and barriers for incorporating multiple technology
use within the infrastructure. |
| 1.850 |
...identify barriers and incentives for implementing delivery by the
administration? |
| 1.931 |
...identify how instructors teaching a collaborative offering interact
with learners from other institutions? |
| LOWER SIDE OF IMPORTANT (x = 2.000 to 2.499a) |
| 2.042 |
...research how instructors are transferring skills from using traditional
campus-based instruction to using distance-based instruction? |
| 2.288 |
...compare traditional face-to-face delivery with multi-media processes? |
| 2.289 |
...compare the benefits and drawbacks of various administrative models
such as? |
| 2.367 |
...compare the benefits and drawbacks of various administrative models
such as lead instructor with instructors-of-record at sites? |
| 2.410 |
...compare traditional face-to-face with computer-based (Lotus Notes,
etc.)? |
| 2.467 |
...compare the benefits and drawbacks of various administrative models
such as facilitator licensed processes? |
| UPPER SIDE OF NEITHER IMPORTANT NOR UNIMPORTANT (x =
2.500 to 2.999a) |
| 2.664 |
...compare traditional face-to-face delivery with television-based? |
| LOWER SIDE OF NEITHER IMPORTANT NOR UNIMPORTANT (x =
3.000 to 3.499a) |
| 3.120 |
...compare traditional face-to-face delivery with telephone-based? |
aScale:
1 = Very important
2 = Important
3 = Neither important nor unimportant
4 = Unimportant
5 = Very unimportant
|
Types of evaluations needed to serve decisions
relative to outcomes, or impacts that distance education
is having
Topics in the category focusing on outcomes of distance education
are important in judging program attainments. Of the seven items
suggested in the category, two were ranked as very important, three were
on the upper side of important and two on the lower side of important (Table
4).
Major interests for evaluation focused on assessing outcomes in formal
higher education courses followed by outcomes reached in K-12. There
is less interest in assessing outcomes of non-formal professional and personal
growth workshops.
Documenting participation and completion rates was viewed as important,
as was identifying effective and fair teacher evaluation processes.
There is less interest in studying the maturation of distance instruction,
and in conducting meta-analysis of the research on different types of distance
delivery modes.
Table 4. Types of evaluations needed to serve decisions relative
to the outcomes of distance education.
| VERY IMPORTANT (x = 1.000 to 1.499a) |
| x |
How important is it to... |
| 0.885 |
...assess how well instructional outcomes are reached in K-12? |
| 1.097 |
...assess how well instructional outcomes are reached in formal higher
education courses? |
| UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a) |
| 1.599 |
...document participation and completion rates? |
| 1.705 |
...assess how well instructional outcomes are reached in non-formal
professional and personal growth workshops (i.e. seminars, video-conferences,
etc.)? |
| 1.791 |
...identify effective and fair teacher evaluation processes? |
| LOWER SIDE OF IMPORTANT (0 = 2.000 to 2.499a) |
| 2.250 |
...study the maturation of distance instruction as instructors design
and deliver educational processed for distance delivery? |
| 2.467 |
...conduct a meta-analysis of the research on different types of distance
delivery modes? |
aScale:
1 = Very important
2 = Important
3 = Neither important nor unimportant
4 = Unimportant
5 = Very unimportant
|
Types of research needed about education in general
Topics in the category focusing on education in general will help the
general educational context within which distance delivery is emerging.
Of the nine items suggested in the category, five were ranked as very important,
three were on the upper side of important, and one was on the lower side
of important (Table 5).
The highest ranked item was assessing how to include training
on adult education theory and practice so distance education instructors
become more action-oriented. Other issues seen as very important
focus on the learner. Identifying if distance education creates changes
in the learning process is followed closely by studying how the change
process is managed by students and identifying how distance education can
facilitate lifelong learning.
Creating a long-term vision about educational systems is very important,
followed by integrating distance education into strategic plans.
Studying how the change process is managed by faculty is more important
than studying how it is managed by administration. Determining if
felt needs are met is also important.
Table 5. Research is needed about education in general
| VERY IMPORTANT (x = 1.000 to 1.499a) |
| x |
How important is it to... |
| 1.104 |
...assess how to include training for faculty to learn about adult
education theory and practice so the distance education instructors become
more action oriented? |
| 1.267 |
...identify if distance education creates changes in the learning process?
If so, how? |
| 1.308 |
...study how the change process is managed by students? |
| 1.359 |
...identify how distance education can facilitate lifelong learning? |
| 1.440 |
...create long-term vision about educational systems that will serve
Nebraskans in 2020? |
| UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a) |
| 1.773 |
...assess how institution are integrating distance education into their
strategic plan? |
| 1.879 |
...study how the change process is managed by faculty? |
| 1.939 |
...find methodologies that will help determine if the education programs
meet felt needs? |
| LOWER SIDE OF IMPORTANT (x = 2.000 to 2.499a) |
| 2.043 |
...study how the change process is managed by administration? |
aScale:
1 = Very important
2 = Important
3 = Neither important nor unimportant
4 = Unimportant
5 = Very unimportant
|
|
