Full text in one page (very long, takes time to load, but easiest for printing)

 Abstract

Research Summary, Conclusions, and Challenges

Introduction

Related Literature

Distance education research
Evaluation approaches
Models to assess priorities for distance education research and evaluation

Methods and Procedures

CIPP framework
The Delphi process
Data interpretation

- Types of research, needs or opportunity assessments, or evaluations needed for planning decisions
- Types of research, or evaluations needed to serve structuring decisions
- Types of assessments needed to serve decisions relative to how distance education is being implemented in Nebraska
- Types of evaluations needed to serve decisions relative to outcomes, or impacts that distance education is having in Nebraska
- Types of research need about education in general

References Cited

Appendix 1: Delphi survey, second round

Appendix 2: Steering committee and authors
 
 

To provide a strong philosophical and policy basis for developing distance education opportunities, distance educators across Nebraska identified and ranked future research and evaluation needs/issues for distance education.  In the area of planning, major interests focused on identifying ways to promote better cooperation among institutions so both technology utilization and distance education programming can be coordinated more effectively.  As the coordination and cooperation for programming and technology use improve, it is important to identify the impact the improved strategy has on learners.  In the area of structuring, major interests appear to concentrate on effective strategies for successful distance learning experiences; the support needed from the educational institution; and, training needs for distance education teachers.  In the area of implementation, the main themes focused on learner issues, instructional delivery, administration and quality control.  In the area of outcome needs, major interests focused on assessing outcomes in formal higher education courses and K-12.  There is less interest in assessing outcomes of non-formal professional and personal growth workshops.   Documenting participation and completion rates were viewed as important, as was identifying effective and fair teacher evaluation processes.  In the area of general education, it is seen as very important to assess how to include training on adult education theory and practice so the distance education instructors become more action-oriented, to identify if distance education creates changes in the learning process, to study how the change process is managed by students, and to identify how distance education can facilitate lifelong learning.  Creating a long-term vision about educational systems is very important as is integrating distance education into strategic plans. 
 
 

In September, 1997, the Nebraska Network 21 (NN21) Distance Education Action Team embarked on the task to provide a strong philosophical and policy basis for developing distance education opportunities in Nebraska.  Basic to this is a solid research foundation focusing on Nebraska’s current needs and issues in distance education.  This research is needed to provide the context within which distance education is developing along with the inputs that are needed to effectively implement distance education opportunities.  As steps are identified and actions are taken to implement distance education, evaluations are necessary so adjustments can be made, and outcomes need to be assessed to find out if the state’s educational needs are being met.

While distance education research and evaluation is emerging across the country, a number of questions are being asked by decision makers in Nebraska.  These questions include:

     What are high priority research and evaluation needs in Nebraska’s educational institutions?
     What kind of collaborations need to be developed to implement distance learning activities economically?
     1. How can educational institutions work together to assess the effectiveness of distance learning opportunities?
     What kind of evaluation processes are needed as institutions work across state boundaries?  national
         boundaries? international boundaries?
     2. What accountability issues need to be addressed for decision makers?

Because there are numerous research and evaluation needs on distance education, priorities need to be established to provide some guidelines for Nebraska researchers/evaluators.  Current research can be directed toward these priorities and proposed research can focus on state-wide needs.  Likewise, evaluations can be directed toward these priorities and evaluations that cut across institutional boundaries can be developed. 

The goal of this paper is to identify research and evaluation priorities for distance education in Nebraska.  Specifically, it focuses on research that helps with planning and structuring decisions as distance education is developing, and evaluations needed to assess the implementation processes and document outcomes.

The study design used a modified Delphi process to help understand distance educators' perceptions about research and evaluation priorities for distance education in Nebraska in 1998.   A five-person steering committee representing local, national and international distance education interests identified relevant topics/issues for a survey instrument.  A 43-person Delphi panel that represented distance educators from Nebraska and surrounding states ranked 98 research and evaluation issues twice.  After discussing the findings at a state-wide distance education conference, 14 participants again ranked all 98 items to establish the research and evaluation priorities. 
 
 

Planning Decisions

Collaboration and Coordination:  The highest interest is in collaboration among postsecondary institutions; there is slightly less interest in collaboration between postsecondary and secondary institutions and even less between secondary and elementary schools.  However, in collaborative efforts, it is important to identify funding formulas that fairly reward all institutional participants. 

Technology coordination within higher education is important, as is connectivity among these institutions.  Technology coordination and connectivity among a grouping of schools identified as pods is also of interest.  There is much less interest in Nebraska Educational Television’s (NET's) role in this coordination.   There appears to be more interest in research on how the technology is used rather than on the processes needed to coordinate and connect the technology.

Distance Learners:  Understanding the characteristics of successful distance learners ranks high, along with potential problems they may have with required equipment.  There is less interest in exploring problems learners have using the technology.

Diagnosing problems learners have with access in terms of time and place is very important, along with how the required use of technology affects their motivation.  Less concern was noted for diagnosing problems with overall student costs.

It is very important to determine learners expectations for credit courses and professional improvement and less important to identify expectations for personal enrichment.  There is more interest in ascertaining opportunities different institutions have to meet learner expectations for obtaining credit courses than for professional improvement or personal enrichment needs. 

It is very important to determine why potential learners fail to take advantage of distance learning opportunities.
 

Structuring Decisions

It is most important to identify strategies that are highly effective for a successful distance learning experience B  including assessing innovative instructional processes to identify what best helps distance students learn.  Determining factors impeding or enhancing the development of a distance education support structure also ranked high.  Also of concern is assessing the financial resources needed and available for course development, along with the time faculty need to develop and teach via distance.  Determining a cost/benefit ratio ranks slightly lower.

Distance education teacher competencies are a great concern.  Research identifying effective teacher competencies ranks very high, as does teacher training needs.  Also important is identifying the types of support/assistance instructors need. 

Less important is an assessment of the pros and cons of different learning models, as well as instructor qualifications for in-state distance learning courses. 

Preference is given to developing models for in-state use.  Assessing models designed for regional, national or international use is viewed as less important. Along with this, instructor qualifications for such regional, national and international models are viewed as less important.

There is less interest in assessing past courses to identify successes and failures as part of a continuing education effort for teachers.  It is also viewed as less important to assess past marketing strategies to identify how to create a market.  There is little interest in defining and identifying terms relating to distance education.
 

Implementation Issues

The most important issue deals with identifying barriers and incentives for using distance delivered education.  In addition, considerable emphasis is placed on comparing classroom-based instruction and distance learning.  The most interest is on application and use of content, followed by knowledge or skills acquired,  learning styles, interaction with other learners and with the instructor, and feeling a part of a learning community. 

There is a great deal of interest in identifying what makes successful collaborative distance education.  Specific implementation issues of concern, relative to instructors and instructional processes, include the barriers and incentives for implementing distance delivery by the instructor, the processes used to customize the educational experience, factors encouraging  educators to work together and the instructors' use of multiple technologies. 

Another important issue deals with identification of current structures blocking distance delivery.  Other administrative issues of importance include factors encouraging or discouraging educators from working together for program development and delivery.  The issue of quality and the maintenance of rigor is important, along with identifying factors that influence how quality is judged from the audience’s perspective.

Addressing how multiple technologies can be used is important, specifically as they relate to learners' perceptions about the advantages and disadvantages, the incentives and barriers for instructors to incorporate multiple technologies and incorporating multiple technology use within the infrastructure.

Of less importance is identifying the benefits and drawbacks of various administrative models, such as cohort designs, lead instructor with instructors-of-record and facilitator licensed processes.  There is also less interest in research into how instructors are transferring skills from the traditional classroom to distance-based instruction.

 There is also less interest in comparing traditional face-to-face delivery with the various distance delivery methods that are television-based, computer-based and telephone-based.

Outcome Needs

Assessing outcomes in formal higher education is very important, as are outcomes reached in K-12.  There is less interest in assessing outcomes of non-formal professional and personal growth workshops. 

Documenting participation and completion rates was viewed as important, as was identifying effective and fair teacher evaluation processes.

There is less interest in studying the maturation of distance instruction and in conducting a meta-analysis of the research on different types of distance delivery modes. 
 

General Education

It’s very important to assess how to include training on adult education theory and practice so distance education instructors become more action-oriented, to identify if distance education creates changes in the learning process, to study how the change process is managed by students and to identify how distance education can facilitate lifelong learning. 

Creating a long-term vision about educational systems is very important, as is integrating distance education into strategic plans.  Studying how the change process is managed by faculty is more important than studying how it is managed by administration.  Determining if perceived  needs are met is also important.
 
 

Important research and evaluation needs for distance education focus on four major themes:

• Cooperation and collaboration among institutions.  To create a cooperative and collaborative environment among institutions across the state, it will be important to create a long-term vision about the educational system so distance education can be integrated into strategic plans.  The emphasis needs to focus on working relationships among higher education institutions to promote inter-campus cooperation, including connectivity among the institutions and coordination of the technology.  It will be important to determine factors that impede or enhance the development of a structure that supports distance education programming, and identifies funding formulas that fairly reward all institutional participants in the collaborative process. 

 

 
 
 

Secondarily, the research needs to focus on the interface between postsecondary institutions and secondary schools with later research on the interface between secondary and elementary schools. 
 

• Designing the educational experience for the distance learner.  Understanding the characteristics of successful distance learners is necessary to identify strategies highly effective for a successful distance learning experience.  It will be important to study if learning occurs differently via distance.  It is necessary to identify processes teachers use to customize the learning experience and assess innovative instructional processes to identify which best help students learn.

 

 
 
 

It will be important to diagnose problems learners encounter with access in terms of time and place, along with how the use of technology affects their motivation. 

It is important to ascertain student expectations for credit courses and why they fail to take advantage of distance learning opportunities.  It is less important to ascertain expectations for professional improvement or personal enrichment.
 

• Teacher preparation.  Emphasis needs to be placed on identifying effective teacher competencies, along with their training needs that support faculty development.  It is necessary to identify the amount and type of support or assistance teachers need, as well as resources required for various distance teaching approaches and course development.

 

 
 
 

It will be important to identify barriers and incentives for instructors to teach via distance, how instructors can be encouraged to work together, how instructors can most effectively use multiple technologies, how rigor is maintained and how teaching quality is judged by different audiences.  It will be important to identify effective and fair teacher evaluations. 
 

• Educational outcomes.  It will be important to determine participation and completion rates and assess outcomes of formal higher education and K-12.  It is less important to assess outcomes of non-formal professional and personal growth workshops.  However, it will be important to determine if perceived educational needs are met and if distance education facilitates lifelong learning.

In September, 1997, the Nebraska Network 21 (NN21) Distance Education Action Team embarked on the task of providing a strong philosophical and policy basis for developing distance education opportunities in Nebraska.  Basic to this is a strong research foundation  focusing on the current needs and issues in distance education across the state.  This research needs to focus on the context within which distance education is developing along with the inputs needed to effectively implement distance education.  As steps are identified and actions are taken to implement distance opportunities, these processes need to be evaluated so adjustments can be made, and outcomes need to be assessed to find out if educational needs across the state are being met.

While research focusing on distance learning is emerging as researchers across the country address specific issues of interest, a number of questions are being asked by decision makers in Nebraska.  These questions include:

     What are high priority research and evaluation needs in Nebraska’s educational institutions?
     What kind of collaborations need to be developed to implement distance learning activities economically?
     How can educational institutions work together to assess the effectiveness of distance learning opportunities?
     What kind of evaluation processes are needed as institutions work across state boundaries? 
          national boundaries?  international boundaries?
     What accountability issues need to be addressed for decision makers?

Because there are numerous research and evaluation needs in distance education, priorities need to be established to provide some guidelines for Nebraska researchers/evaluators.  Current research can be directed toward these priorities and proposed research can focus on state-wide needs.  Likewise, evaluations can be directed toward these priorities and evaluations that cut across institutional boundaries can be developed. 

The goal of this paper is to identify research and evaluation priorities for distance education in Nebraska.  Specifically, it will focus on:

• Research targeted toward the context within which distance education is developing. 
• Research targeted toward inputs required for distance education.
• Evaluation needs that help assess the implementation process.
• Evaluation needs in documenting outcomes.

While the paper focuses on the topic of distance education, the process used to identify the research and evaluation priorities is also of importance.  The literature base was explored for relevant research about distance education, appropriate evaluation approaches and methods used to assess priorities for issues such as distance education. 
 

Distance Education Research

"Distance education" in this paper means education which occurs when the learner and the instructor are in different locations.  Distance education correspondence courses have been used for decades.  Now methodologies are expanding to include new technological advances such as audio- and tele-conferences, satellites, Internet, video and audio tapes and multimedia.  Educational experiences in which instructors travel to distant locations to teach face-to-face in a classroom setting are not considered as distance education.

The literature review and study of prior research were conducted through a review of CRIS and ERIC listings, refereed journals, books and university and government documents.  These sources produced an abundance of articles concerning distance education course presentations and media methods, evaluations of courses and methods along with student outcomes associated with these and students' attitudes toward distance education.

Distance Education Context.  Distance education established its roots as a form of instruction at least 150 years ago (Holmberg, 1986).  An early form of distance education was correspondence study.  As more sophisticated methods and media became available, distance education advanced to audio recordings and educational television programs delivered via satellite or fiber optics.   The Internet has opened the door for computerized courses, as well as supplemental processes  enhancing televised methods.  Distance education underwent other changes in practices, programs and definitions when distance education universities evolved, such as the emergence of Great Britain’s Open University. 

Rapid changes in technology have changed distance education.  New policies are being established that will determine how distance education is employed and used.  The growth and impact of distance education and the opportunities it offers are directly linked to the availability of new technologies.  As technology brings distant sites into an electronic web of information, people throughout the world are pulled together, and a demand for distance education opportunities is seen worldwide (Thach and Murphy, 1994, Hanson et al., 1996).

While the distance education environment is changing, many questions remain unanswered.  These questions concern definitions and theories of how to practice distance education in a collaborative environment.  In this new educational paradigm, research is needed to guide practice in the distance education movement.

 Distance Education Research.  Miller (1993) states distance education has entered a particularly important stage in its development.  He notes four long-term trends, including: 

The simultaneous diversification and convergence of technologies (live, interactive media, computer conferencing, digital technologies). 
Changing relationships with students (learning communities, student interaction, group and individual instruction and empowerment). 
Changing relationships among institutions (consortia, networks).
Educational adaptation (higher education adapting distance education to currents of social change).

Holmberg (1987) suggested the structure or categories of distance education research include:

Philosophy and theory of distance education.

• Distance students, their milieu, conditions and study motivations.
• Subject-matter presentation.
• Communication and interaction between students and their supporting organization (tutors, counselors, administrators, other students).
• Administration and organization.
• Economics.
• Systems (comparative distance education, typologies, evaluation, etc.).
• History of distance education. 

Some research is emerging in the category of administration and organization because administration and management consider the issues and construct the institutional policies that provide structure for successful distance education programs.  The operating practices of a distance education establishment are based on the educational philosophy of the institution, as well as its economic and political restrictions (Verduin and Clark, 1991).  Operational issues occur at all levels of distance education enterprises: local, state, national and international.  As students and teachers are connected across borders, international and national issues surface, as well as state and local issues.  Various management and administrative bodies should consider the issues and construct policies designed to facilitate effective solutions in concert with political and economic policy-making agendas (Colles, Veen and DeVries, 1993). 

 Operational issues include networking, cooperation, coordination and collaboration.  Thach and Murphy (1994) discuss the continuum of collaboration from a local level to an international level, and from student-to-student, class-to-class, institution-to-institution.  The concept of collaboration brings changes to the institution which include changes in structure, policy, faculty reward and skill requirements B elements of autonomy not easily relinquished by individual partners.  Schlosser and Anderson (1994) note that each partner has its own aims, goals and objectives, as well as its own culture of academia.  Roger and Whetten (1982) stress inter-organizational coordination.  Although organizations would prefer to maintain their autonomy, as the environment becomes more complex, organizations become more specialized and this increased specialization leads to a greater need for increased intra-organizational coordination.  With distance education efforts moving toward increased collaboration, policy research is needed to aid decision-making.

Moore (1994) identifies administrative and operational barrier issues to distance education at the federal, regional, state and institutional levels.  These administrative barriers to distance education relate to funding and monitoring processes based on evaluation or research and address accreditation needs.  Further administrative issues are: institutional policy, administrative structures and procedures, institutional support to faculty and students, tuition payment, faculty promotion and tenure, problems of territoriality, ways of rewarding institutions for collaborating and reforming policies with regard to faculty.

Operational issues also involve polices.  Concerns which need to be considered as the demand for distance education increases include academic policy, faculty development and program delivery (Willis, 1989).  Other issues of policy concern include: developing academic policies and procedures for statewide delivery of courses related to credit; telecommunication transmission costs and student support services; resolving faculty development issues (time and workload); resolving instructional development issues (course development and evaluation); developing policies and criteria for the evaluation, selection and use of compatible hardware; exploring external sources for funding distance delivered courses and developing distance education courses for system-wide delivery.  Other concerns focus on "traditional" verses "non-traditional" policy as related to credit transfer, faculty workload and cost sharing for instructional/faculty development. 

From this, Moore (1993) concludes the future of distance education depends on new forms of organization which will reorganize educational resources into a "total delivery system. Educators, administrators and policy makers have yet to come to terms with impact that redistributing educational resources through distance education could unleash.  To do this, teams within and outside of the institution, locally, nationally and internationally will need to be involved. 

Both The American Council on Education and The Alliance: An Association for Alternative Programs for Adults (1996), focus on what they call "Guiding Principles to Distance Learning." One of these principles is organizational commitment.  The principle states that distance-learning initiatives must be backed by an organizational commitment to quality and effectiveness in all aspects of the learning environment.  From this comes nine subprincipals:
 

• Policy is integrated into the mission of the provider.
• The provider makes a commitment to supporting faculty and learner services.
• Support systems are compatible with the learning delivery system.
• Curricular and administrative policies incorporate distance learners' needs. 
• The provider commits to researching and developing distance learning.
• The provider supports faculty and staff with development and resources.
• The provider includes distance learning in staff promotions and funding.
• All aspects of distance learning are regularly evaluated.
• The provider does not distinguish between distance learning and other means of learning in recognizing achievement.

• New forms of assessment and evaluation, including means to insure that the student’s work is original and authentic.
• A set of nationally accepted institutional accreditation standards to insure the quality of distance education.
• A nationally accepted set of teacher certification standards which meet a minimum criterion, including training in distance education theory, methods, and strategies.
• The need for cooperation among business, government, and education sectors.
• Technology training and accessibility for all, not just for progressive students and teachers.

 A report by the Social and Economic Sciences Research Center (SESRC) at Washington State University in Pullman, Washington (Dillman et al., 1995) summarizes the results of a national survey on higher education.  The major findings were: lifelong learning has become a reality for most Americans; getting educated once is not enough in our knowledge-based economy; teaching conducted only in the traditional campus classroom will not meet the public's demand for tailored educational services; distance education strategies have the potential to overcome significant barriers to lifelong learning; although lifelong learning is a reality for most Americans, some people are losing out; and public support exists for universities, and land grant institutions in particular to do more than educate 18-22-year-old undergraduates.  The implications of this survey for institutions of higher education are: colleges and universities have more potential customers than in the past; they need to change how they do business for meeting the demand for lifelong learning, including rewarding faculty for assuming new responsibilities for off-campus instruction and developing new curricula; distance education methods offer one means of meeting the demands for lifelong learning through technological developments and telecommunications and providing access to this education will require new policy measures. 

The authors conclude by stating that public support does exist for universities, particularly for land grant institutions, to do more than just provide undergraduate education.  Areas of education identified as important were off-campus education, education for returning students and applied research on problems.  Institutions of higher education will meet these needs if continuing and distance education are brought to the core of the universities and colleges, if faculty are rewarded and respected for assuming the new responsibilities connected to these and if distance teaching methods are developed to serve new clientele.

Gellman-Dansley (1995) summarizes many of these concerns when she observes that state policy-makers and institutions of higher education are facing a variety of questions.  These include: who is responsible for writing the policy; who will implement the policy; which governing body assures compliance; what must be included in the content; what can be learned from other states and how do colleges/universities currently handle distance learning?

Clearly, it is essential for higher education leaders to recognize and address the need for new policy development/implementation where distance learning courses and programs are concerned.  As this method of higher education transcends the borders at local, state, national and international levels, there is a need for clear policies.  This requires institutions to look at what is available internally, as well as beyond.  Before institutions and faculty in academic departments can fully address issues of distance education cooperatively, it is important for them to understand their own distance learning courses and programs, the changing needs of students, the issues of territorial control and the infrastructure of delivery nationally.  They need to address the above questions and have clear and concise policy direction.  All need to also recognize the proactive/reactive dimensions B that is, sound research can lead and guide distance education, and evolving practice in distance education can force policy decisions.  These dimensions are often interdependent.
 

Evaluation Approaches

A number of diverse evaluation approaches identified in the evaluation literature provide guidelines for specific evaluations.  To bring order to various evaluation models, many people have developed classification schemes to group the models into evaluation approaches (Guba and Lincoln,1981; House,1983; Madaus, Scriven, & Stufflebeam,1983; Popham, 1975; Scriven, 1993; Shadish, Cook, & Leviton,1991; Stake 1975; Worthen & Sanders 1973, 1987).  A specific model may be categorized somewhat differently in classification systems because of the diverse needs evaluation addresses.  The schema by Worthen, Sanders, and Fitzpatrick (1997) was used to structure the current study.

Worthen et al. (1997) sort evaluation approaches into six categories:
 Objectives-oriented approaches:  The central theme focuses on determining which goals and objectives have been attained.
Management-oriented approaches:  The focus is on identifying and meeting informational needs of decision makers.
Consumer-oriented approaches:  The central concern is developing evaluation information on products or services used by consumers.
Expertise-oriented approaches:  Professional experts judge the quality of whatever endeavor is evaluated.
Adversary-oriented approaches:  The central focus is on different evaluators taking opposite views for the evaluation.
Participant-oriented approaches:  Stakeholders are integral in determining the values, criteria, needs and data for the evaluation. 

Because of the nature of the study to identify priorities for research and evaluation for distance education in Nebraska, the management-oriented evaluation approaches were the most appropriate.  Stufflebeam's (1971) CIPP (context, input, process and product) model and Alkin's (1969) UCLA evaluation model are directed primarily toward helping administrators make good decisions.  However, several other evaluation models also have characteristics that serve information needs of educational program managers.  These include Provus's (1971) discrepancy evaluation model, Patton's (1986) utilization-focused model and Wholey's (1983, 1994) practical uses of evaluation in public administration settings.

Stuffelbeam's CIPP model most directly targets the needs of this study.  The CIPP model uses the following framework to help managers and administrators as they face decisions.  Worthen et al. 1997) summarize CIPP as:

Context evaluation:  The objectives are to define the institutional context, to identify the target population and assess their needs, to identify opportunities for addressing the needs, to diagnose problems underlying the needs and to judge whether proposed objectives are sufficiently responsive to the assessed needs. 
Input evaluation:  The objective is to identify and assess system capabilities, alternative program strategies and procedural designs for implementing the strategies, budgets and schedules.
Process evaluation:  The objectives are to identify or predict program design and implementation defects, which provide information for pre-program decisions and to record and judge procedural events and activities. 
Product evaluation:  The objectives are to collect descriptions and judgments of outcomes and to relate them to objectives and to context, input and process information, and interpreting their worth and merit. (p. 99)

According to McKillip (1987), three basic models are used to assess needs: the discrepancy model, marketing model and decision-making model.  The discrepancy, or gap, model emphasizes normative expectations and involves: goal setting, i.e., identifying what ought to be;  performance measurement, i.e., determining what is and discrepancy identification, i.e., identifying difference between what ought to be and what is (Witkin, 1977).  Supporters of the marketing model (Marti-Costa & Serrano-Garcia, 1983; Nickens, Purga, & Noriega, 1980) define needs assessment as a feedback process used to learn about clients= needs.

The decision-making model most closely addresses Nebraska’s distance education situation.  The decision-making model has three stages: problem modeling, quantification, and synthesis (McKillip, 1987).  In the modeling stage, need identification takes place.  During the quantification stage, measurements from need identification are transformed to reflect the decision makers' values and interests.  The final step, synthesis, provides an index that orders options on need and gives a relative standing on the need.  Assumptions in the decision-making model are based on findings from research on:

Utilization:  If results focus on the needs and values of potential users, they will more likely be used (Weiss & Bucuvalas, 1980).
Decision making:  When decision makers are confronted with complex, multidimensional information, biases that attempt to simplify the decision problem result (Kahneman, Slovic, & Tversky, 1982).  Therefore, results need to be simplified.
Applied methodology:  Constructs are not measured perfectly by a single indicator or criterion measure.  Therefore, continual efforts to listen to multiple indicators of need will more accurately measure the construct (Cook & Campbell, 1979).

       1. Records and social indicators:  Methods include unobtrusive observations, using existing records, modifying or creating record keeping systems, rates-under-treatment (RUT), using data from existing data banks and two special techniques, mapping and indirect estimation.
       2. Surveys:  Three kinds of surveys include the written questionnaire, the interview and the critical incident technique.
       3. Group processes:  Group processes where the salient feature is the opportunity for face-to-face interaction include the community group forum, the nominal group technique and the focus group interview.
       4. Specialized survey and group techniques:  Methods that use group processes include DACUM (Developing a Curriculum), the mailed Delphi survey, the group or modified Delphi technique, electronic groups and concept mapping.
        5. Future-oriented procedures:  While some of the methods mentioned in the previous grouping (Delphi, nominal group technique, surveys and interviews, and focus group interviews) can be used with an orientation for the future, the methods unique to this category include strategic planning, scenario development, cross-impact analysis, future wheels and trend analysis.
       6. Causal analysis:  While brainstorming is the simplest way to do causal analysis, three structured methods give more precise information for decision making: fishboning, cause and consequence analysis and fault tree analysis.

Therefore, a modified Delphi technique was the best approach to gather data for identifying research and evaluation priorities for distance education in Nebraska.  The Delphi technique was originally used to try and predict the future B it used a survey in a way to target future problems and tried to foresee solutions.  The appeal of using the Delphi technique in this study is its flexibility of procedure and opportunity for the participants to respond to the findings and alter their responses (McKillip 1987).   Part of Delphi’s success lies in its use of experts in the field in question.  By utilizing the knowledge of experts, combining it and redistributing it, the study opens up doors and forces new thought processes to emerge.  It also allows for respondents to see how closely they responded to the rest of the field of experts and to justify their train of thought. 
 
 

CIPP Framework

The CIPP (context, input, process, product) evaluation model (Stufflebeam, 1971) was used as the framework to identify research and evaluation priorities for distance education in Nebraska.   Initially, descriptions were formulated for context evaluation, input evaluation, process evaluation and product evaluation, along with specific questions providing guidelines for the study.  The descriptions and the questions for CIPP follow.

CONTEXT:  Topics in this category help define objectives for distance education across the state and judge whether proposed objectives are sufficiently responsive to the assessed needs. 
Question: What types of research, needs or opportunity assessments or evaluations are needed to serve planning decisions for implementing distance education in Nebraska?   How important is it to...

...look at the interrelated conditions among higher education institutions in Nebraska in which distance education occurs (i.e., institutional context)?
...survey target populations and assess their needs?
...ascertain opportunities for addressing the needs?
...diagnose problems underlying the needs?
 ...judge whether proposed objectives for distance education are sufficiently responsive to the assessed needs?

INPUT:  Topics in this category facilitate designing distance education programs and procedures.
Question: What types of research, or evaluations are needed to serve structuring decisions for distance education in Nebraska?   How important is it to...

...appraise available resources (i.e., to assess the systems capabilities)?
...investigate alternative program strategies? 
...determine plans that seem to have the best potential for meeting needs (i.e.,     procedural designs for implementing the strategies, budgets, and schedules)?

PROCESS:  Once questions are answered in this category, distance education procedures can be monitored, controlled and refined.
Question: What types of research, or evaluations are needed to serve decisions relative to how distance education is being implemented in Nebraska?   How important is it to identify...

...defects in the procedural design or its implementation?
...how well distance education is being implemented in higher education?
...barriers that threaten the success of implementing distance education in higher education across the state?
...revisions that are needed in implementing distance education in higher education across the state?

PRODUCT:  Questions in this category are important in judging program attainments.
Question: What types of evaluations are needed to serve decisions relative to the outcomes, or impacts that distance education is having in Nebraska?   How important is it to identify...

...what results are obtained?
...how well needs are reduced?
...what should be done with a program after it has run its course?

 Study Participants.  Three groups of distance educators participated in the data collection process for the study.  They contributed their ideas for research and evaluation issues that are relevant in Nebraska at the present time, and/or ranked the importance of the ideas.  The three groups included: a five-member Steering Committee with interests in distance education both locally, nationally and internationally; a Delphi panel comprised of four representatives from state colleges, 10 from the University of Nebraska-Lincoln, three from the University of Nebraska at Omaha, five from elementary/secondary education, three from University of Nebraska Medical Center, four from states other than Nebraska, five who represented special interests related to distance education, four from community colleges, two from the University of Nebraska at Kearney and three from state government; and participants in the state-wide Distance Learning Conference: Communities of Learning that included, but were not limited to, K-12 classroom teachers, technologists, community leaders, extension educators, instructional designers, media specialists and librarians, college/university professors and school board members. 

Basic Input from Steering Committee.  The Steering Committee reviewed, critiqued and approved the proposed plan to use the CIPP framework to assess Nebraska’s research and evaluation priorities for distance education and to use the Delphi process to collect data.   Subsequently, they identified potential members for the panel of experts (i.e., a Delphi panel). 

The Steering Committee brainstormed to identify relevant topics/issues for a survey instrument.  These topics/issues were categorized according to the CIPP framework and specific items were organized into a draft survey instrument.  The committee reviewed and critiqued the items to confirm that they reflected the committee’s thoughts and ideas about potential research and evaluation needs in Nebraska. 

Delphi Panel -- First round.  For the first round, 39 items with sub-topics were identified.  Items were written in question form, followed by a rating scale ranging from very important to very unimportant.  The first round instrument was posted on a World Wide Web page.  Members of the panel were sent a letter of explanation about the study and its purpose, a hard copy of the questionnaire and instructions on accessing and answering the instrument electronically.  Twenty panel members participated in the first round.

Delphi Panel -- Second round.  Mean scores were calculated for each item from the first Delphi Panel response using a 5-point scale where very important =1; important = 2; neither important or unimportant = 3; unimportant = 4; and very unimportant = 5.  The mean score was marked on an importance scale and added to each of the original items.  In the second round, panel members were asked to rate the accuracy of the mean scores using a three-point scale comprised of (a) should reflect More Importance, (b) is an Accurate representation of importance and (c) should reflect Less Importance (Appendix 1).

As a result of the first round responses, 10 new items were added to the second round questionnaire.  Respondents were asked to rate the importance of these items using the same 5-point scale employed in the first round instrument.  Twenty-eight panel members completed the second round instrument.

Distance Education Conference Participants -- Third round.  Frequency distributions were calculated for the accuracy ratings given to each of the original items that were added in round two.  Round two mean scores were adjusted up or down based on the net difference between the proportions of responses indicating the item was judged either more important or less important.  The adjustment factor was calculated according to the following formula:

Adjusted Mean = Mean +/- Probit(Q) x SE

Mean = Mean for item from round one
 Probit = the inverse of the standard normal cumulative distribution function 
S = Standard Deviation for item from round one
SE = Standard Error = (S) x (Error) 

Error = (Square root (nmore + nless)) (S / square root (nround 1 + nround 2))
Q = Absolute Value( PD) + 0.5
PD = (number more important - number less important)/total n [from round two]

For cases where PD was greater than or equal to 50 percent, the value of Q was set at 1, yielding a Probit value of 1.96 (the point two standard deviations above the mean in a normal distribution).

The adjusted means, together with the raw mean scores calculated for the new items added in round two, were added to an instrument for a third round.  The third round instrument again asked for a rating of the accuracy of the mean scores using a three-point scale: (a) should reflect More Importance, (b) is an Accurate representation of importance and (c) should reflect Less Importance. 

Results from round two were presented at the state-wide Distance Learning Conference: Communities of Learning in September, 1998.  After the presentation, 14 conference participants completed the third round instrument. 

Final Scores.  The adjusted round two scores were then re-adjusted up or down to produce final scores using a modification of the round one formula.

Final Mean = Adjusted Mean +/- Probit(Q) x SE

Adjusted Mean = Mean for item from round two
Probit = the inverse of the standard normal cumulative distribution function
S = Standard Deviation for item from round one
SE = Standard Error = (S) x (Error) 

Error = (Square root (nmore + nless)) (S / square root (nround 1 + nround 2 + nround 3))
Q = Absolute Value( PD2) + 0.5
PD2 = (number more important - number less important)/total n [from round three]

For cases where PD2 was greater than or equal to 50 percent, the value of Q was set at 1, yielding a Probit value of 1.96 (the point two standard deviations above the mean in a normal distribution).

Data were grouped according to the CIPP categories B context (planning decisions), input (structuring decisions), process (implementation decisions) and product (outcome decisions).  One additional category included general education items.   After the second round of the Delphi panel, the adjusted means were used as a guide for interpreting priorities. 

An independent distance education consultant first identified the priorities in each of the categories as he saw them emerging from the data.  The principal investigator then independently verified the priorities and presented the findings at the state-wide Distance Learning Conference: Communities of Learning.  Conference participants discussed the findings before they completed the third round of the Delphi data collection process.  The principal investigator then adjusted the initial findings to reflect how the conference participants confirmed or changed the importance ratings of the different items.  And finally, the Steering Committee critiqued the written report to confirm the findings and their interpretations.
 
 

Two issues were of interest in the findings: the items included on the instrument, as well as the rankings of the items.  Of the 98 items ultimately included on the instrument, 94 were ranked as important or very important.  The other four were ranked as being neither important nor unimportant. 

As the data were being analyzed, it became apparent the issues for research and evaluation priorities were, in all likelihood, issues that distance educators felt were important to address as distance education opportunities are implemented in more and more situations.  In other words, research and evaluation issues become synonymous with general issues about distance education that need to be addressed.
 

Types of research, needs or opportunity assessments 
or evaluations needed for planning decisions

Topics in the category focusing on planning decisions help (a) define objectives for distance education across the state and (b) judge whether proposed objectives are sufficiently responsive to assessed needs.  Out of the 27 items suggested in the planning decisions category, 15 were ranked as very important, nine were on the upper side of important, and three were on the lower side of important (Table 1).

Major interests for research and evaluation focus on identifying how to better cooperate and coordinate distance education programs, as well as how to cooperate and coordinate in  technology use.  As these things improve, it is important to identify the impact the improved strategy has on learners.
 

Collaboration and Coordination

The highest interest is on collaboration among postsecondary institutions.  There is slightly less interest on collaboration between postsecondary and secondary institutions and even less interest in collaboration between secondary and elementary schools.  It is important to identify funding formulas that fairly reward all collaborative participants.

Technology coordination among higher education is important, as is connectivity among these institutions.  Technology coordination and connectivity among the grouping of schools identified as "pods" is also of interest.  There is much less interest in Nebraska Educational Television’s (NET's) role in this coordination.  There appears to be more interest in research on how the technology is used rather than on the processes needed to coordinate and connect the technology.
 
 

 Distance Learners

Several items relate to concern about student success.  Understanding the characteristics of successful distance learners ranks high, along with potential problems they may have with required equipment.  There is less interest in exploring problems learners have using the technology.

Diagnosing problems learners have with access in terms of time and place is very important, along with how the required use of technology affects motivation.  Less concern was noted for diagnosing problems with overall student costs.

It is very important to determine learners expectations for credit courses and professional improvement and less important to identify expectations for personal enrichment.  There is more interest in ascertaining opportunities different institutions have to meet learner expectations for obtaining credit courses than for professional improvement or personal enrichment needs. 

It is very important to determine why potential learners fail to take advantage of distance learning opportunities.  Although not stated as a marketing issue, these items could be used in marketing research to develop and refine effective marketing strategies.

Table 1. Types of assessments needed to serve planning decisions for implementing distance education in Nebraska

VERY IMPORTANT (x = 1.000 to 1.499a)
x	How important is it to...
1.029	...diagnose problems learners have with various distance delivery strategies (i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in regard to equipment requirements?
1.130	...assess how technology is organized for connectivity among higher education institutions?
1.145	...assess factors that facilitate or inhibit coordination/cooperation for programming (i.e. course development, mutual course use, credit transfer, etc.) among higher education institutions?
1.152	...identify characteristics of successful distance learners (i.e. self-regulation, independent inquiry, collaborative tendency, familiarity with computer tools, etc.)?
1.154	...assess how technology is organized for connectivity between K-12 and higher education institutions?
1.168	...diagnose problems learners have with various distance delivery strategies (i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in regard to access to the learning opportunity at a convenient time?
1.229	...assess client expectations for access to various educational opportunities for obtaining credit courses and/or degrees?
1.245	...assess how the technology is coordinated between K-12 and higher education institutions?
1.277	...assess how the technology is coordinated among higher education institutions?
1.339	...assess client expectations for access to various educational opportunities for professional improvement?
1.340	...diagnose problems learners have with various distance delivery strategies (i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in regard to access to the learning opportunity at a convenient place?
1.344	...diagnose problems learners have with various distance delivery strategies (i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in regard to motivation or desire to participate?
1.373	...ascertain opportunities that different educational institutions have to meet the client expectations for obtaining credit courses and/or degrees in-state, regionally, and nationally?
1.378	...assess how technology is organized for connectivity among the pods across the state?
1.391	...ascertain why potential learners fail to take advantage of distance education offerings?
UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a)
1.504	...ascertain opportunities that different educational institutions have to meet the client expectations for professional improvement in-state, regionally, and nationally?
1.517	...assess factors that facilitate or inhibit coordination/cooperation for programming (i.e. course development, mutual course use, credit transfer, etc.) between higher education institutions and secondary schools?
1.628	...diagnose problems learners have with various distance delivery strategies (i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in regard to using the technology required in the delivery process?
1.666	...assess client expectations for access to various educational opportunities for personal enrichment?
1.735	...assess how the technology is coordinated among the pods across the state?
1.856	...assess how funding formulas can be changed to fairly reward all institutions in a collaborative distance education project?
1.888	...ascertain opportunities that different educational institutions have to meet the client expectations for personal enrichment in-state, regionally, and nationally?
1.897	...identify how the different educational institutions are relating their distance education programming efforts to their institutional mission?
1.984	...diagnose problems learners have with various distance delivery strategies (i.e. integrative TV, audio-conference, web-based, multi-media, etc.) in regard too overall student costs?
LOWER SIDE OF IMPORTANT (x = 2.000 to 2.499a)
2.118	...assess factors that facilitate or inhibit coordination/cooperation for programming (i.e. course development, mutual course use, credit transfer, etc.) among secondary/elementary units?
2.204	...assess how technology is organized for connectivity between NET and educational organizations?
2.305	...assess how the technology is coordinated between NET and educational organizations?
aScale:  1 = Very important  2 = Important  3 = Neither important nor unimportant  4 = Unimportant  5 = Very unimportant

Types of research or evaluations 
needed to serve structuring decisions

Topics in the category focusing on structuring decisions facilitate designing distance education programs and procedures.  Of the 24 items suggested in the structuring decisions category, nine were ranked as very important, 10 were on the upper side of important, three on the lower side of important, and two fell out as neither important nor unimportant (Table 2).

Major interests for research and evaluation appear to concentrate on effective strategies for successful distance learning experiences, the support needed from the educational institution and training needs for distance education teachers.

It is most important to identify strategies that are the highly effective for a successful distance learning experience B including assessing Innovative instructional processes to identify what best helps distance students learn.  Determining factors impeding or enhancing the development of a structure supporting distance education programming is also ranked high.   Also of concern is assessing the financial resources needed and available for course development, along with the time faculty need to develop and teach via distance.  Determining a cost/benefit ratio ranks slightly lower.

Training needs and competencies of distance learning teachers are a great concern.  High on the list is identifying the types of support/assistance instructors need, training needs of teachers and effective teacher competencies.  Sufficient resources are also viewed as important, as well as resource systems supporting faculty development. 

Less important is an assessment of the pros and cons of different learning models, as well as instructor qualifications for in-state distance learning courses. 

Preference is given to developing models designed primarily for in-state use.  Assessing models designed for regional, national or international use is viewed as less important.  Instructor qualifications for such regional, national and international models are also viewed as less important.

There is less interest in assessing past courses to identify successes and failures as part of a continuing education effort for teachers.

It is also viewed as less important to assess past marketing strategies to identify how to create a market.  This is apparently more of an implementation concern than a structuring concern.

There is little interest in defining and identifying terms relating to distance education.
 

Table 2. Types of assessments needed to serve structuring decisions for distance education in Nebraska

VERY IMPORTANT (x = 1.000 to 1.499a)
x	How important is it to...
0.971	...identify which strategies are the most effective in making a distance education learning experience successful?
0.987	...determine the factors that impede or enhance the development of a structure that will make distance education work programmatically (i.e. competing bureaucracies, etc.)?
1.068	...identify the kinds of support/assistance that is necessary for individual instructors to develop courses?
1.324	...identify factors that contribute to effective teacher competencies in distance education?
1.328	...assess resources needed as well as those available in regard to financial costs to an institution?
1.350	...assess resources needed as well as those available in regard to resource systems that support faculty development?
1.351	...identify training needs of distance education teachers?
1.372	...assess "innovative" instructional processes to identify what best helps distance students learn?
1.402	...assess resources needed as well as those available in regard to time faculty need to develop and teach distance courses?
UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a)
1.608	...determine how competitive interests can cooperate or collaborate?
1.622	...identify expectations for instructor qualifications required for distance education courses that are used in-state?
1.626	...assess resources needed as well as those available in regard to cost benefit?
1.673	...identify the pros and cons of distance education models that are designed primarily for national use?
1.689	...assess pros and cons of different distant education models or methods for addressing various learning objectives (pod's, television based, computer based, telephone based, combinations of the technologies)?
1.693	...assess past courses, workshops, or conferences to identify success and failures for continuing education?
1.697	...identify expectations for instructor qualifications required for distance education courses that are used regionally?
1.705	...assess past courses, workshops, or conferences to identify success and failures for degree programs?
1.711	...identify expectations for instructor qualifications required for distance education courses that are used international use?
1.725	...identify expectations for instructor qualifications required for distance education courses that are used nationally?
LOWER SIDE OF IMPORTANT (x = 2.000 to 2.499a)
2.067	...assess past marketing strategies to identify how to best create a market?
2.124	...identify pros and cons of distance education models that are designed primarily for regional use?
2.272	...identify the pros and cons of distance education models that are designed primarily for international use?
UPPER SIDE OF NEITHER IMPORTANT NOR UNIMPORTANT (0 = 2.500 to 2.999a)
2.791	...identify pros and cons of distance education models that are designed primarily for in-state use?
LOWER SIDE OF NEITHER IMPORTANT NOR UNIMPORTANT (0 = 3.000 to 3.499a)
3.137	...to identify and define terms that relate to distance education (i.e. system, method of delivery, pods, etc.)?
aScale:  1 = Very important  2 = Important  3 = Neither important nor unimportant  4 = Unimportant  5 = Very unimportant

Types of assessments needed to serve decisions relative 
to how distance education is being implemented

Once questions about the topics in the category focusing on implementation are answered, distance education procedures can be monitored, controlled and refined.  Of the 31 items suggested in the implementation category, 12 were ranked as very important, 11 were on the upper side of important, six on the lower side of important, and two were clustered around neither important nor unimportant (Table 3).

While there is some difference between what is more or less important for research or evaluation in the implementation category, this section seems to focus on a few main themes: learner issues, instructional delivery, administration and quality control.

The most important issue deals with identifying learners' barriers and incentives in using distance delivered education.  In addition, considerable emphasis is placed on comparing classroom-based instruction and distance learning.  The most interest is on application and use of content, followed by knowledge or skills acquired,  learning styles, interaction with other learners and with the instructor and feeling a part of a learning community. 

There is a great deal of interest in identifying what makes collaborative distance education offerings successful.  Specific implementation issues of concern relative to instructors and instructional processes include the barriers and incentives for implementing distance delivery by the instructor, the processes used to customize the educational experience, factors encouraging educators to work together and the instructors' use of multiple technologies. 

Another important issue deals with identifying structures blocking distance delivery.  This appears to be an administrative issue.  Other administrative issues of importance include factors encouraging or discouraging educators from working together for program development and delivery.  The issue of quality and the maintenance of rigor is listed as an important item, along with identifying factors influencing how quality is judged from the perspective of different audiences

 Multiple technologies are addressed in three different responses.  The most important relates to learners perceptions about their advantages and disadvantages, followed closely by an identification of the incentives and barriers for instructors to incorporate multiple technologies.   Such incorporation within the infrastructure was also listed as a valuable item.

Of less importance is identifying of the benefits and drawbacks of various administrative models such as cohort designs, lead instructor with instructors-of-record and facilitator licensed processes.  There is also less interest in research into how instructors are transferring skills from the traditional classroom to distance-based instruction.

There is also less interest in comparing traditional face-to-face delivery with the various distance delivery methods that are television-based, computer-based and telephone-based.

Table 3. Types of assessments needed to serve decisions relative to how distance education is being implemented in Nebraska

VERY IMPORTANT (x = 1.000 to 1.499a)
x	How important is it to...
1.128	...identify barriers and incentives for using distance delivery by the learner?
1.134	...identify what makes collaborative distance education offerings successful?
1.142	...identify structures that are in place that are blocking delivering education via distance?
1.181	...compare learner reactions of distance delivery to traditional classroom delivery in terms of being able to apply and use the content?
1.220	...identify barriers and incentives for implementing delivery by the instructor?
1.224	...identify processes distance education instructors are using to customize the educational experience for students along with the benefits and drawbacks of these process?
1.233	...assess learners perceptions about the advantages and disadvantages of using various technologies?
1.296	...compare learner reactions of distance delivery to traditional classroom delivery in terms of knowledge or skills acquired?
1.344	...identify incentives and barriers for incorporating multiple technology use by the learners?
1.364	...identify factors that are encouraging or discouraging educators to work together for creating new educational structures (i.e. models, systems, etc.)?
1.389	...compare learner reactions of distance delivery to traditional classroom delivery in terms of learning styles of the learners?
1.441	...identify incentives and barriers for incorporating multiple technology use by the instructor?
UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a)
1.519	...identify factors that are encouraging or discouraging educators to work together for program development and delivery?
1.526	...compare learner reactions of distance delivery to traditional classroom delivery in terms of interaction with other learners in the course/workshop?
1.529	...compare learner reactions of distance delivery to traditional classroom delivery in terms of interaction with the instructor?
1.611	...identify factors that influence how quality is judged from the perspective of different audiences such as different institutions, accreditation agencies, faculty, general public, and students?
1.616	...compare learner reactions of distance delivery to traditional classroom delivery in terms of feeling a part of a learning community?
1.617	...identify barriers and incentives for implementing delivery by the technical support people?
1.637	...identify indicators that are useful for controlling quality and maintaining rigor?
1.655	...identify what is being done to control quality and maintain rigor?
1.677	...identify incentives and barriers for incorporating multiple technology use within the infrastructure.
1.850	...identify barriers and incentives for implementing delivery by the administration?
1.931	...identify how instructors teaching a collaborative offering interact with learners from other institutions?
LOWER SIDE OF IMPORTANT (x = 2.000 to 2.499a)
2.042	...research how instructors are transferring skills from using traditional campus-based instruction to using distance-based instruction?
2.288	...compare traditional face-to-face delivery with multi-media processes?
2.289	...compare the benefits and drawbacks of various administrative models such as?
2.367	...compare the benefits and drawbacks of various administrative models such as lead instructor with instructors-of-record at sites?
2.410	...compare traditional face-to-face with computer-based (Lotus Notes, etc.)?
2.467	...compare the benefits and drawbacks of various administrative models such as facilitator licensed processes?
UPPER SIDE OF NEITHER IMPORTANT NOR UNIMPORTANT (x = 2.500 to 2.999a)
2.664	...compare traditional face-to-face delivery with television-based?
LOWER SIDE OF NEITHER IMPORTANT NOR UNIMPORTANT (x = 3.000 to 3.499a)
3.120	...compare traditional face-to-face delivery with telephone-based?
aScale:  1 = Very important  2 = Important  3 = Neither important nor unimportant  4 = Unimportant  5 = Very unimportant

Types of evaluations needed to serve decisions 
relative to outcomes, or impacts that distance education is having 

 Topics in the category focusing on outcomes of distance education are important in judging program attainments.  Of the seven items suggested in the category, two were ranked as very important, three were on the upper side of important and two on the lower side of important (Table 4).

Major interests for evaluation focused on assessing outcomes in formal higher education courses followed by outcomes reached in K-12.  There is less interest in assessing outcomes of non-formal professional and personal growth workshops. 

Documenting participation and completion rates was viewed as important, as was identifying effective and fair teacher evaluation processes.

There is less interest in studying the maturation of distance instruction, and in conducting meta-analysis of the research on different types of distance delivery modes. 

Table 4. Types of evaluations needed to serve decisions relative to the outcomes of distance education.

VERY IMPORTANT (x = 1.000 to 1.499a)
x	How important is it to...
0.885	...assess how well instructional outcomes are reached in K-12?
1.097	...assess how well instructional outcomes are reached in formal higher education courses?
UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a)
1.599	...document participation and completion rates?
1.705	...assess how well instructional outcomes are reached in non-formal professional and personal growth workshops (i.e. seminars, video-conferences, etc.)?
1.791	...identify effective and fair teacher evaluation processes?
LOWER SIDE OF IMPORTANT (0 = 2.000 to 2.499a)
2.250	...study the maturation of distance instruction as instructors design and deliver educational processed for distance delivery?
2.467	...conduct a meta-analysis of the research on different types of distance delivery modes?
aScale:  1 = Very important  2 = Important  3 = Neither important nor unimportant  4 = Unimportant  5 = Very unimportant

Types of research needed about education in general

Topics in the category focusing on education in general will help the general educational context within which distance delivery is emerging.  Of the nine items suggested in the category, five were ranked as very important, three were on the upper side of important, and one was on the lower side of important (Table 5).

 The highest ranked item was assessing how to include training on adult education theory and practice so distance education instructors become more action-oriented.  Other issues seen as very important focus on the learner.  Identifying if distance education creates changes in the learning process is followed closely by studying how the change process is managed by students and identifying how distance education can facilitate lifelong learning. 

Creating a long-term vision about educational systems is very important, followed by integrating distance education into strategic plans.  Studying how the change process is managed by faculty is more important than studying how it is managed by administration.  Determining if felt needs are met is also important.

Table 5. Research is needed about education in general

VERY IMPORTANT (x = 1.000 to 1.499a)
x	How important is it to...
1.104	...assess how to include training for faculty to learn about adult education theory and practice so the distance education instructors become more action oriented?
1.267	...identify if distance education creates changes in the learning process? If so, how?
1.308	...study how the change process is managed by students?
1.359	...identify how distance education can facilitate lifelong learning?
1.440	...create long-term vision about educational systems that will serve Nebraskans in 2020?
UPPER SIDE OF IMPORTANT (x = 1.500 to 1.999a)
1.773	...assess how institution are integrating distance education into their strategic plan?
1.879	...study how the change process is managed by faculty?
1.939	...find methodologies that will help determine if the education programs meet felt needs?
LOWER SIDE OF IMPORTANT (x = 2.000 to 2.499a)
2.043	...study how the change process is managed by administration?
aScale:  1 = Very important  2 = Important  3 = Neither important nor unimportant  4 = Unimportant  5 = Very unimportant

 American Council on Education and The Alliance: An association for alternative programs for adults. (1996-97). A credo for going the distance. American Council on Education: Washington, D.C.

Clark, T. (1993). Attitudes of higher education faculty toward distance education: A national survey. The American Journal of Distance Education, 7 (2), 19-33. 

Collis, B., Veen, W. & DeVries, P. (1993). Preparing for an interconnected future: Policy options for telecommunications in education. Educational Technology, 33 (1), 17-24. 

Cook, T. D. & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis of issues for field settings. Chicago: Rand McNally. 

Dillion, C. L. & Walsh, S. M.. (1992). Faculty: The neglected resource in distance education. The American Journal of Distance Education, 6 (3), 5-21. 

Dillman, D. A., Christenson, J. A., Salant, P. & Warner, P. D. (1995). What the public wants from higher education. Work force implications from a 1995 national survey. Social and Economic Sciences Research Center (SESRC). Washington State University: Pullman. 

Gellman-Dansley, B. (1995). A workbook on policy development for distance learning. Boulder, CO: Western Cooperative for Educational Telecommunications. 

Guba, E. G. & Lincoln, Y. S. (1981). Effective evaluation. San Francisco: Jossey-Bass.

House, E. R. (1983). Assumptions underlying evaluation models. In G. F. Madaus, M. Scriven, & D. L. Stufflebeam (Eds.), Evaluation models: Viewpoints on educational and human services evaluation. Boston: Kluwer-Nijhoff. 

Hanson, D., Maushak, N. J., Schlosser, C. A., Anderson, M. L., Sorensen, C. & Simonson, M.. (1996). Distance education: Review of the literature (2nd ed.). Research Institute for Studies in Education. Ames: Iowa State University. 

Holmberg, B. (1986). Growth and structure of distance education. Croom Helm: New Hampshire. 

Holmberg, B. (1987). The development of distance education research. The American Journal of Distance Education, 1 (3), 16-23. 

IANR Ad Hoc Distance Education Committee. February, 1996. A framework for lifelong distance education. IANR, University of Nebraska-Lincoln. 

Kahneman, D., Slovic, P. & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press. 

Madaus, G. F., Scriven, M. & Stufflebeam, D. L (Eds.). (1983). Evaluation models: Viewpoints on educational and human services evaluation. Boston: Kluwer-Nijhoff. 

Marti-Costa, S. & Serrono-Garcia, I. (1993). Needs assessment and community development: An ideological perspective. Prevention in Human Services, 3, 75-83. 

McKillip, J. (1987). Need analysis: Tools for the human services and education. Newbury Park, CA: Sage Publications. 

Miller, G. (1993). American independent study, news from the NUCEA Division of Independent Study. Fall 1993. 

Moore, M. (1994). Administrative barriers to adoption of distance education. The American Journal of Distance Education, 8 (3), 1-4. 

Moore, M. (1993). Is teaching like flying? A total systems view of distance education. The American Journal of Distance Education, 7 (1), 1-10. 

Nickerns, J. M., Purga, A. J. & Noriega, P. P. (1980). Research methods for needs assessment. Washington, DC: University Press of America. 

Patton, M. Q. (1986). Utilization-focused evaluation (2nd ed.). Beverly Hills, CA: Sage. 

Popham, W. J. (1975). Educational evaluation. Englewood Cliffs, NJ: Prentice-Hall. 

Provus, M. M. (1971). Discrepancy evaluation. Berkeley: CA: McCutchan. 

Roger, D. L. & Whetten, D.A. (1982). Interorganizational coordination: Theory, research, and implementation. Ames: Iowa State University Press. 

Schlosser, G. A. & Anderson, M. L. (1994). Distance education, reviews of the literature. Washington, D.C.: Association for Educational Communications and Technology. 

Scriven, M. (1993). Hard-won lessons in program evaluation. New Directions for Program Evaluation, No. 58, 1-107. San Francisco: Jossey-Bass. 

Shadish, W. R., Cook, T. D. & Leviton, L. C. (1991). Foundations of program evaluation. Newbury Par, CA: Sage. 

Sherry, L. (1994). Issues in distance education. [On-line], Available: gopher://oasis.denver.colorado/UCD/dept/IT/sherry/lit.html

Stake, R. E. (1975). Program evaluation, particularly responsive evaluation (Occasional Paper No. 5). Kalamazoo: Western Michigan University Evaluation Center. 

Stufflebeam, D. L. (1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education, 5, 19-25. 

Thach, L. & Murphy, L. (1994). Collaboration in distance education: From local to international perspectives. The American Journal of Distance Education, 8 (3), 5-21. 

Verduin, J. R. Jr. & Clark, T. A. (1991). Distance education: The foundations of effective practice. San Francisco: Jossey-Bass. 

Weiss, C. H. & Bucuvalas, M. J. (1980). Social science research and decision making. New York: Academic Press. 

Wholey, J. S. (1983). Evaluation and effective public management. Boston: Little, Brown. 

Wholey, J. S. (1994). Assessing the feasibility and likely usefulness of evaluation. In J. S. Wholey, H. P. Hatry, & K. E. Newcomer (Eds.). Handbook of practical program evaluation. San Francisco: Jossey-Bass. 

Witkin, B. R. (1977). Need assessment kits, models and tools. Evaluation Technology, 17, 5-18. 

Witkin, B. R., & Altschuld, J. W. (1995). Planning and conducting needs assessments: A practical guide. Thousand Oaks, CA: Sage Publications. 

Worthen, B. R. & Sanders, J. R. (1973). Educational evaluation: Theory and practice. Belmont, CA: Wadsworth. 

Worthen, B. R. & Sanders, J. R. (1987). Educational evaluation: Alternative approaches and practical guidelines (2nd ed.). New York: Longman. 

Worthen, B. R., Sanders, J. R.,& Fitzpatrick, J. L. (1997). Educational evaluation: Alternative approaches and practical guidelines. New York: Longman. 

Willis, B. (1989). Distance education and academic policy: Making it all fit. Tech Trends, 34 (3), 32-33). 
 

ID# _____

 This is the second and final round for the Delphi study focusing on research and evaluation priorities in Distance Education for Educational Institutions in Nebraska.

 Below are new items suggested by those who responded to the first instrument. Please rank how important it is to conduct research or evaluation on items A1 through E1 according to the following scale:

 Scale: 

VI = Very Important
 I = Important
 N = Nice to know but neither important nor unimportant
 U = Unimportant
VU = Very Unimportant

         VI     I     N     U     VU
Comment:

B1. Identify training needs of distance education teachers.

         VI     I     N     U     VU
Comment:

B2. Identify factors that contribute to effective teacher competencies in distance education.

         VI     I     N     U     VU
Comment:

B3. Identify the kinds of support/assistance that is necessary for individual instructors to develop courses.

         VI     I     N     U     VU
Comment:

B4. Assess "innovative" instructional processes to identify what best helps distance students learn.

         VI     I     N     U     VU
Comment:

C1. Identify what makes collaborative distance education offerings successful. 

        VI     I     N     U     VU
Comment:

C2. Identify how instructors teaching a collaborative offering interact with learners from other institutions. 

        VI     I     N     U     VU
Comment:

D1. Identify effective and fair teacher evaluation processes. 

        VI     I     N     U     VU
Comment:

D2. Conduct a meta-analysis of the research on different types of distance delivery modes. 

        VI     I     N     U     VU
Comment:

E1. Assess how to include training for faculty to learn about adult education theory and practice so the distance education instructors become more action oriented. 

        VI     I     N     U     VU
Comment:

MI = should reflect More Importance
 A = is an Accurate representation of importance
LI = should reflect Less Importance 

 A.  What types of assessments, are needed to serve planning decisions for implementing distance education in Nebraska?

Note: Topics in this category will help (a) define objectives for
distance education across the state and (b) judge whether
proposed objectives are sufficiently responsive to the
 assessed needs.
How important is it to . . .

-Among the pods across the state
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Between NET and educational organizations
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI 

-Between K-12 and higher education institutions
  Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Among higher education institutions
Mean = VI x . . . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI

2.  Assess how the technology is coordinated:

-Among the pods across the state
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Between NET and educational organizations
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Between K-12 and higher education institutions
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Among higher education institutions
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

3.  Assess factors that facilitate or inhibit coordination/cooperation for programming  (i.e. course development, mutual course use, credit transfer, etc.) . . .

-Among higher education institutions
Mean = VI . x . . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Among secondary/elementary units
Mean = VI . . . . . . . . . xI . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Between higher education institutions and secondary schools
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

4.  Assess client expectations for access to various educational opportunities for . . .

-obtaining credit courses and/or degrees
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-professional improvement
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
- personal enrichment
Mean = VI . . . . . . . . . xI . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

 5.  Ascertain the opportunities that different educational institutions have to meet the client expectations for . . .

-obtaining credit courses and/or degrees in-state, regionally, and nationally
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-professional improvement in-state, regionally, and nationally
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
- personal enrichment in-state, regionally, and nationally
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

6.  Identify how the different educational institutions are relating their distance education programming efforts to their institutional mission.

Mean = VI . . . . . . . . . I . . x . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

7.  Diagnose problems learners have with various distance delivery strategies (i.e. interactive TV, audio-conference, web-based, multi-media, etc.) in regard to . . .
-equipment requirements
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-access to the learning opportunity at a convenient time
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-access to the learning opportunity at a convenient place
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-overall student costs
Mean = VI . . . . . . . . . I . x . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-motivation or desire to participate
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-using the technology required in the delivery process
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
MI         A         LI

8.  Ascertain why potential learners fail to take advantage of distance education offerings.

Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

 9.  Identify characteristics of successful distance learners (i.e. self-regulation, independent inquiry, collaborative tendency, familiarity with computer tools, etc.)

Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

B.  What types of assessments are needed to serve structuring decisions for distance education in Nebraska?

Note: Topics in this category would facilitate designing distance
          education programs and procedures. 

10.  Assess resources needed as well as those available in regard to:

-Resource systems that support faculty development?
   Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Time faculty need to develop and teach distance courses?
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
- Financial costs to an institution?
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Cost benefit?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

11.  Identify and define terms that relate to distance education (i.e. system, method of delivery, pods, etc.)?

Mean = VI . . . . . . . . . I . x . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

12.  Assess the pros and cons of different distance education models or methods for addressing various learning objectives . . .

-POD’s
-Television-based
-Computer based
-Telephone based
-Combinations of the technologies
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI

13.  Identify which strategies are the most effective in making a distance education learning experience successful.

Mean = VI . x . . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI

14.  Identify the pros and cons of distance education models that are designed primarily for . . .

-in-state use?
Mean = VI . . . . . . . . . I . x . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-regional use?
Mean = VI . . . . . . . . . I . . x . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI
-national use?
Mean = VI . . . . . . . . . I . . . . x . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-international use?
Mean = VI . . . . . . . . . I . . . . . . . x . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

15.  Identify expectations for instructor qualifications required for distance education courses that are used . . .

-in-state?
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-regionally?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-nationally?
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-internationally?
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

16.  Determine the factors that impede or enhance the development of a structure that will make distance education work programmatically (i.e. competing bureaucracies, etc.)?

Mean = VI . . . x . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

17.  Assess past courses, workshops, or conferences to identify success and failures for . . .

-degree programs?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI 
-continuing education?
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

18.  Assess past marketing strategies to identify how to best create a market?

Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

19.  Determine how competitive interests can cooperate or collaborate?

Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

C.  What types of assessments are needed to serve decisions relative to how distance education is being implemented in Nebraska?

Note: Once questions are answered in this category, procedures
 can be monitored, controlled, and refined. 

20.  Compare traditional face-to-face delivery with various distance delivery methods?

-face-to-face with computer-based (Lotus Notes, etc.)
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-face-to-face with television-based
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-face-to-face with telephone-based
Mean = VI . . . . . . . . . I . . . x . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A        LI
-face-to-face with multi-media processes
Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

21.  Identify processes distance education instructors are using to customize the educational experience for students along with the benefits and drawbacks of these processes.

Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

22.  Assess the learners perceptions about the advantages and disadvantages of using various technologies.

Mean = VI . . . x . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

23.  Compare learner reactions of distance delivery to traditional classroom delivery in terms of . . . 

- knowledge or skills acquired
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-interaction with the instructor
 Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-interaction with other learners in the course/workshop
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-feeling a part of a learning community
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-being able to apply and use the content
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-learning styles of the learners
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

24.  Identify factors that influence how quality is judged from the perspective of different audiences such as different institutions, accreditation agencies, faculty, general public, and students.

Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

25.  Identify what is being done to control quality and maintain rigor?

Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

26.  Identify indicators that are useful for controlling quality and maintaining rigor?

Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

27.  Identify factors that are encouraging or discouraging educators to work together for. . .

-program development and delivery
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-creating new educational structures (i.e. models, systems, etc.)
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

28.  Identify barriers and incentives for using distance delivery by the learner?

Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

29.  Identify barriers and incentives for implementing delivery by the . . .

- instructor?
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI
- administration?
Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
- technical support people?
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

30.  Identify structures that are in place that are blocking delivering education via distance?

Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

31.  Identify incentives and barriers for incorporating multiple technology use . . .

-within the infrastructure
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-by the instructor
   Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
- by the learners
Mean = VI . . . . x . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

32.  Compare the benefits and drawbacks of various administrative models such as . . .

-lead instructor with instructors-of-record at sites
Mean = VI . . . . . . . . . xI . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-cohort designs
Mean = VI . . . . . . . . . xI . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-facilitator licensed processes
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI

33.  Research how instructors are transferring skills from using traditional campus-based instruction to using distance-based instruction.

Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI

 D.  What types of evaluations are needed to serve decisions relative to the outcomes, or impacts that distance education is having in Nebraska?

Note: Questions in this category are important in judging program attainments. 

34.  Document participation and completion rates.

Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI

35.  Assess how well instructional outcomes are reached in . . .

-K-12
Mean = VI . . . . . x . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI 
-Formal higher education courses
Mean = VI . . x . . . . . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-Non-formal professional and personal growth workshops (i.e. seminars, video-conferences, etc.)
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI

36.  Study the maturation of distance instruction as instructors design and deliver educational processes for distance delivery.

Mean = VI . . . . . . . . . I . . x . . . . . . N . . . . . . . . . U . . . . . . . . . VU
 MI         A         LI

E.  What types of research is needed about education in general?

Note: Topics in this category will help the general educational
context within which distance delivery is emerging. 

37.  Create a long-term vision about educational systems that will serve Nebraskans in 2020?

Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

38.  Assess how institutions are integrating distance education into their strategic plans?

Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

39.  Find methodologies that will help determine if the education programs meet felt needs.

 Mean = VI . . . . . . . . . I . x . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

40.  Identify if distance education creates changes in the learning process?  If so, how?

Mean = VI . . . . . . x . . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

41.  Identify how distance education can facilitate lifelong learning?

Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
  MI         A         LI

42.  Study how the change process is managed by . . .

-students
Mean = VI . . . . . . . x . I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-faculty
Mean = VI . . . . . . . . x I . . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI
-administration
   Mean = VI . . . . . . . . . I x . . . . . . . . N . . . . . . . . . U . . . . . . . . . VU
   MI         A         LI

Steering Committee
 

Arnold J. Bateman, Director of Learning Center Coordinator/Assistant to the Vice Chancellor, 
     University of Nebraska-Lincoln
Shirley L. Baugher, Professor & Chair, Family & Consumer Sciences, University of Nebraska-Lincoln
Gwen C. Nugent, Director, Educational Telecommunications, University Television, University of
     Nebraska-Lincoln
Sandra K. Scofield, Director, Math & Science Initiative, University of Nebraska-Lincoln
Ward Sybouts, Professor (Emeritus), Educational Administration & Professor Curriculum & Instruction,
     University of Nebraska-Lincoln

Authors

Principal Investigator & Primary Author:  S. Kay Rockwell, Professor, Agricultural Leadership, Education, & 
     Communication, University of Nebraska-Lincoln 

Evaluation assistant: Jack W. Furgason, Program Evaluation Assistant, Agricultural Leadership, Education, & 
     Communication, University of Nebraska-Lincoln 

Distance education consultant: Leroy V. Rockwell, Professor Emeritus, Broadcasting, University of Nebraska-Lincoln

Statistical consultant: David B. Marx, Professor, Biometry, University of Nebraska-Lincoln

Graduate assistants: Julia Kotsiopulos
                              Alan Nietfeldt
                              Jolene Schauer