While the paper focuses on the topic of distance education, the process used to identify the research and evaluation priorities is also of importance.  The literature base was explored for relevant research about distance education, appropriate evaluation approaches and methods used to assess priorities for issues such as distance education. 

Distance Education Research

"Distance education" in this paper means education which occurs when the learner and the instructor are in different locations.  Distance education correspondence courses have been used for decades.  Now methodologies are expanding to include new technological advances such as audio- and tele-conferences, satellites, Internet, video and audio tapes and multimedia.  Educational experiences in which instructors travel to distant locations to teach face-to-face in a classroom setting are not considered as distance education.

The literature review and study of prior research were conducted through a review of CRIS and ERIC listings, refereed journals, books and university and government documents.  These sources produced an abundance of articles concerning distance education course presentations and media methods, evaluations of courses and methods along with student outcomes associated with these and students' attitudes toward distance education.

Distance Education Context.  Distance education established its roots as a form of instruction at least 150 years ago (Holmberg, 1986).  An early form of distance education was correspondence study.  As more sophisticated methods and media became available, distance education advanced to audio recordings and educational television programs delivered via satellite or fiber optics.   The Internet has opened the door for computerized courses, as well as supplemental processes  enhancing televised methods.  Distance education underwent other changes in practices, programs and definitions when distance education universities evolved, such as the emergence of Great Britain’s Open University. 

Rapid changes in technology have changed distance education.  New policies are being established that will determine how distance education is employed and used.  The growth and impact of distance education and the opportunities it offers are directly linked to the availability of new technologies.  As technology brings distant sites into an electronic web of information, people throughout the world are pulled together, and a demand for distance education opportunities is seen worldwide (Thach and Murphy, 1994, Hanson et al., 1996).

While the distance education environment is changing, many questions remain unanswered.  These questions concern definitions and theories of how to practice distance education in a collaborative environment.  In this new educational paradigm, research is needed to guide practice in the distance education movement.

 Distance Education Research.  Miller (1993) states distance education has entered a particularly important stage in its development.  He notes four long-term trends, including: 

The simultaneous diversification and convergence of technologies (live, interactive media, computer conferencing, digital technologies). 
Changing relationships with students (learning communities, student interaction, group and individual instruction and empowerment). 
Changing relationships among institutions (consortia, networks).
Educational adaptation (higher education adapting distance education to currents of social change).

Holmberg (1987) suggested the structure or categories of distance education research include:

Philosophy and theory of distance education.

• Distance students, their milieu, conditions and study motivations.
• Subject-matter presentation.
• Communication and interaction between students and their supporting organization (tutors, counselors, administrators, other students).
• Administration and organization.
• Economics.
• Systems (comparative distance education, typologies, evaluation, etc.).
• History of distance education. 

Some research is emerging in the category of administration and organization because administration and management consider the issues and construct the institutional policies that provide structure for successful distance education programs.  The operating practices of a distance education establishment are based on the educational philosophy of the institution, as well as its economic and political restrictions (Verduin and Clark, 1991).  Operational issues occur at all levels of distance education enterprises: local, state, national and international.  As students and teachers are connected across borders, international and national issues surface, as well as state and local issues.  Various management and administrative bodies should consider the issues and construct policies designed to facilitate effective solutions in concert with political and economic policy-making agendas (Colles, Veen and DeVries, 1993). 

 Operational issues include networking, cooperation, coordination and collaboration.  Thach and Murphy (1994) discuss the continuum of collaboration from a local level to an international level, and from student-to-student, class-to-class, institution-to-institution.  The concept of collaboration brings changes to the institution which include changes in structure, policy, faculty reward and skill requirements B elements of autonomy not easily relinquished by individual partners.  Schlosser and Anderson (1994) note that each partner has its own aims, goals and objectives, as well as its own culture of academia.  Roger and Whetten (1982) stress inter-organizational coordination.  Although organizations would prefer to maintain their autonomy, as the environment becomes more complex, organizations become more specialized and this increased specialization leads to a greater need for increased intra-organizational coordination.  With distance education efforts moving toward increased collaboration, policy research is needed to aid decision-making.

Moore (1994) identifies administrative and operational barrier issues to distance education at the federal, regional, state and institutional levels.  These administrative barriers to distance education relate to funding and monitoring processes based on evaluation or research and address accreditation needs.  Further administrative issues are: institutional policy, administrative structures and procedures, institutional support to faculty and students, tuition payment, faculty promotion and tenure, problems of territoriality, ways of rewarding institutions for collaborating and reforming policies with regard to faculty.

Operational issues also involve polices.  Concerns which need to be considered as the demand for distance education increases include academic policy, faculty development and program delivery (Willis, 1989).  Other issues of policy concern include: developing academic policies and procedures for statewide delivery of courses related to credit; telecommunication transmission costs and student support services; resolving faculty development issues (time and workload); resolving instructional development issues (course development and evaluation); developing policies and criteria for the evaluation, selection and use of compatible hardware; exploring external sources for funding distance delivered courses and developing distance education courses for system-wide delivery.  Other concerns focus on "traditional" verses "non-traditional" policy as related to credit transfer, faculty workload and cost sharing for instructional/faculty development. 

From this, Moore (1993) concludes the future of distance education depends on new forms of organization which will reorganize educational resources into a "total delivery system. Educators, administrators and policy makers have yet to come to terms with impact that redistributing educational resources through distance education could unleash.  To do this, teams within and outside of the institution, locally, nationally and internationally will need to be involved. 

Both The American Council on Education and The Alliance: An Association for Alternative Programs for Adults (1996), focus on what they call "Guiding Principles to Distance Learning." One of these principles is organizational commitment.  The principle states that distance-learning initiatives must be backed by an organizational commitment to quality and effectiveness in all aspects of the learning environment.  From this comes nine subprincipals:
 

• Policy is integrated into the mission of the provider.
• The provider makes a commitment to supporting faculty and learner services.
• Support systems are compatible with the learning delivery system.
• Curricular and administrative policies incorporate distance learners' needs. 
• The provider commits to researching and developing distance learning.
• The provider supports faculty and staff with development and resources.
• The provider includes distance learning in staff promotions and funding.
• All aspects of distance learning are regularly evaluated.
• The provider does not distinguish between distance learning and other means of learning in recognizing achievement.

• New forms of assessment and evaluation, including means to insure that the student’s work is original and authentic.
• A set of nationally accepted institutional accreditation standards to insure the quality of distance education.
• A nationally accepted set of teacher certification standards which meet a minimum criterion, including training in distance education theory, methods, and strategies.
• The need for cooperation among business, government, and education sectors.
• Technology training and accessibility for all, not just for progressive students and teachers.

 A report by the Social and Economic Sciences Research Center (SESRC) at Washington State University in Pullman, Washington (Dillman et al., 1995) summarizes the results of a national survey on higher education.  The major findings were: lifelong learning has become a reality for most Americans; getting educated once is not enough in our knowledge-based economy; teaching conducted only in the traditional campus classroom will not meet the public's demand for tailored educational services; distance education strategies have the potential to overcome significant barriers to lifelong learning; although lifelong learning is a reality for most Americans, some people are losing out; and public support exists for universities, and land grant institutions in particular to do more than educate 18-22-year-old undergraduates.  The implications of this survey for institutions of higher education are: colleges and universities have more potential customers than in the past; they need to change how they do business for meeting the demand for lifelong learning, including rewarding faculty for assuming new responsibilities for off-campus instruction and developing new curricula; distance education methods offer one means of meeting the demands for lifelong learning through technological developments and telecommunications and providing access to this education will require new policy measures. 

The authors conclude by stating that public support does exist for universities, particularly for land grant institutions, to do more than just provide undergraduate education.  Areas of education identified as important were off-campus education, education for returning students and applied research on problems.  Institutions of higher education will meet these needs if continuing and distance education are brought to the core of the universities and colleges, if faculty are rewarded and respected for assuming the new responsibilities connected to these and if distance teaching methods are developed to serve new clientele.

Gellman-Dansley (1995) summarizes many of these concerns when she observes that state policy-makers and institutions of higher education are facing a variety of questions.  These include: who is responsible for writing the policy; who will implement the policy; which governing body assures compliance; what must be included in the content; what can be learned from other states and how do colleges/universities currently handle distance learning?

Clearly, it is essential for higher education leaders to recognize and address the need for new policy development/implementation where distance learning courses and programs are concerned.  As this method of higher education transcends the borders at local, state, national and international levels, there is a need for clear policies.  This requires institutions to look at what is available internally, as well as beyond.  Before institutions and faculty in academic departments can fully address issues of distance education cooperatively, it is important for them to understand their own distance learning courses and programs, the changing needs of students, the issues of territorial control and the infrastructure of delivery nationally.  They need to address the above questions and have clear and concise policy direction.  All need to also recognize the proactive/reactive dimensions B that is, sound research can lead and guide distance education, and evolving practice in distance education can force policy decisions.  These dimensions are often interdependent.
. 
Evaluation Approaches

A number of diverse evaluation approaches identified in the evaluation literature provide guidelines for specific evaluations.  To bring order to various evaluation models, many people have developed classification schemes to group the models into evaluation approaches (Guba and Lincoln,1981; House,1983; Madaus, Scriven, & Stufflebeam,1983; Popham, 1975; Scriven, 1993; Shadish, Cook, & Leviton,1991; Stake 1975; Worthen & Sanders 1973, 1987).  A specific model may be categorized somewhat differently in classification systems because of the diverse needs evaluation addresses.  The schema by Worthen, Sanders, and Fitzpatrick (1997) was used to structure the current study.

Worthen et al. (1997) sort evaluation approaches into six categories:
 Objectives-oriented approaches:  The central theme focuses on determining which goals and objectives have been attained.
Management-oriented approaches:  The focus is on identifying and meeting informational needs of decision makers.
Consumer-oriented approaches:  The central concern is developing evaluation information on products or services used by consumers.
Expertise-oriented approaches:  Professional experts judge the quality of whatever endeavor is evaluated.
Adversary-oriented approaches:  The central focus is on different evaluators taking opposite views for the evaluation.
Participant-oriented approaches:  Stakeholders are integral in determining the values, criteria, needs and data for the evaluation. 

Because of the nature of the study to identify priorities for research and evaluation for distance education in Nebraska, the management-oriented evaluation approaches were the most appropriate.  Stufflebeam's (1971) CIPP (context, input, process and product) model and Alkin's (1969) UCLA evaluation model are directed primarily toward helping administrators make good decisions.  However, several other evaluation models also have characteristics that serve information needs of educational program managers.  These include Provus's (1971) discrepancy evaluation model, Patton's (1986) utilization-focused model and Wholey's (1983, 1994) practical uses of evaluation in public administration settings.

Stuffelbeam's CIPP model most directly targets the needs of this study.  The CIPP model uses the following framework to help managers and administrators as they face decisions.  Worthen et al. 1997) summarize CIPP as:

Context evaluation:  The objectives are to define the institutional context, to identify the target population and assess their needs, to identify opportunities for addressing the needs, to diagnose problems underlying the needs and to judge whether proposed objectives are sufficiently responsive to the assessed needs. 
Input evaluation:  The objective is to identify and assess system capabilities, alternative program strategies and procedural designs for implementing the strategies, budgets and schedules.
Process evaluation:  The objectives are to identify or predict program design and implementation defects, which provide information for pre-program decisions and to record and judge procedural events and activities. 
Product evaluation:  The objectives are to collect descriptions and judgments of outcomes and to relate them to objectives and to context, input and process information, and interpreting their worth and merit. (p. 99)

According to McKillip (1987), three basic models are used to assess needs: the discrepancy model, marketing model and decision-making model.  The discrepancy, or gap, model emphasizes normative expectations and involves: goal setting, i.e., identifying what ought to be;  performance measurement, i.e., determining what is and discrepancy identification, i.e., identifying difference between what ought to be and what is (Witkin, 1977).  Supporters of the marketing model (Marti-Costa & Serrano-Garcia, 1983; Nickens, Purga, & Noriega, 1980) define needs assessment as a feedback process used to learn about clients= needs.

The decision-making model most closely addresses Nebraska’s distance education situation.  The decision-making model has three stages: problem modeling, quantification, and synthesis (McKillip, 1987).  In the modeling stage, need identification takes place.  During the quantification stage, measurements from need identification are transformed to reflect the decision makers' values and interests.  The final step, synthesis, provides an index that orders options on need and gives a relative standing on the need.  Assumptions in the decision-making model are based on findings from research on:

Utilization:  If results focus on the needs and values of potential users, they will more likely be used (Weiss & Bucuvalas, 1980).
Decision making:  When decision makers are confronted with complex, multidimensional information, biases that attempt to simplify the decision problem result (Kahneman, Slovic, & Tversky, 1982).  Therefore, results need to be simplified.
Applied methodology:  Constructs are not measured perfectly by a single indicator or criterion measure.  Therefore, continual efforts to listen to multiple indicators of need will more accurately measure the construct (Cook & Campbell, 1979).

       1. Records and social indicators:  Methods include unobtrusive observations, using existing records, modifying or creating record keeping systems, rates-under-treatment (RUT), using data from existing data banks and two special techniques, mapping and indirect estimation.
       2. Surveys:  Three kinds of surveys include the written questionnaire, the interview and the critical incident technique.
       3. Group processes:  Group processes where the salient feature is the opportunity for face-to-face interaction include the community group forum, the nominal group technique and the focus group interview.
       4. Specialized survey and group techniques:  Methods that use group processes include DACUM (Developing a Curriculum), the mailed Delphi survey, the group or modified Delphi technique, electronic groups and concept mapping.
        5. Future-oriented procedures:  While some of the methods mentioned in the previous grouping (Delphi, nominal group technique, surveys and interviews, and focus group interviews) can be used with an orientation for the future, the methods unique to this category include strategic planning, scenario development, cross-impact analysis, future wheels and trend analysis.
       6. Causal analysis:  While brainstorming is the simplest way to do causal analysis, three structured methods give more precise information for decision making: fishboning, cause and consequence analysis and fault tree analysis.

Therefore, a modified Delphi technique was the best approach to gather data for identifying research and evaluation priorities for distance education in Nebraska.  The Delphi technique was originally used to try and predict the future B it used a survey in a way to target future problems and tried to foresee solutions.  The appeal of using the Delphi technique in this study is its flexibility of procedure and opportunity for the participants to respond to the findings and alter their responses (McKillip 1987).   Part of Delphi’s success lies in its use of experts in the field in question.  By utilizing the knowledge of experts, combining it and redistributing it, the study opens up doors and forces new thought processes to emerge.  It also allows for respondents to see how closely they responded to the rest of the field of experts and to justify their train of thought.