Education 900D

Education 900D

Survey Methods in Educational Research

University of Nebraska-Lincoln, Lincoln, NE 68588-0515

5-640 pm Thursday Professor: Dr. Donna McAlister-Kizzier

E532 East Nebraska Hall Office: 529A Nebraska Hall

Jan 11, 1999 to May 8, 1999 Phone: 402 472-3647

Call No. 3309; CH 2 Internet: dkizzier@unlinfo.unl.edu

Prerequisite: Educ 800 or Equivalent; EdPs 459/859 or equivalent

Web Page: http://www.unl.edu/kizzier/900D/900D.html

Office Hours: TBA and by appointment

Course Description

This course examines the basic principles of survey research as a data collection procedure.

Recommended References

Fink, Arlene. The Survey Kit, available from Sage Publications (805-499-0721), 1995. ISBN 0-8039-7288-8.
Salant, Priscilla and Don A. Dillman. How to Conduct Your Own Survey, John Wiley & Sons, Inc. 1994. ISBN 0-471-01267-X. Available at Nebraska Bookstore.
McAlister-Kizzier, Donna L. 900D Readings, 1999. Notebooks/CD-ROM available for checkout.

Objectives

The student should be able to:

understand the purpose of survey research
identify populations
develop and administer a questionnaire
design all components necessary for survey research
understand, analyze and report data

Assignments

Group Forum 30%
One article with abstract suitable for inclusion in readings notebook
may be incorporated into Group Forum requirements, at option of student 10%
Final project 60%

Student Responsibilities

Attend class
Keep up to date on readings and assignments
Active participation in class activities
Demonstrate mastery of course objectives through performance

Instructor Responsibilities

Design class activities to actively involve all learners and stimulate growth and challenge
Provide prompt, thoughtful feedback to students on class projects
Be available to students for feedback/expert guidance
Provide a respective class setting that values diversity in gender, age, handicap, race, religion, marital status, veteran status, national or ethnic origin.

Resource Notes:

Textbooks. Students may choose from two text resources. The Salant & Dillman book is reader friendly, economical, succinct and is highly recommended by former students as a quick reference of key technical points in survey construction. The Fink Survey Kit is a series of nine booklets; each booklet addresses a specific survey topic in an easy-to-follow, relevant format. While the Fink kit is intended for the academic, the Salant & Dillman book is written for a more general audience.

Readings notebooks/CD. The instructor has selected readings to serve the diverse needs of students. Consider readings as a resource file. Students are not expected or encouraged to read all the resource material provided for this course; rather students should assess individual needs to determine the most appropriate readings. Two key factors should guide students in making decisions regarding what to read: (1) the nature of final projects and (2) the need to be grounded for forums/class discussions. Most readings are preceded by a short abstract to assist in making reading choices.

Primary, secondary, tertiary. To provide guidance, this syllabus divides readings into primary, secondary and tertiary resources. Primary readings contact basic information on the topic, while secondary and tertiary provide increasing levels of scholarly depth.

Session Activities & Assignments

January 14 Introduction to Course

Overview of Survey Research (Outline/Readings #1, Kizzier)

Forum Group Organization

Primary

Fink #1. The survey handbook

Salant & Dillman. Chap 1. Practical surveys

Secondary

Ruckmick. The uses and abuses of the questionnaire procedure. Pp. 32-41. RB 63-67.

Jick. Mixing qualitative and quantitative methods: triangulation in action, pp. 602-611. RB 69-78.

McFee. Triangulation in research: two confusions. Pp. 215-219. RB 79-86.

Tertiary

Howe & Eisenhart. Standards for qualitative (and quantitative) research: a prolegomenon. Pp. 2-8. RB 87-96.

______________________________________________________________________________________

January 21 Overview of Survey Research (continued)

Pre-Survey Decision Points (Outline/Readings #2, Kizzier)

Selecting the Topic

Defining Problems

Identifying Type of Study

Data Collection Methods

Primary

Fink #5. How to design surveys

Salant & Dillman. Chap 2. Cornerstones of a quality survey

Salant & Dillman. Chap 3. Deciding what information you need

Salant & Dillman. Chap 4. Choosing a survey method

Secondary

Goetz & Lecompte. Conceptualizing the research process: theory and design. Pp. 33-62. RB 111-126.

Hillestad. The research problem. Pp. 1-21. RB 127-137.

Cates. Questions, statements and hypotheses used in research studies. Pp. 12-22. RB 139-150

Oppenheim. Chap 2. Analytic survey designs. Pp. 21-37. RB 151-163.

Tertiary

Sproull. Using electronic mail for data collection in organizational research. Pp. 15-168. RB 165-174

Dunnington. New methods and technologies int eh organizational survey process. Pp. 512-530 with abstract by Ford. RB 175-185

________________________________________________________

January 28 Survey Design Issues (Outline/Readings #3, Kizzier)

Format/Design of Instrument

Issues in Designing Questions

Primary

Fink #3. How to conduct self-administered and mail survey

Fink #2. How to ask survey questions

Salant & Dillman. Chap 6. Writing good questions

Salant & Dillman. Chap 7. Questionnaire design

Narins. Guidelines for creating better questionnaires. Pp. 8-9. RB 218a-218b

Secondary

Converse & Presser. Survey questions: handicrafting the standardized questionnaire. Pp. 9-31. RB 219-231.

Birdie, Anderson & Niebuhr. Questionnaires: design and use. Pp. 23-41. RB 234-245

Alreck & Settle. Questionnaire construction. Pp. 158-190. RB 247-256

Alreck & Settle. Creating Item Scales. Pp. 113-142. RB 266.1-266.31

Oppenheim. Chap 7. Questionnaire planning. Pp. 100-117. RB 289-301

Oppenheim. Chap 8. Question wording. Pp. 119-149. RB 307-322

Tertiary

Fowler. How unclear terms affect survey data. Pp. 218-231. RB 323b-323o

Sudman & Bradburn. Format of the questionnaire. Pp. 229-260. RB 324-340

Rasinski, Mingay & Bradburn. Do respondents really mark all that apply? Pp. 400-406. RB 340.1-340.9

Boser. Surveying alumni by mail: effect of booklet/folder questionnaire format and style of type on response rate. Pp. 149-159. RB 271-281.

Sudman & Bradburn. The social context of question asking. Pp. 1-19. RB 341-350

______________________________________________________________________________________

February 4 Format/Design of Instrument (continued)

Issues in Design Questions (continued)

Peer critique of preliminary problem statements (whole class/small groups)

Assignment: Be prepared to report on problem statements for peer feedback

______________________________________________________________________________________

February 11 IRB procedures—short primer, if necessary (Kizzier)

Interviewing techniques (Outline/Readings #4)--Forum 1(Team 1)

Primary

Fink #4. How to conduct interviews by telephone and in person

Disney, Khati, Gubbels, Vincent & Richter. Discussion questions—interview techniques. RB 352.1-352.2

Pope, Keasler, Williams & Egland. Interviewing techniques. RB 352.3-352.9

Weisberg, Krosnic & Bowen. An introduction to survey research, polling and data analysis, with abstract by Rupiper. Pp. 103-150. RB ________.

Secondary

Key. The telephone interview as a data collection technique. RB 352.9a-352.9e

Canell. Overview: response bias and interviewer variability in surveys. Pp. 1-23. RB 353-365

Bauman & Adair. The use of ethnographic interviewing to inform questionnaire construction. Pp. 9-23. RB 401.16-401.24

Beed, Terence & Stimson (Cleland). The use of panel discussions in qustionnaire design. Pp. 105-111. RB 366-371

Clark & Schober. Asking questions and influencing answers. Pp. 15-48. RB 373-392

Sobal. Disclosing information in interview introductions: Methodological consequences of informed consent. Pp. 348-361. RB 393-401

Tertiary

Loftus. Leading questions and the eyewitness report. Pp. 560-572. RB 402-402.7

Groves, Cialdini & Couper. Understanding the decision to participate in a survey. Pp. 475-495. RB 402.8-402.21

Pol. A method to increase response when external interference and time constraints reduce interview quality. Pp. 356-359. RB 405-406

Lavrakas. Supervision II: structuring supervisory work, telephone survey methods: sampling, selection and supervision. Pp. 122-136. RB 406.1-406.10

Sanchez. Effects of questionnaire design on the quality of survey data. Pp. 206-217. RB 406.11-406.22

Suchman & Jorden. Interactional troubles in face-to-face survey interviews. Pp. 232-241. RB 406.24-406.35

Oksenberg, Coleman & Camnell. Interviewers’ voices and refusal rates in telephone surveys. Pp. 97-111. RB 406.35-406.52

Edwards, R. (1993). An education in interviewing, placing the researcher and the research. In Renzetti, C. M. & Lee. R. M. (eds). Research sensitive topics. Sage: Newbury Park. Abstracted by Janet McDonald, June, 1996. RB 406.52-406.61

____________________________________________________________________________________

February 18 NEAR Center, (Craig Enders, Director; Bancroft Hall)

Overview of services

Design tips for surveys

Common problems

Preparation for working with NEAR

Data Analysis & Coding Issues (Outline, Readings #5)--Forum 2 (Team 2)

Primary

Various handouts from NEAR center, readings highlights and statistical summaries

Fink #8. How to analyze survey data

Salant & Dillman. Chap 9. From questionnaires to survey results

Secondary

Borg & Gall. Processing research data. Pp. 843-857. RB 418-436

Orlich. Designing sensible surveys. Pp. 132-151. RB 437-447

Narusis. Designing forums for studies. Pp. 60-69. RB 449-461

Tertiary

Myers. How to use local census data. Pp. 52-54. RB 462.1-462.3

Bracey. Sense, non-sense and Statistics. RB 463.4

Slakter, Wu & Suzuki-Stakter. *.**. And ***. Statistical nonsense at the .00000 level. RB 465-466.

Smith & Prentice. Exploratory data analysis. Pp. 349-389 (edited by Keren & Lewis). RB 466.1-466.21

Babbie. Social statistics. Survey research methods. Pp. 283-304, with abstract by Swain. Pp. 466.22.1-466.22.23

____________________________________________________________________________________

February 25 Kizzier out of town this week (OSRA, Atlanta)

Critique of real survey instruments. In-class small group activity.

______________________________________________________________________________________

March 4 Attitudinal Items (Outline/Readings #7)

Assignment: Purpose and problem statements complete; at student option, instructor will review.

Primary

Fink #2. How to ask survey questions

Babbie. Index and scale construction. Pp. 253-278. RB 489-502

Secondary

Gordon. Scaling theory. Pp. 15-42. RB 504-519

Fishbein & Ajzen. A theory of reasoned action. Pp. 41-77. RB 520-520.18

Bentler & Speckart. Models of attitude-behavior relations. Pp. 452-464. RB 520.19-520.29

Tertiary

Harrison, Thompson & Rodger. Robustness and sufficiency of the theory of reasoned action in longitudinal prediction. Pp. 25-40. RB 520.30-520.38

Osgood, Suci & Tannenbaum. The dimensionality of the semantic space. Pp. 31-75. RB 520-545

Example of semantic differential scale. RB 547

Carabetta & Paquette. Using the randomized response technique to educational research. Pp. 47-53. RB 553-561

Example of Guttman scale (attitude toward divorce, etc.) RB 562-562.7

Wandzilak, Ansorge, & Potter. Utilizing ‘undecided’ option with Likert items: associated

measurement problems. Pp. 51-58. RB 562.8-562.15

March 11 Demographic Items (Outline/Readings #8)

Census example, RB 567-579

Waggoner, Census may add multiracial category. Pp. 3A

Grand Island Independent. Panel rejects multiracial box for Census

Assorted handouts

March 18 Peer critique of student instrument drafts (Outline/Readings #9)

Assignment: Bring drafts of instruments for final project.

March 21-28 Spring Break—No Class

______________________________________________________________________________________

April 1 Ethical Issues (Outline/Readings #6)--Forum 3 (Team 3)

Mid-term peer feedback for instructor

Primary

Assorted handouts/IRB materials/summaries from ethics forums

Krathwoth, Ethics and legal constrants, Pp. 658-676..RB 466.22-466.33

Babbie. The ethics of survey research. Pp. 347-357. RB 467-480

AERA (1992). Ethical Standards of the American Educational Research Association. Educational Researcher, with abstract by Howell, pp. 23-26. RB 480a-480g.

OPPR Reports. The Belmont report: ethical principles and guidelines for the protection of human subjects of research. April 18, 1979. Pp. 1-8. RB 480.60-480.67

Secondary

Rosenthal & Rosnow. Ethics and values in human subject research. Pp. 231-250. RB 480.29-480.41

Howe & Doughterty. Ethics, Institutional Review Boards and the changing face of educational research. Pp. 16-21. RB 480.68-480.73

Robinson & Gross. Counseling research: ethics and issues. Pp. 331-333. RB 480.1-480.5

Iammarion, O’Rourke, Pigg & Weinberg. Ethical issues in research and publication. Pp. 101-106. RB 480.21.1-488.21.5

Sudman. Mail surveys of reluctant professionals. Pp. 349-360. RB 488.21.6-488.21.11

Tertiary

Kearney, Hopkins, Mauss & Weisheit. Self-generated identification codes for anonymous collection of longitudinal questionnaire data. Pp. 370-378. RB 480.42-480.48

Fisher & Fyrberg. Participant partners: college students weigh the costs and benefits of deceptive research. Pp. 417-426. RB 488.21.21-488.21.16

Floyd, Schroeder & Finn. Only if I’m first author: conflict over credit and management scholarship. Pp. 734-747. RB 488.21.17-488.21.23

Bauchamp. Ethical issues in funding and monitoring university research. Pp. 5-16. RB 480.22-480.28

Sieber. Planning ethically responsible research: a guide for students and Internal Review Boards—strategies for ensuring confidentiality. Pp. 53-63. RB 480.29-480.38

Adair, Duschendo & Lindsay. Ethical regulations and their impact on research practice. Pp. 59-71. RB 480.39-480.52

Gray & Cooke. The impact of institutional review boards on research. Pp. 36-41. RB 480.53-480.58

Koay example of sensitive questioning. RB 480.59

Mishkin. On parallel tracks: protecting human subjects and animal. Pp. 36-43, with abstract by Dydingco. RB 480.74-480.82

______________________________________________________________________________________

April 8 Peer critique of student instrument drafts

Pilot test (Outline/Readings #10)

Assignment: bring 4 copies of rough draft of survey instrument for peer critique

Primary

Sudman & Bradburn. Questionnaires from start to finish. Pp. 281-286. RB 587-590

Secondary

Converse & Presser. The tools at hand. Pp. 48-75. RB 591-606

Oppenheim. Questionnaire design, interviewing and attitude measurement. Pp. 47-64. RB 607-619

______________________________________________________________________________________

April 15 Cover letters (Outline/Readings #11)

Follow-up Procedures (Outline/Readings #12)

Primary

Assorted communications/wording handouts

Pearl & Fairley. Testing for the potential for non-response bias in sample surveys

Pp. 553-560. RB 631-634

Secondary

Berdie, Anderson & Neibuhr. Questionnaires: design and use, how to stimulate response, pp. 42-63. RB 638-649

Fox, Crask & Jonghoon. Mail survey response rate: a meta-analysis of selected techniques for inducing response. Pp. 468-491. RB 651-667

Tertiary

Reagan, Aaronson, Pinkleton & Ramo. Differentiating telephone surveys from telemarketing to increase response rates (1995). Communication research reports 12(2). Pp. 170-177 with abstract by Sargent. RB 667.1-667.10

Singer E., Von Thurn & Miller (1995). Confidentiality assurances and response: a quantative review of the experimental literature. Public Opinion Quarterly (59) 66-77. with abstract by Shippy. RB 667.11-667.18

______________________________________________________________________________________

April 22 Sampling/Sampling Error (Outline/Readings #13)

Reporting Data (Outline/Readings #16)--Forum 4 (Team 4)

Primary

Fink #6. How to sample in surveys

Fink #9. How to report on surveys

Salant & Dillman. Chap 5. When and how to select a sample

Salant & Dillman. Chap 10. Reporting survey results

Secondary

Deming. Some criteria for judging the quality of surveys. Pp. 83-90. RB 672.1-673

Freed, Hess & Ryan. A summary of sampling techniques. Pp. 449-459. RB 673-685

Borg & Gall. Populations and samples. Pp. 213-241. RB 686-714

Assorted reporting summaries (matrices, tips, etc.). RB—throughout section

Hillstad. Writing the report. Pp. 96-136. RB 739-758

Stock, M. Writing about research. Pp. 135-164. RB 765.1-765.18

Best & Kahn. The research report. Pp. 61-80. RB 759-760.19

Tertiary

Chase-Landale, Mott, Broos-Gunn, & Phillips. Children of the national longitudinal survey of youth: a unique research opportunity. Pp. 913-931. RB 71.1-714.15

Robson. Designing small surveys. Pp. 121-145. RB 714.16-714.31

Moore, Burnell & Moore. Approved practices in reporting quantitative research. Pp. 1-24. RB 760.20-760.28

Hensen. Writing for publication: some potential mistakes. Pp. 781-784. RB 761-764

Fine & Kurdick. Reflections on determining authorship credit and authorship order on faculty-student collaborations. Pp. 1141-1147. RB 764.3-764.8

Winston. A suggested procedure for determining order of authorship in research publications. Pp. 515-518. RB 764.9-764.13

Tabor & Tung. Packet addressing report sections and style. RB 765.21-766

Tuffe. The visual display of quantitative information. Pp. 91-121. RB 766+

Tuffe. With abstract by Kautz. Visual explanations: Images and quantities, evidence and narrative. Pp. 27-53. RB 767-767.31.

_________________________________________________________________________________________________

April 29 Administration Procedures (Outline/Readings #14)

Reliability & Validity Analysis (Outline/Readings #15)

Primary

Fink #7. How to measure survey reliability and validity

Salant & Dillman. Chap 8. Setting your survey in motion and getting it done

Salant & Dillman. Chap 11. Advice, resources and maintaining perspective

Secondary

Strong. Chap 5. Registration of a copyright claim. Pp. 92-116. RB 715.1-715.15

Kearney, Hopkins, Mauss & Weisheit. Self-generated identification codes for anonymous collection of longitudinal questionnaire data. Pp. 370-378. RB (Ethical Issues Section, 480.42)

Henderson & Fitz-Gibbon. How to measure attitudes, validity and reliability of attitude instruments. Pp. 131-157. RB 719-734

Janing. Abstract of McBurney. Research Methods (1994, 3^rd ed). Chap 6: Validity. Belmont: Brooks/Cole. Pp. 119-140. RB 734a-734l

Agnew & Pyke (1987). The science game: an introduction to research in the social sciences. Chap 13: Validity—the reach of research. Prentice-Hall, Inc., Englewood Cliffs, New Jersey. Abstract and Pp. 204-214. RB 734m-734u

Janing, Judy. Notes for significance presentation w/statistics organizer. RB 734v-734ab

Tertiary

Geisinger. The metamorphosis of text validation. Pp. 197-222. RB 734.1-739.12.

Alrick & Settle. Questionnaire construction. Pp. 158-190 (see in Survey Design Issues section)

Turner. Buros Institute summary w/related articles (Plake, Conoley, Kramer & Murphy). RB 714.32-714.62

______________________________________________________________________________________

May 6 8:15-10:15 pm, Final Exam Scheduled (may reschedule with unanimous student approval)

Final Course Assessment

Course Wrapup/Closure

Assignment:

Final project due (accepted earlier)

Final Project Criteria

Your task is to produce a model of a survey study. This model should include the following elements, valued as noted below. Also refer to the grading rubric, which provides weighing of each criteria, base don total points assigned.

Survey design (15 points)
Survey instrument
Cover letter/appropriate communication component
Follow-up
Other appropriate components
Statement of purpose and research questions. Students should clearly identify which items relate to specific research questions/hypotheses (10 points)
Each of the following required components are worth 5 points each:
Sampling design
Validity procedures
Reliability procedures
Procedures to analyze data—include statistical procedures you plan to use for each research question; you may wish to consult statisticians or NEAR center staff
Procedures used to pilot the instrument (within the context of the class assignmetn) and additional plans (if any) to further pilot test the instrument (5 points)
Procedures you plan to use to administer the survey (mail? Telephone? Interview? Email? Mixed methods?)
Ethical/IRB procedures

Use a formal report format. Quality of content is more important than quantity. The instructor will evaluate the porject based on individual student research objectives. If you feel a statement of your project goal is necessary, please include one. For example, in this class, some students are preparing proposals for doctoral or masters committees while others are developing surveys for use in their work settings or as part of a larger quantitative or qualitative study. The type of project produced for such varied goals would differ yet be assessed based on appropriate survey research design.

The following rubric will guide your final project assessment, based on the criteria outlined earlier. The left-hand column provides total points possible for each criteria:

Criteria	Exemplary = A	Good = B	Adequate = C	Inadequate = D	Absent/Poor = F
Design	14-15	12-13	11	9-10	0-8
Problem Stmt.	10	8-9	7	6	0-5
Methods Criteria (7)	5	4	3	2	0-1
Total Points	56-60 (>94%)	50-55 (84-94%)	44-49 (64-74%	<37 (<64%)

Please provide the instructor with two copies of your final report. Also please provide a signed statement regarding whether or not you give the instructor permission to use your project as a referen e in future course resource files, some of which may be placed on a class retricted-use web site.

Abstract Guidelines

Locate an article or reading pertinent to one of the topics covered in class. The reading should be suitable for the class reading resource. Photocopy the article/chapter/resource and attach a complete citation for the reading. Write a brief summary of the reading and a critique. Include a recommendation regarding how you think the reading would enhance the resources for this class.

Provide the instructor with two copies of your abstract and reading, along with a signed statement regarding whether permission is granted for your abstract to be included a s a resource in future classes. Examples of abstracts can be found in the course resources.

At a minimum, your abstract would include:

A brief summary of the reading
Your critique of the reading
A complete citation of the reading
Your recommendation regarding where and how you see the reading best used in the course; identify the pertinent course topic to which your selection is related; provide your rationale for recommending your reading.

Issues Forum Guidelines

Each team will select a topic on which to lead a 60-90 minute discussion. The discussion should be conducted in the form of an organized open forum, with the team leading the discussion.

The presenting team is responsible for:

Grounding themselves in the topic by reading assigned readings plus at least one pertinent outside resource per group member (may be human resources or written). The reading can be used to meet the reading/abstract requirement for the class, if abstract is added and abstract criteria are met.
Leading the discussion, using appropriate strategies to engage non-presenting classmates into the discussion as much as appropriate for the topic; asking classmates high level questions is almost always necessary for this exercise (questions that will make class members think critically about the topics/apply information to case site/or apply to personal lives)
Generating pertinent issues/questions for class discussion
Inviting outside "experts" as appropriate to assist in the topic discussion
Contacting the instructor as needed for assistance in identifying resources, understanding content, expertise, etc.
Sharing pertinent resources you have found with classmates
Placing all materials in a file for evaluation by the instructor (copies of additional resources, outline of presentation, specific topic, team members names, other presentation materials (copies of transparencies, etc.)

The non-presenting teams are responsible for:

Being prepared with at least three high-level questions/issues for the presenting group, derived from the assigned reading
Participating actively in the forum discussion

Evaluation criteria (each weighed 5 points each; 30 points total)

1. Well grounded on the issue
2. High-level issues generated as appropriate to topic
3. Engaged classmates in discussion as appropriate to topic
4. Well-selected resources
5. Meaningful learning experience
6. Tailored for relevance to peers
7. Resource file (required) (no points assigned)

The following rubric will guide your forum assessment, based on criteria 1-6 listed above.

Criteria	Exemplary=A	Good=B	Adequate=C	Inadequate=D	Absent/Poor=F
Criteria # 1-6	5	4	3	2	0-1
Total Points	28-30 (>94%)	25-27 (84-94%)	22-24 (74-83%)	19-21 (64-74%)	<18 (<64%)