Evaluation in extension used to focus primarily on judging a program’s merit or worth. Additionally, the methodology associated with earlier forms of evaluation was portrayed as basically a quantitative activity. In today’s increasingly complex and demanding world, evaluation must deal with issues of accountability, good management, knowledge building and sharing, organizational learning and development, problem identification and policy formation. As the scope of evaluation expands, qualitative approaches and multiple methods are becoming increasingly necessary. Concurrently, today’s evaluator in extension finds that he or she needs to fulfill multiple roles and be familiar with numerous methods. This manual is designed to cover the expanding field of evaluation as it applies to extension and to provide you, the evaluator, with a methodological toolbox containing a broad array of methods and suggestions as to their appropriate use.

• Identify and describe the program you want to evaluate.

Identifying the program phase and type of evaluation study needed

A needs assessment focuses on identifying needs of the target audience, developing a rationale for a program, identifying needed inputs, determining program content, and setting program goals. A needs assessment asks questions about what exists and what is needed:

A baseline study establishes a benchmark from which to judge future program or project impact. A baseline study asks questions about what exists:

A formative, process, or developmental evaluation provides information for program improvement, modification, and management. A formative evaluation asks descriptive questions:

A summative, impact, or judgmental evaluation focuses on determining overall success, effectiveness, and accountability of the program. It helps make major decisions about a program’s continuation, expansion, reduction, and/or termination. A summative evaluation asks questions about what happened:

A follow-up study examines long-term effects of a program. A follow-up study asks questions about long-term impacts:

• Is there an important decision to be made on the basis of the evaluation?
• Is there a commitment to use the evaluation findings?
• Will important program decisions be made regardless of evaluation findings?
• Is there a legal requirement to carry out an evaluation?
• Does the program have enough impact or importance to warrant formal evaluation?

• What program will be evaluated, what it includes and excludes.

Clarify evaluation questions, issues, indicators and criteria

List questions, issues and criteria from all sources consulted.

Organize material into a manageable number of categories. Match level of program with indicators appropriate for that level -- remember that it is not possible for an evaluation to address all areas of interest.

Come to agreement with stakeholders on the degree of incompleteness that is acceptable, given monetary and time constraints.

Focus the scope of the evaluation to the crucial and practical.

• Examine various evaluation models and available literature.

Program Levels	Indicators
End results		Changes in participants’ personal and working lives as a result of program participation.
Practice and behavior changes			Changes in participants’ practices as a result of program participation.
Knowledge, attitude, skill and aspirational Changes (KASA)				Changes in participants’ knowledge, attitudes, skills and aspirations as a result of program participation.
Reactions					How participants and clients reacted to the program.
Participation						Who participated and how many.
Activities							Activities that participants were engaged in through the program. The kinds of information and methods used to interact with program participants.
Inputs								The personnel and other resources used during the program.

Quantitative Methods measure a finite number of pre-specific outcomes and are appropriate for judging effects, attributing cause, comparing or ranking, classifying and generalizing results. Quantitative Methods are:

• Suitable for large-scale projects.
• Useful for judging cause and effect.
• Accepted as credible.
• Applicable for or generalizing to a larger population.

Qualitative Methods take many forms including rich descriptions of people, places, and conversations and behavior. The open-ended nature of qualitative methods allows the person being interviewed to answer questions from his or her own perspective. Qualitative Methods are appropriate for:

• Understanding the context in which a program takes place.
• Complex problems and process issues.
• Clarify relationships between program objectives and implementation.
• Identifying unintended consequences of a program.
• Gathering descriptive information.
• Understanding operations and effects of programs.
• In-depth analysis of program impacts.

Multiple Methods combine qualitative and quantitative methods within one evaluation study. This combination can be used to offset biases and complement strengths of different methods. When using multiple methods, care should be taken to ensure that the selected methods are appropriate to the evaluation questions and that resources are not stretched too thinly. Multiple Methods are appropriate for:

• Understanding complex social phenomenon.
• Allowing for greater plurality of viewpoints and interests.
• Enhancing understanding of the both the typical and unusual case.
• Generating deeper and broader insights.

Validity - The data collection instrument measures what it is supposed to measure and data collected are relevant to the specific situation or audience.

Reliability - The data collection instrument measures consistently, yielding the same results with the same groups of people under the same conditions.

A large array of methods exist which can be used in evaluation. We will cover the following:

• Establishing the need for a program – use census data, media feature stories, maps, or service and business statistics.
• Describing how the program was carried out and who it reached – use program documents, log books minutes from meetings, enrollment records, media releases.
• Assessing results – use public records, local employment statistics, agency data, and evaluation of similar programs.

• In most cases, it is readily available.
• It can be obtained with minimal cost and effort.
• Data with a wide variety of characteristics are available.
• It can be accessed on a continuing basis.
• It can have high credibility.

• Data tend to be descriptive and may require the evaluator to sort, discriminate and correlate.
• Some figures may represent estimates or projections rather than actual accounts.
• It may not reveal values, reasons or beliefs underlying current trends.
• Local community data are frequently limited and not always current.

• It may present a biased view of reality.
• Testing Information and Knowledge

• It can provide an indication of knowledge level and other changes related to a particular program.

• It to relatively easy to implement.

• It can be carried out in a group setting.

• It tends to be low-cost.

• Adults often resist attempts to test their knowledge.
• Knowledge gain may be unrelated to behavior changes.
• Valid and reliable tests require special skills and time to develop.
• It is not appropriate for less literate audiences.

1. Construct a test consisting of questions that focus on the specific subject matter presented during the program.
2. Phrase questions in a non-critical manner.
3. Give the test to program participants either orally or in written form.
4. If responses are given in a group discussion, be sure to elicit the full range of responses.
5. Individual responses can be in a written format or discussed informally.
6. Grade the responses or compare then against answers that have been identified as correct.
7. Summarize and analyze the degree of comprehension achieved through the program.

• It permits fairly complex questions.
• It allows for anonymity of respondents.
• Cost is moderate.
• It is easy to reach a large number of people.
• Surveys are useful when the population is widely dispersed.

• A survey does not easily prove a cause-and-effect relationship.
• Surveys are difficult to use in cross-cultural settings.
• Using surveys requires a fairly literate population.
• It can be difficult to find an accurate and up-to-date list of potential respondents.

Key questions that need to be answered before carrying out a survey:

• How many people are required for a valid survey?
• Who should answer the questionnaire -- a few or only a select group?
• How should the sample be selected?
• How high should the response rate be?
• How accurate will the results be?

• Paid or unpaid people to carry out the survey.
• Your time frame and budget.
• A person experienced in survey work available to assist you.
• Available facilities such as telephone access.

• They can be used when respondents are widely dispersed geographically.
• They tend to have a high response rate.

• They can address more complex questions than mail questionnaires.
• They provide a quick and efficient source of data.
• Selection of a specific respondent is easier to control than with a mail survey.
• The interviewer can explain questions to respondents.

• Questions must be clear and concise.
• Surveys can be time consuming.
• Cost per response is comparable to mail surveys.
• They require interviewing skills and a trained supervisor.
• Bias may result as households without telephones or with unlisted numbers are excluded.
• Timing of calls is critical and may introduce bias.
• The telephone interviewer’s voice or mannerisms may introduce bias.
• They require a reliable telephone system and an adequate location for conducting interviews.

1. Find suitable facilities and equipment necessary to implement the survey.
2. Decide on a sampling design, including the method of respondent selection within a sampling unit. Choose the method to generate a pool of telephone numbers that will be used in sampling.
3. Prepare survey material: An advance letter if names and addresses are available; the questionnaire; a cover sheet to record identification number and a call-sheet; help sheets for the interviewer.
4. Train interviewers: background information about the survey; basics of telephone interviewing; how to use equipment; and how to fill out questionnaires and call-sheets.
5. Develop an interview schedule. Assess when you will be likely to contact respondents, during working or non-working hours. For most surveys, approximately 50 minutes is sufficient to complete an interview. Decide how to handle refusals.
6. Make calls. Decide on the number of calls to make to each number. In local surveys six to seven calls are customary. Make callbacks

Sample call sheet for telephone interviews

A call-sheet is used for each number chosen from the sampling frame. The interviewer records information that allows the supervisor to decide what to do with each number that has been processed. Call sheets are attached to questionnaires after an interview is completed.

• Can be used with a large sample size.
• Can be used with a widely dispersed population or one that is not accessible by telephone or personal interviewing.

• Provides a visual display of questions.
• Is free of interviewer bias.
• Enables respondents to give thoughtful answers and control the pace and sequence of response.
• Are relatively inexpensive.

• The questionnaire must be short and carefully designed.
• The response rate is highly dependent on the number of contacts made with the respondent and the timing of the mailing.
• There is little control over the completeness of the response.
• Those who reply may not be representative of the target population.
• Pretesting of the questionnaire is necessary to avoid costly mistakes.
• A mail survey requires from four to six weeks to collect data.
• It requires a literate population and a reliable postal system.

1. Prepare survey material. Design a written questionnaire, using an identification number on the cover of each questionnaire to track returns.
2. Pretest instrument to assure validity and reliability.
3. Select sample population
4. Develop a mailing schedule: a) Two weeks before mailing the survey, send an advance letter; b) mail the questionnaire including a cover letter and a stamped, self-addressed envelope; c) send a postcard a week or so later, thanking those who responded and reminding those who did not to return their surveys; d) three weeks after mailing the first questionnaire, send a follow-up letter stating that a response has not been received, including a replacement questionnaire and a stamped, self-addressed envelope. In developing the mailing schedule avoid holidays, especially the month of December. For most purposes, a 60 to 90 percent return rate is considered satisfactory.

Personal Surveys

• It can be used with a highly dispersed population.

• It is suited for populations where a representative sample cannot be drawn.

• It can be used where there is a low literacy rate.

• There is a high degree of control over who answers the survey.

• The interviewer can increase the willingness of respondents to answer questions.

• Visual aids can be used to facilitate understanding of survey questions.

• Questions can be fairly complex.

• It can be expensive and time-consuming.
• Interviewers must be carefully selected and receive adequate training.
• It requires a good supervisor.

• It requires more material than a telephone or mail survey.

1. Develop survey material: a) an advance letter if names and addresses are available; b) name tags for interviewers; c) an introductory letter explaining the purpose of the survey to be left with the respondent; d) an interviewer’s instruction manual; e) sampling information for interviewers; and f) the questionnaire.
2. Identify and train a staff of interviewers.
3. Mail letters describing the procedure and telling residents to expect a visit from an interviewer.
4. Notify public officials about the survey.
5. Conduct interviews. A supervisor should be available by telephone while the survey is being carried out to handle any problems that may arise.
6. The supervisor should meet regularly with interviewers to edit questionnaires and answer any questions interviewers may have. Costly errors, misunderstandings, and cheating by interviewers can be detected at this time.
7. After interviews are completed, the questionnaires are returned to the survey supervisor.

• Maintain a neat appearance.
• Follow the sampling plan to locate respondents.
• Be considerate and honest with the respondent.
• Understand the purpose of the study.
• Ask questions exactly as written.
• Record responses accurately.
• Be familiar with the research instrument.
• Follow sampling instructions.
• Check work for completeness.

• Introduce yourself, show your credentials.
• Remind respondent of the notification letter he or she received a few days earlier.
• Explain the purpose of the survey.
• Assure the respondent that his/her answers will remain confidential.
• Explain how respondents were chosen.

• Choose respondents following the sampling criteria.
• Conduct the interview or select a mutually convenient time to return.
• To avoid distractions, try to conduct the interview without an audience.
• Remind participants that the interview is voluntary and their responses are confidential.
• Establish rapport by expressing appreciation of the respondent’s responses and willingness to participate.
• Read questions as they appear in the questionnaire and record answers accurately.
• Do not express your opinions.
• If an answer to an open-ended question is incomplete or appears irrelevant, probe to get a clearer response.
• If a respondent refuses to answer a question, do not insist to get an answer. It may jeopardize the entire interview.
• Creating Quality Surveys by Avoiding Errors

A group-administered questionnaire is handed directly to each participant in a group at the end of a workshop, seminar or program. Respondents answer the questions individually and hand them back to the person conducting the evaluation.

• The questions have a direct relationship to recognized goals or objectives.
• It’s low in cost and easy to administer.
• It’s easy for respondent to complete.
• It gives immediate feedback.
• It can sample the total population.
• It can be used as a basis for discussion.

• Program participants may not be representative of the wider population, therefore findings may not be generalizable.
• Questionnaires usually cover a single topic or issue.
• It takes time away from the program.

1. The questionnaire is prepared following the guidelines for constructing a survey instrument. However, the objectives and instructions for completing the questionnaire are explained to the participants by the instructor, supervisor or agent. He or she also should assure participants that anonymity will be maintained.
2. The questionnaire is distributed to each participant to be filled out individually.
3. The questionnaire is collected and checked for completeness

Questionnaire Design

• Make a list of what you want to know and how the information will be used.
• Check to make sure the information is not already available somewhere else.
• Eliminate all but essential questions.
• As you write questions try to view them through the eyes of the respondents.

1. The title of the questionnaire should appeal to the respondents.
2. The type used should be large and easy to read.
3. The questionnaire should appear professional and easy to answer.
4. The introduction should identify the audience and the purpose of the survey and give directions on how to complete the questionnaire.
5. Questions should not appear crowded.
6. Each question should be numbered and sub-parts of a question should be lettered.
7. Questions should be arranged in a logical order with general questions preceding more specific ones. Easy-to-answer questions come first, followed by increasingly complex, thought-provoking, or sensitive questions. Personal or potentially threatening questions should be placed at the end.
8. Be explicit about what is required to answer each question.
9. Sufficient space should be left for answering open-ended questions.
10. Clearly indicate where branching occurs and where general questions resume.
11. Key words should be boldface or capitalize to avoid the possibility that they are misread.
12. Request for demographic information should be included near the end of the questionnaire.
13. For mail surveys, remind respondents to return the questionnaire and provide an addressed, postage-paid envelope.
14. The questionnaire should end with a "Thank You."

Telephone questionnaires: Telephone questionnaires dependent on oral communication, so special attention must be paid to designing a questionnaire that will assist the interviewer as much a possible in holding the respondent’s attention. Design and construction of the questionnaire are based on utility rather than aesthetics.

Designing a Questionnaire Cover Letter

1st paragraph:

The objective of these questions is to measure consumer preference. The first question lacks focus, consumers may like a particular brand, but may not buy it because of its high price.

The first question could be interpreted in weeks, months, years, or by date.

A respondent may answer the first question ambiguously. For example, "I have two boys and a girl. They are 5, 7, and 10 years old." It is not possible to determine the ages of each child from this response.

Knowledge questions include what people know and how well they understand something

Beliefs, attitudes and opinions include people’s perceptions, their thoughts, their feelings, their judgments or their ways of thinking.

Behavioral questions ask people about what they have done in the past, what they do now or what they plan to do in the future.

Attributes are a person’s personal characteristics, such as age, education, occupation or income. Attribute questions ask respondents who they are, not what they do.

Closed-Ended questions have pre-determined categories of responses from which the respondent can choose. When asking closed-ended questions make sure that all alternative response categories have been included.

• Are easy to answer.
• Do not require a lot of time.
• Require less interviewer training.
• Reduce interviewer bias.
• Can be used to compare and quantify individual responses.
• Facilitate data entry and analysis.

• Response categories must be known before questions can be written.
• Response categories may not be inclusive of all potential/likely responses.
• Response categories may be superficial or biased.
• Response categories may be interpreted differently by different respondents.
• Response is difficult if the response list is too long.
• They force participants to make choices.

1. Have you or members of your family ever taken classes at the Regional Education Center in this county? _____Yes _____No

2. To what extent do you agree or disagree with the new zoning code?

1. Strongly disagree
2. Mildly disagree
3. Neither agree or disagree
4. Strongly agree

• Stimulate free thought.
• Provide a chance for respondents to express feelings and opinions.
• Allow respondents to express themselves in their own terms.
• Can provide vignettes and material for explanatory quotes.
• Are excellent for exploratory studies.
• Can provide material to develop close-ended questions.

• Difficult to analyze.
• Require more time to answer.
• Depend on respondent recall.
• Require greater interviewing skill.
• Lack response categories to help clarify questions.
• Handwritten responses may be illegible.

1. How do you plan to use the information acquired during this training?
2. What do you think should be done to improve the 4-H program in this county?

A focus group is a small group, typically 8 to12 people who are relatively homogeneous, which is selected to discuss a specific topic in a non-threatening atmosphere. The focus group is moderated and recorded by a skilled interviewer. A focus group measures community needs and issues; citizens’ attitudes, perceptions and opinions on specific topics; and impacts of a particular program on individuals and communities.

• It is easy to setup.
• It is fast and relatively inexpensive.
• It is can reduce the distance between project personnel and intended beneficiaries.
• It stimulates dialogue.
• It can generate ideas for evaluation questions to be included in other survey methods.

• It is easily misused.
• It is requires special moderator skills.
• Data interpretation is tedious.
• Avoiding bias can be difficult.
• Capturing major issues that emerge can be difficult.
• Results may not be generalizable to the target population.

Good evening and welcome to our session tonight. Thank you for taking the time to join our discussion of county educational services. My name is _______ and I represent ____________. Assisting me is _________ from _________. We are attempting to gain information about educational opportunities in the community. We have invited people who live in several parts of the county to share their ideas.

You were selected because you have certain things in common that are of particular interest to us. You are all employed outside the home and you live in the suburban areas of the county. We are particularly interested in your views because you are representative of others in the county.

Tonight we will be discussing non-formal educational issues in the community. These include all the ways you gain new information about areas of interest to you. There are no right or wrong answers but rather differing points of view. Please feel free to share your point of view even if it differs from what others have said.

Before we begin, let me remind you of some ground rules. Please speak up, but only one person should talk at a time. We’re tape-recording the session because we don’t want to miss any of your comments. If several are talking at the same time, the tape will get garbled and we’ll miss your comments. We will be on a first- name basis tonight, and in our later reports, there will not be any names attached to comments. You may be assured of complete anonymity of responses. Keep in mind that we’re just as interested in negative comments as positive comments, and at times the negative comments are the most helpful.

Our session will last about an hour and we will not be taking a formal break. Well, let’s begin.

Let’s find out some more about one another by going around the room one at a time. Tell us your name and where you live.

2. Avoid dichotomous questions – those that can be answered with a yes or a no.

7. Consider using standardized questions (explain this - what does this mean?)

Note: These questions will be distributed to all advisory committee members during the focus group session.

1. Let’s find out some more about one another by going around the room. Tell us your name, where you live and what first comes to mind when you hear the words "Michigan State University."

   (Facilitator’s notes: Record key words and return to some of these later)

2. What are you hearing people say about Extension agriculture and natural resources programs in your community? How has Extension’s work changed in the recent past?

   (Facilitator’s notes: Probe positive and negative comments.)

3. Think back to an experience you have had with MSU Extension that was outstanding. Describe it.
4. Think back to an experience you have had with MSU Extension that was disappointing. Describe it. How could Extension change its programming?
5. Michigan State University Extension has adopted an Area of Expertise (AOE) team approach to Extension work. Have you taken advantage of the Area of Expertise (AOE) teams? What have been your impressions of the AOE team performance during the past year?
6. How can MSU Extension’s field crop AOE team improve its future program offerings? Could you suggest ways to improve Extension field crop programs?

   (Facilitator’s notes: Encourage each participant to respond. Refrain from probes until each participant has a chance to react.)

7. Do you have any final comments, recommendations or thoughts about either Michigan State University Extension or its agriculture and natural resources programs?

    

8. Have we missed anything?

• Is low-cost.
• Requires little time.
• Can encourage local participation.
• Can decrease outsider bias.
• Can encourage participation of frequently overlooked groups.
• Offers flexibility in method selection.

• Seasonal bias.
• Accessibility bias.
• Elite bias.
• Hypothesis confirming - selective attentiveness.
• Concreteness bias - confusing specificity with generality.
• Consistency bias - premature formation of coherence in data.
• May not be generalizable.

1. Identify goals of RRA and questions to ask.
2. Identify resources available for RRA – time, skills, staff, clientele, formal organizations, informal groups, firms, transportation, telephone/mail, media.
3. Review existing documentation.
4. Identify data needs, type of analysis needed.
5. Identify possible sources of information.
6. Identify, adapt and/or create data collection methods.
7. Field test methods.
8. Adjust questions, sources of information and approaches.
9. Plan when and where you’ll visit.
10. Begin data collection while remaining flexible to the situation. Adapt methods and adjust questions and activities as warranted.
11. Record data as collected in a systematic fashion.
12. Continually analyze data by verifying responses, deepening understanding, and making distinctions and connections between responses.

• It is good for addressing how and why questions.
• It is gives concreteness to problems and solutions.
• It is can be used to study evolutionary or decision-making processes.
• It is provides in-depth information on a single setting, group or organization.
• It is can be tailored to specific situations.
• It is can provide background information as a guide for further study.
• It is contributes insight into relationships and personal feelings.
• It is draws out underlying assumptions and general knowledge.
• It is can be used as a supplement to other methods.

• It is time consuming and requires a large amount of data.
• It is may not be generalizable to a larger population
• It is may provide data on only one or two aspects of a problem.
• It is requires good observational, recording and reporting skills.
• Information may be subjective because of investigator bias.

A detailed and systematic recording of evidence before and after a producer participates in a comprehensive financial farm management program could provide valuable insights into program impact that might be useful in expanding the program to a larger group of producers.

1. Review what is expected from the case study.
2. Define preliminary questions and hypotheses. Hypotheses suggest relationships between variables: e.g., Farmers participate in a program because they derive some benefit from participation.
3. Identify and define the boundaries of the case.
4. Assess the ability of the evaluator to carry out a case study: the ability to ask questions, to assimilate large amounts of new information without bias, adaptability and flexibility.
5. Anticipate key problems, events, attributes and persons that may be encountered.
6. Form initial plan, including role of on-site observer.
7. Arrange for access and negotiate plan of action.
8. Discuss arrangements for maintaining confidentiality.
9. Make preliminary observations of activities.
10. Identify informants and sources of information.
11. Develop record keeping system, files, tapes develop; coding system; arrange for protected storage of data.
12. Rework priorities based on emerging attributes, problems, events, audiences, etc.
13. Reconsider issues or other theoretical basis to guide data collection.
14. Make observations, interview, gather logs, use surveys, etc.
15. Select vignettes, special testimonies, illustrations.
16. Classify raw data; begin interpretation of data.
17. Redefine issues and case boundaries, renegotiate arrangements with hosts, if needed.
18. Gather additional data, using triangulation to validate key observations, review raw data for various possible interpretations.
19. Search for patterns in data.
20. Seek linkages between program arrangements, activities and outcomes.
21. Draw tentative conclusions, organize according to issues, and organize final report.
22. Review data, gather new data -- deliberately seek disconfirming evidence.
23. Describe setting within which case study took place.
24. Draft report and reproduce material for audience. Consider the report as if it were a story; look for ways the story is incomplete and fill in missing information.

Semi-structured interviews with project participants and other key informants begin with an interview guide that lists topics to cover and open-ended questions to ask. Probing techniques are used to solicit answers and raise new topics that reflect the people’s perspectives, beliefs, attitudes and concerns.

• They are useful in complex situations in which answers to questions can not be predetermined.
• They can be used to generate hypotheses to guide an evaluation.
• Respondents are not confined by pre-selected choices when answering questions.
• The structure of the interview is not predetermined but develops as the interview unfolds.
• Additional questions can be asked to clarify issues or explore new areas.

Disadvantages of semi-structured interviews:

• They may not be viewed as valid by those more familiar with quantitative methods.
• They require an interviewer who knows when and how to probe for more complete answers and can recognize emerging and relevant topics.

• Identify topics and develop open-ended questions.
• Remember your appearance and mannerisms have an impact on the interview. Dress appropriately and speak in a non-threatening manner and use easy-to-understand terms.
• Select a respondent following the chosen sampling criteria.

• Conduct the interview or select a mutually convenient time to return.
• To avoid distractions, try to conduct the interview without an audience.
• Explain the purpose of the interview to the respondent and remind participants that the interview is voluntary and his/her responses are confidential.
• Establish rapport by beginning with a general conversation on a neutral subject that might interest the respondent and share some personal background and express appreciation for the respondent’s responses.
• Begin with simple questions that do not require long answers or a lot of reflection, then move on to more complex and sensitive questions.
• Record answers verbatim.
• Do not express your personal opinions.
• If an answer is incomplete or appears irrelevant, probe to get a clearer response.
• If a respondent refuses to answer a question, attempt to get an answer but avoid doing something that might jeopardize the interview.

• Observation takes place in its natural setting.
• PO is unstructured and flexible.
• It can be readily combined with other methods.
• It’s useful for small units such as a neighborhood, a classroom or a group.
• It’s useful in assessing long-term effects of programs or practice on local residents.
• It may uncover behavioral patterns, social processes or problematic issues that participants are not aware of.

• It rarely provides enough information for an evaluation and must usually be combined with other methods such as interviewing.
• It requires an evaluator with well developed observational skills.
• The evaluator has less control over the situation.
• The presence of an evaluator may change the behavior of the group being observed.
• Observations may not be generalizable.
• The observer may lose objectivity as a result of being a participant.
• Time is often a limiting factor.
• It may not be suitable for large and/or heterogeneous groups.

1. Participant observation (PO) as used in evaluation is motivated by the need to solve practical problems, not to construct theory. Therefore, the evaluator using this technique should enter the field with a initial conceptual framework. The framework should include preliminary issues and the possible relationships between them.
2. Define the main concepts in the framework, e.g. learning style, leadership.
3. Identify sources of information.
4. Select the site in which participant observation is to be carried out. Selecting two or more sites allows for comparative analysis of data. An informal sampling technique is used in PO. The site selected should be representative of the type of program or organization being observed, the organization must be willing to accept the PO evaluator, the PO evaluator must be able to enter into activities under observation, timing is crucial for one time activities, seasonal events, or those with a daily routine.
5. Arrange for access and develop arrangements for maintaining confidentiality.
6. Assemble tools for observation: notebook, pen, camera, tape recorder, etc. How data is recorded depends on the situation. You may want to take notes on the spot or you may want to make notes after finishing your observations. Photographs and tape recorders assist in recording, but in some instances may be intrusive and influence the situation being observed.
7. Begin observation. You do not need to observe everything that is going on, but rather should focus only on those aspects of the activity pertinent to the evaluation. The following questions may help guide your observations:
   1. What is the setting of the scene you observed?
   2. What is going on?
   3. Where are you in relation to the scene you observed?
   4. What is the content of the situation being observe? – e.g. time of day; weather conditions; approximate number of participants; participants’ age, gender, ethnicity, and class; relationships (if any) among participants; people’s position in relationships to each other; other activities going on around the scene.
   5. Are you a participant in the activity? If so, how did your participation affect your observation?
   6. What is the age, gender, ethnicity, and class of those you are observing?
   7. If you asked more than one person, did you get the same answer?
   8. What does the scene you are observing make you think about? What puzzles you? What do you think you understand?
   9. Is the activity similar to or different from other types of activities you have observed in similar settings?
8. Can the activities that you observe be linked to any theoretical frameworks?

• It has high credibility as a source of information.
• It’s useful to justify budgets, demonstrate the value of a program, and/or assist in getting the most outcome from program inputs.
• It yields useful information for donors and funders.

• It may be difficult to quantify costs or benefits in monetary terms.
• It may be difficult to account for opportunity costs, hidden costs and/or assumed costs.
• It may be difficult to account for indirect benefits of the program.
• Bias may occur when assigning monetary value to costs and benefits.
• Bias may occur through underlying and untested assumptions.

Steps to Benefit/Cost Analysis

1. Define the program parameters.
2. Develop a list of costs and benefits from various sources. Program costs include direct costs, implied or indirect costs, and implicit or assumed costs. Program descriptions, professional literature, your own knowledge, and information compiled during initial phases of analysis. Program benefits are the positive outcomes or consequences resulting from the program or project. They include direct benefits and those that accrue over time. When determining costs and benefits, make sure that costs and benefits are measured at the same level.

   A. The cost equation: Cost = L + K + I – i

   L = labor: The cost per hour for labor including salary and fringe benefits. Fringe benefits vary but normally fall within 22 to 35 percent of full salary. The full labor hourly formula (L) is: (S+S.35)/260/8 where S = salary and S.35 = fringes, 260 = workdays per year and 8 = hours per workday.

   K = direct costs: Direct program costs budgeted for or assigned to the program, e.g., supplies, correspondence, communications, travel and per diem expenses, equipment and audiovisuals. If costs are shared between projects, the total is calculated from a cost/share equation. Opportunity costs are defined as opportunities that participants have lost in order to participate in the program. Opportunity costs are included in direct costs to the participants, the presenters or the stakeholders, depending on the level of analysis.

   I = indirect costs: Costs indirectly associated with the participants but directly associated with the program e.g., administrative costs such as facility rental, photocopying, report costs, telephone and prorated equipment and supplies costs.

   i = discount amortization: Measurable returns over time (both positive and negative). Discount amortization is not included if returns can not be traced over time.

   B. The benefit equation: B = Cr + DB + IB)

   Cr = cost reductions attributable to program activities

   DB = direct benefits - the primary outcomes experienced by participants and others directly involved in the program. They are typically derived from program objectives.

   IB = indirect benefits - secondary or intangible outcomes of the program or project experienced by participants, non-participants or society in general. These outcomes or consequences can be positive or negative.

3. Compare costs with benefits, either directly by subtracting costs from benefits or as a ratio of benefit to cost. The first equation provides a means of comparing costs with benefits within a given program; the second allows comparison between programs.

Benefits Estimate Worksheet Cost Estimate Worksheet

Step 7. Sampling for Evaluation

A sample is a set of respondents selected from a larger population for the purpose of a survey. When done properly, the sample represents the characteristics of the population as a whole. Sampling saves time, money, materials and efforts without sacrificing accuracy and precision.

Characteristics of population – addresses the amount of variability in the population to be sampled. A relatively homogeneous population may permit a smaller sample size. Conversely, a more heterogeneous one may require a larger population size.

Sampling error - the difference between an estimate taken from the population and that taken from the sample when the same method is used to gather the data. Sampling error is larger when the sample size is small. It is therefore advisable to use the largest sample size possible given the constraints on time, money and materials.

Degree of precision - measures the degree to which an estimate approximates the estimate obtained from the total population, assuming the same method of data collection was used. In designing a sample, the evaluator may begin by defining the degree of precision desired.

Margin of error - It is a matter of choice depending on the objectives of the inquiry. If we want to be relatively safe about our conclusions, then a 5 percent margin of error is acceptable (see appendix). In general, more subjects are needed for a .01 alpha test than a .05 alpha test, and a two-tailed test requires a larger sample size that a one-tailed.

Confidence level - the probability that a value in the population is within a specific, numeric range when compared with the corresponding value computed for the sample. Generally, a 95 percent confidence level will give the security needed to draw conclusions for the larger population based on the sample.

Cost - A small sample size reduces cost.

* n = population size; s = sample size

Random or probability sampling is based on random selection of units from the identified population. Random sampling techniques include:

Simple random sample - all the individuals in the population have an equal and independent chance of being selected as a member of the sample. A random numbers table is sometimes used with a randomly selected starting point to identify numbered subjects (see appendix).

Systematic sampling - all members in the population are placed on a list for random selection and every nth person is chosen after a random starting place is selected.

Stratified sampling – is used to ensure that certain subgroups in the population will be represented in the sample in proportion to their numbers in the population. Each subgroup is separately numbered and random selection is used for each subgroup. A definite rationale should exist for selecting any such subgroup.

Cluster sample - the unit of sampling is not the individual but rather a naturally occurring group of individuals such as a classroom, organization or community.

Matrix sample - one sample of people receives a given sampling of questions and another sample of people receives another sampling of questions.

Purposive sample is chosen to include a wide variety of people on the basis of a number of specifically chosen and critical characteristics. Purposive sampling does not rely on random selection of units.

Accidental sample - sample consists of individuals who are available at the time. This is the weakest type of sample. Generalizations to the larger population can not be made.

Reputational sample people are selected to respond to a survey or an interview based on a judgment of who is and who is not a "typical" representative of the population.

Analysis and interpretation of qualitative data are not simple technical processes like the analysis of quantitative data. Analysis of qualitative data is the process of bringing order to the data and organizing what there is into patterns, categories and basic descriptive units. Interpreting qualitative data is the process of bringing meaning to the analysis, explaining patterns, and looking for relationships and linkages among descriptive dimensions. The evaluator and/or stakeholders then make judgments about assigning value or worth to what has been analyzed and interpreted.

• It begins as soon as data collection begins.
• It is an iterative process that continues throughout data collection.
• Issues of validity and reliability are expressed in terms of clarity, verifiability and replicability.

• The words used by the participants and the meaning of those words.
• The context. Interpret the comments in light of the context.
• The internal consistencies and inconsistencies. Determine the cause of inconsistencies.
• The frequency or extensiveness of comments.
• The intensity of comments.
• The specificity of response.
• Dominant themes.
• Types of Qualitative Data Analysis

Content Analysis^: A coding or classifying technique that investigates pattern of information and the meaning of data within a specific conceptual framework.

Content analysis is a research method that uses a set of procedures to make valid inferences from text such as newsletters, meeting minutes, correspondence, interview transcripts, etc. The inferences are about the sender(s) of the message, the message itself, or the audience of the message. Content analysis can be used for many purposes, such as auditing communication content against objectives, coding open-ended questions in surveys, describing attitudinal and behavioral responses to communications, revealing the focus of individual, group, institutional or societal attention toward something. A central idea in content analysis is that many words of text are classified into much fewer content categories. Each category may consist of one, several or many words. Words, phrases or other units of text classified in the same category are presumed to have similar meanings. Content analysis procedures create quantitative indicators that assess the degree of attention or concern devoted to cultural units such as themes, categories or issues. The investigator then interprets and explains the results using relevant theories. It involves three steps:

1. Measurement consists of counting the occurrences of meaning units such as words, phrases, content categories or themes. Counting generates results that allow for more precise comparisons among texts. We also want to know how much more (or less) attention is devoted to some issue than to others. Quantitative analytical procedures often reveal similarities and differences among texts that would be difficult, if not impossible, to detect otherwise.
2. Representation deals with the fact that in content analysis essential syntactic or semantic features of the languages or text are omitted, as it is difficult to encode or represent the richness of language. One way that the meaning of words, phrases, or other textual units is represented is through classification into a set of categories. In assigning meaning to categories, not all connotations or nuances of meaning are pertinent. Examples: kind, state.
3. Interpretation

To make valid inferences from the text, it is important that the classification procedure be reliable in the sense of being consistent: Different people should code the same text in the same way. Reliability problems in content analysis usually grow out of the ambiguity of word meanings, category definitions, or coding rules. Three types of reliability are pertinent to content analytic analysis: stability, reproducibility, and accuracy.

Stability refers to the extent to which the results of content classification are invariant over time, i.e., whether content will be coded in the same way if it is coded more than once by the same coder.

Reproducibility refers to the extent to which content classification produces the same results when the same text is coded by more than one coder.

Accuracy refers to the extent to which the classification of text corresponds to a standard or norm. It is the strongest form of reliability, but usually not available and done. Sometimes, it is used to train coders, though.

The classification procedure must also generate valid variables, that is, it must measure or represents what the investigator intends it to measure. As happens with reliability, validity problems also grow out of the ambiguity of word meaning and category or variable definitions.

Face validity (weakest) consists of the correspondence between the investigators’ definitions of concepts and their definitions of the categories that measured them. A category has face validity to the extent that it appears to measure the construct it is intended to measure.

A measure has construct validity to the extent that it is correlated with another measure of the same construct. Thus, construct validity entails the generalizability of the construct across measures or methods. There is no simple right way to do content analysis, investigators must judge what methods are most appropriate for their purpose. Large portions of text, such as paragraphs and complete texts, usually are more difficult to code as a unit than smaller portions, such as words or phrases, because large units typically contain more information and a greater diversity of topics. Hence, they are more likely to present coders with conflicting cues.

1. Define the coding units: Words, word sense (code different senses of words with multiple meanings or code phrases that constitute a semantic unit), sentences (when interested in words or phrases that occur closely together), themes, paragraphs, whole text.
2. Define the categories, which involves two decisions: First, whether the categories are mutually exclusive, and second, how narrow or broad the categories are to be.
3. Test coding on a sample of text.
4. Assess reliability.
5. Revise the coding rules.
6. Return to step three until coders achieve sufficient reliability.
7. Code all the text.
8. Assess achieved reliability. Coders are subject to fatigue and are likely to make more mistakes as the coding proceeds. Also, as the text is coded, their understanding of the coding rules may change in subtle ways that lead to greater unreliability.

Data analysis focuses on organizing and reducing information and making logical or statistical inferences; interpretation, on the other hand, attaches meaning to organized information and draws conclusions. All interpretations, to some extent, are personal and idiosyncratic. Therefore, not only interpretations but also the reasons behind should be made explicit. Useful interpretation methods include the following:

1. Determining whether objectives have been achieved.
2. Determining whether laws, democratic ideals, regulations, or ethical principles have been violated.
3. Determining whether assessed needs have been reduced.
4. Determining the value of accomplishments.
5. Asking critical reference groups to review the data and to provide their judgments of successes and failures, strengths and weaknesses.
6. Comparing results with those reported by similar entities or endeavors.
7. Comparing assessed performance levels on critical variables to expectations of performance or standards.
8. Interpreting results in light of evaluation procedures that generated them.

One method of bringing multiple perspectives to the interpretation task is to use stakeholder meetings. Stakeholders can be supplied in advance with the results, along with other pertinent information such as the evaluation plan and list of questions, criteria, and standards that guided the evaluation; that way, the meeting can be devoted to discussion rather than presentation. At the meeting, findings are systematically reviewed in their entirety, with each participant interpreting each finding, using questions such as: What does this mean? Is it good, bad or neutral? What are the implications? What, if anything, should be done?

Scales of measurement refers to the type of variable being measured and the way it is measured. Different statistics are appropriate for different scales of measurement. Scales of measurement include:

Measure of central tendency:

The purpose of central tendency is to report a single summary score or category that best describes a set of observations. Mean, median and mode are the most common measurements of central tendency and are used to compare one group with another, identify some behavior that is unknown, or compare a group to a standard.

The mean is used for interval variables. It is the arithmetic average of all observations. You calculate mean by totaling all observations (scores or responses) and dividing by the number of observations. The mean is sensitive to "outliers" or extreme values in the observations. When your data has a few extremely small or large observations, the data are "skewed."

Example: 15 participants received the following scores: -2, -1, 1, 4, 4, 4, 7, 7, 7, 7, 7, 8, 8, 8, 9.

The mean of the scores (3 X/n) is:

(-2)+(-1)+(1)+(4)+ (4)+(4)+(7)+(7)+(7)+(7)+(7)+(8)+(8)+(8)+(9)/15 = 5.1

The median is most appropriate for ordinal variables. The median is the middle observation. Half of the observations are larger and half are smaller. The median is not as sensitive to the outliers as the mean.

Examples: Observation 1 = 6,8,13,18,25. The median is 13, because half the scores fall above this number and half fall below; Observation 2 = 1, 4, 7, 8, 10, 11, 21, 22. The median is determined by summing the middle two numbers, 8 and 10, and dividing by 2. The median is 9.

The mode is used for nominal variables. It is the observation or category that occurs most frequently. The mode can be used to show the most "popular" observation or value. A distribution can be either unimodal or bimodal.

Distribution A is unimodal or has a single mode of 83, with 18 responses.

Distribution B is bimodal or has two modes, 61 and 88, with 21 responses each.

• The distribution is approximately symmetrical.
• You are interested in numerical values.

• You are concerned with the typical score.

• The distribution is skewed.
• You have ordinal data.

• The distribution has two or more peaks.
• You want the prevailing view, characteristic or dominant quality.

Chi-Square (x²) is the most popular of all non-parametric inferential statistical methods. Chi-square tests for differences between categorical variables (e.g., nominal or ordinal data). There are both "one-way" and "two-way" chi-square procedures.

Example of one way chi-square: A sample group is asked a question about political party preference, assuming the question on the instrument form requires a categorical response (Democratic, Republican, Independent, etc.). The one-way chi-square would test for differences in popularity between the political party categories relative to the sample’s response to the question.

Example of two-way chi-square: Used if two categorical variables are to be compared. If the same group above were split into male and female, thus creating a new variable, "sex of the respondent," then this categorical variable could be compared (or "cross-tabulated") to political party choice. In this way, comparisons between the sexes on political preference may be evaluated (e.g., significantly more males are Republican and more females are Democratic).

Both the one-way and two-way chi-square procedures result in a chi-square value and associated significance (probability) level. Chi-square is a non-parametric statistic and as such requires no parametric data assumptions. The data must be categorical in nature.

t-Test is used to test the difference between two means even when the sample sizes are small. The significance of the t statistics depends upon the hypothesis the researcher plans to test. If you are interested in determining whether there is a significant difference between two means, but you do not know which of the means is greater, use the two-tailed test. If you are interested in testing the specific hypothesis that one mean is greater than the other, use the one-tailed test. Data should satisfy parametric assumptions: 1) the sample is selected from populations that are nominally distributed; 2) there is homogeneity of variance -- i.e., the spread of the dependent variable within the group tested must be statistically equal; and 3) data are of continuous form with equal intervals of quantity measurement. Dependent variables must be interval or ratio-type data.

T-test for matched pairs: if both groups of data are contained in each data record, the appropriate t-test is for matched pairs. An example of an appropriate use of the t-test for matched pairs might be to compare pretest and posttest scores where each person took a pretest (variable 1) and a posttest (variable 2). Both values are contained in each data record.

T-test for independent groups: If each case in the data file is to be assigned to one group or the other based on another variable, use the t-test for independent groups. For example, to compare reading scores between males and females, split the reading scores into two groups, depending on whether the person is male or female (each record in the date file is assigned to one group or the other).

Degrees of freedom: (this is not a complete description) The degrees of freedom (d.f.) reflect sample size. When two independent samples are being considered, d.f. are equal to the sum of two sample sizes minus 2; i.e., d.f. =(n1 + n2 –2).

Measures of variance indicate the spread or dispersion of the group and include range, variance and standard deviation.

Range is the difference between the largest and the smallest scores in a distribution.

Example: Scores of 3, 6, 8, 10, 14, 17. The range is 14 points. The scores range from 3 to 17.

Variance is the mean of the squares of the deviation scores. Calculate the difference (deviation) between each score and the mean of the scores, square the deviations, sum the squares and divide the sum by the number of scores minus 1.

Standard deviation measures the spread of data about their mean and is an essential part of any statistical test. It is calculated by taking the square root of the variance. This transforms variance into the same unit of measurement as the raw scores. Standard deviation is expressed in terms of "one standard deviation above the mean" or the like. If the standard deviation is 11 and the score is 63, then one standard deviation is above the mean is 74, two standard deviations is 85 and so forth. The value of this figure becomes apparent when we understand the relationship between standard deviations and percentiles in a normal curve. The area contained within +1 and - 1 standard deviations of the mean includes approximately 69 percent of all scores on the distribution. Therefore, in our earlier example 68 percent of all scores were between 52 and 74.

Another way of assessing the meaning of the standard deviation is to compare scores with percentiles. It is known that, in a normal distribution, 97.7 percent of the cases are below two standard deviations above the mean. So when a raw score for one case is found to be two standard deviations above the mean, we know that the case scored higher than 97.7 percent of all other cases.

Evaluators have a responsibility to report their findings to stakeholders and other audiences who may have an interest in the results. Communication with stakeholders should occur throughout the evaluation process to help ensure meaningful, acceptable and useful results. (Continue this)

Reporting results: A variety of reporting procedures may be used.

• Reports that are short, concise and to the point are the ones that get attention.

• Craft the style and content of the evaluation report to fit the intended audience.

• Avoid technical terms that your audience may not know.

• Use a conversational tone.

• Use a combination of long and short sentences.

• Read report aloud to check for confusing ideas and sentences.

• Write in an active voice.

• Use a logical structure for your documents.

• Allow sufficient time for writing drafts and getting feedback and proofreading.

A Final Step: Evaluating the Evaluation

• Conceptual clarity - Is the evaluation well focused and the purpose, role, and general approach clearly stated?

Program Chain of Events					Matching Level of Evidence
Program & decision impacts				To what extent and in what ways could the program improved? To what extent were informed, high-quality decisions made?
Practice and program changes			To what extent did intended use occur? Were recommendations implemented?
Stakeholder’s knowledge and attitude changes		What did intended users learn? How were users’ attitudes and ideas affected?
Reactions of primary users	What do intended users think about the evaluation? What’s the evaluation’s credibility? believability? relevance? accuracy? potential utility?
Stakeholder participation		Who was involved? To what extent were key stakeholders and primary decision makers involved throughout?
Evaluation activities			What data were gathered? What were the focus, the design and the analysis? What happened in the evaluation?
Inputs				To what extent were resources for the evaluation sufficient and well managed? Was there sufficient time to carry out evaluation?