Towards Wellbeing in Forest Communities: A Source Book for Local Government

Tool 3: Community evaluation
Steps 7-8

Box 32. Tips and options

Supply accurate information
Assure anonymity
Use methods that are interesting, innovative and not time consuming
Provide incentives for community members to participate in the assessment.

Step 7. Analyse and present results

Combine the results by programme, community, or focus group. Average and compare the results. For instance, officials may want to compare the overall success of a programme implemented in several communities by comparing how women and men faired across all the sites. To do this, first add together the scores of all women who participated in the evaluation. Then divide the votes by the total number of participants. This provides an average. Repeat the same steps for the men. This procedure allows comparison of programmes across groups with different numbers of respondents.

Box 33. Example of scores from focus groups

Scores from two communities on impact of several education and health programmes. Scores range from 0 (very negative impact) to 1 (very positive or good impact).

Sector	Project	Community A			Community B
Sector	Project	Elite	Women	Youth	Elite	Women	Youth
Education	Free school uniform	0.94	0.75	0.92	1.00	0.58	0.95
	Operational school funds	*	-	0.15	0.25	0.47	0.60
	Free books	0.22	0.14	0.88	0.94	0.28	0.80
	Scholarship	N/A	N/A	N/A	0.81	0.47	0.50
Health	Health insurance	0.72	0.9	0.69	0.56	0.31	0.65
	Immunisation	1.00	1.00	0.92	1.00	0.64	0.80
	Supplementary food for children < 5 years	0.72	0.93	0.73	1.00	0.14	0.85

* In the discussion with the elite this project was not mentioned, so no score was given.
N/A = not applicable.

Use a colour code to visualise differences (see Figure 19). This information is useful for decision makers concerned with impacts in specific locations or target populations.

Figure 19. Comparison of normalised focus group evaluations in one community in Malinau.

Photo by Michaela Haug

Organise a presentation of the evaluation for community members and government officials. Present the information in a format that is easy to understand and meaningful to community members as well as to government officials. Record comments made during the presentation. The comments are very important because they provide information about where improvements are needed. Some comments might be very specific, but consistent patterns may emerge. For example, in Malinau, two common problems identified with government programmes were the lack of information about the programme and the lack of follow up, like providing planting or livestock breeding material without technical assistance (Box 34 gives examples of comments made by participants during the discussions).

Be careful that the results are not misinterpreted. For example, percentages can be misleading if only a few people were involved in a focus group. For instance, if 100% of the people agreed on a positive evaluation, but there were only two people in the group, then this high percentage score has little significance.

Box 34. Examples of comments given in community evaluation, Malinau

EDUCATION

Free school uniforms:
- Number of uniforms not sufficient
- Size—some uniforms did not fit
School operational funds:
- Information about programme not clear
- No control over implementation
- Not appropriate to local conditions (prices in Malinau are relative high)
Free school books:
- Number not sufficient and not all books are provided
Scholarships:
- Information about programme not clear
- Amount provided does not consider difference in expenditure at different levels of education
- No control over implementation
- Scholarships from the district government not yet received.

HEALTH

Health insurance:
- Not all people received insurance cards (unclear eligibility)
- Service at health centres not satisfactory
- Procedure to use insurance too bureaucratic
Immunisation:
- Implementation erratic
- Some children got infections from vaccination
- No additional general medical service provided during visits of health staff to provide immunisation.
- Supplementary food for babies under 5 years old
- Amount not enough and delivery erratic.

Step 8. Use results to improve programmes

The results of the evaluation should be channelled into the local government’s evaluation and planning cycle. Theoretically, a number of offices ought to be interested in this information, such as the agencies responsible for the evaluated projects, the planning agency, the agency responsible for fiscal accountability, and the representatives elected by the people. Alternatively, communities and organisations outside of government can use the results to draw attention to weak programmes or call for change. Responsiveness is more likely where there is pressure from both outside and inside the government.

It is important that the evaluation results are communicated to a broad audience, including community members, local government and other local institutions. The communication strategy can include the following actions:

Distribute the results to participants and other community members; post the results in a public place so that everyone can see and discuss them;
Organise a community meeting to present the results; ask the community to brainstorm on how they might use the results;
Facilitate a discussion of the results as part of the annual community planning (see Tool 4);
Organise presentations where community members can share their results with local government and other communities; invite external audiences such as regional government, and environmental and development organisations;
Involve the local press; provide them with written materials; suggest that participants talk about the exercise on a radio programme.

Box 35. Tips for getting started

Focus group discussion for evaluating government programmes is a good way of improving communication between local government and communities, but might be perceived as sensitive or even threatening. Thus, the initial approach is important to avoid misunderstandings. To achieve the best results with this approach think about the following points to get started.

Organise informal discussions or meetings with local government and institutions to present the concept. The discussion should include: what is the aim of the survey, which sectors’ impacts are interesting/relevant, what are key agencies to involve. It is important to discuss these with key decision makers to ensure good understanding, support and involvement. During this phase, a form can be developed to make documentation (and reporting) of the results easier.
Train the staff who will conduct the focus group discussions. One way of training is to conduct a training session with all the survey team members. If limited time is available, on-the-job training can be done by gradually involving the team members in facilitating the focus group discussions.
Prepare for going to the field. If government officials will conduct the focus group discussions, scheduling and budgeting may need (considerable) time. Timing of the survey will have to consider best timing for the communities that will be visited, as well as lead time for government officials to deal with bureaucratic requirements and obtaining budgets. During this time, communities should be informed (by letter or other appropriate means) that the evaluation is going to take place. The information should introduce the aim of the survey and explain the use of focus groups.

For more information on focus groups and community-based evaluation, see:

Kreuger, R.A. 1988 Focus groups: a practical guide for applied research. Sage, London.

Morgan, D.L. 1997 Focus groups as qualitative research (2nd ed.). Sage, London.

Morgan, D.L. and Kreuger, R.A. 1993 When to use focus groups and why. In: Morgan D.L. (ed.) Successful focus groups. Sage, London.

Stewart, D.W. and Shamdasani, P.N. 1990 Focus groups: theory and practice. Sage, Newbury Park, CA, USA.

Wollenberg, E., Limberg, G., Iwan, R., Rahmawati, R. and Moeliono, M. 2006 Our forest, our decision. A survey of principles for local decision-making in Malinau. CIFOR, Bogor, Indonesia.

Photo by Kristen Evans

< back

next >