What does it mean to measure the success of your community alcohol project?
By “measuring the success” of a project, we mean evaluation. Evaluation helps you assess whether your project is effective or not. It can also help you find out if your project is heading in the right direction and whether it’s achieving what it set out to do.
Why evaluate a program?
Evaluation can help you to:
- be accountable and prove that you did what you set out to do. People you may be accountable to include the community, funding bodies and other interested parties
- find out what worked, what didn’t and why
- identify the strengths and weakness of your project
- repeat your success and learn from your mistakes
- create a platform for future planning
- check your progress
- secure funding for future projects.
How do I evaluate a project?
Even though the evaluation section is at the end of this plan an alcohol community project guide, you should be thinking about evaluation methods while you’re planning your project. Leaving evaluation to the end means you could end up missing out on valuable information that arises during the project. Being aware of evaluation opportunities throughout the project gives you a better chance of measuring your levels of success. There are five key steps when considering evaluation.
Step 1. Work out what it is you want to achieve
When you’re thinking about evaluation, take a step back and review the situation. You should have a very clear idea of what it is you want your project to achieve, and how you are going to do it. You can identify this by asking the following questions:
- What problem/s are you trying to solve?
- What would make change happen?
- How do you plan to make change happen?
- What results do you want to see?
Answering the above questions helps you create a framework for your evaluation process.
Step 2. How to develop an evaluation plan
Once you’ve figured out what you want to achieve, you can start to draw up an evaluation plan. Your plan should include these questions:
- What results do you expect from the project?
- How will you measure your outputs (indicators)?
- How will you gather the evaluation information?
- Who will be responsible for collecting the evaluation information?
- How will you use the information to improve the future project?
We have developed a template to help you with your evaluation plan.
What are indicators in evaluation?Indicators are signs of progress. They’re used to find out if your project is achieving its goals and objectives. There should be at least one indicator for each project outcome.
Examples of indicatorsProject reach – If you want to measure how well your project is reaching your target audience and if it’s up to scratch, these indicators could include:
- Number of participants.
- The proportion of the target market participating in the project/program.
- Drop-out rate.
Participant satisfaction indicators could include:
- Did the participants feel comfortable that their concerns were listened to?
- Were the venue, times and dates of any project activities appropriate?
- Were topics relevant to participants?
Program implementation indicators could include:
- Was media coverage achieved?
- Were all activities implemented?
- Were materials for target groups easily understood and appropriate?
Short-term change indicators could include:
- Changes in knowledge, attitudes and intended behaviour.
- Increase in confidence, self-esteem, coping skills.
- Increased social networks and improved relationships.
Long-term change indicators could include:
- Community engagement.
- Increased mental and physical wellbeing.
- Increased opportunities such as education and employment.
Adapted from: Western Australian Centre for Health Promotion Research, 2010, My-Peer Toolkit, www.mypeer.org.au
Measures or assesses the activities in your project and the quality of your project. For example, is the project reaching the target group?Are participants satisfied with the project? Are all the activities of the project being implemented?
Measures and assesses the immediate effects of a program.
Impact evaluation is related to the aims/objectives of the project. For example, measuring the change in behaviors of the target group, and the level of community involvement before, during and towards the end of the project.
Measures and assesses the longer-terms effects of the project. Outcome evaluation is related to the goal/s of the project. For example, you would be looking at whether the project had an effect on the target group over a long period of time.
- What do you need to find out?
- What is the best way of doing this?
- Who should collect the data?
Some of the most common ways to collect data include:
Surveys and questionnairesThese are useful for getting a broad understanding of the views and experience of the participants, the wider community, agencies etc. Surveys and questionnaires can be rolled out by phone, mail, face-to-face or online. If you want to ask a lot of questions, develop yes or no questions. If you want more detailed responses, be aware that the survey will take participants longer to fill out and potentially less people will participate.
InterviewsIn-depth interviews are more time consuming so choose your participants wisely. Choose a variety of people who’ll offer different views and insights into the project. Interviews can be done over the phone or face-to-face.
Focus GroupsFocus groups are also great for getting in-depth information. Ideally, you’ll want a trained facilitator to conduct the sessions as they know how to effectively engage and manage group interactions.
Feedback formsThese are ideal for finding out how useful participants found your event/training/workshop. Use this form to ask participants what they found the most and least useful and what could have been better.
ObservationsThe aim of this method is to observe changes in people and/or their environment. Observational data collection methods can also record aspects of activities and events such as the:
- people taking part
- frequency and duration.
Diaries and logsAsk key people to keep a diary or a log of their involvement with the project. Make sure they record the processes and accomplishments of the project/activity. These can also be used to record decision making processes and statistics, such as how many people attended an event.
Media reportsGather and review media reports on the area (For example, you could see whether there has been an increase in positive reports about reducing alcohol-related harm in the area.)
Evaluation workshops and meetingsHost workshops with people who are involved in your project using pictures, photographs as well as spoken word, to get feedback from participants.
Step 4. How to analyse the informationOnce you’ve collected your data it’s time to start analysing it. This process involves three steps:
- Preparing your data so it’s all together in one place and easy to read.
- Analysing your data to work out what it’s telling you. Ask yourself these questions:
- Does the information you’ve collected show that you’ve reached your goals?
- Does the data show up any problems/issues that need to be tackled? (Keep your eye out for any unexpected outcomes, both good and not so good).
- Interpreting your data means to reflect upon the analysis and develop an understanding that combines all the information you have collected.
This can be a lengthy process, particularly if you’ve used a combination of data collection methods, which means you’ll have to repeat the process several times. This is OK because you can cross check your findings from various sources. You might also discover that the findings differ between different sources. Don’t be concerned, because this is worth investigating.
Step 5. Sharing and using your findingsThe findings of your evaluation aren’t meant to sit in a report and gather dust. Sharing your findings can help people associated with the project to recognise any problems or issues that may be preventing the project from progressing. It can help people learn from any mistakes and identify the successes. You can share the findings in a number of ways and formats, depending on the audience. Ways in which you can share your findings include:
- A full report – for funding bodies, your community group, or organisation.
- A brief report – for the media or a meeting with your local MP.
- A PowerPoint presentation – for presenting at a conference or a community meeting.
- A short You Tube video – to engage with a wider audience.
When preparing your findings make sure you answer the following key questions:
- Were the project aims and objectives achieved?
- How were they achieved?
- Were they achieved for all involved?
- What were the outstanding achievements of the project?
- How well was the project, and the process, managed?
- What lessons were learnt?
- What was the experience like for participants, the community group/organisation and the community?
- How did the project contribute to individuals, partnership organisations, the community?
- What unanticipated outcomes occurred? (positive and negative)
- What recommendations can you propose?
Information in this section was drawn from the following sources:
- Taylor M, Purdue D, Wilson M, Wilde P, 2005, Evaluating community projects: A practical guide. Joseph Rowntree Foundation, York.
- South Australian Community Health Research Unit, n.d., Using evidence to improve outcomes for communities: Evaluating your project, Practice brief Issue 1, South Australian Community Health Research Unit, Flinders University.
- Live Longer 2011. Community Health Action Pack, Australian Government, Department of Health and Ageing, Canberra
- Creative Victoria, 2015, Evaluating community arts and wellbeing, Creative Victoria, Melbourne. Available from: http://creative.vic.gov.au/Research_Resources/Resources/Evaluating_Community_Arts_and_Wellbeing