Part 1: Getting involved in evaluation as a person with disability
‘It should be a priority to ensure that people with disability know about, and be included in, decisions about all parts of the evaluation … evaluations need to be about giving people with disability choice and control over their own lives ... and evaluations are about people with disability knowing what is best for themselves’.
Your voice matters
As a person with disability, you are an expert on your own life and experiences. When you take part in an evaluation, it will reflect more diverse ideas and experiences. It will also help the evaluation be correct and clear about how well a policy, program or service is working.
You have the right to take part in decisions about policies, programs and services that affect your life. You have the right to tell people about barriers and share your ideas about how to remove them.
The experiences and ideas of people with disability can lead to stronger findings. These can be used to improve how things work for all Australians.
What you can expect
There are many reasons to evaluate: to learn, measure outcomes and show the value of something. Evaluation show us what is working and how we can improve.
We have written a checklist to help people with disability, their families and allies. It will give you some examples of things to think about before and during an evaluation.
If you decide to take part in an evaluation, your role and the questions you answer will depend on how and when you are involved. For example, you might:
- help plan or design the evaluation
- be a participant – someone who answers the evaluation questions
- look at people’s answers and give advice about what has been learned.
Respect your privacy
Evaluators and evaluation commissioners must respect people’s privacy. This includes yours. They should tell how they will store, share and destroy personal information.
Support to make decisions
Everyone has the right to make choices about their own life. If you are thinking about taking part in an evaluation, you can ask the evaluators for any extra information to help you decide. If you need support to make decisions, the evaluators can work with you and the people you trust. For example, your:
- guardian or person who helps you make decisions
- friends or family.
This is known as supported decision-making.
It is important that people with disability have the information they need to decide if they want to take part in the evaluation and understand what they’re agreeing to.
You can learn more about supported decision-making by going to the Supported Decision Making Hub.
Other support you might need
It is important to know that you might be asked about topics that affect your life directly. These topics might be uncomfortable or upsetting. Before you take part in an evaluation, it is good to find out if there is someone you can contact if you feel this way. The evaluators and evaluation commissioners should help you feel safe and supported to speak about any concerns. You can ask what support is available if you do not know or if it is not clear.
Deciding to take part
To decide if you want to be part of an evaluation, it is important to think about:
- if you want to be a part of the evaluation
- why you want to take part in the evaluation
- if there are other people taking part who have different experiences
- your role and what is expected of you
- how much time you will need to give
- if you will be paid and how you will be paid
- what supports are available
- what accommodations or adjustments will be made
- how your information will be used.
‘This resource will provide the understandings necessary for researchers to do evaluation effectively and respectfully in collaboration with people with disability.’
Being involved and recognised in evaluation
Your time, ideas and knowledge are important. It is important that this is shown. There are many ways this can happen, someone might:
- pay you – this is called remuneration
- formally acknowledge your work – for example, including your name in a written report
- say ‘thank you’ – this is more likely if someone fills out an online survey or something similar that does not include meeting or working with others.
It is important that the evaluators are clear from the beginning about how they will show they value the work of people with disability. They also need to be clear about how much of your time is needed.
You can tell the evaluators if you think you will need more time to contribute. You can tell evaluators if they have left out some of the costs for you to take part. You can ask evaluators if they can help with these things.
If you are being paid, you might receive this money:
- directly into their bank account
- as a gift card.
You can learn more about having your time and work recognised in Fact sheet 4: Having your time and work recognised.
How people with disability can take part
If you decide to take part in an evaluation, you might be asked to take part at different stages and in different ways. This Toolkit explains evaluation in 3 simple stages: plan, do and use.
You can learn more about collaboration in Fact sheet 2: Collaborating in evaluation.
1. Plan the evaluation
In the first stage, evaluation funders or commissioners will decide what the evaluation will look like. They will think about:
- why they need the evaluation
- what they need to evaluate
- how they will evaluate it
- who will take part in the evaluation
- how much the evaluation will cost
- who will run the evaluation
- how long the evaluation will take.
You might help make these decisions or you might be one of the people they ask to run their evaluation.
The evaluation stays in the planning stage after the evaluators have been chosen. This is because more decisions need to be made about how to run the evaluation.
Every evaluation should start with being clear about what is being evaluated. A common tool to explain what is being evaluated is a ‘theory of change’ or ‘program logic’. You might be asked to help make this.
A theory of change is a clear and logical explanation of how and why a desired change is expected to happen in a specific context.
Program logic is a structured representation that links a program’s activities to its intended outcomes, showing how and why the program is expected to achieve its goals.
You can learn more about how to do this through a video made by the Australian Institute of Family Studies. The video is a step-by-step guide to program logic.
In the planning stage, you might be asked about:
- what is the problem
- what group of people the problem affects
- how the problem affects them
- what the goal of the policy, program or service is
- how the policy, program or service tries to reach its goal
- what is already known about the policy, program or service
- the connection between the organisation that runs the policy, program or service and the community it affects
- what needs to happen for the policy, program or service to change.
The National Disability Research Partnership have some tips to help understand policy, program, service and impact in Australia. It also explains who in governments is involved in making decisions. You can use it for support if you’re thinking about taking part in an evaluation that looks at government policy, program or service.
2. Do the evaluation
This stage is where evaluators will ask people to participate. They will ask people for information and then look at what they can learn from that information. This is known as data collection and analysis. The ‘data’ refers to the feedback heard from the people involved. It might include your feedback. Data can be things like interview notes, focus group recordings, written submissions, survey results or video recordings.
Data collection requires evaluators to work out what information they need to collect and how to collect it. Evaluation data often includes both stories and numbers, as combining different types of information makes the findings stronger.
If you are taking part in this stage, you will be giving feedback and sharing your experiences about the evaluation topic. You may be asked about:
- how the policy, program or service helped
- what did and didn’t work
- how the policy, program or service could be better
- what might have happened if there was no policy, program or service.
You may be one of many people asked to share ideas. There might be other participants who don’t agree or who have a different opinion. The people who organise the evaluation should make sure everyone has a chance to talk about their ideas. They should also support everyone to share their ideas privately or in a different way, if they would like.
You can ask for support if you need more time or information to answer any of the questions. You can ask to check how your answers have been recorded to make sure they show what you wanted to say.
Analyse the data
To analyse means to study something in detail to understand more about it. If you are involved in this stage, you are helping to analyse the data that has been collected.
Once all the data has been collected, you might be asked to help summarise the information and look for patterns. These are the outcomes. When you are taking part in this stage, you might be asked to look for:
- patterns in the numbers and things people have said
- any changes over time or for different groups of people.
3. Use the evaluation
Once the evaluators have finished doing the evaluation, they need to decide how to tell people what the evaluation found. The most common way to do this is through a written report, but that is not the only way.
When evaluators share what they found out, they should explain:
- what happened
- what feedback people shared
- the findings and what they mean
- recommendations or next steps.
They should share this information clearly and in ways that people can use to make better decisions.
If you take part in this stage, the evaluators might ask you to think about:
- what you think would make the policy, program or service better
- if you think the findings show what is happening in the real world
- which recommendations you think are the most important to tell decision makers
- any ideas about how to share the information in the report.
To help make sure the evaluation is useful and leads to change, evaluators might ask you to help share the findings and recommendations. If you want to do this, this might include:
- your friends and family
- people you work with
- service providers you use
- other forums and groups.
You might also want to think about whether you need support to do this and what kind of support you need.
Part 3 of this Toolkit includes a number of fact sheets and checklists to support you in evaluation.