Introduction to the Toolkit

Copyright notice

This document, Toolkit for Engaging with People with Disability in Evaluation, is licensed under the Creative Commons Attribution 4.0 International Licence, with the exception of:

Australia’s Disability Strategy 2021-2031 logo and branding
any third-party material
all images and/or photographs.

More information on this CC BY license is set out at https://creativecommons.org/licenses/by/4.0/legalcode

ISBN: 978-1-76007-585-9

Online ISBN: 978-1-76007-584-2

Copyright

This work is copyright. You may copy, print, download, display and reproduce the whole or part of this work in unaltered form for your own personal use or, if you are part of an organisation, for internal use within your organisation, but only if you or your organisation:

do not use the copy or reproduction for any commercial purpose; and
retain this copyright notice and all disclaimer notices as part of that copy or reproduction.

Apart from rights as permitted by the Copyright Act 1968 (Cth) or allowed by this copyright notice, all other rights are reserved, including (but not limited to) all commercial rights.

Requests and inquiries concerning reproduction and other rights to use are to be sent to the Corporate Communication Branch, Department of Health, Disability and Ageing, GPO Box 9848, Canberra ACT 2601, or via e-mail to copyright@health.gov.au.

Acknowledgements

Acknowledgement of Country

The Department of Health, Disability and Ageing acknowledges First Nations peoples as the Traditional Owners of Country throughout Australia, and their continuing connection to land, sea and community. We pay our respects to them and their cultures, and to all Elders both past and present.

Acknowledgement of people with disability

This Toolkit for Engaging with People with Disability in Evaluation would not have been possible without:

people with disability
the families and carers of people with disability
representative organisations
Australia’s Disability Strategy Advisory Council.

We want to thank everyone for their meaningful work.

‘Nothing about us, without us’

Foreword

Australia’s Disability Strategy Advisory Council provides advice to all levels of government on how well Australia’s Disability Strategy 2021–2031 (ADS) is working. We take action to achieve ADS vision for an inclusive society where people with disability are:

treated equally
supported to live a good life.

We look to drive action and improvements to achieve the vision for an inclusive Australian society that makes sure all people with disability can fulfil their potential as equal members of the community. Achieving this vision requires people with disability and governments to work together with people with disability to understand how well ADS is working and how it can be improved.

In ADS consultations, people with disability said that a guide to working with people with disability in evaluation would be an important resource. We proudly support this Toolkit for Engaging with People with Disability in Evaluation (the Toolkit), as a needed step toward embedding the voices of people with disability in all stages of policy, practice and evaluation.

Evaluation can tell us many things. Importantly, they allow us to understand:

how useful our actions are
what the impact of our actions is
who our actions support
if our actions are achieving the outcomes we want.

Evaluation can be a useful tool to share stories about how people feel. It can also share stories about how different policies and programs affect people’s lives.

The Toolkit tells us what an evaluation is and how people with disability can get involved. For the people who plan and run evaluations, it gives advice on how to make evaluation accessible and inclusive. It recognises that people with disability are part of every community and that we are the experts on our own lives. It also acknowledges the important role we can play in improving the programs and services that affect our lives.

We know a more inclusive society needs change that will only happen when the voices of people with disability are amplified. Together, we can make sure our voices are heard and valued in the evaluation of the plans, projects and programs that matter to us.

We look forward to seeing how people with disability, evaluators and the organisations commissioning (or funding) evaluations use the Toolkit to make sure all people are a part of saying what has worked and what needs to be different.

‘Evaluation should not be a tick box. It should be a mirror, a compass, and a platform. This Toolkit helps ensure our evaluations reflect reality, guide inclusive action, and amplify voices often left unheard’.

(Sisaleo Philavong – ADS Advisory Council)

Executive Summary

1 in 5 Australians identify as a person with disability. This figure can be much higher in some communities. Recognising people with disability can be a part of many communities like First Nations, LGBTIQA+ and culturally and linguistically diverse communities.

ADS sets out a vision for an inclusive Australian society that makes sure people with disability can fulfil their potential as equal members of community. Achieving this vision requires a whole-of-community response, inclusive of government, business, the non-government and services sectors, community and individuals.

The intent of this Toolkit is to support people with disability to be confident to engage in evaluation and make sure evaluations are accessible and inclusive.

The Toolkit talks about why people with disability need to be involved in evaluation activities. It aims to help anyone commissioning, funding or conducting an evaluation to remove barriers to equal participation, support the vision of ADS and uphold the rights of people with disability.

To make this Toolkit, we worked with the disability community to understand how we can create inclusive and accessible evaluation. This included people with disability, their representative organisations, academics, the public and government colleagues.

A report on consultation activities and findings has been published as a companion to this Toolkit. You can access the report on Australia’s Disability Strategy Hub.

Who this toolkit is for

This Toolkit will support greater inclusion of people with disability by providing guidance and advice. There are 2 primary audiences for the Toolkit:

people with disability who would like evaluation to include their experiences and understand what is involved in an evaluation
anyone funding or planning an evaluation to be confident in undertaking inclusive evaluation and to take action to make evaluation activities accessible.

This Toolkit can also be used as a resource to assist families, carers and allies of people with disability to understand evaluation and support the voices of people with disability to be heard.

People who run evaluations of Australian Government programs should also refer to the Commonwealth Evaluation Policy and Evaluation Toolkit.

Toolkit structure

The Toolkit is divided into 3 parts:

Our approach to disability and evaluation

What is disability?

ADS acknowledges that people use different words to talk about disability. Each person with disability has a way of talking about disability and about themselves that they like best.

We also acknowledge that there isn’t a definition of disability that everyone agrees on and the way disability is explained and understood varies between different people, cultures and communities. In line with the United Nations Convention on the Rights of Persons with Disabilities (UN CRPD), disability can be broadly defined as an evolving concept that results from the interaction between a person with impairment(s) and attitudinal and environmental barriers that hinder their full and effective participation in society on an equal basis with others.

ADS takes a strengths-based approach and uses person-first language to talk about disability. This means ‘people with disability’ is often used in this Toolkit.

What is evaluation?

Evaluation is about looking at a policy, program or service to tell us if it is working. It tells us what can be improved and helps us to learn from what has happened.

The people planning and paying for an evaluation are known as evaluation commissioners or evaluation funders. Someone who is hired (or commissioned) to check how well something is working and suggest ways to improve it is an evaluator.

Who is involved

Different people can be involved in an evaluation, depending on how big or small it is. They include:

community stakeholders – people involved in or impacted by the policy, program or service who can share their experiences and feedback
evaluators – someone who is hired (or commissioned) to check how well something is working and suggest ways to improve it
evaluation commissioners – the people planning and paying for an evaluation. They are also known as evaluation funders.

Ways to collect data

To answer questions about a program, an evaluator will collect data. Data includes many kinds of information including stories and numbers.

There are many ways to collect data. An evaluator will be interested to talk with people and listen to their views.

Examples of evaluations

There is no one way to evaluate something. A good evaluation is designed to match what is needed. This means it collects the best data to answer the evaluation questions. It is not about doing things in the same way each time.

Some people who commission an evaluation have money available for a large evaluation. They might hire lots of evaluators who might talk with people all over the country and collect a lot of data. But not every evaluation needs to be big. For example, a smaller evaluation can work well for a local program.

Some evaluations are about value for money. They use amounts of money and numbers to explain if the activity has made a difference. This kind of evaluation is used to decide whether to keep funding a policy, program or service.

Examples

Below we mention some examples of existing evaluations.

Western Australia for Everyone

The Western Australian Government commissioned an independent evaluation of the Western Australia for Everyone: State Disability Strategy 2020-2030 (the Strategy).

The evaluators collected data through:

co-design workshops
conversations with people with disability
surveys with a range of people, including disability sector organisations.

The evaluation found that after 3 years, the Strategy was still important and relevant for supporting the inclusion of people with disability in Western Australia.

Be Connected

The Department of Social Services funded an independent evaluation of the Be Connected Program. The evaluation found that for every $1 spent there was $4 created in social value by the program. The evaluation found that the program made a real difference with older Australians. The program supported them to build confidence in their digital skills and stay safe online.

Link-Up

The National Indigenous Australians Agency co-commissioned a monitoring and evaluation strategy with the National Link-Up Leadership Group. Link-Up Services support First Nations peoples who were removed from their families and communities as children, as well as their descendants. Descendants are people who come after someone in their family tree. For example, if you have children, they are your descendants.

The program helps them to reconnect with family, community, culture and Country. The monitoring and evaluation strategy is used by 8 Aboriginal community-controlled organisations. These organisations are independent and run Link-Up Services.

Payment by Outcomes Trial

The evaluation of the payment by outcomes trial for social enterprise was split into 2 parts. The first part looked at the program design and how it is being rolled out. The second part looked at the program findings. The evaluation used storytelling to share the voices of people with disability. Its approach centred around people with disability providing important insight into making employment more inclusive.

Glossary

Auslan – the name for Australian Sign Language and the main language of Australia’s Deaf community.

Accessible – environments, facilities, services, products and information that people can use and interact with in a way that suits their needs.

Barriers - attitudinal and/or societal actions, physical and/or environmental factors, and policy, and/or systemic issues that create a disabling effect. Such as inadequate time and resources to allow people with disability to contribute meaningfully.

Bias – an unfair outlook or perception towards someone or something based on opinion or experience.

Carer – someone who provides supports to a person with disability, often a family member. Some people with disability prefer the term over support person.

Citizens’ Jury – a panel of diverse people who have been randomly selected. They aim to be fair, well-informed and thoughtful in how they explore topics.

Co-design – a design process where stakeholders are equal partners and take leadership roles in the design of products, services, systems, policies, laws and research.

Co-facilitation – is when 2 or more people work together to lead or guide a group activity, workshop or meeting. They share responsibilities like planning, presenting and supporting participants.

Co-production – an extension of co-design, where stakeholders are involved in the development and design, decision-making, implementation and evaluation of products, services, systems, policies, laws and research.

Collaboration – working together with others to achieve a shared goal. It involves sharing ideas, helping each other and combining strengths.

Cultural safety –respects, supports and empowers the cultural rights, identity, values, beliefs and expectations of a particular culture while providing quality services that meet their needs. 

Descendants – people who come after someone in their family. For example, if you have children, they are your descendants.

Disability – an evolving concept that results from the interaction between a person with impairment(s) and attitudinal and environmental barriers that hinder their full and effective participation in society on an equal basis with others.

Discrimination – is when people are treated unfairly or different because of who they are. It often results in people being excluded, hurt or denied opportunities.

Easy Read – is a way of presenting written information to make it easier to understand. It typically uses simple words, short sentences, accessible fonts and layouts and images or photos to assist in illustrating the information.

Ethics – is more than just knowing what’s right or wrong – it is about doing what is fair, respectful and responsible in how we treat others and make decisions.

Evaluation – is a structured and fair way to see how a policy, program or service is working. It looks at how relevant, efficient, effective and sustainable it is. An evaluation helps to strengthen accountability and learning. 

First Nations people – First Nations people include Aboriginal and Torres Strait Islander people who identify as such and are accepted as such by their community.

Human rights approach – disability is a natural part of human diversity that must be respected and supported in all its forms. People with disability should enjoy all human rights and fundamental freedoms on an equal basis with others, including respect for their inherent dignity; freedom from all forms of violence and abuse; and to participate fully in the community. 

Inclusive – making sure everyone feels welcome, respected, and able to take part – no matter their background, identity or ability. It is about treating people fairly and making sure no one is left out.

Indigenous Data Sovereignty – the right of First Nations peoples to exercise ownership over Indigenous Data. Ownership of data can be expressed through the creation, collection, access, analysis, interpretation, management, dissemination and reuse of Indigenous data.

Intersectionality – recognises that each person’s identity is made up of many parts — like age, gender, race, disability and religion. These parts work together, not in isolation, shaping unique life experiences.

Ownership – is when people with disability have a level of control and power over an evaluation’s aims, processes and actions.

Partnership – means working together to achieve a common goal.

Person-centred – puts an individual at the centre of all actions and decisions.

Program logic – is a structured representation that links a program’s activities to its intended outcomes, showing how and why the program is expected to achieve its goals.

Reasonable accommodations and adjustments – article 2 of the Convention on the Rights of Persons with Disabilities defines reasonable accommodation as ‘necessary and appropriate modification and adjustments not imposing a disproportionate or undue burden, where needed in a particular case, to make sure persons with disabilities the enjoyment or exercise on an equal basis with others of all human rights and fundamental freedoms.’

Remuneration – is when people are paid for their work.

Self-determination – refers to a person’s right to control and make decisions about their own life. But it can also mean different things to different groups of people.

Stakeholder – is someone who has an interest in or is affected by a plan, project, or decision. This can include people who help make it happen, benefit from it, or are impacted by its results.

Stigma – is a negative belief people share about certain traits. It usually isn’t true and can mean people with these traits are treated unfairly.

Social model of disability – recognising attitudes, practices and structures can be disabling and function as barriers preventing people from fulfilling their potential and exercising their rights as equal members of the community. The social model seeks to change society to accommodate people with disability.

Supported decision-making – processes and approaches that assist people to decide, by giving them the tools they need to make the decision for themselves, including accessible information. Supported decision making does not mean deciding for or on behalf of another person. 

Tender – the process of asking organisations to formally submit an offer to provide a good or service.

Theory of change – is a clear and logical explanation of how and why a desired change is expected to happen in a specific context.

Trauma-informed – is when frameworks and strategies make sure that the practices, policies and culture understand, recognise and respond to the effects of trauma and minimise, as far as possible, the risk that people may be re-traumatised.

Universal design – Disability Inclusion Bill 2024 definition: universal design means the design of an environment that can be accessed, understood and used to the greatest extent possible by all people regardless of the nature of their attributes, including their age, size, ability or disability.

Acronyms

ADS Australia's Disability Strategy 2021-2031

Cth Commonwealth

DDA Disability Discrimination Act 1992 (Cth)

DRO Disability Representative Organisation

DHDA Department of Health Disability and Ageing

LGBTIQA+ Lesbian, Gay, Bisexual, Transgender, Queer/Questioning, Intersex, Asexual. The + stands for all other identities not encompassed in the short acronym.

MSC Most Significant Change

NDIS National Disability Insurance Scheme

NDRP National Disability Research Partnership

UN United Nations

UN CRPD United Nations Convention on the Rights of Persons with Disabilities

WCAG Web Content Accessibility Guidelines