Research, evaluation and monitoring

Family violence prevention activity is, by nature, a long-term endeavour. Activity must, therefore, be supported by a long-term theory of change based on evidence. Progress must also be monitored to ensure the work is on track and continues to add to the evidence base. The Family Violence Reform
Rolling Action Plan notes that:

A strong and effective family violence evidence base is key to delivering long-term, sustainable reform of our family violence system. It tells us what is working, what needs to be adjusted, and where to focus our efforts for the greatest effect.

Research and evidence

Stakeholders told us that evidence in addressing the gendered drivers of violence to prevent that violence from occurring is relatively strong. As Free From Violence stated:

While we have a sound evidence base for the prevention of violence against women (outlined in Change the Story), we know less about what causes, and what works to reduce, other forms of family violence including elder abuse, violence against people with a disability, adolescent violence and violence within LGBTI communities.

Stakeholders confirmed that these gaps remain [relates to action 1]. They also highlighted building knowledge around what works to prevent violence for multicultural communities and how to engage men and boys to prevent violence against women as areas for further development.

Table 1 presents a snapshot of more recent evidence collections we found for diverse population groups. All these documents indicate that evidence of what works to prevent violence among these groups is limited and must continue to be built on. Pursuing these and other knowledge gaps must include evaluating and monitoring progress of existing initiatives (see section below) and undertaking research to fill the gaps. The family violence research agenda includes primary prevention of family violence and violence against women as a research priority. However, we have been told that no funding has been provided to support this research priority. While Respect Victoria has an accepted role in research, its current level of funding means it has some limited capacity to undertake select research but not to be a major research funding body. Therefore, the role of others with the capacity to fund research in line with the research agenda – such as universities, the sector and industry – will continue to be vitally important.

Reviewing funding arrangements and organisational capacity for research is beyond the scope of this review, but we note that Respect Victoria is considering ways to enhance research coordination and partnerships to improve collective research capacity in line with the research agenda.

Furthermore, while building the evidence base through research is essential, there are numerous actions that can be taken now to reduce gender inequality and other forms of inequality and discrimination against people from various marginalised groups, and to ensure broader primary prevention efforts are more inclusive. For example:

• Changing the Landscape contains six high-level essential actions to tackle the underlying drivers of violence against women and girls with disabilities, including a number of more specific actions, such as ensuring that prevention or gender equality initiatives include women and girls with disabilities and improving representation of women and girls with disabilities in the media and popular culture.
• Pride in Prevention suggests some initial priority areas for intervention to address family violence in the LGBTIQ+ context, one of which is ‘supporting families’. For example, it suggests that ‘existing programs for new parents and other family support programs could be expanded to incorporate positive messaging around having a child that is LGBTIQ’.

Table 1: Recent efforts to collate Australian evidence for what works to prevent violence for different cohorts

Progress and outcomes monitoring

Given the long-term nature of primary prevention work, and the expected long road to seeing a drop in levels of violence, it is essential to monitor progress in addressing the drivers of violence to ensure efforts are on the right track and to recalibrate the implementation approach as needed. We note the important role of Respect Victoria’s three-yearly progress reporting to parliament in supporting strong oversight and visibility of overall progress. As discussed in section 1, we also highlight the importance of having a clear theory of change at the outset – one that underpins the work program and supports the system architecture – against which expected short-, medium- and long-term outcomes can be assessed.

While a series of primary prevention outcomes and indicators are outlined in the Free From Violence Outcomes Framework – and reflected in the broader Family Violence Outcomes Framework – these do not capture the more immediate changes that can be expected.

A revised Free From Violence outcomes framework was to be published in the Free From Violence second action plan, but this didn’t occur [relates to action 2]. Instead, the second action plan states that ‘(t)he Department of Families, Fairness and Housing, with Respect Victoria, will also refresh the outcomes in the early phase of the Second action plan to develop short- and intermediate-term outcomes, indicators and measures to accompany the existing long-term outcomes’. It is positive that there is a commitment to developing phased outcomes, and we understand that this work is underway. Our Watch’s Counting on Change guide to prevention monitoring sets out broad short-, medium- and long-term indicators and measures that are appropriate for measuring population-level progress in addressing the drivers of violence against women (see Figure 16). This, along with Tracking Progress in Prevention, a report that operationalises Counting on Change, may assist as Victoria’s work on outcomes progresses.

Using these measures and indicators to effectively track progress requires the availability of data. Counting on Change and Tracking Progress in Prevention identify where there are gaps in existing data, and how these may be addressed to ensure more effective tracking of progress to address the drivers at the population level.

There are some major population-level data sources that support ongoing monitoring. The National Community Attitudes towards Violence against Women Survey (NCAS), conducted every four years, highlights areas of progress and areas where there are problematic attitudes that need to be targeted through primary prevention activities. ANROWS also performs periodic deep dives into NCAS findings that are of particular concern, such as the 2017 NCAS finding that four in 10 Australians mistrust women’s reports of sexual violence. These deep-dive reports provide more nuanced recommendations for primary prevention. Helpfully, Respect Victoria has produced guidance to support prevention specialists to use NCAS data.

Respect Victoria’s creation of the Prevention of Family Violence Data Platform provides a repository of data organised according to agreed indicators in the Free From Violence outcomes framework. Over time, this platform will provide trends to help assess the collective impact of Victoria's prevention activity. The data platform heavily relies on four-yearly data sources such as the NCAS and the Australian Bureau of Statistics’ Personal Safety Survey and General Social Survey, with 18 out of 27 indicators relying on measures from such sources, and there are four indicators where there are currently no measures available (see Table 2). Attitudinal data by local government area is not available, although we note that Victoria funded a larger sample size in the 2021 NCAS survey to allow for more nuanced analysis of the data.

We are not aware of any processes to collate data from the broad range of program evaluations occurring across the state but suggest that putting such processes in place will play an important role in building the evidence base and monitoring progress. We note that Respect Victoria’s Monitoring and Evaluation Strategic Framework provides high-level advice to those involved in implementing the Free From Violence strategy, with the potential to be applied more broadly. It ‘outlines a strategic and transparent approach to assessing the strategy’s progress towards its objectives and outcomes and identifying its impacts. It also enables learning to be shared and capability for monitoring and evaluation to be enhanced across organisations.’ Setting up these data monitoring and reporting components of the infrastructure sets Victoria in good stead as we keep refining our approach to primary prevention. We suggest as research, evaluation and monitoring processes are further developed, increased focus should be given to the following:

• supporting those leading specific prevention programs through the iterative process of learning what changes they might expect to see as a result of their program, or what other measures they should use to assess whether programs have been implemented in such a way that maximises their effectiveness
• incorporating short-term measures into data reporting such as the process measures described in Counting on Change
• facilitating the ability to drill down into the data to conduct finer grained analyses (for example, by local government area or region and/or by other demographics, if possible), including by funding larger sample sizes for the NCAS
• as part of ongoing monitoring and reporting of the short-term measures, consider including an assessment of progress against each of the Change the Story essential actions and other essential actions for other forms of family violence
• considering how data gathered through research and ongoing monitoring will be used to adjust the course of action if needed [relates to action 1].

The importance of monitoring – example: Respectful Relationships

An example of the importance of measuring progress and effectiveness at the program level and refining implementation accordingly is the Respectful Relationships initiative. Respectful Relationships is a key plank in Victoria’s approach to primary prevention and is backed by evidence:

Evaluations of the whole-of-school approach in primary and secondary schools in Australia have shown the potential for respectful relationships education to challenge gender stereotypes, develop a culture of equality among staff and students and support attitude change among students.57

However, for this to be realised, evidence shows some key elements are required, including a true whole-of-school approach, ongoing professional learning for staff, and sustained commitment including through funding.

Positively, the implementation of Respectful Relationships in Victorian schools has been evaluated, indicating support for the initiative, appreciation for the available guidance and resources, and some positive signs of impact for school communities (see Figure 17 for some of these positive indicators).

Figure 17: Statistics included in the public-facing Respectful Relationships evaluation report

The evaluation also raises some issues around sustainability of the approach, with the level and duration of support schools require (for example, from the regional workforce and from lead schools) varying [relates to action 9]. The evaluation also flags significant variation in implementation between schools and evidence that not all schools are progressing the intended whole-school approach in full.

During our consultations, feedback from several stakeholders indicated they had seen a loss of momentum, that there is a lack of necessary support at the school level to support implementation, and that the Respectful Relationships regional workforce – which is responsible for supporting schools to implement the whole-school approach, as well as supporting MARAM and information sharing reforms in schools – is stretched too thinly.

We acknowledge the significant disruption of COVID-19 to normal school operations to students’ learning, and that schools are necessarily focusing on supporting students’ mental health and helping them catch up on learning. But concern has been expressed about how focus on effective implementation of Respectful Relationships will be regained, particularly as schools continue to face staffing challenges due to COVID-19.

This is a major implementation risk, and we understand that the only ongoing monitoring in place is through the regional workforce, which can feed any implementation issues up to the Department of Education and Training’s central office [relates to action 10]. However, we suggest that given the workforce workload issues already raised, these regional staff are unlikely to have the capacity to effectively monitor implementation fidelity in all schools. It is a positive sign that all Victorian Government schools are now signed on to Respectful Relationships, but process indicators need to be monitored to determine whether implementation is occurring effectively and whether the intended outcomes can be expected to be realised.

The initiative is a central part of Victoria’s primary prevention approach and therefore the integrity of its implementation is critical. Despite the concerns raised, there was strong praise for regional departmental staff who are driving the initiative and are clearly committed to working with Victorian students to prevent family violence. There are also some recent developments that may help to regain a focus on embedding Respectful Relationships in schools:

• the commitment in the National Plan to End Violence against Women and Children 2022–2032 to strengthening Respectful Relationships in the national curriculum
• the introduction of a new Child Safe Standard, ‘Children and young people are empowered about their rights, participate in decisions affecting them and are taken seriously’, which requires schools to offer students access to sexual abuse prevention programs in an age-appropriate way and to document this accordingly. The department has advised that schools could comply with this standard by delivering Respectful Relationships and using the optional Resilience, Rights and Respectful Relationships and Building Respectful Relationships teaching and learning materials.

With careful planning and ongoing implementation monitoring, there is an opportunity for Victoria to leverage these developments and harness the goodwill of key stakeholders to ensure more Victorian students will never go on to experience family violence.