Evaluating the impact of cultural intelligence training

The challenge

Barts Health NHS Trust has set its sights on becoming a truly inclusive organisation. To support this, they have created a training programme, ‘Leading and Engaging Inclusively with Cultural Intelligence’ – cultural intelligence is the ability to work and relate effectively in culturally diverse environments.

They commissioned Kaleidoscope Health and Care to deliver an evaluation toolkit to understand the impact of the cultural intelligence training programme, which is being rolled out to 12,000 staff across the Trust.

We were asked to create a robust online survey that the Trust could circulate and analyse. The Trust wanted to use the findings to see what effect the training was having and how it could be improved. The objectives were to:

• understand whether cultural intelligence training affects culture change at Barts Health NHS Trust
• develop a sustainable evaluation approach that can be used to measure continuing culture change
• generate formative learning about what does and doesn’t work well to support positive culture change.

Our approach

Our approach to co-designing with Barts colleagues, from start to finish, was especially effective. It meant we could be confident the outputs were accessible, relevant and would support the impact of the training programme, building a more inclusive organisation.

We began by creating the ‘process and impact evaluation toolkit’. We started by running an in-depth workshop to understand and define their needs, objectives and what work or decision the project would inform.

Theory of change

We developed a theory of change with a mixed group of staff, including people involved in Barts’ inclusion strategy work and cultural intelligence trainers. This enabled us to understand how and why the desired cultural change was expected to happen as a result of the training.

Evaluation framework

To co-design an evaluation framework and a set of accessible survey questions we took the following steps:

• conducted a rapid desktop review of literature on cultural change training interventions
• carried out follow-up interviews, with data collection staff and cultural intelligence trainers.

To test the survey, we ran a pilot with 300 people and refined the survey questions to improve accessibility and reduce ongoing analysis.

Evaluation toolkit

We worked with Barts colleagues to identify the most useful format in which to hand over the evaluation. We developed an evaluation toolkit, including an analysis dashboard for them to easily process survey results and highlight the key metrics, and also provided a ‘how to’ guide.

Results

Thanks to our approach, and commitment to co-design, Barts now has the tools, data and capability to continue to efficiently and effectively assess their impact against their inclusive ambitions.

Kaleidoscope designed a robust high-quality survey for us to understand the impact of our cultural intelligence programme. It was lovely to work with a team who are interested and curious about the outcomes. The quality of Kaleidoscope’s work was good – as expected – but the programme management and collaboration was especially valuable, professional and also enjoyable.

Henry Ireland, Programme Manager, Barts Health NHS Trust