Putting the theory of change in theories of change
Many researchers, me included, feel that creating a theory of change is a crucial element of any evaluation, not only for evaluations explicitly labelled ‘theory of change evaluations’ but also for those involving counterfactual approaches (RCTs, QEDs) and implementation and process evaluations. However, I think the canonical approach to defining a theory of change omits the most important element, the bit that explains why an intervention is believed to work.
Traditional theory of change template
The traditional structure for a theory of change (see figure 1 below) comprises inputs (resources the intervention requires), activities (what’s actually done as part of the intervention), outcomes (what you’re directly aiming to change, also known as proximal outcomes) and impacts (wider, more distal changes you’re hoping the intervention will lead to or contribute to).
Figure 1: Traditional structure for a theory of change
Revised theory of change template
The crucial element this structure that is missing is ‘mechanism’ (see figure 2 below). The mechanism is what causally links the activities that are being undertaken with the outcomes you hope to achieve. The mechanism is not simply an intermediate outcome(s) (ie it’s not just a more finely grained series of linked outcomes), it is what is believed to cause the outcomes to change, the active ingredient.
Figure 2: Revised structure for a theory of change
Examples of mechanisms
To illustrate what I mean by mechanisms, I’ll compare two interventions that have similar activities and outcomes but very different mechanisms (figure 3 below). The first is Minding the Baby, an intervention for vulnerable mothers developed by Yale University and trialled by the NSPCC. The programme involved nurse practitioners who went into the homes of new mothers every week for two years and supported them with the care of their infant. The aim of the intervention was to improve mother-child attachment. The second intervention is a mentoring intervention for young people at risk of being involved in violence. This intervention involves weekly meetings between a mentor and young person, where the mentor supports the young person emotionally and practically with the aim of reducing the behavioural difficulties exhibited by the young person.
Figure 3: Two comparable interventions
Although the two interventions have similar activities (regular support from a trained individual who is not a family member) and similar outcomes (broadly increased positive behaviours), the intended mechanisms are very different (see figure 4 below). The mechanism underlying Minding the Baby was based on social learning theory, an extension of operant conditioning. The theory is that rewards and punishments influence behaviour when observed happening to someone else not just when being experienced directly. In the case of Minding the Baby, the idea was that the mother would see actions performed by the nurse leading to better soothing or improved positive engagement with the child and start to adopt those behaviours herself.
In contrast, the mentoring intervention is believed to work through role modelling, where the young person identifies/admires the mentor and this changes their goal values (what they see as worth trying to achieve) and their goal expectations (their beliefs about how likely it is that they can achieve their goals). Thus the two mechanisms are thought to act in very different ways, one within a reward/punishment system (probably associated with the basolateral amygdala in the brain) and the other via social cognition (more likely to be associated with the dorsolateral prefrontal cortex).
Figure 4: Two comparable interventions with different mechanisms
Benefits of theories of change with mechanism
If you can identify the mechanism of change for an intervention, it supports all the good things that theories of change help with, such as facilitating a shared understanding of what is being evaluated, sense checking how realistic the outcomes are, and getting a handle on the dose and timescales needed to realistic observe change. It also helps guide how you might measure or even test elements of the mechanism to inform the understanding of any impact observed or the failure to detect an impact (eg did young people not identify with their mentors or did identification not lead to changes in goal values and expectancies, etc).
Developing theories of change with mechanisms
In terms of developing a theory of change, the Platonic ideal would be for the relevant outcomes and behaviours to be chosen, research undertaken or existing research reviewed to understand the causes of those outcomes and behaviours, potential ways of influencing the causes identified (through theory and practical knowledge) and then the intervention designed taking into account all that preparatory work. In other words, the theory of change is not bolted on when it comes time to evaluate an intervention, the theory of change is integral to the development of the intervention in the first place.
In reality, developing a theory of change can be an exercise in backward engineering, taking the activities and outcomes that have already been defined and attempting to divine the relevant mechanism. In terms of a process, I recommend having a first workshop that isn’t intended to produce the model but instead aims to:
describe background to intervention
describe context of intervention
describe what intervention is trying to achieve (shorter/longer term)
describe how it works/ typical session
describe journey of a successful participant
describe journey of a participant it didn't work for
describe any potential backfires
describe views/ theories on why the intervention works
The next step is to go away and review the relevant research evidence and theoretical context and combine this with the information gleaned from the workshop to create a draft theory of change, which is presented in a second workshop for discussion and debate. Ideally, this process would continue until everyone (developers and evaluation team) is happy the theory of change is accurate and the right mechanisms have been identified. Again, in reality this may need to be a time-limited process, but as with all applied, real-world research, it is better to know what you are aiming to do and compromise where needed than start with a sub-optimal goal.
A final thing to note is that this process can feel uncomfortable because it is challenging to be asked why you think your intervention should change particular outcomes. However, just because the mechanisms can’t be immediately identified, doesn’t mean that the intervention is doomed to failure. Intervention developers may have a very good intuitive sense of which activities work and why they work but find it difficult to articulate them, which is where well-informed and sensitive evaluators can add value to the process beyond what might be expected from the JD.