Theories of change, again
The Evaluation Task Force has just published a very useful set of guidance, which provides an introduction to many aspects of the evaluation process. The module on theories of change (ToC) uses the example of a text message intervention reminding patients of GP health check ups, which the ToC indicates will lead to the outcome of an increase in people booking check ups which in turn leads to the impact of improved blood pressure and reduced cardiovascular incidents.
As well as being a clear example of the common format of a ToC, it also illustrates to me how much richer it would be if a mechanism was explicitly included in the standard ToC template. As it stands, the ToC gives no indication why a text reminder will cause people to book a health check up. There are a number of possible models which could link the activity with its outcome, including (but not limited to) the following:
- information deficit model - this assumes that people are motivated to book a check up but lack the information about when they should do it and the text fills this gap
- cognitive deficit model - this also assumes that people are motivated to book a check up but need a prompt to remember to do it because they cannot keep the requirement cognitively accessible
- deference to authority model - this doesn't assume that people are motivated to book a check up per se, but they will do so when they feel someone in authority indicates it is something they should do
- social observation model - this assumes that people are aware they ought to book a check up but only do so because they believe their behaviour is being observed and there will be some consequence of not doing so
- social norm model - this assumes that people are not intrinsically motivated to book a check up but the message indicates that it is something that people generally do and they want to conform with the social norm
So why is it useful to be explicit about the mechanism that is thought to link the intervention and its outcomes? There are at least four reasons:
- it helps understand why an intervention does it does not work as quantitative and qualitative methods can be used to investigate the connection; this in turn can help distinguish between theory failure, programme failure and implementation failure (if the intervention is found not to be effective)
- it can help estimate the effect size and the time period within which changes in outcomes can be expected to be observed
- it can help identify who in a population the intervention might be expected to work for
- it can help identify possible backfire effects, ie depending on how the intervention works, it may make some people less likely to book a check up or it may mean that the people who do book a check up are unlikely to follow advice or take medication that result from the check up
When I look at a ToCs, I would like it to answer at least three questions: what happens during the intervention, what is the intervention aiming to achieve, and why should the intervention work. Without including a mechanism, ToCs fail to answer the final, and arguably the most important, one of these.