I’m relatively new to the field of evaluation – I’ve only been practising for 18 months after an early career foray into health and education research.
I absolutely love working as an evaluator, and have been incredibly lucky to work on some exciting and innovative projects during my time at Urbis.
This piece is partly inspired by some of the frustrations I’ve had adapting from life as a policy researcher with a high degree of freedom to one where I have to fit within the confines of an industry that I think inherently constrains innovation. I offer few answers, but some thoughts on how we can work, as evaluators, commissioners and as an industry to build a courageous climate in evaluation.
Before I begin proper, two caveats. The first is that I do not intend to define innovation. This is for three reasons: I don’t think it’s actually necessary to do so for the logic of my arguments; I don’t think it’s helpful to explicitly define a concept which is based on the idea of stretching current conceptual understandings; and, finally, because I’m genuinely not sure on what definition I could arrive at even after contemplating this issue for months.
My second caveat is that I write from my professional perspective as an evaluator, that is, from within an independent, commercial evaluation consultancy working almost-exclusively with public sector clients. That said, I think my thoughts are transferrable to a diversity of clients and commissioning contexts.
But why care about innovation anyway? It’s of value to the evaluation, to the field, and to the evaluator. Innovation can help to deliver new insights on a program or policy. Often innovation emerges as a response to a specific problem in a specific situation, and the innovation consequently is of high value. New ideas can lead to new processes that lead to more efficient and effective evaluation approaches. This is beneficial to the evaluation community by furthering the sector, both in a practical sense and in an academic sense, where knowledge is developed for knowledge’s sake (something often regarded as frivolous, but I think vital to living in a more interesting world).
Finally, I want to work on fun and interesting evaluations, not operate as the evaluation machine. Innovation helps me, us as evaluators, to engage more deeply in the work, and to have a purpose for getting up in the morning to go to work. In turn, higher engagement with the evaluation process by the evaluator leads to better work.
But there are significant structural barriers in the evaluation industry as it currently stands which hinder innovative practice. In the context of evaluation, I think these barriers occur at three key points in the innovation cycle: the generating of ideas, the selecting of ideas, and the implementation of ideas.
The first barrier to innovation is perhaps the most obvious – why make the effort to do something new when the way you are doing things already ‘works’? I think this is a particular concern in the evaluation commissioning world, as independent evaluators tend to conservatism in order to win jobs. If there is a general acceptance of the methods and processes that work, there is no incentive to act as a first mover proposing new ideas when tendering for a job. This is especially the case when we work in a fixed fee world, where there is little financial incentive to do something different – the large financial pay offs that exist in some private sectors simply don’t exist here.
When it comes to selecting ideas, even if evaluators take the risks of proposing new concepts, the commissioner might not take the risk of accepting them. At the moral level, this is because public servants act as delegates of the community to provide services and evaluations. In the private sector, innovators negotiate openly with investors to literally buy in to their ideas. The public have not bought in to the idea of public servants gambling with their money on innovations, which inherently carry risks. By and large, public servants respect this by taking a risk averse approach.
And the risks are significant. Particularly, accountability unbalances the risk/reward ratio of innovation. Sadly, we rarely get rewarded when things go right, and this is the case in evaluation. This creates a situation where if we do the status quo, it works and nobody comments. We do the new innovation, it works and nobody comments. We do the new innovation, it fails, and a furore is unleashed. Thus the risks are high, but the rewards of innovation are uncertain and not necessarily held personally in the same way that backing of innovation in the private sector leads to potential financial reward. Often the perception of risk is enough to put people off, even if the likelihood of the risk eventuating is minimal.
Effective evaluation addresses program and policy implications in a way that is of benefit, not just to evaluation commissioners but also to all stakeholders in the program. Evaluation findings need to be acceptable, and this often means that the evaluation process has to have legitimacy in their eyes too. Therefore, even in situations where we have courageous clients, it may be the case that they don’t have courageous stakeholders, stymying the innovation process once again.
So how do we overcome these barriers? As evaluators, we have a key role to play in selling our ideas to commissioners of evaluation. From the outset of tendering, we have to act in ways that avoid hubris and reassure clients that our approaches will, on balance, deliver benefits that outweigh innovation risks. Explain why you think the approach will work, and be willing to compromise if necessary to minimise risks while maintaining the benefits of the evaluation. Compromise is important as we learn from clients, who are the subject matter experts in the evaluation. Often they will identify risks to the innovation that we won’t recognise because they operate at a different level of knowledge, and we need to accept that.
Commissioners should have faith that the evaluator is acting in their best interest, and as masters of method, offer their perspectives on how best to minimise risk in evaluation approaches. Increasingly it seems that commissioners do understand the importance of innovation, but are incentivising it in an unhelpful way. For example, many Victorian tenders now require evaluators to complete a separate section outlining their specific innovative contributions. I think this is an unhelpful lens through which to view innovation. Innovation is not always necessary – sometimes we don’t need new, we just need things done well. And sometime innovation is the path to ensuring that evaluation is done well. It’s important then, to view innovation as part of the process of evaluation improvement, not an ‘add on’ to be considered separately.
From a practical perspective, there may be changes in approach needed to accommodate innovation. Risk is a consistent theme in innovation, and risk requires risk management. Innovative approaches often need more project management in order to succeed, and to reassure commissioners and stakeholders throughout the process.
Often this reassurance comes in the form of partnership with others. Many of the things that we as evaluators view as innovations are accepted practice in other disciplines. One of the key strengths of evaluation in my mind is that it draws on theory and practice from a range of other fields to deliver the best answers to evaluation questions. Harness the power of others, and your innovation might not seem so risky after all.
My final thought is that amongst all this, there is a key role for us as evaluators, and the AES, as advocates for the potential of evaluation. We need to make innovation the norm in evaluation. We need to support client to become more courageous.
Click here to download a copy of the presentation
This is an adapted version of a paper presented at the Australasian Evaluation Society (AES) conference, held in Melbourne on 6 September 2015. This article originally appeared on LinkedIn.