Evidence gathering for in-depth pieces of work is a fundamental aspect of scrutiny practice. Typically inquiries will be looking to offer credible solutions to complex issues that can be adopted with confidence. The work needs, therefore, to be underpinned by the robust collection and use of evidence.
Like any research process it’s important to set out the underpinning principles.
So, here’s our approach.
It is important to say at the start that much of what we do was learnt from the Outreach Team at the National Assembly for Wales following a learning day we spent with them. I’ve written about it here.
Aim
Overall our aim is to ensure that scrutiny councillors have the evidence they need to make balanced judgements.
This aim is supported by five principles that are set out below.
1. Set out the working assumptions
Each in depth inquiry starts with a ‘how’ question and, at the beginning of the process, the councillors agree their lines of inquiry.
There will be some ideas in councillors’ minds about what the issues are and these can be stated as part of the lines of inquiry – these assumptions can then be tested as evidence is collected.
2. Gather a range of perspectives
The issues being dealt with are by their very nature complex and contested (not much point having an inquiry if they are not). There is unlikely to be a ‘scientifically’ correct answer. Instead councillors will need to make a considered judgement having listened to different points of view. A good process, therefore, is one that allows councillors to hear the views of key stakeholders (for want of a better word).
It is important to hear from Council officers and external experts but also important to hear from those who experience or observe issues first hand or their representatives.
A key question at the start of the process then is ‘who do we need to hear from?’
Another is ‘who do we need to hear from who we don’t normally hear from?’
3. Always have the councillors in the room
We try to ensure that councillors are directly involved in any face-to-face evidence gathering even if this means departing from more ‘scientific’ approaches for things like focus groups.
We think that councillors are better able to make a judgement if they can see the people making the arguments – it is entirely legitimate that evidence is ‘felt’ as well as logically assessed. Within a political process decisions are made with hearts as well as heads.
Councillors also feel more engaged with the work when it is done in this way (as opposed to simply responding to reports written by officers).
We also use ‘behind the scenes’ meetings where councillors can meet with those who may wish to talk frankly without their views being published or individually attributed.
4. Produce digestible data
At the end of an in-depth inquiry councillors will have a mountain of evidence to sift through. As officers our job is to support the process of organising and interpreting what this evidence is saying.
For our inquiries we break down the evidence into single statements as we go along and then (using QDA Miner) code each statement according to who said it, which session it came from, and the lines(s) of inquiry that it related to.
This allows us to present the evidence easily against each line of inquiry to see the ‘story’ for each – particularly the areas of difference and agreement between evidence givers.
Producing evidence as statements is a skilled process and is of course subject to interpretation by the scrutiny officer. As a check we go back to evidence givers to ensure that they are happy with what we have captured. Changes can easily be made as the intention is not to capture a verbatim record of a meeting but an accurate representation of the views of evidence givers.
5. Make it shareable
Because we believe in open data as a principle, because scrutiny is a democratic function and because scrutiny inquiries in different councils often cover the same ground we think it’s important to share.
We produce and publish an ‘evidence summary’ from each session and, at the end of the inquiry, publish an ‘evidence pack’ that includes all of these summaries and any other relevant material. While this is primarily a document for councillors to refer to when developing their final report we also hope that it will be of wider use.
We are also experimenting with publishing the evidence pack in a spreadsheet format – a more friendly form of open data.
You can find an example of an evidence pack and a data spreadsheet from an inquiry into school governance here. (The final report from that work is here)
Leave a Comment