*All views and statements represent those of the authors and do not necessarily reflect those of Accountable Now.
This blog was originally published by CIVICUS.
The Resilient Roots Accountability Initiative is working with 15 partners to help them design and rollout year-long accountability projects and document the factors which seem to help boost or hinder their accountability. We want to track changes in accountability over the course of the initiative, and to do so we need to measure our starting point!… But this proved to be much more complicated than we expected. Read on for some raw reflections about what we learned on the way!
How to do an accountability baseline in a comprehensive, yet efficient and comparable way, you ask? Well, you make a survey. Or to be exact, you make two similar surveys that cover various aspects of accountability (such as voice, responsiveness, trust, communication, etc.), addressed to the core of any organisation: their primary constituents and their staff.
For the majority of questions, the accountability baseline survey used the Net Performance Analysis (NPA)methodology, which involves respondents choosing a score on a scale from 0 to 10, from “totally disagree” to “completely agree”. The NPA then helped us generate a single number for each question which allows for easy comparison across constituent groups, between organisations, and over time. Comprehensive and comparable: check!
The exact method used to administer these surveys varied from one organisation to the next (based on the age of respondents, access to the internet, geographical location, etc), and included a mix of in-person/over the phone interviews and online surveys via the web and mobile phone applications. To do this, Resilient Roots hired independent consultants in almost all pilot project countries to undertake the surveys in the local language, and help reduce bias in responses (as opposed to organisations carrying out these baselines themselves).
Questions were standardised across all pilot project organisations, but the language in the surveys was adapted by each partner to fit the local context, make it less NGO-sounding and more accessible to its constituents. Then came the (11!) translations, one of the most time-consuming parts of the baseline measurement process. If you have ever tried to use “accountability” in another language, then you know the struggle of having to find translation for a term that simply does not exist outside of the English-speaking world. Now add words like “primary constituents” and “resilience” to the mix, and you have a buzzwords soup for a survey.
We approached these steps on a case-by-case basis, which made it a very laborious and slow process. But we wanted this baseline to be a real shared effort between our partners and the Resilient Roots team, and here taking our time proved to be more rewarding than efficiency.
So, we managed to create and implement a baseline survey that was indeed comprehensive, replicable, and comparable, and (to a lesser extent) efficient… That is until we got to the data analysis part, where we quickly realised how overwhelming this phase would be. The single, most important lesson learned from this is to “start at the end”. If we had started by spending more time thinking about the analysis, what we wanted to do with this information and what systems (read: complex equations on excel or powerful programming languages like “R”) we needed to set up to help us get there, this would have saved us lots of time and effort!
And it did not stop there! Once we had some preliminary results, we then had to figure out how to share these findings with the pilot project organisations, in a constructive and learning-oriented manner. After much debating, the best we came up with was an eight-page report (we tried!) with follow up calls. Of most importance for us was to visualise the data without generalising the findings or missing the nuance the NPA can give. So after we graphed, and pie-charted, and density-plotted, disaggregated and tabled, I think we got there!
Yet, a big part about understanding the results of this baseline survey does not depend on how many pretty graphs you make, but how vulnerable and open you are to both good and bad feedback from the people you work with. More importantly, it is about making serious commitments to address and respond to the feedback you receive. Accountability is (to a large extent) about organisational culture and how we “practice” sustainable development. These changes take time, and as a sector we have lots of work to do on this front! For Resilient Roots and the pilot project organisations, this baseline was our point of departure for setting off on this journey.
In sum, this accountability baseline measurement has been truly illuminating – though quite challenging and a somewhat burdensome process. But we have learned a lot on the way and we will continue to improve and adapt our methodology based on these learnings. Now, we feel readier than ever to support these 15, very exciting accountability pilot projects organisations make the best out of their efforts to increase their primary constituent accountability!
If you think, “hmm this is interesting, I want to know more about this methodology” then you are in luck! During the Global Accountability Week taking place 12-16 November, we will publish a much more comprehensive guide about how your organisation can measure its own accountability baseline. Stay tuned!
For more information on how Resilient Roots measures changes in accountability, read this article that focuses on the second component and in particular, the approach of the Resilient Roots team! In it you will find an overview of our methodology and some recommendations of things to consider when setting up your own mechanisms for measuring accountability. We are still working to improve our methodology and invite you to use it yourself, share your experiences with us and provide feedback on our approach. Please also share any similar approaches that you are aware of, or that you have used in your organisation with Resilient Roots ([email protected]).
Share this article!
Accountable Now is a global membership platform. We support our members – and civil society at large – to be more transparent, responsive, impact-focused, and locally led. Interested in becoming a member?
Accountable Now is a member of Accountability Lab’s global translocal network.