Chapter 3. Evidence as cliché
p. 39-52
Texte intégral
“For […] there are experimental methods and conceptual confusion […]. The existence of the experimental method makes us think we have the means of solving the problems that trouble us; but problem and method pass one another by.”
Ludwig Wittgenstein (1958)
1In 2018, I was invited by the Cochrane Database of Systematic Reviews to review an important manuscript. It was a systematic review on the impact of decentralised governance on health services. That is, on the impact of shifting the governance of health services – or aspects of it – from a higher, more powerful, distant level of government (e.g. a national government) to a lower, less powerful, proximate level of government (e.g. subnational governments). Or a transfer of responsibility from any government to community groups, to health facility managers, or even to private for-profit entities. Or a transfer of responsibility from the headquarters of a large organisation (e.g. a national ministry of health) to subunits of itself (e.g. located in different regions of a country). I was happy to have been asked. The protocol for the review was published in 2013, and I had read it with a mix of excitement and scepticism (Sreeramareddy & Sathyanarayana, 2013).
2Cochrane places evidence from randomised controlled trials – or experimental studies more broadly – atop a hierarchy of methods to generate the best evidence for decision-making. The authors had proposed in their protocol to review evidence only from experimental and quasi-experimental studies on decentralised governance. As policies to decentralise the governance of health services are implemented globally, the authors planned to review evidence on its impact anywhere in the world. But knowing that decentralised governance (as a policy intervention) is not readily amenable to such methods of inquiry, I wondered where the authors would find the studies to include in their review. As a policy intervention, decentralisation is typically implemented as part of all-encompassing political reforms within a country, with inevitable flow-on effects in sectors such as health and education (Liwanag & Wyss, 2017).
3A randomised controlled trial would require that researchers find a way to deliberately separate decentralised (experimental) from centralised (control) provinces or districts. It was a move that I thought would be unduly intrusive and thus not tolerated within a complex political process; hence, unlikely to occur. It would require high levels of cooperation from politicians and high-level policy actors whose interests and incentives tend to vary too frequently to last the duration of a randomised controlled trial. I was therefore not surprised to note that the authors of the systematic review on the impact of decentralised governance on health services were able to identify only one eligible study. But even that study was not really eligible. As I wrote in my peer-review report (Cochrane has an open peer-review process, and I agreed to being acknowledged by name in the final publication):
[…] the authors define decentralisation and centralisation only in relation to governments. The challenge of such a limited definition is twofold. (1) Decisions of governments to (de)centralise the governance of services between levels of government are made in such a way that it is hard, if not impossible, to subject to experimental evaluation – hence it is not surprising that the authors found no such study that meets their inclusion criteria. (2) The only study that meets the authors’ inclusion criteria does not really meet their inclusion criteria – this study examined decentralisation within an organisation (and not from one level of government to another). The authors have the option of rewriting or reframing their review and how they define (de)centralisation in a broader sense that goes beyond what happens between [or] among governments, to [include what happens within] the governance structure within organisations.
4When I wrote these open peer-review comments, I felt that most (if not all) experimental studies of decentralised governance would potentially
amount to large-scale tampering with health system governance, unaccountably, from a distance. I still think so. However, I have since changed my mind on the potential to find such studies. Over time, I have come to the view that a well-known health policy intervention – “performance-based financing” (e.g. an arrangement in which health service providers are financed by a government or an entity that pays for health services based on how well they achieve pre-set performance targets) – which has repeatedly been subjected to experimental and quasi-experimental inquiry (Witter et al., 2012; Oxman & Fretheim, 2009; Eijkenaar et al., 2013, Suthar et al., 2017; Das et al., 2016), is a misnomer. In fact, it is decentralised governance in disguise, just like interventions such as purchaser–provider split (e.g. when a government or an entity that pays for health services decentralises the responsibility for service delivery to other entities, which may be for-profit entities), and community engagement in governance (e.g. when community groups are assigned or take on some of the responsibility of supporting the demand or supply of health services within their community). As with decentralisation, performance-based financing, purchaser–provider split and community engagement in governance have been used as strategies in efforts to improve health service delivery around the world. Whether or not trials of paying for performance in health care are useful, or should have been conducted, they exist. They should have been considered as eligible for inclusion in the proposed Cochrane review on decentralisation. But I did not make the link at the time.
5A major appeal of randomised controlled trials is the dubious promise of simplicity; the ability to reimagine a complex social intervention that is emergent and contingent on context as one that is simple, tangible and decontextualised. Randomised controlled trials were designed for such simple interventions. The problem with attempting to simplify complex interventions is that the evidence generated by doing so often at best has the relevance of a cliché, typically appealing to the foreign gaze. A cliché in the sense that the evidence never means what it is claimed to mean, but replaces or obstructs thought in ways that may appeal to someone listening from afar with too little detail to think about it adequately. The further away one is from what is being evaluated, the simpler it seems, and the simpler one’s perceived needs concerning evidence appear. This is a tendency at work in the evidence on performance-based financing, an intervention whose name deceptively simplifies a complex intervention that involves extensive decentralisation of health system governance, which is itself an inevitably messy political process. This simplification uncouples performance-based financing from decentralisation, from context and complexity, thus limiting the learning that ought to accumulate on the effects of decentralisation in health systems.
6With pilots or full-scale programmes in at least 32 out of the 46 countries in sub-Saharan Africa, performance-based financing is one of the most widely implemented health policy measures in the region (Gautier et al., 2018, 2019). Its spread is backed by a well-nurtured “nexus of strongly dedicated diffusion entrepreneurs”, working in, funded by, or supported through bilateral and multilateral development banks and agencies, especially the World Bank (Gautier et al., 2018). Trials have played a central rhetorical role in legitimising performance-based financing initiatives across Africa. Even though the results of those trials are rarely impressive, decisions to scale up the initiative within a country or to spread it to other countries have often been made “before research results were made available, or in spite of them” (Gautier et al., 2019). Even the champions of these performance-based financing initiatives in sub-Saharan Africa seem to be aware of the clichéd use of evidence from trials to supposedly evaluate their impact on health system performance. In any case, the literature on performance-based financing does not position decentralisation as its core feature. Instead, it is framed in terms of a relatively marginal and often adaptable feature – incentives given to health facilities or peripheral governing entities to improve the quantity and quality of their services.
7In the implementation science literature, programmes or interventions are sometimes described as having core elements and adaptable elements (Fixsen et al., 2009; Eboreime et al., 2020). Core elements are features that are directly or primarily responsible for programme impact. Adaptable elements are features that are modified to align with contextual nuances. The core feature or element of performance-based financing is often framed as the incentive to improve performance. But at the core of the initiative is the transfer of power, resources and responsibilities from central to peripheral actors. This transfer may occur between national and subnational governments, between a government and health facilities, or between a government and community groups (e.g. community health committees) (Oxman & Fretheim, 2009). Thus, the core feature is decentralisation – performance-based financing is decentralised governance by another name. Without decentralisation reforms, health facilities, subnational governments or community groups cannot receive, use or make decisions based on performance incentives.
8I put this conceptual confusion down to the problem of gaze: the foreign gaze. The framing of performance incentives as being at the core of performance-based financing makes it amenable to evaluation through randomised controlled trials. The alternative is much more difficult. Asking directly for the decentralisation of health system governance is tantamount to asking for a – likely unwelcome – wholescale retooling of health systems, involving a complex, contested, threatening and long process of reforms and negotiations. The “simplification” of performance-based financing lends it, in turn, to the generation of simple and apparently compelling evidence on its effectiveness through randomised controlled trials, making it “marketable” to a funder or policy actor at a distance – to the foreign gaze. The language of performance-based financing offers decentralisation through the backdoor; after all, rather than an extensive reform, it is quite a specific intervention. The core element (decentralisation) thus becomes a relatively silent consideration.
——
9Much like decentralisation (Dwicaksono & Fox, 2018; Sumah et al., 2016; Cobos Muñoz et al., 2017; McCoy et al., 2011; Casey, 2018), efforts to quantify the effects of performance-based financing on health system performance have yielded mixed results (Witter et al., 2012; Oxman & Fretheim, 2009; Eijkenaar et al., 2013, Suthar et al., 2017; Das et al., 2016), and inevitably so. It is a non-problem. What would raise concerns is if the results of the evaluation of such complex interventions were not mixed. In spite of repeated efforts (including the use of trials in the case of performance-based financing) to demonstrate their effectiveness, or lack thereof, it has proved to be an impossible, and perhaps, unnecessary endeavour. After all, their effectiveness could not possibly be proven one way or another, or proven once and for all. They are complex social (and/or political) interventions with effects resulting from the interacting and varying behaviours and interests of the diverse individuals and groups who design and implement them, and those who are their targets or intended beneficiaries. Their effects also depend on their design: decentralisation or performance-based financing in one place is necessarily different from an intervention that carries the same label elsewhere.
10While their effects vary from place to place and from time to time, there may be tendencies and identifiable patterns in how these complex social interventions and phenomena perform over time when introduced or activated in a particular place or setting (Pawson & Tilley, 1997; Pawson et al., 2005). But even those tendencies are always contingent on context. For example, in a setting where X exists, and people have experienced Y and so reason in a particular way Z, favourable outcomes result from decentralised governance or performance-based financing. Hence, for a policymaker, the question is not so much whether to decentralise governance (in settings where they have the power to do so) or to implement performance-based financing, but rather, how will it work in a setting where X does not exist, but rather there is A, and where instead of Y, people have experienced B, and so are likely to reason in way C rather than Z when the policy is introduced.
11Understanding the knowledge and evidence needs on complex interventions and phenomena in terms that acknowledge their complexity should be the starting point of inquiry, not the conclusion. Too often, it is the other way around. The studies, often experimental, randomised controlled trials, have been set up and conducted in multiple places, frequently at great cost, only to conclude after their results accumulate over time that the evidence is mixed. Of course, the evidence is mixed. It is a misuse of the experimental method. But the practice persists. The question is why? In the case of performance-based financing, once you see it as decentralised governance, the question becomes even more difficult to answer. Decentralised governance is an ongoing process that involves continual negotiations and learning amid tensions. It is never complete. Any evidence on its effectiveness is at best tentative. Generously interpreted, it is a thought-stopping cliché: “the evidence says yes” or “the evidence says no” – as if the evidence can possibly be definitive, final. At worst, it is a disingenuous, cynical (if sometimes useful) excuse for tampering with health system governance.
12If the question is “Should a country adopt performance-based financing?”, these randomised controlled trials cannot answer it. Nor can they answer the question “What kind of performance-based financing should a country adopt?” Nor the question “How should a country modify a performance-based financing initiative to suit its context?” So what are they good for? Why does evidence from these trials remain so important even though the results, whether positive or negative, demonstrating effectiveness or not, say little about what is actually an effort to reform a system? Who is the audience of these randomised controlled trials? The foreign gaze? Is saying “the evidence from randomised controlled trials is X” perhaps simply an easier way to convince funders and unsuspecting, distant governments who will accept the evidence as unthinkingly as policy entrepreneurs who have been told and have come to believe that these trials are the arbiters of truth?
13Using randomised controlled trials to assess performance-based financing initiatives is like judging a cake by the icing on top of it. The cake is the core, with its underlying layers of decentralisation reforms and processes, on top of which the “performance incentive” rests. In these randomised controlled trials, it is the whole package that is being evaluated, although the evidence is typically presented as if it was about the performance incentives alone. When the evidence is mixed, it is often because the context asserts itself, again and again. To understand why evidence from randomised controlled trials could be considered useful at all, one can only infer from the rhetoric implicit in such trials. This implicit premise is that there are benefits to simplifying a complex intervention, and to wishing away context and complexity, such that even when context is taken seriously, the contextual aspects considered are those that readily lend themselves to simplification.
14These wishful assumptions relate, in part, to the origin story of performance-based financing. Early evidence of its use in international development and global health came from post-conflict states undergoing or considering sweeping governance reforms (Bertone et al., 2018; Paul et al., 2018). The first scale-up effort was in the very atypical reformist setting of post-conflict Rwanda. It was evaluated in a randomised controlled trial that showed success in improving health system performance (Basinga et al., 2011), a result that has since been challenged and has hardly been replicated elsewhere despite repeated efforts (Paul et al., 2018). However, outside such atypical settings with ongoing governance reforms onto which performance-based financing can position itself as icing on the cake (for example, in Rwanda, Burundi and Zimbabwe), it is rare for national governments to devote significant domestic funds or local resources to implement or scale up performance-based financing initiatives (Paul et al., 2018; Mayaka Ma-Nitu et al., 2018). Those funds have typically come from outside – notably, as loans from the World Bank (Gautier et al., 2018, 2019; Paul et al., 2018).
15In the absence of ongoing reforms or national or subnational governments' willingness to undergo such reforms, efforts to introduce or scale up performance-based financing (usually accompanied with randomised controlled trials), may therefore require unwelcome tampering with health system governance. And given that existing governance arrangements are typically entrenched, context reasserts itself in the (in)effectiveness of such efforts. Tampering may cause unintended consequences. Nonetheless, masquerading a necessarily “organic” reform (that is, decentralised governance) as a “surgical” intervention (that is, performance-based financing) may also work as a deliberate backdoor strategy to introduce an important and desirable reform into a health system that powerful interests in the system would otherwise have resisted. But this coy (if sometimes beneficial) strategy is easily undermined by the appetite of the foreign gaze for simple rather than complex interventions.
16I experienced this appetite first-hand in Nigeria in 2013. I was working at the National Primary Health Care Development Agency in Abuja, which was implementing Nigeria’s performance-based financing initiative. I had volunteered to help during the fieldwork for a study (thankfully not a randomised controlled trial). There was suboptimal uptake of services in pilot health facilities for the initiative, and the World Bank wanted to know why. In the study, we consistently found that where local decentralised governance structures (community health committees) were active, service uptake was high, and where they were not, service uptake was low (Mabuchi et al., 2017). But this finding was not reflected in the recommendations in the draft report shared with the rest of the team by the lead World Bank consultant. The report focused on the reasonable but much less compelling idea of using transportation vouchers to improve uptake. I raised this glaring omission. The consultant replied that it would be corrected. In the final report, there was hardly a mention of community health committees; the focus remained on transportation vouchers (Mabuchi & McCune, 2015).
17The foreign gaze had held on to a tangible, surgical intervention – something simple that could be readily sold to a funder looking or acting at a distance, something that could be proven, once and for all, to have worked. This is how I interpret that experience: when you are looking from a distance, you see “concrete” things, like funds and performance incentives, things that can come from outside, and surgically (or magically) make things better – things like transportation vouchers. It is harder to see things that are organic, that require on-the-ground retooling, negotiations, fixing, learning – things like community health committees or decentralised governance more broadly. Transportation vouchers are tangible: they can be measured and evaluated, implemented in the same format from place to place replicable, like a travelling model. It is a solution easily imagined as scalable. It is discrete. It can be presented as something “new”. It appears attractive at a distance.
18Yet interventions should begin from the premise that people are already making efforts and learning how to solve their own problems. At close range, an example of what you may see community health committees and similar entities do is respond in various ways to an unsatisfied demand for health and other social services (Ben-Ner & Van Hoomissen, 1991). In his book, Exit, Voice and Loyalty (Hirschman, 1970), the German social scientist Albert Hirschman theorised about the range of conceptual options available to people when they find the cost, quality, quantity or distribution of public goods and services unsatisfactory. He described “exit” as choosing to obtain services elsewhere or under a different arrangement (for example, in a nearby community or in a for-profit health facility); “voice” as choosing to form or use a coalition of users (for example, a community health committee) to advocate for improved provision and oversight of services from government; and “loyalty” as choosing to contribute or generate resources to support their services (for example, through financial donations from high-income community members).
19The presumption that people anywhere are waiting with folded arms for a saviour is one that is only easily made and sustained at a distance. People select from among many diverse options available to them (Figure 2). They may use their “voice” to demand improved government financing of existing public sector provision or better regulation of private sector provision. Local actors may come up with new for-profit entities, and coalitions of users may come up with non-profit responses to meet unsatisfied demand in the community, providing channels of “exit”. Coalitions of users may “exit” into new arrangements, transforming into a self-provision coalition as they co-produce or augment existing public, for-profit or non-profit services. These efforts – all of which are emergent, deeply contingent on context and involve lots of learning on-the-go – may also fail (Ben-Ner & Van Hoomissen, 1991). But the range of decentralised agency potentially on display is ignored by parachute interventions such as performance-based financing.
Figure 2 | Potential decentralised responses to unsatisfied demand for primary health care services.

Source: Abimbola (2020), inspired by Ben-Ner & Van Hoomissen (1991).
20If performance-based financing were reframed as a form of decentralised governance, how would it be studied? Before I came to this understanding myself, I conducted and published an evidence synthesis on how decentralised governance influences health system performance (Abimbola et al., 2019). But in that review, I left out the literature on performance-based financing. Looking back now, this omission leaves me with deep regret and appreciation for the many such potential opportunities for learning that we so easily miss when we frame interventions or reform efforts in a way that makes their adaptable elements seem like the core elements. As a result, we do not optimise potential learning that may otherwise occur across settings and/or interventions. We fail to draw insights from seemingly different interventions which, on closer examination, belong in the same “core elements” family (Abimbola, 2018). Which does incalculable harm to our ability to solve the problems that trouble us in global health and development.
21Nonetheless, the evidence synthesis I conducted on decentralisation began with a premise of complexity. It acknowledged that what is useful evidence is not whether decentralisation “works”, but how, for whom, and under what circumstances it works or not (Pawson & Tilley, 1997; Pawson et al., 2005). It also depends on what we mean when we say "works". It acknowledged that what is called “decentralisation” is often limited by a focus on its top-down connotations as an “intervention”. That decentralised governance may also be seen as a common phenomenon: as how things are, regardless of a formal policy to enact (de)centralisation as an intervention. There are de facto ways in which health systems are decentralised: for example, regardless of government mandate, community groups may govern their local health system through the exercise of local agency. One cannot study decentralisation only as an “intervention”. This conceptualisation of decentralisation as both phenomenon and intervention allowed me to cast a wider net on studies that could be included in the evidence synthesis, enriching the range of potential sources of learning.
22While none of the included studies was a randomised controlled trial, in retrospect I realise that trials (of performance-based financing) could have met the inclusion criteria. But such trials are rare, precisely because decentralised governance involves iterative social and political decision-making processes that resist randomisation. Trials assume standardised interventions across sites, while decentralisation involves continuous local learning and adaptation. Unfortunately, the “surgical” appetite of the foreign gaze means that researchers who are inclined to gain understanding from the bottom up, to study, support and engage in long-term organic processes of change, may feel the need to apologise for their choice (although it is the better one), or to even justify why they “have not developed a traditional intervention” (Barasa et al., 2020). These researchers risk being seen as “academic lightweights, producing nothing of substance”, who “answer questions which are dull, not novel (little contribution to the scientific literature), or not generalizable (focused on local issues)” (Oliver et al., 2019).
23Trials do not entirely preclude asking nuanced questions, but make them much more difficult to ask. In the context of a trial, such questions are an afterthought (when embedded within the trial), are limited (by the assumptions of simplicity which are necessary to conduct a trial), or are wrong (e.g. when asked as a binary, such as whether something is good or bad, or whether it works or not) (Liu et al., 2019; Lewin et al., 2009). Yet what is really important are nuanced questions of process or more fundamental questions of appropriateness, of fairness, of justice, or overarching systems. Or of the ongoing, iterative, long-term effects of health system interventions, processes and reforms (and their outcomes) and day-to-day phenomena (such as feedback, adaptation and learning) that trials are ill-equipped to capture: What does a system need to improve? Are performance incentives (beyond salaries) necessary? Why? Are there locally informed strategies to address these issues? Do they require local political engagement? How do you support ongoing local political process to better generate desirable change?
24Much like randomised controlled trials, performance-based financing has generated serious debate (Paul et al., 2018; Mayaka Ma-Nitu et al., 2018). Both debates are linked. The opportunity and transaction costs of implementing performance-based financing are cited by those challenging it. They also cite trials showing its failure, just as the other side cites trials showing its success and makes the case that any failure is due to “context”. This is a thought-stopping clichéd debate that shows the limits of trials. Notably, in their defence of performance-based financing initiatives, a group of local health system practitioners across six African countries do not cite evidence from trials. They argue that it is a “reform approach” in “constant evolution” “over time”, which builds capacity at different levels of decentralised governance, to improve “coordination, decentralisation, accountability […] including community engagement in […] governance” (Mayaka Ma-Nitu et al., 2018). Viewed from a local pose, performance-based financing is decentralised governance.
——
25In 2020, I visited the Cochrane website to check the status of the review on decentralisation. I wanted to see the direction the authors had taken in response to my and others’ peer-review comments. Unfortunately, I found a notice, dated 2019, stating that the editors of the Cochrane Database of Systematic Reviews had “withdrawn it from publication” because “this protocol has not been successfully converted into a full Cochrane review within established timelines due to lack of resources to complete the review” (Sreeramareddy & Sathyanarayana, 2019). The two authors are based in Malaysia, which may explain their limited resources. I thought, what a loss. By broadening the scope and redefining its parameters, their systematic review – preferably one that would not simply ask if decentralisation “works”, but how, for whom, and under what circumstances it works or not: a realist review or synthesis – would have been an opportunity to deepen and enrich the literature on the impact of decentralisation on health systems and services.
26I am left wondering what the results would look like of a systematic review on decentralisation that includes evidence – both qualitative and quantitative – on performance-based financing initiatives. It could be an extension of the evidence synthesis I conducted, or a revision of the planned systematic review that may never be completed, due to lack of resources and Cochrane’s preference for experimental studies. The result would look different, with “performance incentives” featuring as one among many contextual factors that may enable or constrain a range of outcomes, such as quality, equity and efficiency. The literature is poorer for the lack of (and for not normalising) such a complexity-informed realist review or synthesis (Pawson et al., 2005). This is one of the many uncounted costs of randomised controlled trials – they can obscure conceptual connections and therefore limit opportunities for learning. We must find ways to count this cost, the cost of unrealised learning too.
27What are the costs of randomised controlled trials in health systems, global health, and international development research? In the example that I have presented here, in part due to the rhetorical, if clichéd, advantage of these randomised controlled trials in feeding the appetite of the foreign gaze, a policy measure that was designed to strengthen decentralised governance is largely misnamed (as performance-based financing), misvalued (using evidence from randomised controlled trials), and mismarketed (like a Trojan horse) to governments as an excuse to tamper with health system governance (although sometimes desirably). The literature on performance-based financing in health systems should be part of the literature on decentralised health system governance. The fact that it is not, limits the learning that could have taken place between them. The cost of simplification – aided by randomised controlled trials – is that it, perhaps unwittingly, limits learning.

Le texte seul est utilisable sous licence Creative Commons - Attribution - Pas d'Utilisation Commerciale - Pas de Modification 4.0 International - CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.
La recherche interventionnelle en santé des populations
Concepts, méthodes, applications
François Alla, Linda Cambon et Valéry Ridde
2024
Population health intervention research
Concepts, methods, applications
François Alla, Linda Cambon et Valéry Ridde
2023
Investigación de intervenciones en salud poblacional
Conceptos, métodos y aplicaciones
François Alla, Linda Cambon et Valéry Ridde
2024