Version classiqueVersion mobile

The Transformation of Research in the South

Rigas Arvanitis
David O’Brien

Managing research for impact

Zenda Ofir

Texte intégral

1 Context

1Amid diminishing resources and growing competition among nations, research supported by public funds is facing worldwide calls for renewal and change. Demands to ensure (and assess) “impact” and “value for money” are becoming more commonplace, yet research incentive systems remain powerful barriers to change. Incentives are determined by how research is conceptualized and valued, and in the current closed, “science-centric” view of research — and in spite of the proliferation of funding opportunities to address “grand challenges” in research — scientists continue to place value almost exclusively on narrowly defined notions of technical credibility and impacts in the scientific domain. As a result, there has been very little emphasis on how to manage research programs and grant portfolios for impact beyond the scientific domain. This will have to change although it will inevitably place an additional burden on researchers, and especially on those working in resource-poor institutions in the South.

2Evaluative approaches can support a shift toward managing for impact. It can be much more than a charge aimed solely at satisfying bureaucratic demands for compliance or accountability. It has potential to become an indispensable part of the effective and empowering management of research programs and grant portfolios toward impact.

2 Empirical Approach and Main Findings

3A systems perspective on research — Science tends to deal with complexity by examining things in their constituent parts, often to the detriment of understanding their interrelationships, and the effects of their interactions. Research enterprises must be understood using a holistic, broad, long-term view cognisant of how people (structured as agents through teams, programs, organizations, or networked relationships) interact with one another, with their histories, and with the contexts in which they operate. Experiences in practice, including the results of multiple evaluations of research and its uptake beyond the scientific domain, have highlighted that use-focused basic and applied research that aims to find solutions to complex real-world problems amid evolving societal dynamics compels a complex systems perspective on the whole research-to-impact endeavour — including with respect to the conceptualization, and hence assessment, of “research quality” or “research excellence.”

4This view of the research enterprise has several important implications. First, there are relationships between science and society that influence perceptions around the integrity, relevance, and legitimacy of the research (e.g., information may be scientifically relevant, without being decision relevant). Second, research performance trajectories are not linear, and different types of contexts and other dynamics must be accounted for when conducting, managing, and evaluating research for impact. Third, pathways to research use, influence, and impact are often convoluted and unpredictable, complicating understanding of the contributions of multiple collaborations and networks across sectors, worldviews, and disciplines. This understanding of the research enterprise has important implications for efforts to manage research for societal impact.

5The feasibility of managing research toward impact — Efforts to do high quality research that has an impact on society must be fully cognisant of the unpredictability and messiness of decision-making environments. This is most evident in the policy environment, which is widely seen as the most challenging in this regard. Policies develop out of ongoing interactions among groups of people and organizations interested in a specific issue, and policy choices are made when streams of information and possible solutions are brought together. As such, many argue that managing research toward impact is neither desirable nor feasible.

6Yet studies over the past two decades have shown that it is possible even in complex policy environments to plan and conduct work in a manner that will enhance the possibility of research uptake and use. A perfunctory nod to decision makers through the dissemination of materials, or a one-off interaction at a conference or workshop will usually not be enough. Instead, many case studies echo the need for early stakeholder engagement and for contextualized, nuanced approaches to knowledge translation. A large-scale IDRC study, Knowledge to Policy (Carden 2009), as well as a swathe of other recent empirical analyses point to practical steps that researchers can take to increase their prospects of influencing policy and practice. Devising and executing a strategy for influential research may turn on a few recognizable practicalities — as long as the complex systems nature of the research enterprise is acknowledged.

3 Main messages for policy and practice

7Value and provide incentives to context-sensitive, engaged, boundary-spanning research — A systems perspective on the research enterprise means that context-sensitive, boundary-spanning, engaged scholarship across disciplinary, ideological, sector, and stakeholder boundaries should be encouraged and rewarded. But to structure incentives and rewards systems accordingly, research program managers must understand and be explicit about the extent to which each research effort is expected to focus on working toward impact. Such an approach to managing research programs requires stratification and mapping of programs, and/or of research outputs, based on the types and attributes of the research and, as a result, expectations of impact. The resulting maps can, for example, account for : critical influences on performance ; the extent to which the research is multi, inter, or transdisciplinary ; pioneering work in a new domain ; and/or its relevance to frameworks such as the Sustainable Development Goals. Maps can highlight the composition and balance of a portfolio, and enable assessments of progress and performance that are both nuanced and useful to all involved. They can also help establish simple measures to nudge relevant research initiatives toward impact. For example, when the National Research Foundation in South Africa initiated the Centres of Excellence funding support program in 2004, its Service Level Agreements required these university-based centres to focus on impact outside the academic environment. These expectations played an important role in how the centre directors planned and conducted their work, which in turn led to a much greater focus on, and success in, managing for impact.

8Do not seek “impact” without a strong focus on “quality” that is appropriately defined — Research excellence is most often defined as encompassing both research quality and research effectiveness. From a systems perspective, these concepts are indeed interconnected and overlapping — how the attributes and importance of quality are defined and perceived by different stakeholders will have an influence on their interest in using the research. “Quality research” must be a precondition for working toward impact, yet this notion is often lost amid efforts to achieve and evaluate research impact.

9A complex systems perspective of research demands reconceptualization of what is meant by “quality.” A recent effort in this regard is the Research Quality Plus (RQ+) Assessment Framework, developed and tested by IDRC as part of their pursuit of a useful and more nuanced approach to defining and assessing research quality (Figure 1). The so-called “RQ+ approach” to research assessment acknowledges that research quality has meaning only in context, that an assessment of scientific merit alone is not enough to confirm the quality of the research, and that scientists can, where appropriate, be held accountable for positioning the research for impact. It espouses making sure that a holistic approach to quality is promoted from inception of a research effort. It focuses not only on technical integrity, but also on the legitimacy of the research from key stakeholders’ perspectives, on the importance of the research, and importantly on its positioning for uptake and use toward impact. It accounts for contextual influences on research performance, and applies classification, mapping, rating, and assessment of performance based on qualitative and quantitative data captured in a set of research performance rubrics.

10In its test phase, the RQ+ approach highlighted several advantages of efforts to manage research toward impact. It brings clarity on what is meant by research quality or excellence ; provides the flexibility needed to tailor the assessment frameworks and indicators to organizational mandates and values ; cultivates common understanding among participants and partners ; enables comparative assessment within and between portfolios ; and improves the quality of assessments. Importantly, it can be prospectively applied during planning processes (which is ideal), during implementation, and retrospectively during summative evaluations.

Figure 1 – Research Quality Plus (RQ+) Assessment Framework, developed and tested by IDRC (Ofir 2010, Ofir and Schwandt 2012).

Figure 1 – Research Quality Plus (RQ+) Assessment Framework, developed and tested by IDRC (Ofir 2010, Ofir and Schwandt 2012).

11Establish clear boundaries for accountability for impact — It is important to draw distinct accountability boundaries for planning and evaluation. A three-spheres framework for research (spheres of control, influence and interest ; Figure 2) is a useful non-linear way to conceptualize and organize the outputs, outcomes, and impacts that may flow from research initiatives, and to establish the boundaries for stakeholders’ accountability for results. The three spheres highlight that the progression from the inception of the research process to the eventual impact of the new knowledge is non-linear. Theory-based and mixed-method evaluation designs using comparative case studies, process tracing, and contribution analysis can be used to trace research contributions to outcomes and impact in the spheres of influence and interest. These approaches tend to be time-consuming and have to be carefully built into evaluation systems to ensure that they provide meaningful and timely results. However, they are also rigorous, useful for learning about how to manage for impact, and valuable for tracing examples of instrumental, conceptual, and tactical use of research.

12A key point is that researchers cannot be held accountable for the impact of their research, but they can be held accountable for conducting and positioning the research in a manner that enhances the chance of impact. The work in their “sphere of control” thus has to be done mindful of the need for impact. This notion is applied in the RQ+ Assessment Framework.

13Managers and funders of research should encourage a comprehensive approach to positioning research for impact ; funding allocations should recognize the complexity of positioning research for impact, and the expertise required for this purpose ; preconditions that can facilitate uptake of the research should be identified from experience ; and organizing frameworks such as the three spheres and theories of change can help encourage researchers to think beyond outputs and short-term outcomes, and work accordingly — yet ensure that they are not held accountable for achieving impact.

Figure 2 – A three-spheres framework for research (spheres of control, influence and interest).

Figure 2 – A three-spheres framework for research (spheres of control, influence and interest).

14Use adaptive management and maturity models to steer programs and grants portfolios toward impact — Efforts to manage research program and grant portfolios toward impact can make use of systematic “theory-based evaluation” approaches to facilitate decision-making and adjustment in strategy and tactics. Well-designed external reviews and evaluations can play an important role in providing targeted, context-sensitive information about progress and impact, and about reasons for success or failure. But it is even more important and empowering to combine these external evaluative activities with regular reflection and structured self-evaluation by the management team (with partners and stakeholders if feasible) based on useful quantitative and qualitative monitoring information. Such information can be used as part of risk management, and support research or program performance. It can help the team to track and understand the trajectory along which a certain project or program is making progress toward high-quality outputs and appropriate impacts. Much of science is by its very nature about iterative processes of experimentation, observation, reflection and learning, and adjusting. Researchers and managers in this domain are therefore extraordinarily well positioned to ensure that evaluation is an empowering, meaningful activity on their road to impact.

15In this process, the RQ+ Assessment Framework can be usefully expanded into a Research Quality (Plus) Maturity Model (Table 1) that can guide adaptive as well as strategic management from the inception of a research project or program. As demonstrated in Table 1, a maturity model is a set of structured levels that describe how well the behaviours, practices, and processes of an organization or partnership can reliably and sustainably produce required outcomes. It can be used to trace a program or portfolio trajectory toward outstanding research quality or research quality maturity. It can be expanded to encompass research performance and effectiveness within the spheres of control, influence, and interest, and can help identify weaknesses, strengths, and risks, and guide management strategies at various levels toward research excellence.

Table 1. Selected research-quality rubrics in a RQ+ Maturity Model for program and grant portfolio management.

Table 1. Selected research-quality rubrics in a RQ+ Maturity Model for program and grant portfolio management.

16This study focused on a few key issues to encourage urgent consideration of systems perspectives on research management and evaluation, and fresh thinking about how to manage research for impact. This can help the research community to arrive at a conceptual platform that can sharpen understanding of how to do this. Most importantly, the study aimed to stimulate debates that can lead to changes in practice by those powerful actors who design and manage incentives and rewards systems for research performance. Only then will research systems be transformed to enable much more effective efforts to address the challenges faced by societies around the world.


Related Resources

Carden, Fred. 2009. Knowledge to Policy. Sage Publications, IDRC.

Ofir, Zenda. 2010. External Evaluation : The Policy Influence of LIRNEasia : Final Report. International Development Research Centre (IDRC), Canada.

Ofir, Zenda and Schwandt, T.S. 2012. Understanding Research Excellence at IDRC : Final Report. International Development Research Centre (IDRC), Canada.

Table des illustrations

Titre Figure 1 – Research Quality Plus (RQ+) Assessment Framework, developed and tested by IDRC (Ofir 2010, Ofir and Schwandt 2012).
Fichier image/jpeg, 19k
Titre Figure 2 – A three-spheres framework for research (spheres of control, influence and interest).
Fichier image/jpeg, 17k
Titre Table 1. Selected research-quality rubrics in a RQ+ Maturity Model for program and grant portfolio management.
Fichier image/jpeg, 129k


School for Public Leadership
University of Stellenbosch
South Africa

© IRD Éditions, 2019

Conditions d’utilisation :

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search