Open Science and Research Assessment. Trends and State of Play in Europe
p. 129-134
Texte intégral
1I am very pleased to join the discussion on how important it is to change and reform research assessment in order to make open science a reality. However, let me also stress a point the other way around: open science is also the main driver for this new way of assessing researchers and research that is more comprehensive and that pushes us towards better science. The open science movement has brought us here today and we must acknowledge that.
2I am going to give you an overview of the current situation of open science in research assessment from an institutional point of view, so I am going to focus on European universities. To do that I am going to tell you the results of the work we have been doing at the Expert Group on Open Science at the European University Association. We have been following the trends and the evolution of open science for about a decade, and our Expert Group has itself been evolving in following those different trends. In fact, we have just published our new vision and new key priorities for the years to come, and research assessment is key within that.
3We have been following this evolution through different surveys, asking universities whether they were implementing open science, and these surveys have also evolved with time. We started in 2014, just asking publications about open access. Then when we incorporated FAIR data and in 2017 we included research assessment. What I am going to tell you about is a combination of the two last reports and surveys from the EUA1, and it is worth noting that those surveys were answered by university leaders, or by responsible open science units, or departments at the universities, so it is not a bottom‑up approach and it is not the way academics in universities think about things, but it is a top‑down approach from the university leaders. The other thing I want to note is that all these results and all these surveys were done just before the European Commission started this discussion with the stakeholders, so it is going to be very interesting to see what happens after the debate we are currently having.
4Let us start with open science. The good news is that open science is more and more present within the strategic priorities of European universities and in fact a huge majority of them have specific open science institutional policies or are developing one. However, when we go deep into the detail of how these open science policies are organised, there is an important difference between the different open science areas. For instance, open access to publications is given the highest level of importance and of implementation, whereas evaluation is not seen as important.
Fig. 11: Limited role of OS in university academic assessment approaches

Source: European University Association et al., “Open Science in university approaches to academic assessment”, December 2021, https://www.eua.eu/resources/ publications/999:open-science-in-university-approaches-to-academic-assessment. html.
5This is also reflected in the limited role of open science in the academic approaches that universities have. When we asked universities what open science elements were included in these academic assessments most of the respondents responded with elements that were pretty much related to open access of articles and books. Moreover, nearly one‑half of respondents did not incorporate open science aspects in their academic assessments at all. When we asked specifically about research career assessment the results showed that open science and open access are given the least importance among the activities for research career assessment, whereas the most the two most important activities are, first, research publications and, second, attraction of research funding.
6When we asked for the assessment methods for a research career we found two main aspects that are followed by most of the institutions. Assessment is therefore a combination of the metrics based on the number of publications and citations and the peer review assessment. The main metrics used are those of journal impact factor and h‑index. This is the reality at European universities right now because there is a time lag between the open science strategy and open science being incorporated in the research assessment. Also, the main difficulty is that university leaders and administrators like to use this metrics-based approach for research assessment. When we go deep into the details of this main difficulty, most of them are related to practical implementations, so they have seen that incorporating this reform is a complex issue and they perceive that they have a lack of institutional capacity to do it. There are also concerns about increases in costs. There is also another very important concern which is that universities perceive there to be resistance to research reform from researchers. This is a key point. It is very important to bring researchers into the conversation. We cannot just have a top‑down approach. We need things to be bottom‑up as well in order to create and to move all the reforms we need.
7Let me share with you the struggles we have, and I have myself as a vice president in a university, trying to change things and struggling with the balance between what we want to do and what we can actually achieve. There is always this dilemma. It is difficult to navigate through the context, the environment we are in, the lack of incentives, the lack of guidelines, the lack of different policies, the differences between countries and between disciplines, and the question of how we as academics are we going to do this without damaging researchers, particularly early‑career researchers. However, when we ask about the future most of the respondents say that open science is going to be more and more present, including within academic assessment.
8There are many organisations rethinking how we can improve academic assessment and I am going to give you some examples of some of them. I am going to start at an institutional level. I am going to give you examples of three universities that started working on these issues some years ago. I am going to start with Ghent University, which published that vision statement in 20162 looking towards a more comprehensive evaluation system. That vision was followed by a rethink and reshape of the evaluation system of academic staff, which is now seen more as a professional development and not just as a control and evaluation process. There is another very good example from Utrecht University3, where the medical school embraced an open science strategy, incorporating a more qualitative and descriptive portfolio of what is going to be taken into account in order to make assess researchers and groups.
9I will also explain to you a little bit about we are doing at the Open University of Catalonia with our own Open Knowledge Action Plan that we approved in 20184. We decided to incorporate research assessment as a way to move forward to open science. As a first step, we created a taskforce that did a lot of work to see whether we could accomplish what DORA5 advocated in not using journal impact factor for assessing researchers and research. After that, we signed it and we started implementing a more narrative curriculum in our internal evaluations, discarding the journal impact factor in our internal course.
10There are also some very good initiatives at the national level. There is the Dutch Universities initiative that has incorporated a very interesting way of including asymmetry, for instance, in the assessment6. In Finland, some years ago, the Responsible Research network published a nice recommendation set stressing the need for change in assessment in order to incorporate the new ways of producing and communicating in science7. Also, universities in Norway came out with the evolution of the open science career assessment matrix8. The way they did it is very interesting, taking into account the different disciplines and a bottom‑up approach, and they set up a toolbox giving us some indicators, both quantitative and qualitative, and included a reflection on this.
11All those examples are also present in the work that DORA, the EUA and SPARC Europe have been doing9. These are the individual initiatives that have led us to be heard by the European Commission and now we are gaining momentum in convincing all the actors that these changes have to be made. It is very important that the new European Research Area Policy Agenda for the coming years has a specific action for reform for research assessment. The Scoping Report is also an important step10. We are now working all together in this process with the Commission as a facilitator in order to create the critical mass that we need. Let me end by saying that change is just ahead. We have room for creating more and more actions in order to move to a more comprehensive, more transparent and fairer way of assessing academic research for better science and I think the opportunity is here.
To go a step further
12“The European Commission, with a set of stakeholders, are now drafting a global agreement on the reform of research evaluation. They have also opened an expression of interest for those institutions that want to be part of this change. More than a hundred institutions have already volunteered, with the aim of ending up with an agreement with specific actions to be taken, and hopefully also some support measures which are needed.”
13“We know what we do not want, but we are also pretty much aware that open science is not going to be a universal solution for all. It should be adapted to the context, to the country, to the university, to the researchers and to the disciplines. At the UOC, we started small, and we are going step by step and with joint action learning between all the universities, between all the stakeholders. That is why these UNESCO recommendations are so important, because the global scale is very, very important for us in order to give this real momentum there. Open access of research publishing is easier because we have been discussing open access to papers for many years, and we know how to incorporate them, and we know how to open this access. We should now learn how to do the other things.”
14“Not everybody is aware of the limits of metrics measuring research, such as h-index, within the academic community. We should open this debate more, raise awareness and bring everybody in, in order to know all of these drawbacks.”
Notes de bas de page
1 European University Association et al., “Open Science in university approaches to academic assessment”, December 2021, https://www.eua.eu/resources/publications/999:open-science-in-university-approaches-to-academic-assessment.html.
2 “Vision Statement for Evaluating Research at Ghent University”, 8 November 2016, https://www.ugent.be/en/research/research-strategy/evaluation/research-evaluation-principles.pdf.
3 “Recognition and Rewards”, Utrecht University, accessed 16 March 2022, https://www.uu.nl/en/research/open-science/tracks/recognition-and-rewards.
4 Universitat Oberta de Catalunya, “Open Knowledge Action Plan”, May 2019, https://www.uoc.edu/portal/en/coneixement-obert/pla-accio/index.html.
5 American Society for Cell Biology, “Declaration on Research Assessment (DORA)”, DORA, 2012, https://sfdora.org/.
6 VSNU et al., “Room for Everyone’s Talent”, November 2019, https://www.nwo.nl/en/position-paper-room-everyones-talent.
7 Working group for responsible evaluation of a researcher, “Good Practice in Researcher Evaluation: Recommendation for the Responsible Evaluation of a Researcher in Finland”, 2020, https://doi.org/10.23847/isbn.9789525995282.
8 Universities Norway, “NOR-CAM – A toolbox for recognition and rewards in academic careers”, 1 June 2021, https://www.uhr.no/en/front-page-carousel/nor-cam-a-toolbox-for-recognition-and-rewards-in-academic-careers.5780.aspx; original matrix : Working Group on Rewards under Open Science, “Evaluation of Research Careers fully acknowledging Open Science Practices Rewards, incentives and/or recognition for researchers practicing Open Science”, July 2017, https://ec.europa.eu/research/openscience/pdf/os_rewards_wgreport_final.pdf.
9 Saenen, Bregt, Anna Hatch, Stephen Curry, Vanessa Proudman and Ashley Lakoduk, Reimagining Academic Career Assessment: Stories of innovation and change, January 2021, https://eua.eu/resources/publications/952:reimagining-academic-career-assessment-stories-of-innovation-and-change.html.
10 European Commission, “Towards a Reform of the Research Assessment System: Scoping Report” (LU: Publications Office of the European Union, 2021), https://data.europa.eu/doi/10.2777/707440.
Auteur
-
Pastora Martínez Samper
Open University of Catalonia – UOC

Le texte seul est utilisable sous licence Creative Commons - Attribution 4.0 International - CC BY 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.
Livre blanc — Une Science ouverte dans une République numérique
Direction de l’Information Scientifique et Technique - CNRS
2016
White Paper — Open Science in a Digital Republic
Scientific and Technical Information Department - CNRS
2016
Livre blanc — Une Science ouverte dans une République numérique — Guide stratégique
Études et propositions en vue de l’application de la loi
Direction de l’Information Scientifique et Technique - CNRS
2017
White Paper — Open Science in a Digital Republic — Strategic Guide
Study Review and Proposals for Implementing the Act
Scientific and Technical Information Department - CNRS
2017
Connecting the Knowledge Commons — From Projects to Sustainable Infrastructure
The 22nd International Conference on Electronic Publishing – Revised Selected Papers
Leslie Chan et Pierre Mounier (dir.)
2019
Actes des Journées européennes de la science ouverte
Open Science European Conference – OSEC 2022
Open Science European Conference
2022
Proceedings of the Paris Open Science European Conference
OSEC 2022
Open Science European Conference
2022