Version classiqueVersion mobile
OpenEdition Books

White Paper — Open Science in a Digital Republic — Strategic Guide

Scientific and Technical Information Department - CNRS

Freedom to analyse scientific results

What is text and data mining?

Texte intégral

1Data mining is a recent concept that first appeared in 1989 under the name Knowledge Discovery in Databases (KDD), also known in French as Extraction de Connaissances à partir des Données (EDC).

2The term “text and data mining” appeared for the first time in the field of marketing at the beginning of the 1990s. This concept, as applied in marketing, is closely linked to the notion of the “one-to-one relationship” (Michael Berry and Gordon Linoff, creators of Data Mining in Marketing), i.e. the customisation of the relationship between a company and its clients.

  • 1

3These Guidelines focus on the application of TDM to science, but TDM is practised in many sectors of activity such as, for example1:

  • direct marketing: in this area, TDM techniques are used, for example, to segment customer databases and predict customers’ purchasing intentions in order to optimise the marketing pitch;

  • communication: anti-spam filtering of emails and the global Echelon system for the interception of private and public communications (SIGINT) developed by the United States, the United Kingdom, Canada, Australia and New Zealand in the framework of the UKUSA Agreement, both use TDM techniques;

  • banking and finance;

  • insurance and health;

  • medical and pharmaceuticals.

4TDM has been adopted in the sciences in the last few years, with the development of open archives such as arXiv or HAL, so as to optimise the exploration of their databases, which are constantly growing in volume.

5The notion of text and data mining applied to the scientific field is today widely used to designate a variety of tools and activities. We therefore propose analysing the notion of TDM from a legal, technical and economic perspective in order to answer the following questions:

  • What is TDM?

  • What are the operations involved in TDM?

  • In what areas can TDM be applied?

  • How can we measure the efficiency of TDM?

TDM and the law

The consecration of a legal right to TDM via an exception

6Twofold exception. Article 38 of the Act (Petite Loi) establishes a right to text and data mining by introducing an exception to copyright and to the right of database producers, under the following terms:

The Intellectual Property Code is modified as follows:

1° After the second subparagraph of 9° of Article L. 122-5, a 10° shall be inserted as follows:

“10° Digital copies or reproductions made from a lawful source, in view of the exploration of texts and data for public research needs included in or associated with scientific results for the needs of public research, excluding any commercial purpose. A decree lays down the conditions under which the exploration of texts and data is implemented, as well as the terms for storage and communication of the files produced on conclusion of the research activities for which they were produced; these files constitute the research data”;

2° After 4° of Article L. 342-3, a 5° shall be inserted as follows:

“5° Digital copies or reproductions of the database made by a person with lawful access, in view of text and data mining included in or associated with scientific results in a research framework, excluding any commercial purpose. The storage and communication of technical copies resulting from processing, on conclusion of the research activities for which they were produced, are carried out by organisations appointed by decree. Other copies or reproductions are destroyed.”

7Lack of a definition. The text does not define the concept of data exploration or mining. When both terms are used in the same text, a comment is called for: the use of the term “exploration” of texts and data in the first part of the text, introducing an exception to copyright, and then of “mining” (fouille) in the second part, creating an exception to the right of the database producer, may raise difficulties of interpretation. The implementing decree could start by specifying that the concepts of exploration and mining cover the same practices.

The implementing decree could start by specifying that the concepts of exploration and mining cover the same practices. The decree should follow the European Union Directives in containing an article entitled “Definition”.

8Scope of the notion. Although the actual concepts of text and data mining and exploration are not defined, the text lays down limits and a framework for this practice:


Article 38

Legal basis

An exception to copyright and the right of the database creator: the right to copy and reproduce material digitally for the purpose of TDM

Scope of TDM

Mining of text and data included in or associated with scientific texts

Who benefits from the exception?



TDM limited to the needs of scientific research/a research framework

Non-commercial purposes

Lawful source/lawful access to texts and data subject to TDM

Introducing a TDM exception in the proposed Directive on Copyright in the Digital Single Market

9Preliminary reports. The White Paper Open Science in a Digital Republic referred to many reports, some of them commissioned by the European Commission, which advocated the revision of Directive 2001/29/EC “On the harmonisation of certain aspects of copyright and related rights in the information society” and the introduction of a right to TDM:

    • 2 CSPLA report, p. 8.

    the Sirinelli Report for the French Higher Council for Literary and Artistic Property (CSPLA) Rapport de la mission sur la révision de la directive 2001/29/CE sur l’harmonisation de certains aspects du droit d’auteur et des droits voisins dans la société de l’information (Report of the mission on the revision of Directive 2001/29/EC on the harmonisation of certain aspects of copyright and related rights in the information society) of December 2014 states that “demands for the creation of new exceptions to copyright relate in particular to the activities known as text and data mining (TDM)”;2

  • the study by Wolf & Partners in March 2014, entitled Study on the Legal Framework of Text and Data Mining,3 for the European Commission;

  • a group of experts of the European Commission also published, in April 2014, a report entitled Standardisation in the area of innovation and technological development, notably in the field of text and data mining;4

  • the Reda Report: this report, adopted by the European Parliament on 9 July 2015, “stresses the need to properly assess the enablement of automated analytical techniques for text and data (e.g. ‘text and data mining’ or ‘content mining’) for research purposes”;

  • the European Commission press release of 9 December 2015 presenting the measures to improve access to online content and the Commission’s vision of an overhauled copyright. In this context, “the Commission intends to work on key EU exceptions to copyright” and, in particular, “will revise EU rules to make it easier for researchers to use ‘text and data mining’ technologies to analyse large sets of data”.

10Proposed Directive. The proposed Directive on Copyright in the Digital Single Market (COM(2016) 593 final) was published by the European Commission on 14 September 2016.

  • 5 Proposal for a Directive of the European Parliament and of the Council on Copyright in the Digital (...)
  • 6 Press release of 14 September 2016.
  • 7 Introducing a TDM exception to copyright under English law, based on Article 5, 3a) of Directive 20 (...)

11Through this proposed Directive on Copyright in the Digital Single Market (COM(2016) 593 final),5 the Commission “proposes modern EU copyright rules for European culture to flourish and circulate … The proposals will also bring tools for innovation to education, research and cultural heritage institutions”.6 The objective of this Directive is to adapt the provisions relating to copyright to the increasing use of digital technologies, in particular in the field of scientific research, noting the multifaceted application of the provisions of the InfoSoc Directive and in particular the exceptions.7

12Defining TDM. Article 2 of the Directive offers a definition of the notion of text and data mining:

“text and data mining means any automated analytical technique aiming to analyse text and data in digital form in order to generate information such as patterns, trends and correlations”.

13Exception. Article 3 introduces an exception to copyright and an exception to the right of the database creator in favour of text and data mining in the following terms:

Article 3 – Text and data mining

1. Member States shall provide for an exception to the rights provided for in Article 2 of Directive 2001/29/EC, Articles 5(a) and 7(1) of Directive 96/9/EC and Article 11(1) of this Directive for reproductions and extractions made by research organisations in order to carry out text and data mining of works or other subject-matter to which they have lawful access for the purposes of scientific research.

2. Any contractual provision contrary to the exception provided for in paragraph 1 shall be unenforceable.

3. Rightholders shall be allowed to apply measures to ensure the security and integrity of the networks and databases where the works or other subject-matter are hosted. Such measures shall not go beyond what is necessary to achieve that objective.

4. Member States shall encourage rightholders and research organisations to define commonly-agreed best practices concerning the application of the measures referred to in paragraph 3.

14This exception can be analysed according to the following criteria (criteria used for an analysis of the French text):

  • 8 Draft Directive, Article 2: “‘Research organisation’ means a university, a research institute or an (...)


Proposed Directive

Legal basis

An exception to copyright and the right of the database creator: the right to make reproductions or extracts for the purpose of TDM

Scope of TDM

TDM on works or other objects

Who benefits from the exception?

Research organisations (the notion is broadly defined in Article 2 of the proposed Directive8)


TDM limited to the needs of scientific research

Non-commercial purposes

Legal access to the source material of TDM

15The Commission justifies the general principles behind this text in the following manner:

Text and data mining:

  • Option 1 consisted in self-regulation initiatives from the industry.

  • Other options consisted in the introduction of a mandatory exception covering text and data mining.

    • In Option 2, the exception covered only uses pursuing a non-commercial scientific research purpose.

    • Option 3 allowed uses for commercial scientific research purpose but limited the benefit of the exception to some beneficiaries.

    • Option 4 went further as it did not restrict beneficiaries.

  • Option 3 was deemed to be the most proportionate one.

16The Commission specifies that the purpose of this text is to provide a legal clarification and a framework of fair competition so that European researchers can use innovative techniques for data analysis, thus enabling them to more quickly find innovative solutions in response to major challenges such as global epidemics and climate change, and promoting cross-border and interdisciplinary collaborations. This exception supports European competitiveness by promoting Open Science.9

17Carlos Moedas, European Commissioner for Research, Science and Innovation, has justified the need for this exception in the following manner:

Science needs a copyright law that reflects the reality of the modern age. We must remove barriers that prevent scientists from digging deeper into the existing knowledge base. This proposed copyright exception will give researchers the freedom to pursue their work without fear of legal repercussions, and so allow our greatest minds to discover new solutions to major societal problems.”

18France and Europe are developing a legislative arsenal authorising the use of automated analysis techniques, but it is also interesting to look at the provisions adopted by other countries.

Shifting limits to the notion of TDM in the legislation of different countries

19The table below presents an analytical reading of the notion of TDM in British, American and Japanese law, which have each introduced a legal right to TDM.




Characteristics of the TDM

United Kingdom

Act of Parliament
Article 29 A introduced in 2014
10 in the Copyright, Designs and Patents Act (1988)

29A Copies for text and data analysis for non-commercial research
(1) The making of a copy of a work by a person who has lawful access to the work does not infringe copyright in the work provided that—
(a) the copy is made in order that a person who has lawful access to the work may carry out a computational analysis of anything recorded in the work for the sole purpose of research for a non-commercial purpose, and
(b) the copy is accompanied by a sufficient acknowledgement (unless this would be impossible for reasons of practicality or otherwise).
(2) Where a copy of a work has been made under this section, copyright in the work is infringed if—
(a) the copy is transferred to any other person, except where the transfer is authorised by the copyright owner, or
(b) the copy is used for any purpose other than that mentioned in subsection (1)(a), except where the use is authorised by the copyright owner.
(3) If a copy made under this section is subsequently dealt with—
(a) it is to be treated as an infringing copy for the purposes of that dealing, and
(b) if that dealing infringes copyright, it is to be treated as an infringing copy for all subsequent purposes.
(4) In subsection (3) “dealt with” means sold or let for hire, or offered or exposed for sale or hire.
(5) To the extent that a term of a contract purports to prevent or restrict the making of a copy which, by virtue of this section, would not infringe copyright, that term is unenforceable.

Legal basis: Exception to copyright for the purpose of “computational analysis”
Scope: Works and all related data
Beneficiary: /
Lawful access
Non-commercial purpose
For the sole purpose of research
Attribution of authorship
No copy may be transferred to any other person/no copy may be transferred or licensed by contract

United States

Federal Agency Data Mining Report Act of 2007: Article 42 U.S. Code § 2000ee-3

The term “data mining” means a program involving pattern-based queries, searches, or other analyses of 1 or more electronic databases, where—
A. a department or agency of the Federal Government, or a non-Federal entity acting on behalf of the Federal Government, is conducting the queries, searches, or other analyses to discover or locate a predictive pattern or anomaly indicative of terrorist or criminal activity on the part of any individual or individuals;
B. the queries, searches, or other analyses are not subject-based and do not use personal identifiers of a specific individual, or inputs associated with a specific individual or group of individuals, to retrieve information from the database or databases; and
C. the purpose of the queries, searches, or other analyses is not solely—
i. the detection of fraud, waste, or abuse in a Government agency or program; or
the security of a Government computer system.11

Legal basis: Positive right to conduct software-based searches
Scope: Models, searches, or other types of analysis of one or more electronic databases
Beneficiary: Federal agencies
Limits: Predictive analysis concerning terrorist or criminal activity
No use of personal identifiers

United States

Court ruling
Authors Guild v. Hathi Trust, 755 F.3d 87 (2d Cir.2014).

The court held that the HDL’s first use—creation of a full-text searchable database—was fair. It found that use “quintessentially transformative” because “the result of a word search is different in purpose, character, expression, meaning, and message from the page (and the book) from which it is drawn.” The court further held that the copies were reasonably necessary to facilitate the HDL’s services to the public and to mitigate the risk of disaster or data loss. In addition, it held that the full -text search posed no harm to any existing or potential traditional market for the copyrighted works. The court also held that the second use—access for the print -disabled—was fair. It concluded that providing such access was a valid purpose under the first statutory factor, even though it was not transformative. The court held that it was reasonable for the defendants to retain both text and image copies because the text copies were required for text searching and text -to-speech capabilities, and the image copies provide an additional method by which many disabled patrons can access the works. Finally, the court held that the fourth factor favored fair use given the insignificance of the present-day market for books accessible to the handicapped.12

Legal basis: The reproduction and use of digital books for their conversation, for text searches or for assisting the visually impaired is not an infringement of copyright, as it corresponds to “fair use”
Scope: Works/databases
Beneficiary: Parties to the dispute
Limits: Four criteria for fair use:
- non-commercial purpose
- the nature of work protected by copyright
- the portion of the work used
- the use must have no financial effect

United States

Court ruling
Authors Guild v. Google, INC.
16 October 2015

In sum, we conclude that (1) Google’s unauthorized digitizing of copyright-protected works, creation of a search functionality, and display of snippets from those works are non-infringing fair uses. The purpose of the copying is highly transformative, the public display of text is limited, and the revelations do not provide a significant market substitute for the protected aspects of the originals. Google’s commercial nature and profit motivation do not justify denial of fair use.
(2) Google’s provision of digitized copies to the libraries that supplied the books, on the understanding that the libraries will use the copies in a manner consistent with the copyright law, also does not constitute infringement.
Nor, on this record, is Google a contributory infringer.

Legal basis: Google’s legal right to make certain passages from books available in digital format on the basis of fair use
Scope: Works
Beneficiary: Parties to the dispute
Limits: Four criteria for fair use:
- non-commercial purpose
- the nature of work protected by copyright
- the portion of the work used
- the use must have no financial effect
With this ruling, the United States has given its researchers a significant advantage by granting them the possibility of digitising very large lawfully accessible datasets, of sharing the corpora and developing search functions and data-processing algorithms


Japan Copyright Act – Article 47 septies introduced in 2009

For the purpose of information analysis (“information analysis” means to extract information, concerned with languages, sounds, images or other elements constituting such information, from many works or other much information, and to make a comparison, a classification or other statistical analysis of such information; the same shall apply hereinafter in this Article) by using a computer, it shall be permissible to make recording on a memory, or to make adaptation (including a recording of a derivative work created by such adaptation), of a work, to the extent deemed necessary. However, an exception is made of database works which are made for the use by a person who makes an information analysis.15

Legal basis: The text provides for an “information analysis” exception for purposes of comparison, classification, and statistical analysis
Scope: Information of any kind
Beneficiary: /
Limit: Information analysis is not restricted to public research, or to non-commercial purposes

A comparative analysis of legal provisions

  • 16 The comparative table can be found in Annex 1.

20A comparative analysis of French, British, American and Japanese legal provisions, as well as of those in the proposed Directive, inspires the following remarks in the light of the four analytical criteria used (the legal basis, scope, beneficiaries and limits of TDM):16


Comparative analysis

Legal basis

An exception to copyright and the right of the database creator is favoured in each of the countries that have legislated to authorise TDM.
The United Kingdom prefers the expression “computational analysis” to “text and data mining”.


Different national legal codes offer different degrees of scope for TDM:
- Japanese law is the broadest, stipulating that TDM can be practised on any kind of information;
- English law and the proposed Directive reserve it for works and all associated data;
- French law refers to “text and data included in or associated with scientific texts”.


Neither English nor French law specifies any particular beneficiary of these provisions. However, the beneficiaries concerned are indirectly the public research organisations; TDM is limited to purposes of public research.
The proposed Directive expressly limits the use of TDM to research organisations.


All the legal systems mainly specify three limits. TDM should be:
- limited to the needs of scientific research;
- used for non-commercial purposes;
- applied to texts and data with lawful access.

21This comparative analysis shows that there are certain similarities between the different legislative frameworks governing TDM in the countries analysed.

22In order that the French legislation should not fall short of the provisions in other countries and the proposed Directive, we recommend that the implementing decree clarify the concept of “texts and data included in or associated with scientific texts” to make it as broad as possible.

Definition of the notion of TDM and the scope of its application. Clarification of the concept of “texts and data included in or associated with scientific texts” to make it as broad as possible.

TDM techniques

23From a technical point of view, TDM can be analysed in relation to:

  • the necessary technological objects involved;

  • the necessary technical operations;

  • its general features.

The technological objects necessary for TDM

24To mobilise text- and data-mining practices, researchers need:

  • input data;

  • processing tools;

  • output data, when processing tools are applied to the input data.

25The raw material: Input data. Technically speaking, TDM could be described as a process for the extraction of knowledge from selected texts and data, based on mechanisms for the identification of hitherto unknown structures that are scientifically valid and exploitable. Each scientific field (astrophysics, molecular biology, sociology of organisations, marine geology, linguistics, etc.) has developed its own arsenal of resources and techniques for data collection: sensors, probes, satellites, sequencers, cameras, digitisation, simulation, chemical analysis, and so on, which lead to the accumulation of huge datasets (Big Data).

  • 17 CNRS STI Strategic Orientation Plan, Les publications scientifiques : une augmentation continue et (...)
  • 18 CNRS STI Strategic Orientation Plan, Données et publications : une course poursuite (Data and publi (...)

26Overall scientific production is also experiencing spectacular growth due to the increase in scientific activity around the world, exacerbated by pressure to publish. Global annual production of scientific publishing has progressed significantly; for example, between 1996 and 2012 it rose from 1,134,000 to 2,250,000 articles per year on Elsevier’s Scopus database.17 However, “despite its rapid growth, publication is not increasing in proportion to the production of research data”: more than 90% of data remain stored on local hard disks and are consequently not shared.18

27The process of scientific analysis requires the use of research data in the broadest sense, concerning both scientific publications and the raw data, and including tables, images, statistical data, sounds, other bodies of text, and in general all the information needed by the researcher.

28High Energy Physics and the LHC (Large Hadron Collider) provide an extreme example of this new scientific practice centred on data: a technical device built by engineers – the collider – is associated with instruments designed by researchers – the detectors – to study the particles emitted during the billions of collisions produced. Something like 15 petabytes of data are produced every year. Such a volume of data requires both colossal technical processing resources and a data management infrastructure capable of providing maximum availability to research teams.

29But, before they can be made available, the data must be prepared. For example, it is easy to understand that the major earth and ocean observation programmes, which are usually conducted cooperatively as international operations, require processes for preparing the data before any data mining can be undertaken. These activities, grouped under the term “digital data curation”, include the selection, verification, standardisation, annotation, reformatting, enrichment and structuring of the data collected; the ultimate goal is to have qualified data that can then be used for scientifically valid data mining.

30The leading digitisation programmes for old books thus include processes to create information-rich metadata, facilitating the extraction of knowledge.

31The data repositories resulting from these curation processes are then made available to researchers through thematic access portals such as the International Virtual Observatory (astronomy), the World Observatory of Biodiversity, or the portal of the World Climate Data and Monitoring Programme.

32Processing capabilities obviously play an essential role in the development of TDM practices. The capabilities of computer centres are constantly improving, and they are organised to perform grid computing at the international scale, and to respond to the challenges of processing the massive flows of data generated by research into the climate, the environment, health and astronomy. At the end of the processing chain, there is a need for tools capable of representing and/or displaying complex data, which require high levels of computing power.

  • 19 CNRS STI Strategic Orientation Plan, p. 16.

33Analytical software. In addition, the research practices of the different scientific communities are undergoing change, moving towards more collaborative research involving multiple actors, and are based on the latest technologies for processing and exploiting large bodies of data produced in a shared environment.19

34TDM technologies aim to reveal relationships between the data items analysed, detect links of cause and effect, establish models, and validate their reproducibility. To do this, depending on the types of data and the proposed objectives, TDM is used alongside techniques from descriptive statistics, data analysis (exploratory statistics) or informatics (artificial intelligence).

35Text mining uses the same techniques as data mining but it is first necessary to process the textual data with language technologies to make them compatible with the methods used in data mining. The use of data structures adapted to the properties of the texts and semantic algorithms is specific to text mining.

36The data-analysis methods used (factorial, discriminant, principal components, multiple components, correspondences, etc.) bring out the different internal dimensions of the datasets, revealing the parameters that show how the data is organised. The classification methods (clustering, unsupervised learning) can identify groups of elements. The purpose of regression methods and supervised learning (artificial intelligence) is to predict the evolution of certain behaviours as a function of other variables.

37TDM software contains features comprising all or part of this processing chain, ranging from access through to the preparation of data, including the application of selected computing algorithms (learning), the exploitation of output, the exploitation of models, and the visualisation of results.

38The software specific to text-mining processes natural language by grammatical labelling, syntax rules, ontologies, learning from labelled corpora, etc. Based on the corpora of documents thus structured, different analytical algorithms can be used: automatic classification, analysis of trends, rules of association, etc.

39Many tools and techniques for the automatic analysis of texts and data have been developed by French research laboratories and are already used by researchers.

40At the French Alternative Energies and Atomic Energy Commission (CEA), TDM is widely used in particle physics, nuclear physics and astrophysics.20 Using smart software to analyse interoperable archives is fundamental to European projects such as the European Virtual Observatory,​, or EUROPLANET,​.

41In other areas, TDM is used by CEA teams for automatic language processing, in particular for the compilation, visualisation and analysis of networks of citations (​aan/​index.php) or to improve the performance of specialised search engines (​ or​acl/​).

42At INRA, the Bibliome research team (MaIAGE-INRA Unit) has developed many TDM applications for the texts of scientific articles in areas of interest to INRA. Some of these projects also use references (PubMed), patents (EspaceNet) and professional journals (e.g. Perspectives Agricoles).

43The Lisis UMR (“joint research unit”, in this case run by INRA, ENPC and UPEM) (​), associated with the IFRIS, is developing a digital platform entitled CorTexT for the analysis of textual corpora (​) in the framework of research in the human and social sciences.

44The CIRAD is also concerned, through the IATE UMR (CIRAD, INRA, SupAgro, University of Montpellier II), and develops innovative methods and tools for the processing of data and knowledge. The objective is to propose decision-support methods and tools for use in the overall management of biomass transformation processes. These methods and tools must be capable of collecting, representing and managing different types of data and knowledge, including imperfect data (for example, unreliable or imprecise data, and so on), expert opinions, and process engineering models. In addition, the tools proposed must be capable of taking multiple criteria into account, as well as the preferences and arguments of the actors in the agricultural sector.

45At IRSTEA, scientists from the TETIS UMR use innovative TDM solutions in the framework of research conducted in collaboration between IRSTEA, INRA and AgroParisTech: exploring scientific articles to identify new research themes by the enrichment of semantic resources (termino-ontological resources for each speciality, thesauruses, etc.) or improved tools for epidemiological surveillance in animals. In addition, TDM can identify relationships within heterogeneous datasets in very voluminous corpora that include both scientific and non-scientific texts (datasets and databases, images, etc.), which has led to the discovery of new and complementary knowledge (​index.php/​fr/​).

  • 21 These examples are drawn from a note by the Association of STI managers of research organisations ( (...)

46The GESTE UMR (IRSTEA, ENGEES), in Strasbourg (​), currently has scientists working on the problem of emerging pollutants in water and changes in the behaviour of individuals and artisans to reduce emissions of these pollutants. This is a major issue in terms of health and the environment, still insufficiently understood in terms of either the risks or the solutions. The use of TDM will enable risk-mapping, in both its scientific and societal aspects, clarification of chronology, and the identification of sub-themes as well as key institutions and actors.21

47INRIA has also developed GROBID (GeneRation Of BIbliographic Data),22 an automatic learning (or machine learning) tool available in open source. This application can extract, analyse and restructure scientific publications in raw formats (e.g. PDF) into a Text Encoding Initiative (TEI) format.

48Other tools. Software such as Alceste23 (developed by the company IMAGE and the CNRS) and Calliope24 (developed by Astefo) are specialised in lexical analysis. The Gargantext project provides an example of such an analysis. It can be used to analyse texts according to a paradigm other than complete words:

It smees that teh oredr of the leettrs in a word has no imtorpance. The frist and the last leettr must be in the rghit plcae. The rest can be in a total disrodre and can stlil be read with no proelbm. This menas that we don’t raed each leettr in itslef, but the word as a whole. A chagne in the frmae of rference and we trnasopse the rselut to the text itslef: the orede of the words is unmiprtoant compared to the conetxt, which cuonts much mroe: countertexted with Gargantext.”25

49Weka is one of the most widely used TDM applications in the world. It was originally developed by the University of Waikato in New Zealand, and enables data to be viewed and analysed rapidly. Finally, many commercial software applications have been produced by companies, such as RightsDirect,26 KNIME27 or RapidMiner.28

50Scientific data and articles, which are the primary source of information for analysis by TDM, may be imported from platforms for the sharing of scientific and technical information (such as digital libraries, databases, open archives, search engines, etc.). Some platforms are attempting to facilitate free access to scientific production such as, in France, HAL, Gallica, NAKALA, ISIDORE, OpenEdition and Persée.29

51ISTEX. Lastly, value-added services are being developed on the digital platform for access, sharing and enrichment of scientific and technical information (STI) of the ISTEX project (Excellence Initiative of Scientific and Technical Information – ANR-10-IDEX-004-02). These services are designed to enable users to analyse large bodies of data automatically, in particular via the following services:

  • data enrichment;

  • machine-induced datasets;

  • nano-publications;

  • collaboration patterns;

  • influence patterns;

  • semantic analysis;

  • impact analysis;

  • automatic document generation.

52Many other platforms for sharing scientific and technical information are listed by the CNRS and described in order to promote initiatives for pooling knowledge and free access to scientific production.30

53Results. Once the TDM process has been applied to the datasets, a result is generated automatically. This result is presented differently depending on the analytical technique used, which depends in turn on the analytical approach chosen by the researcher (“user-generated content”).

54The result is new knowledge. This can be of several different types and exploited in several different ways:

  • recommendation systems seek to predict behaviour by analysing and filtering the information contained in the data;

  • in science, the links identified between data items are used to recommend new research hypotheses.

55The result produced becomes part of the scientific approach as new analytical data.

The technical operations behind TDM

56It is also important to understand the technical processes underlying every processing operation in TDM.

57Atilf. In a note dated 14 May 2014, Jean-Marie Pierrel, representing the University of Lorraine (acting on behalf of the Conference of University Presidents), who is responsible for the development of value-added services as part of ISTEX and Director of Atilf (Analysis and Computer Processing of the French Language), explained the technical operations carried out by the laboratories in the framework of the development of TDM. In his note, Mr Pierrel makes a distinction between two phases:

1. Work carried out on computers:

  • work is copied from sub-corpora onto computers;

  • linguistic annotation of the data;

  • detection of terms and construction of a terminological reference framework:

    • the detection of named entities and construction of a reference framework of named entities;

    • annotation and lemmatisation of texts, and construction and management of glossaries;

    • tagging of bibliographical databases.

  • use of these annotations in the procedures for selecting articles following a user request;

  • mapping on the basis of the elements annotated.

2. The distribution of the results of searches using TDM

  • in scientific publications;

  • as the dissemination of the results of TDM.

58Illustration of TDM. The following diagram illustrates the steps in the processing of information in the framework of TDM applied to direct marketing, whose principles can be transposed to the scientific field:31


59Summary diagram of TDM operations. Considering all these elements, the processes for dealing with information by TDM are complex and can be summarised in three stages. The technical operations carried out in each of the steps are detailed below:

60Each of these steps needs to be clearly defined and secure, standards and formats must be defined, and a procedure for carrying out the TDM operations must be drafted and possibly standardised.

“Macro” functions within TDM

61Taking a functional macroscopic view of TDM provides another way of grasping the concept as regards its objectives.

  • 33 Yannick Toussaint, Extraction de connaissances à partir de textes structurés (Extracting knowledge (...)
  • 34 Fidelia Ibekwe-Sanjuan, Fouille de textes : méthodes, outils et applications (Text mining: Methods, (...)

62The development of TDM techniques opens new perspectives for the analysis of large volumes of texts and data (Big Data). TDM uses an inductive approach to analyse all this information grouped in corpora. Text mining encompasses a set of practices and a variety of methods that are difficult to treat in a unified way.33 Yet it is possible to paint a comprehensible picture in broad strokes.34

63The vast increase in the number of scientific articles in digital format has made a wide variety of information available: tables, images, statistical data, sounds, corpora of text, and so on. The latter may be written in very different styles (language, expressions, etc.). All these styles and genres of texts to be analysed thus require that researchers using TDM first create coherent corpora of data. The purpose of TDM is to achieve a specific objective. Data are not mined to give a general insight into the contents of a corpus, but as a way of exploring them to answer a specific question.

64TDM therefore analyses a set of data according to a criterion of novelty (what emerges when all the texts in a corpus are cross-referenced, for example?) or a criterion of similarity (what theme recurs in all the texts of the corpus?). It is then for the researchers to find the meaning behind all the cross-references among the data. After selecting and transforming the data (according to their coding, in particular), the researcher uses TDM tools to arrive at an interpretation and assessment. Each of the stages of TDM is therefore essential, from pre-processing and formatting of data through to the conclusions.

65Let us take an example to better understand these techniques. INRA is currently conducting a study with the help of text mining in order to identify new species of fish suitable for captive breeding, i.e. species that could be adapted for aquaculture. Considerable amounts of data relating to fish (breeding, feeding, living environment) are analysed using text-mining technologies. Many of these data come from the ISTEX corpus. The goal is to identify the standard features of aquaculture fish in order to better understand the phenomenon of domestication, as well as to identify those species of fish most similar to aquaculture species (fundamental research). The species identified could then be used in domestication experiments (applied research). The use of the results could open new opportunities for fish-farming.

The economics of TDM

66TDM can generate both financial and social benefits:

  • lower costs and higher productivity;

  • more innovations in products and services.

67In addition, it is also important to analyse the impact of TDM on the world of publishing, in order to defuse any potential conflict while clarifying and securing the practices.

Reduced research costs and more numerous discoveries

68The economic potential of TDM lies primarily in the reduction of research costs and the possibility of more numerous discoveries.

69By reducing processing costs and saving time, TDM makes it possible to produce many more new research articles, thus enriching databases, etc. in a “virtuous circle”.

70The Jisc,35 a British organisation in charge of digital services for research, has estimated that the use and development of TDM for research would increase the productivity of public research without additional cost. TDM would provide the equivalent of 4.7 million extra hours of research work per year, for the whole of the United Kingdom.

71In the same vein, TDM enables the development of interdisciplinary research activities. Corpora could be constructed covering a range of subjects, to give access to the latest progress in biochemistry for a biological study, or to reveal historical changes in sociological analyses.

More innovation

72A large proportion of this new knowledge can be converted into business innovation. For example, in the field of healthcare, an analysis using PubMed36 came up with new hypotheses for research. A group of researchers extracted data from a corpus of medical articles. They looked for links between “drug–disease” and “drug–phenotype”.37 While browsing through all these articles, they identified genes that could be responsible for diseases because of the toxicity of certain medicinal drugs. This enrichment of the data could enable researchers and doctors to rapidly diagnose new links between a “drug” and a “disease”. From an economic point of view, this discovery could enable great progress in medical research, with limited labour costs.

73The economic benefits resulting from the use of TDM therefore arise essentially from the role it can play as a source of incremental innovation. In this respect, the Jisc38 emphasises that TDM “unlocks”39 knowledge, which must inevitably provide both economic and social benefits.

74Furthermore, TDM is a scientific and digital technique that requires considerable human capital if it is to be applied correctly. Text and data mining can therefore provide opportunities for specialised technical employment, primarily for engineers and IT specialists. In addition, research using TDM requires infrastructures and services whose creation and production will also contribute to economic growth.

Economic issues of TDM and the world of publishing

75The economic issues raised by TDM go hand in hand with those facing the world of publishing. TDM is having a profound effect on the traditional notion of copyright. While there will inevitably be questions, there should be no doubt about the advantages of this practice compared to the current publishing model.

  • 40 Digital Republic Bill, NOR: EINI1524250L/Bleue ETUDE D’IMPACT, 9 December 2015 : ”Focus 1: Economic (...)

76Some publishers have already protested that their profession will quite simply disappear in the light of an analysis of Article 17 of the Digital Republic Bill (Article 30 of the Petite Loi), which establishes a right of access to scientific publications after expiry of an embargo period. This argument was not retained in the results of the Impact Study of Article 17, published by the French National Assembly on 9 December 2015.40

77This warning has also proved unfounded in the United Kingdom, where an exception to copyright was introduced in favour of text and data mining in 2014, as well as in the United States, where the practice has also been given legal status (the case Authors Guild v. Google – 14 November 2013).

78Furthermore, the supposed threat from TDM to the profits of the world of publishing does not seem to exceed the possible benefits to society of intensive exploitation of data. The European Commission therefore agrees about the long-term benefits of TDM and has inserted an exception in favour of TDM in the proposed Directive on Copyright in the Digital Single Market.



2 CSPLA report, p. 8.



5 Proposal for a Directive of the European Parliament and of the Council on Copyright in the Digital Single Market, 14.09.16, COM(2016) 593 final,

6 Press release of 14 September 2016.

7 Introducing a TDM exception to copyright under English law, based on Article 5, 3a) of Directive 2001/29/EC of 22 May 2001 on the harmonisation of certain aspects of copyright and related rights in the information society (the InfoSoc Directive). An analysis of the texts of the Directive in their English and French versions makes it clear that the scope of the exception proposed is not the same depending on the language used.

8 Draft Directive, Article 2: “‘Research organisation’ means a university, a research institute or any other organisation the primary goal of which is to conduct scientific research or to conduct scientific research and provide educational services:
(a) on a non-for-profit basis or by reinvesting all the profits in its scientific research; or
(b) pursuant to a public interest mission recognised by a Member State; in such a way that the access to the results generated by the scientific research cannot be enjoyed on a preferential basis by an undertaking exercising a decisive influence upon such organisation.”






14 “Comment l’affaire Google Books se termine en victoire pour le Text Mining”, 21 October 2015,


16 The comparative table can be found in Annex 1.

17 CNRS STI Strategic Orientation Plan, Les publications scientifiques : une augmentation continue et forte (Scientific publishing: Growing constantly and rapidly), p. 15.

18 CNRS STI Strategic Orientation Plan, Données et publications : une course poursuite (Data and publication: A downhill race), p. 15.

19 CNRS STI Strategic Orientation Plan, p. 16.

20 See the work of the Strasbourg Astronomical Data Centre at

21 These examples are drawn from a note by the Association of STI managers of research organisations (EPRIST) on text and data mining, Le TDM comme outil innovant de recherche scientifique (TDM as an innovative tool for scientific research),








29 List (with descriptions) of STI sharing platforms:



32 ©Jisc CC BY-NC-ND

33 Yannick Toussaint, Extraction de connaissances à partir de textes structurés (Extracting knowledge from structured texts), Document Numérique, 8(3), 2004, pp. 11–34.

34 Fidelia Ibekwe-Sanjuan, Fouille de textes : méthodes, outils et applications (Text mining: Methods, tools and applications), coll. Systèmes d’information et organisations documentaires, Hermès, 2007, 352 p.



37 A collaborative project between CTD and Pfizer: Manual curation of 88,000 scientific articles text-mined for drug–disease and drug–phenotype interactions”,

38 page 38

39 Jisc, p. 38.

40 Digital Republic Bill, NOR: EINI1524250L/Bleue ETUDE D’IMPACT, 9 December 2015 : ”Focus 1: Economic impact on institutional scientific publishing in France[:]
The impact of this measure on the economic equilibrium of French institutional scientific publishing, which essentially consists of human and social science publishers, must be assessed relatively, insofar as the majority of their current turnover comprises subsidies from establishments or laboratories. The journals, moreover, represent only 18% of their publishing production on average, while between 40% and 60% of the overall sales figure associated with these journals is achieved from the publications for the year, which would remain subject to embargo at maturity of the proposed measure, ensuring that these players would be only marginally affected.”

© OpenEdition Press, 2017

Conditions d’utilisation :


Open access