Chapter 15: Towards reproducible research in an interdisciplinary framework: issues and propositions regarding the transfer of the conceptual framework and the replication of models
Entrées d’index
Keywords : change, data, experiment, knowledge, measurement, replicability, reproducibility, simulation, transfer, validation
Texte intégral
Introduction
1The notions of reproducibility and replicability are currently the object of intense debates involving all scientific circles, whether by way of authoritative journals1, blogs and research notebooks, or more generally through social networks. The entire scientific world seems to be mobilised to identify, with more or less certainty2, the “good and bad pupils” in this domain. At the origin of this movement, recent and multiple scandals seem to touch one discipline after another (medicine3 4and psychology5 6, for instance) and reveal doubtfully scientific practices, which suddenly inspire fear that they may, at least in certain disciplines, have become standard procedure7 8. This phenomenon has led researchers to question themselves more closely about its causes, but also about how to monitor and reverse this tendency. Thus strong collective initiatives, private and public, have been put in place (Reproducibility Initiative9, for example, on Open Science Framework10). Moreover, this situation has fostered a return to a more general reflection on the terms, practices and tools of scientific research, which, together with the evolution of technical means, has produced an explosion of available solutions11.
2The conclusion of the studies effected within the TransMonDyn project make it possible to establish a conceptual framework and a descriptive grid common to all of the twelve transitions examined (cf. chapitre 2), and have then given rise to multiple responses with regard to modelling (cf. the chapters of part 2). Yet, one is bound to wonder about the validity of the results obtained, in terms both of ‘reproducibility’, with respect to the models proposed, and of the critique one might apply to the conceptual framework in view of the results obtained. Now that the models have reached varied stages of development, this chapter supports an overall reflection, illustrated by some examples, so as to contribute to the scientific and public debate on questions of reproducibility and equifinality, as these may apply in an interdisciplinary context. In this chapter, we have chosen to concentrate uniquely on the transfer of the conceptual framework and models of the Agent-Based Models (ABM) type, so as to bring a perspective both theoretical and practical on the question of scientific reproducibility.
Validating methods and results by their reproducibility: which ones, why and how?
Introduction to the notions of repeatability, reproducibility and replicability
3These notions have been long debated within each discipline. There is no single and unique theoretical or practical understanding of these terms, given that the objects to which they apply may be so different. The most current definitions of these notions are grounded in the vocabulary of metrology and computer sciences12. They are therefore related to the overall verification of measurement. In the case of repeatability, it is a question of doing repeated measurements of a single object, or similar objects, in specific conditions. When reproducibility is the issue, these measurements are taken using different instruments and in variable conditions (place, operator), which must then be specified. Replicability has been studied in the computer science domain and designates the capacity to reproduce the results of an analysis identically13.
The question of reproducibility in social sciences
4These definitions are essentially centred on measurement and calculation in a computer environment, and not on evaluation of analysis based on a corpus of data that are often collected in a heterogeneous manner. In social sciences, databases are most often developed on the basis of data collected by enquiry or field investigation. Even if rigorous protocols are established, the ‘measurements’ depend largely on personal expertise, and not on the sensor of a machine. Moreover, they depend completely on field work and are therefore, by definition, treated in variable conditions. This excludes from the start the principle of strict repeatability. Nevertheless, there are ways of approaching reproducibility in social sciences.
5In archaeology, for instance, as in other disciplines, these questions have been invigorating research for more than half a century with respect to the collection of data and therefore the enrolling of archaeological information. This enrolling enables the ‘measurement’ of facts such as the size of a series of objects, the density of objects discovered in a given place, or the distance that separates objects of the same type. Depending on the mode of investigation, protocols may have been specified to assure collection in the most homogenous conditions possible, or at least well enough described to attempt subsequently to balance the measurements taken14 1516. For the settlement systems studies, this type of approach has allowed comparisons to be made between a number of zones studied17 1819. The measurement, however, is neither direct nor effected by specific instruments of measurement, but rather by human ‘sensors’ whose ‘calibration’ is in reality difficult to control. Prior to the measurement itself, an initial ‘subjective’ act will consist in identifying an archaeological object in order to ‘collect’ it, while a second act will involve putting this object in relation to others to establish a measurement – for instance, the density of objects in a given place. When the same person returns several times to an area to verify his discoveries and his calculation of the density of objects – as is indeed a regular practice – a first step is taken on building a framework of repeatability. The conditions may be modified, however, without necessarily being able to know which parameters could have varied and what effect the taphonomic process20 may have had on the objects observed – or absent from the observations. In fact, only the sum of accumulated knowledge makes it possible to obtain an acceptable consensus regarding the phenomenon studied and authorises acknowledgement of the principle that an archaeological site is the basic datum for studying the settlement system (cf. below).
To reproduce measurements or a reasoned analysis?
6
7Putting in relation the archaeological ‘sites’ which will compose the system under study implies hypotheses which do not rely uniquely on archaeological observation but on theories formalised in other disciplines21. Indeed, archaeology is founded on the study of material traces, and not all human relations necessarily leave these – a fact that does not rule out their existence. Thus there is, in the relating of archaeological sites, an important component of interpretation founded on reasoning, implying knowledge beyond material traces alone. It will be possible, for example, to bring to bear the gravitational model to estimate the value of a flow between two places for ancient periods on the basis only of the material traces left in each. This example shows that the reproducibility or repeatability of a result does not depend uniquely on a ‘measurement’ but also on reasoned analysis. This situation does not merely oppose the past and the present because it is valid in other social sciences where the measure is not always a tangible observation, as in the case of a ‘sense of belonging’ or of ‘well-being’, for example.
8Nevertheless, since the reasoning and the hypotheses issued from the domain under study are chosen, formalised and organised toward ‘facilitating’ 22an interrogation by way of the ‘virtual laboratory’ (simulation), we can consider that the concepts of replicability, reproducibility and repeatability are applicable to the model implementation.
Toward the question of equifinality
9The distinguishing criterion of falsifiability, as propounded by Popper, is a powerful idea which has marked the scientific community’s practices, including modelling in human science23. Still, it should be remembered that its application has been designed above all in a framework of logical demonstration whose many weaknesses have been made evident for a long time24 25. In itself, this would pose no problem, if it did not serve to justify the separation between ‘science’ and ‘non-science’ still defended in certain publications. According to Popper’s scheme, it would indeed be tempting to consider the human sciences as incapable of formulating revisable and durable theoretical frameworks. For Jean-Claude Passeron26, the social sciences construct knowledge in a completely different fashion from the physical sciences, adopting other types of scientificity. While adopting this specificity of the construction of knowledge particular to the social sciences, Denise Pumain27 proposes thinking about knowledge construction, at once cumulative28 and rich with fruitful contributions, supported by the diversity of disciplinary points of view within the social sciences.
10Accordingly, in an analysis of a number of studies by geographers and archaeologists applying the law of Zipf29 to analyse the hierarchical organisation of settlement systems, Lena Sanders shows the interest of adopting an approach combining multiple points of view30. The comparison of the results suggests that an approach on different levels of abstraction authorises a more subtle interpretation than one on a single level, albeit the most universal. Moreover, the different disciplinary practices favour a multiplicity of approaches permitting better explication of the recurrence of a rank-size law in the organisation of a settlement system.
11It is therefore possible to work on the elaboration of methods aiming to consolidate the construction of a form of knowledge which, if possible, sacrifices neither the originality nor the diversity of the points of view engaged. According to Denise Pumain31 :
Nous pourrions ainsi, tout en produisant des formalismes nouveaux, illustrer la question de la complexité d’une façon bien plus éclairante [...]. La complexité d’une notion serait mesurée par la diversité des regards disciplinaires nécessaires à son élaboration, à l’intelligibilité des objets ou des processus étudiés, selon un objectif donné de précision des énoncés et des contextes.
(We could thus, even while producing new formalisms, illustrate the question of complexity in a much more illuminating manner […]. The complexity of a notion would be measured by the diversity of the disciplinary views necessary for its elaboration, for the intelligibility of the objects or processes studied, in keeping with a stated goal of precision of the affirmations and the contexts.)
12To take account of the varied perspectives has been a deliberate choice of the TransMonDyn collective. This diversity both in procedures and formalisms has been a source of richness. Besides the result of each model as such, it is important to stress its mediating role between thematicians and modellers in order to produce constructive feedback on the common conceptual tool. If each model developed has been able to contribute specific forms of knowledge regarding each case studied, it has also contributed to confronting and accumulating knowledge and experiences. The simulation models, for instance, authorise the development and confrontation of hypotheses in a dynamic form, spatial and temporal, within veritable ‘virtual laboratories’.
13Nevertheless, a critical point remains: the question of equifinality. If its definition appeared first in biology32, this notion is also employed by geographers33 and archaeologists34. It designates, in a general way, the possibility of obtaining, for an open system, an identical final state by following varied trajectories and initial conditions. This signifies, in the case of a simulation model, that interplays of hypotheses or different parameters can lead to identical results. Equifinality highlights the prudence necessary to observe in modelling a particular phenomenon. On the other hand, in a context of comparative analysis, a determination of equifinality may be an advantage favouring critical distance on the objects conceptualised and applied (cities, for instance, are defined differently by archaeologists and geographers). Such distance favours the identification of the respective interests of the different approaches, and perhaps the combination of several views in analyzing the same object. In such a procedure, the model becomes a veritable instrument of mediation, offering everyone the opportunity of presenting his point of view in a framework intelligible to all, because it is, ideally, transparent and not confused by his own disciplinary baggage.
14Models should be able to be exchanged, shared, discussed, on the premise that only a model which approaches the ideal of transparency makes it possible to engage a discussion on honest bases. Indeed, our starting point is the principle that even a ‘bad model’, which would not permit reproduction of an observed situation and/or verification of an initial theory, remains a vehicle of interesting information when it comes to fine-tuning the construction of our analyses in the social sciences. It becomes, in our view, a ‘good model’ if its content makes it possible to understand the logical reasoning which leads to the result observed, offering the possibility of reproducing it.
What about the problem of data?
15If one places oneself within the framework of reproducibility as defined by the computer sciences, the data introduced in the social sciences may rapidly become an obstacle, especially for the more ancient periods. Indeed, each case study is founded on a set of data for the most part heterogenous and dependent on the conditions of investigation. From the point when one focuses on societies over the very long term, this heterogeneity between different case studies is increased, as much from the quantitative as from the qualitative point of view. Thus, for example, the simple fact to wish to identify a potential change of regime in the settlement system relies on extremely variable approaches, which range from the observation of a statistical fact established on a large set of data to a theoretical postulate based on sparse data (cf. chapter 2). Some parameters thus rely on mere theoretical hypotheses reconstructed on the basis of analogies and founded on a small number of data. It is therefore appropriate to be as precise as possible regarding the degree of uncertainty attached to the data – precision all the more necessary because the context is interdisciplinary. Similarly, explaining the mode of construction of the entities introduced in the model, as well as their inter-relations, is also crucial in an interdisciplinary context. Tools such as a common conceptual framework, or easy to understand arrow diagrams, allow for elaboration of this meta-information in the form of a progression, which can serve, notably, for the history of the construction of the model. This meta-information facilitates the transmission of the work achieved, as well as its evaluation.
16From this perspective, we can specify how comparison of transition processes, similar in appearance, may function in widely different eco-cultural contexts. Indeed, what matters more than the original data or the validity of the results to explain a transition phenomenon is the cross-checked evaluation of the reasoning, its strengths and weaknesses, undertaken on the basis of each case study in order better to understand and interpret a mechanism. We consider, in fact, that a cross-checked evaluation helps to distinguish the fundamental factors from the contextual factors at work in the process of transition. In this way, two obstacles are avoided: the over-simplification in a universal model which would tend to impoverish the interpretation, and the over-specificity of an ad hoc model which does not allow comparison and the possibility of comprehending global phenomena. The time allotted to the TransMonDyn project has not permitted comparison, from an overall point of view, of all the procedures followed in constructing the different models, but it has been possible to identify some reliable directions, and we present two of these in what follows, one concerning the transfer of the conceptual framework, and the other dealing with some methodological and technical points.
Transferring the conceptual framework of TransMonDyn to other spatio-temporal contexts
17The TransMonDyn collective undertook to engage the question of transitions in settlement systems on the basis of twelve case studies on extremely varied chronological and spatial scales but with the aid of a common conceptual framework, defined collectively. This framework relies on an approach at once systemic and ontological, permitting description of two distinct settlement regimes (cf. chapter 2, figure 3). Identification of differences between the two systems (Regimes 1 and 2) provides identifying the existence of a transition, which is then described in a harmonised manner according to five complementary dimensions (cf. chapter 2, figure 4):
way of living, use of space,
movement, transport, type of mobility,
way of feeding, relationship to the environment,
social organisation, form of social interaction,
power type and organisation, administration, territorial mesh.
18If this procedure of the TransMonDyn collective has established a common language for the description of the transitions, the operative aspect of this conceptual framework in other contexts remains to be verified. The point is to see whether the reasoning is reproducible in other case studies. With this objective, we have tested the transfer of the conceptual framework to settlement systems relating to intra-urban space, which corresponds to types of spaces different from those studied within the TransMonDyn project (from the regional to the worldwide scale). The case studies are drawn from three PhD theses that began inside the project35 3637, associated with different research questions (table 1). The aim is thus to test the relevance of the conceptual framework in various contexts.
Table 1: The three case studies used to test the reproducibility of the conceptual framework.
Case study | Research questions |
Beauvais: from the 19th to the 20th century | How does the shape of the street network evolve? How this evolution could be characterised using graph theory? |
Noyon: the urban space over its entire span of existence (1st-21st century) | What are the functional and spatial changes of the intra-urban space over the long term? How to confront the qualitative and quantitative observations? |
Tours: the urban space over its entire span of existence (1st-21st century) | What is the spatial logic(s) of the localisation of activities over the long term (centre/periphery, neighbourhood, etc.)? |
Development of a procedure for the transfer of the conceptual framework
19Generally speaking, our procedure is founded on two principal steps attached to a hypothesis of transition in the settlement system (figure 1). First, a major change in the intra-urban spatial structure is identified on the basis of expert knowledge and formalised data. We then hypothesise that this change corresponds to a transition in the settlement system. Finally, the hypothesis is tested by a protocol set up specifically for this procedure. It is based on the descriptive grid of regime/transition, which is made up of the five dimensions considered in TransMonDyn (figure 1: a-e). Each of these dimensions is tested by expert knowledge or by data, and three results are possible: (1) a change is identified in the dimension in question, (2) no change is detected, (3) no determination can be made because of a lack of data and expert knowledge. If less than three dimensions experience a change, and/or more than one dimension does not undergo one, then we have chosen to reject the hypothesis of the existence of a transition.
20This validation method, founded on a rigorous application of each dimension of the grid describing transitions, allows for comparison of the results obtained in the three case studies.
Figure 1: General diagram of the transfer procedure of the conceptual framework.

Application to three intra-urban case studies: Beauvais, Noyon and Tours
21The application of the proposed procedure to the three case studies is presented in figure 2, where the items depicted on a grey background refer, in a detailed way, to the general logic of the procedure presented in figure 1. The steps of identifying a change, hypothesising the existence of a transition and testing this hypothesis are there completed by elements specific to the context applied: the case study and the data. The processes connecting these items are represented by solid arrows: the construction and exploration of the changes, the different tests of the hypothesis. Finally, the interrogations relative to the transfer of the conceptual framework are represented by dotted arrows in the form of feedback loops. Even if this procedure follows a common principle, each of the three experiments takes a specific course. In figure 2, it is possible to follow the trajectory of each, thanks to the colour assigned to the respective case study.
Figure 2: Diagram of the application of the conceptual framework to three intra-urban case studies: Beauvais, Noyon and Tours.
22For each case study, the exploration of data of different kinds makes it possible to identify one or more major changes in the spatial structure of the settlement system considered.
23In the case of Tours, for instance, an important change is identified between 400 and 600, when a second urban centre developed, linked to the cult of St Martin, several kilometres distant from the existing city. The establishment of this bipolarity, well known historically, is validated by exploration of the database introduced in the PhD thesis, constituted of objects (buildings, streets…) localised in space and situated in time. Indeed, from the year 400, the appearance of a group of religious buildings has been identified, which becomes denser and extends itself little by little to the west of the city enclosure. The issue, then, is to test the hypothesis that this change in the spatial structure of the city can be interpreted as a transition in the settlement system. In the cases of Noyon and Beauvais as well, changes have been identified, and it is the statistical exploration of the data and of significant clues (comparison of morphological indexes, graphs, multi-dimensional statistics…) that makes it possible to highlight the changes constituting ‘candidates’ for the hypothesis of a transition.
24For Beauvais and Tours, among all the major changes potentially able to occur during the existence of the system, and thus susceptible of being interpreted as a transition, a single one was selected to be tested. By contrast, for Noyon, all the changes identified throughout the city's existence were considered as hypotheses of a transition.
25Once the hypotheses were proposed, they were submitted to the test previously described (figure 1). We develop here, on the basis of the case of Beauvais, an example of the procedure for validating the hypothesis of the existence of a transition. An initial attempt revealed that between the supposed Regime 1 (before World War II) and Regime 2 (after the reconstruction of the city, in 1960), only two primary dimensions ((a) way of living and (b) movement) underwent a partial change (construction of larger dwellings, reconfiguration of housing blocks and the street network). The other dimensions indicate that the change took place long before World War II. In fact, the late industrialisation of Beauvais during the second half of the nineteenth century entailed a collection of major mutations, notably in travel modes (abandonment of the stagecoach from 1876 with the opening of a direct railway line to Paris), in urban land use patterns and in the forms of social organisation (decline of craft activity and development of manufacturing for instance). These findings specific to the case study allowed us to invalidate the initial hypothesis of a transition between World War II and 1960, and to redefine the chosen temporal limits by situating Regim 1 in the period preceding the industrialisation of the city (first half of the nineteenth century). The transition thus turns out to extend over a longer period (between the second half of the 19th century and 1960) and is verified this time by at least four of the five dimensions of the grid (the dimension (e), power type and organisation, administration, territorial mesh, is not verifiable due to the absence of expert knowledge).
Confrontation of the results of the three experiments
26With regard to Beauvais, the validation of the hypothesis of transition proved unsuccessful on the first test, but adjusting the temporal boundaries permitted identification of a new hypothesis of transition. Questioning the initial hypothesis provided interesting feedback on the case study and, finally, on the initial research question (table 1). In fact, the transfer of the conceptual framework revealed in Beauvais a phenomenon of hysterochrony38 in the street network: because of its material nature, it resisted the change in the settlement system for a long time, and its transformation took place only after the massive destruction of World War II (that is, almost a century after the beginning of the city’s industrialisation).
27In the case of Tours, the hypothesis of the existence of a transition between 400 and 600 was validated. This moment of change in the settlement system is therefore indicated more precisely according to the dimensions of the grid. By contrast, when one sets this result opposite the research question related to the case study (table 1), one finds that the change in the logic of localisation of activities in the intra-urban space is not the driving force of the transition observed. At the moment when the transition takes place, the logics of localisation of activities (for instance, are they situated at the periphery or in the city centre?) do not experience a major change, even when one employs a wider temporal window. For example, the installation of the tomb of St Martin, at the origin of the development of the second centre, still conforms to the former ancient logic of the localisation of necropolises close to, but outside, the cities. Contrary to what one might have expected, the transfer of the conceptual framework shows that a transition of the spatial structure of the settlement system can be observed without change in the principles of localisation of urban activities, a reflection of the functioning of the city.
28Whether one is dealing with Beauvais or Tours, the exercise has thus made it possible to question the temporal differences existing between, on the one hand, the dynamic of the settlement system as a whole and, on the other hand, the changes peculiar to the object of study: hysterechrony for the structure of the street network, discrepancy for the logic of localisation of activities.
29In the case of Noyon, the test for the existence of a transition was carried out for three periods of change of the intra-urban system (at the beginning of the 7th century, during the 12th century, and at the end of the 18th century). For the first two, the existence of a transition was validated. With the application of the conceptual framework, it was possible to specify the socio-spatial dimensions of the changes and, in so doing, to bring to light temporalities a priori not highly visible. Indeed, according to an initial interpretation of the data, the changes appeared rather to take the form of fairly rapid spatial and functional ruptures of the intra-urban system. Now, systematic application of the five dimensions of the conceptual framework suggests, on the contrary, that these changes are transitions in the system, characterised by relatively long durations. This study has therefore led to a redefined understanding of the city's rhythms of change. For the third period, the change identified at the end of the 18th century, the hypothesis of a change in the form of a transition was not validated, because the dimensions (b) movement, transport, type of mobility and (c) way of living, relationship to the environment, were set aside. Three factors may be at the origin of this rejection of the hypothesis of transition: (1) the data analysis grid may not correctly capture recent urban developments (urban sprawl, increasing functional specialisation of neighbourhoods, etc.); (2) exploration of the data may not have been sufficient to identify the transitions, because the methods used (observation of maps of different periods and factorial analyses covering all the periods) reveal changes which do not necessarily affect the settlement system. The latter can, indeed, continue to reproduce itself in the same structures, within the same regime, while profound changes have affected the city from a political or religious point of view; (3) the change is actually not a transition in the sense defined in the project.
30Beyond the results particular to each of the research studies, the transfer of the conceptual framework of TransMonDyn is effective in all cases. It appears, therefore, that the concept of transition as conceptualised and modelled in the collective project is reproducible. Most significantly, such transfer is possible at another geographic level than those considered in TransMonDyn. In terms of reproducibility, it should be stressed that application of the conceptual framework necessitates highly specialised knowledge of the object of study in all the fields covered by the five dimensions of the grid, as is not always the case. Nevertheless, from a comparative perspective, the experiment undertaken shows that the use of this conceptual framework is fruitful for the purpose of conducting a collective interdisciplinary research.
Means for achieving good reproducibility for the models
Implementing a procedure for reproducibility
31As we have seen, the conceptual framework defined for characterising a transition seems to be efficient in other contexts and on other scales than those studied in the TransMonDyn project. If most of the twelve case studies have been the object of specific modelling, the choice of formalisms and modes of implementation has remained extremely open, and the comparative perspective has remained at the centre of the collective interrogation. It is in this spirit that in this chapter we propose some avenues that might be explored.
Reproducibility, free access and simulation in theory
32After having engaged in a general way the question of reproducibility/replicability in the social sciences, it is necessary to see what meaning these terms take on when one puts them to work concretely in the construction of, and experimentation with, the simulation models developed. At the outset a question arises: is technical reproduction a necessary prelude to the reproducibility of a model? Ian P. Gent, in his ‘recomputation manifesto’39, affirms this for the computer sciences: these are distinguished by their capacity to reproduce or replicate the execution and behaviour expected of a source code40. The implementation and objectives of models are broadly different in social sciences, as we have seen above. In practice – even if one knows that the two are intrinsically linked – it would therefore be appropriate to dissociate the ‘cogs and wheels’ component of the simulation model (which depends on computer science) from its ‘thematic’ component.
33The “cogs and wheels” aspect relies on a ‘source code’, which must in any case satisfy the criteria of replicability of the computer sciences. The source code contains information on the way in which the rules of the model are organised and invoked. The methods for exploring scenarios (with their inter-connections) are also described by source code accompanying the models. With regard to the ‘thematic’ aspect, it is the very exploration of the model in a perspective of interpretation of different variants which is highlighted, for it allows the reasoning of the thematician to be enriched, particularly where there are few to none observable data (cf. chapter 3).
34This thematic exploration can also be formalised in computer terms (in the form of source codes) to perform sensitivity analyses or to evaluate the impact of the rules scheduling. If these materials are not released by the modellers, the results associated with the ‘thematic’ aspects remain purely hypothetical, for it is then impossible to verify their container41 or to rerun them exactly as they first were. The source codes are therefore vectors of the reasoning generated throughout the modelling process, as much for the ‘thematic’ aspects as for the ‘cogs and wheels’ ones. Because of the intertwining between the development of the reasoning and the computer substrate utilised, the question of reproducibility arises at the reasoning level.
35Beyond simple computer verification – permitted when the creators reveal the source code of the model – the practice of replication thus implies a critical re-evaluation of the underlying reasoning which has produced the construction and exploration of the simulation model42, as well as an analysis of the successive modelling choices and dead-ends43. For almost twenty years, and despite the efforts of different members of the ABM community, this procedure has faced difficulty in becoming generalised. In reality, it is often difficult to gain access to the necessary information: according to a survey44 covering all the publications related to social agent-based models in 2008, the proportion of model sources delivered with the publication is still below 20%. A scientific review45 dealing with the totality of scientific material accompanying the publications in the journal JASSS46 between 2001 and 2012 (diagram, table, equations, source codes, etc.) documents the reigning anarchy and stresses the urgency of a clarification of the practices associated with model publication.
36The point in the following sections is, on the one hand, to show how replicability can be implemented in a gradual manner using various technical solutions and, on the other hand, to illustrate by an example how this procedure helps providing a space for collective discussion.
The replicability of the code and the execution of a model
37The implementation of replicable models, the first step towards reproducible research, is not an easily achievable goal. We propose (figure 3) to identify several steps leading progressively to the realisation of this objective. Two objects are to be taken into account simultaneously in this progression – on the one hand, the simulation model (A) and, on the other hand, the exploration (B) associated with this model. Today all the techniques for implementing such a progression exist. Indeed, just as the appearance of the modelling platform NetLogo in social sciences has facilitated the independence of researchers with regard to computer modelling, efforts are being made nowadays to make these techniques, formerly reserved to computer scientists alone, more accessible to a broader audience.
Figure 3: The levels of replicability of a simulation model and its exploration.

Each level (from top to bottom) permits an advance in the procedure of reproducibility.
38In the first place, the steps of sharing the content of the model (stages A.1. to A.5.) are facilitated by the open platforms of model dissemination, such as the internet NetLogo-model-sharing site ‘Modelling Commons’47, which permits direct execution of the model online and its downloading for local execution (A.4.). The source code and its description are accessible to the user, something which allows it to be analysed, modified and run (A.5.). An initial discussion can thus take place on the basis of this source code and the results produced by its execution. The source code must be accompanied by documentation taking account of the reasoning and the choices decided on (A.1. to A.3.). If this does not allow for the complete replicability of the model, leaving partly implicit its computer implementation, it remains necessary to provide a synthetic view of the model. In the absence of an official standard for description of the model, one may refer to the ODD protocol (Overview, Design concepts and Details) developed by the team of the ecologist Volker Grimm48. This formalism of description, widely recognised and understood within the community of multi-agent modellers, enables models to be described along three axes: (1) Global overview, (2) Conception and (3) Details49. This protocol – sufficiently generic to be adaptable to each agent-based model50 – is nevertheless very heterogeneous in the specification of the descriptions it expects, thus permitting a simple static description (A.2.) or a more precise dynamic one (A.3.). The data utilised by the model (A.6.) can also be associated, just as information on the software dependencies required (simulation software version for example) (A.7.).
The reproducibility of the exploration of the model
39The execution of the model permits deduction of its results (B.1.), and its exploration forms a part of its process of construction. There are, therefore, in the life of the model prior to publication, a number of explorations, which will have given rise to a number of successive modifications of the source code of the model. It is by way of this back-and-forth, in an abductive procedure, that the questions are formulated and implemented, that the explorations are implemented to attempt to respond to them, and that the reasoning progresses (B.2.). The often stochastic nature of agent-based models (cf. chapter 3) reinforces the need for thorough exploration.
40In these conditions, the replicability of the explorations cannot be envisaged without using tools and techniques which permit the formalisation of the experimental modalities and the parameter values to employ51. The stage of replicability sometimes requires the use of exterior calculation resources, superior to those of a classic computer, for example with the software OpenMOLE52 which takes advantages of distributed computed environment 53.
The reproducibility of results: reproducible data and analyses
41A final step towards the ‘complete’ reproducibility of a model consists in rendering its results reproducible. The simplest method is to communicate the totality of the output data, which are then used to analyse the model. Given the amount of experiments which will have been undertaken and the stochastic nature of models requiring numerous replications, this mass of data may be large and difficult to manage. To mitigate this difficulty, it is possible to communicate ‘random seeds’54 used in the experimentation, which, with the aid of the same versions of software, permit identical reproduction of the simulations. Reproducing and publishing the raw data, like the outputs of the simulation, is a first step, but one must still be able to reproduce as well the manner in which they were introduced and exploited in order to analyse them. This can be solved by using, for example, ‘computational notebooks’ which aim at recording all the elements relative to the exploration within a single document.
A proposition for an application to the Young model
42In order to illustrate concretely all of the steps taken up in the preceding section, we propose an application to one of the models presented and developed in Chapter 3, the Young model55 56.
Publishing the model
43An initial description of the model was published (figure 3, A.1.) by Young57 in an article which, however, does not give all clues on the model details: ‘[the order of execution of the model’s four steps] is not explicitly addressed, however, in Young’s article, something which renders its reproducibility difficult’(cf. chapter 3). The description and analysis of it provided in chapter 3 have made it possible to specify some of the points left indistinct by the author of the initial version and to obtain a more precise and standardised description (A.3.). This work permitted the authors of chapter 3 to develop – in NetLogo – a version of the model whose functions and parameters they mastered more fully, in order to carry out their thematic exploration of the process of colonisation. Such developments are in line with the procedure of reproducibility as exposed by U. Wilensky. It has thus been possible to publish the new computer implementation of Young’s model on a suitable public platform for depositing software58, and thereby to capitalise on the experiment conducted in the TransMonDyn project and put the model at the disposition of other researchers. The platform used59 permits, on the one hand, to run simulations in an interactive fashion on the Internet (A.4.)60 (figure 4) and, on the other hand, public consultation of the source code61 (A.5.). These possibilities offer the opportunity of communicating the content of the model in a dynamic and scientific manner, and make it possible to propose modifications of the code, thanks to an appropriate online interface. The author of the software deposited is then in a position to accept these or not. Finally, this type of publication responds to a pedagogic objective by rendering comprehensible the concepts introduced by the manipulation of the model’s parameters.
44The source code having been published, it is thus possible to download it to execute the model locally on one’s own computer. It that case, the ‘software dependencies’ – the version of the NetLogo software to be used, for example – are indicated directly in the online documentation62, thereby satisfying the final criterion of reproducibility (A.7.). Respect for the latter principle tends to assure the durability of the model by reducing the problems of computer frameworks obsolescence.
45In our example, we note that we have no need to furnish the input data (A.6.), because the Young model, as implemented, does not use them. If it did, one could make use of the same ‘software repository’ which hosts the source code to store the data, or, alternatively, host them on a dedicated external platform63.
Figure 4: Online executable version and source code of the Young model.

Publishing the exploration in reproducible form: a collaborative tool
46Chapter 3 has presented a ‘heuristic’ exploration of the model. Here we will try to develop and generalise that exploration, while aiming at the reproducibility of each of the steps taken. The results presented in figure 5 enable illustration of the procedure which was followed in this example of exploration and constitute a first step in the reproducibility of the model by making available synthetic charts showing its behaviour (stage B.1. of figure 3). Moreover, the sources of this analysis are available64 (stage B.2.), rendering its reproduction possible.
47The first step consists in exploring the space of the parameters: how do the different indicators react to a change in the value of the parameters? The exploration design is a full factorial experiment – that is, the design of the experiment causes each parameter to vary independently of the others and systematically explores the totality of possible combinations. Four parameters are explored here: probability of birth (reproduction), of death, of moving, and probability of death by ‘overcrowding’. The exploration is ‘rough’, as the scale of variability of each of these parameters is fairly broad (0, 0.33, 0.66 and 1). Each point in the figure 5(1) represents a simulation: it lies at the intersection of two indicators, the share of the space occupied and the size of the population at the end of the simulation. The high values of the two indicators express the success of the colonisation.
48In the case of a probability of death fixed at 0.33 (figure 5(1)), these two indicators do not appear to satisfy the conditions of a ‘successful’ colonisation, below a probability of moving of 0.66. For such values (cf. the last two panels of figure 5(1)), one also notes that for a probability of death by overcrowding of 0.33 (blue points), the ‘best’ results appear to be situated between a probability of moving of 0.66 (strong population, but less occupation of the space) and 1 (better occupation, but smaller population). With this type of exploration, it is therefore possible to isolate the sub-spaces of parameters that are seemingly interesting to analyse more thoroughly by other methods.
49Figures 5(2) and (3) give an example of this by synthesising here an optimisation computation. The algorithm searches in the space of the parameters the values necessary to attain objectives fixed in advance. It is a question, for example, of identifying the combination of parameter values permitting the maximisation of both the size of the population and the portion of the space occupied at the end of the simulation. The results of the exploration form clusters (marked A, B, C and D) readily identifiable in figure 5(2). Cluster D, which corresponds to weak populations and an inferior occupation of the space, is unsatisfactory. Clusters B and C each produce the best results for only one of the two objectives. Cluster A is the one which maximises both the population and the space occupied. These clusters of simulations are described with the aid of combinations of the four parameters (figure 5 (3))65. This method of exploration proceeding from the ‘rough’ to the refined makes it possible to adjust the parameters and better to understand the behaviour of the model.
50This application example illustrates the way in the social sciences may make the most of their modelling experience to achieve the objective of cumulativity of knowledge within a formalised framework, as advocated by D. Pumain66.
Conclusion
51The methodological and technical propositions advanced in this chapter undoubtedly deserve to be tested and discussed, but they make it possible to open perspectives in a debate which the social sciences cannot ignore. We have intentionally centred our presentation on the conceptual framework and the agent-based models, but it is obvious that this discussion should be opened up and pursued, for it concerns all the steps of modelling and all the types of formalisms. Indeed, we have not here taken up the question of the formalisation of the conceptual model of data and of theoretical hypotheses, but it has been the object of numerous discussions within the project.
52Another point seems important to us but continues to elude thorough treatment: this is the articulation between the conceptual framework and the models from the point of view of reproducibility. If the conceptual framework is indeed common to all twelve of the case studies, the proposed models are using different implementation frameworks. It would be interesting, in the future, to achieve harmonization of both the different models and conceptual framework. Such a process would allow modelling to fully play its role as mediator within an interdisciplinary project.
53In this chapter, it seemed convenient to us to dissociate the technical (computer) part of the reproducibility from its theoretical part (the transfer of the conceptual framework). Reuniting those two aspects would nevertheless be both interesting and innovative. This proposition would participate in the reflection on scientific reproducibility by suggesting an evolution of practices in an interdisciplinary framework, as far as possible, taking into account the complexity and the rapid evolution of the techniques used, and the additional time necessary for a reproducible research. Such research, based on explicitness and transparency, would contribute to the capitalization of knowledge with a complementary added-value from that of classical approaches.
Figure 5: Synthetic results of the steps of an exploration of Young’s model.

NB: In part (1), where the points are partially transparent, greater opacity indicates a superposition of numerous points. In part (3), the charts are ‘violin plots’: the width of each bar describes the relative density of the simulations of one class (marked in part (2)) for each value of the corresponding parameters.
Notes de bas de page
1 http://www.nature.com/news/reproducibility-1.17552
2 The question has been raised even of the replicability of ‘studies of replicability’, as is attested by certain animated and very recent debates concerning publications in psychology (http://projects.iq.harvard.edu/psychology-replications/ and https://hardsci.wordpress.com/2016/03/03/evaluating-a-new-critique-of-the-reproducibility-project/), or in economics (Camerer 2016, http://retractionwatch.com/2016/03/03/more-than-half-of-top-tier-economics-papers-are-replicable-study-finds/)
3 Begley C. Glenn, Ioannidis John, ‘Reproducibility in Science: Improving the Standard for Basic and Preclinical Research’, Circulation Research, 116(1), 2015, pp. 116-126.
4 Ioannidis John, ‘Why most published research findings are false’, 2005. http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124
5 Baker Monya, ‘Over half of psychology studies fail reproducibility test’, Nature, 2015. http://www.nature.com/doifinder/10.1038/nature.2015.18248
6 Yong Ed, ‘Replication studies: Bad copy’, Nature, 485(7398), 2012, pp. 298-300.
7 Begley C. Glenn, Ioannidis John, ‘Reproducibility in Science...’, op. cit.
8 Kaplan Robert M, Irvin Veronica L, ‘Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time’, PLOS ONE, 10(8), 2015, p. e0132382.
9 http://validation.scienceexchange.com
11 Such as ROpenSci (https://ropensci.org/), Dataverse (http://dataverse.org/), or Zenodo (http://zenodo.org/)
12 ‘Au sens le plus général, une science computationnelle est une science qui utilise l’ordinateur comme une machine à modéliser et à simuler au moyen de computations’, p.18 (In the most general sense, a computational science is a science which uses the computer as a machine for modelling and for simulating by means of computation’) (Varenne Franck, ‘Les simulations computationnelles dans les sciences sociales’, Nouvelles perspectives en sciences sociales, (5), 2010, pp. 17-49)
13 Stodden Victoria C, ‘Trust your science? Open your data and code’, Amstat News, 409, 2011, pp. 21-22.
14 Raynaud Claude, ‘Archéologie du paysage autour de l'Étang de l’Or (Hérault), choix, contraintes et méthode de prospection’, Archéologie en languedoc, 2(3), 1989, pp. 59-83.
15 Trément Frédéric, Archéologie d’un paysage, les étangs de Saint-Blaise (Bouche du Rhône), Maison des Sciences de l’Homme, Paris, 1999
16 Nouvel Pierre, ‘La prospection terrestre, Méthodes et objectifs’, ArchéoThéma, 2012, pp. 28-29.
17 Leeuw (van der) Sander Ernst, Favory François, Fiches Jean-Luc, Archéologie et systèmes socio-environnementaux: études multiscalaires sur la vallée du Rhône dans le programme Archaeomedes, 27, Paris, CNRS Editions, 2003
18 Bertoncello Frédérique, Fovet Élise, Tannier Cécile, Gandini Cristina, Lautier Laurence, Nouvel Pierre, Nuninger Laure, ‘Configurations spatiales et hiérarchiques du peuplement antique: des indicateurs quantitatifs pour une confrontation interrégionale’, in Variabilités, Environnementales, Mutations Sociales : Nature, intensités, échelles et temporalités des changements, ed. F. Bertoncello, F. Braemer, Antibes, France, APDCA, 2012, pp. 175-190.
19 Nuninger Laure, Bertoncello Frédérique, Favory François, Fiches Jean-Luc, Raynaud Claude, Girardot Jean-Jacques, Sanders Lena, Mathian Hélène, ‘Peuplement et territoire dans la longue durée : retour sur 25 ans d’expérience’, in L’archéologie en mouvement : hommes, objets et temporalités - 23-25 juin 2010, ed. S. Archambault De Beaune, H.-P. Francfort, Paris, CNRS éditions Alpha, 2012, pp. 152-159.
20 Natural or anthropogenic phenomena intervening following the initial deposit of an object or sediment and implying its decomposition, destruction or fossilisation.
21 Favory François, Nuninger Laure, Sanders Lena, ‘Intégration de concepts de géographie et d’archéologie spatiale pour l’étude des systèmes de peuplement’, L’Espace géographique, (4), 2012, pp. 295-309.
22 Varenne Franck, ‘Epistémologie des modèles et des simulations: tour d’horizon et tendances’, Physique et interrogations fondamentales, 11, 2008.
23 The concept of falsifiability (or refutability) was developed in the 1930s, particularly by Karl Popper (Popper 1959). It consists in declaring an affirmation ‘falsifiable’ if an observation or the results of an experiment permit the exact contradiction of that initial affirmation. Falsifiability makes it possible to distinguish scientific theories, and the theories most falsifiable are those which must be privileged. To attempt to falsify a set of hypotheses to produce an expected result in a simulation model remains, in numerous fields, more important for authors than an accumulation of positive results.
24 Chalmers Alan F, Qu’est-ce que la science ?, Paris, La Découverte, 1987
25 Among the geographers: Besse Jean-Marc, ‘Problèmes épistémologiques de l’explication’, colloque Géopoint, ‘L’explication en géographie’, Groupe Dupont, Université d’Avignon, 2000, pp. 11-18.
26 Passeron Jean-Claude, Le Raisonnement sociologique: Un espace non poppérien de l’argumentation, Paris, Albin Michel, 2006
27 Pumain Denise, ‘Cumulativité des connaissances’, Revue européenne des sciences sociales. European Journal of Social Sciences, XLIII(131), 2005, pp. 5-12.
28 Denise Pumain refers to this definition of Berthelot: ‘Par cumulativité, on entend classiquement la possibilité d’intégrer les résultats d’un grand nombre d’observations et d’expériences dans l’unité d’un modèle susceptible de les déduire’ (By cumulativity is classically meant the possibility of integrating the results of a great number of observations and experiments within the unity of a model capable of deducing them) (Berthelot 1996).
29 Zipf George Kingsley, Human behavior and the principle of least effort, Cambridge, England, Addinson-Wesley Press, 1949
30 Sanders Lena, ‘Regards scientifiques croisés sur la hiérarchie des systèmes de peuplement: de l’empirie aux systèmes complexes’, Région et Développement, (36), 2012, pp. 127-146.
31 Pumain Denise, ‘Cumulativité des connaissances...’, op.cit.
32 Ludwig Von Bertalanffy en 1949 ‘[...] qualifia d’“équifinalité”, [...] la capacité de l’embryon à atteindre un même état final de développement quelles que soient les conditions initiales et les modalités intermédiaires particulières de son développement’ ([…] termed “equifinality”, […] the capacity of the embryo to attain the same final state of development, whatever the initial conditions and the particular intermediary steps of its development) (Pouvreau, 2013).
33 O’Sullivan David, ‘Complexity science and human geography’, Transactions of the Institute of British Geographers, 29(3), 2004, pp. 282-295.
34 Premo Luke S., ‘Equifinality and explanation: the role of agent-based modeling in postpositivist archaeology’, Simulating Change: Archaeology into the Twenty-First Century. University of Utah Press, Salt Lake City, 2010, pp. 28–37.
35 Hachi Ryma, ‘Explorer l'effet de la morphologie des réseaux viaires sur leurs conditions d'accessibilité, une approche empirique fondée sur la théorie des graphes’, PhD thesis, Université Paris I - Panthéon-Sorbonne, 2020
36 Gravier Julie, ‘Deux mille ans d’une ville en système : Proposition d’une démarche appliquée au cas de Noyon ’, PhD thesis, Université Paris I - Panthéon-Sorbonne, 2018
37 Nahassia Lucie, ‘Formes spatiales et temporelles du changement urbain : analyser la localisation des activités à Tours sur 2 000 ans’, PhD thesis, Université Paris I - Panthéon-Sorbonne, 2019
38 We recall that ‘hystéréchronie’ designates the ‘modalité spatio-temporelle qui traduit les décalages, retards et temps de réponse qu’on observe dans la dynamique des formes, entre la constatation d’une “cause” et la manifestation de ses “effets”’ (the spatio-temporal modality which expresses the gaps, delays and response times that one observes in the dynamic of forms, between the establishment of a ‘cause’ and the manifestation of its ‘effects’) (Dictionnaire de l’Archéogéographie).
39 Gent Ian P, ‘The recomputation manifesto’, arXivpreprint, 2013. http://arxiv.org/pdf/1304.3674
40 The source code is the totality of the rules and functions which define the functioning of a computer programme.
41 By ‘container’ here is meant the computer description of these materials within a source code. For instance, in a simulation, the ‘container’ to verify is the source code which describes the different variants and scenarios. This source code thus reflects the thematic choices applied by the modellers..
42 Wilensky Uri, Rand William, ‘Making Models Match: Replicating an Agent-Based Model’, Journal of Artificial Societies and Social Simulation, 10(4), 2007, p. 2.
43 See Banos Arnaud, ‘Pour des pratiques de modélisation et de simulation libérées en Géographie et SHS’, Habilitation thesis, Université Paris 1 Panthéon-Sorbonne, 107 p, 2013, and Rey-Coyrehourcq Sébastien, ‘Une plateforme intégrée pour la construction et l’évaluation de modèles de simulation en géographie’, PhD thesis, Université Paris I - Panthéon-Sorbonne, 2015
44 Heath Brian, Hill Raymond, Ciarallo Frank, ‘A survey of agent-based modeling practices (January 1998 to July 2008)’, Journal of Artificial Societies and Social Simulation, 12(4), 2009, p. 9.
45 Angus Simon D, Hassani-Mahmooei Behrooz, ‘“Anarchy” Reigns: A Quantitative Analysis of Agent-Based Modelling Publication Practices in JASSS, 2001-2012’, Journal of Artificial Societies and Social Simulation, 18(4), 2015, p. 16.
46 Journal of Artificial Societies and Social Simulation.
47 Lerner Reuven M, Levy Sharona T, Wilensky Uri, Encouraging Collaborative Constructionism: Principles Behind the Modeling Commons, 2010
48 Grimm Volker, Revilla Eloy, Berger Uta, Jeltsch Florian, Mooij Wolf M, Railsback Steven F, Thulke Hans-Hermann, Weiner Jacob et al., ‘Pattern-oriented modeling of agent-based complex systems: lessons from ecology’, Science, 310(5750), 2005, pp. 987-991.
49 Bouquet Fabrice, Sheeren David, Becu Nicolas, Gaudou Benoit, Lang Christophe, Marilleau Nicolas, Monteil Claude, ‘Formalisme de description des modèles agent’, in Simulation spatiale à base d’agents avec NetLogo. Volume 1 : introduction et bases, ed. A. Banos, C. Lang, N. Marilleau, Londres, ISTE, 2015
50 Schmitt Clara, ‘Modélisation de la dynamique des systèmes de peuplement : de SimpopLocal à SimpopNet.’, PhD thesis, Université Paris I - Panthéon-Sorbonne, 2014
51 Thiele Jan C, Kurth Winfried, Grimm Volker, ‘RNETLOGO: an R package for running and exploring individual-based models implemented in NETLOGO’, Methods in Ecology and Evolution, 3(3), 2012, pp. 480-483.
52 Reuillon Romain, Leclaire Mathieu, Rey-Coyrehourcq Sébastien, ‘OpenMOLE, a workflow engine specifically tailored for the distributed exploration of simulation models’, Future Generation Computer Systems, 29(8), 2013, pp. 1981–1990.
53 In order to distribute the calculation over a large number of distant computation servers connected throughout Internet.
54 In computer science, chance is thus never completely random: it is based on a ‘random seed’, that is, a number which will permit the generation, in a predictable fashion, of the totality of random numbers subsequently called for. By preserving this ‘seed’, one can therefore regenerate the same random numbers, and thus identically replicate the execution of a simulation.
55 Young David A, ‘A New Space-Time Computer Simulation Method for Human Migration’, American Anthropologist, 104(1), 2002, pp. 138-158.
56 It should be noted that the model implemented by Young (M1_FS-Young) as described in this chapter also contains an implementation of the model of Fisher-Skellam. We will not comment here on the behaviour of this alternative model.
57 Young David A, ‘A New Space-Time Computer Simulation…’, op. cit.
58 We have made an Internet repository using the software repository platform GitHub.
59 NetLogoWeb (http://www.netlogoweb.org): this Internet site actually permits execution in a simple navigator of every NetLogo model accessible online.
60 In a recent navigator: http://www.netlogoweb.org/web?https://cdn.rawgit.com/TransMonDyn/YoungModelExploration/master/Model/M1_FS-Young.nlogo
61 At this address: https://github.com/TransMonDyn/YoungModelExploration/blob/master/Model/M1_FS-Young.nlogo
62 https://github.com/TransMonDyn/YoungModelExploration/tree/master/Model
63 For example, Figshare: https://figshare.com/
64 https://github.com/TransMonDyn/YoungModelExploration/tree/master/Analysis
65 This chart should be read as follows: class A is characterised by a probability of moving superior to 0.75, by a very low probability of death, by a probability of death by overpopulation especially situated between 0.10 et 0.5, and by a probability of birth superior to 0.5.
66 Pumain Denise, ‘Cumulativité des connaissances…’, op. cit.
Auteurs
UMR 6266, Univ. de Rouen
Géographie-cités UMR 8504, Univ. Paris 1 Panthéon-Sorbonne
Chrono-Environnement UMR 6249, MSHE C.N. Ledoux USR 3124, CNRS / Univ. Bourgogne Franche-Comté.
Géographie-cités UMR 8504, Univ. Paris 1 Panthéon-Sorbonne
Géographie-cités UMR 8504, Univ. Paris 1 Panthéon-Sorbonne
Géographie-cités UMR 8504, Univ. Paris 1 Panthéon-Sorbonne
Le texte seul est utilisable sous licence Licence OpenEdition Books. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.
Quatre ans de recherche urbaine 2001-2004. Volume 2
Action concertée incitative Ville. Ministère de la Recherche
Émilie Bajolet, Marie-Flore Mattéi et Jean-Marc Rennes (dir.)
2006
Quatre ans de recherche urbaine 2001-2004. Volume I
Action concertée incitative Ville. Ministère de la Recherche
Émilie Bajolet, Marie-Flore Mattéi et Jean-Marc Rennes (dir.)
2006