6. Operationalisation, Research Design, and Methodology
Texte intégral
1This chapter focuses on the methodological criteria guiding the empirical testing of the three hypotheses introduced earlier (H1, H2, H3). For each of them, we first account for the operationalisation of related theoretical concepts, accounting for their chosen indicators. Second, we move to discussing aspects of research design, making sense of our selection of cases and units of analysis. Third, we move to addressing questions related to inference, also acknowledging potential threats to the internal validity of empirical conclusions, and seeking to control for them. Fourth, we consider measurement, construct-validity, and reliability issues. The above four steps are replicated in sequence for H1, H2, and H3.
6.1 Hypothesis 1
2H1: If salience is manifested in relation to an FTA chapter when negotiations are ongoing (independent variable), political ratification obstacles (PROs) will be posed by veto players after signing (dependent variable).
Concept operationalisation
3With respect to the independent variable, we regard i) salience as the explicit (a) manifestation of concerns that the way an FTA chapter is expected to address a given issue-area is undesired or insufficient. Salience can also take the form of (b) open criticism of the expected terms of an agreement, resulting from the anticipation of a utility loss on the part of would-be veto players. (Thomson and Stokman 2006: 41-42) As for the dependent variable, we conceive ii) PROs as the (a) manifested intention to veto an FTA on the part of EU institutional actors (Council, EP, individual Member States) or as (b) an invitation to reject an FTA by actors lacking direct veto powers (NGOs, unions, business groups). PROs are also found in the (c) open criticism to an FTA after signing, even if the intention to resort to a veto is not explicitly voiced.1
Design
4The testing of H1 relies on a multiple-case parallel design, “in which several instrumental bounded cases are selected to develop a more in-depth understanding of the phenomena than a single case can provide” (Chmiliar 2012: 583). We run empirical observations over six different trade-policy domains. These have been isolated and selected, on the one hand, as issue-areas representative of the “regulatory turn” characteristic of new-generation FTAs, for they go beyond an exclusive tariff approach to market integration (Meunier and Czesana 2019). On the other hand, these are among the sectors the literature has identified as particularly contentious in post-Lisbon EU trade, and emerge as particularly politicised based on our preliminary exploration of recent FTAs (Duina 2019). Thus, their contentiousness makes them relevant for the purpose of H1. Examined cases qualify as “parallel” for they have been selected a priori, based on their expected theoretical pertinence (Chmiliar 2012). Examined policy domains are: i) Investment protection; ii) Trade and sustainable development (TSD); iii) Trade and labour; iv) Regulation and NTBs; v) Sanitary and phytosanitary (SPS) measures; vi) Geographical indications (GIs).2
5The units of analysis in each case are (would-be) veto players. At an official level, we considered the Council, the EP, and individual Member Governments. Private-sector veto players include advocacy NGOs, business organisations, social partners like trade unions, as well as the European Economic and Social Committee (EESC), all officially engaging with the Commissions in trade-policy auditing. Private veto players and social partners were purposely sampled via the EU Transparency Register, relying on research filters to ensure representative samples of stakeholders for each of the above six cases (European Commission 2020b).
6Our units of observation consist of statements, position papers, and reports issued by the above categories of actors. For the independent variable, one document (if available) from each institution or organisation in the sample was selected, and coded dichotomously as salience: 1/0 (i.e. “yes/no”) for each of the six cases examined.3 The same process was repeated under the dependent variable, coding as PROs: 1/0 (i.e. “yes/no”). We performed our coding through the software Nvivo, which allows for the systematic organisation and numerical analysis of textual material, by means of its matrix-query function. Numerical results from the qualitative coding of textual entries are displayed in Table I below.
Table I: Numerical results of the empirical testing of H1 (expressed in percentage). Salience (1/0) constitutes the independent variable; PROs (1/0) the dependent variable. Cases in which salience was associated with PROs are highlighted in red. Control cases are highlighted in green.
Issue-Area | (IV) Salience: 1 (%) | (IV) Salience: 0 (%) | (DV) PROs: 1 (%) | (DV) PROs: 0 (%) |
Investment Protection | 65 | 35 | 78 | 22 |
TSD | 80 | 20 | 62 | 38 |
Trade and Labour | 75 | 25 | 80 | 20 |
Regulation and NTBs | 25 | 75 | 28 | 72 |
SPS Measures | 75 | 25 | 100 | 0 |
GIs | 0 | 100 | 0 | 100 |
Inference
7In testing H1, we control for the possibility of a spurious relationship on three grounds. First, time order between cause and effect is ensured. Documents under the independent variable pre-date those coded to the dependent variable (Buttolph Johnson et al. 2016: 170). Second, the criterion of covariation is met by two of the examined cases (Regulation and GIs) (169). Covariation allows us to observe that not only is the presence of salience associated with PROs, but also its absence or low occurrence is associated with no or low PROs. Third, we seek to address threats to internal validity. In this respect, we acknowledge how, in principle, nonprobability purposeful sampling implies selection bias. Yet, when it comes to specialised interest groups engaging in EU trade dialogues on given issue-areas, the population is small enough to be examined directly with adequate accuracy, without relying on probabilistic generalisations (Trochim and Donnelly 2008: 48-49). For instance, in testing the emergence of PROs related to trade and labour, we sampled all labour-related organisations officially accredited to consult with the Commission (largely unions). Thus, for the sake of sampling, we are not interested in whether labour issues are salient to EU stakeholders in general, but rather whether stakeholders working on each examined sector are observed to have posed PROs. Finally, we acknowledge that the inclusion of six cases for empirical testing qualifies as a small-N design. Yet, an in-depth approach to their analysis allows us to improve the testing of related inferences. As Trochim (1985) observes, the in-depth analysis of a case, and variables within it, can help control for threats to internal validity. Specifically, in-depth case studies run a fine-grained analysis of the causal mechanisms of interest, allowing for “pattern specificity” making possible alternative explanations less likely (580).
Construct validity and reliability
8The consistency and systematicity of textual observations under H1 are improved by relying on “concept mapping,” which allows for the “translation of an idea or construct into something real and concrete” (Trochim and Donnelly 2008: 56). This seeks to ensure construct validity at three main levels. First, as discussed above, operative definitions of the theoretical concepts in H1 have been narrowly devised, looking at previous theories in political science that focus on issues of salience and propose specific empirical proxies for this notion (Thomson and Stokman 2006). Second, a number of specific textual indicators have been attached to each operationalised concept, in order to account for the type of entries included in our coding. Third, the replicability of our measurements has been ensured by handing the textual material to a second coder, who re-coded the entire dataset. The resulting inter-coder reliability (ICR) was calculated by looking at the percentage of coding entries on which the author and the second coder agreed (Feng 2014), and amounts to 85%.
6.2 Hypothesis 2
9H2(a): If official-level veto players mobilise against the terms of an FTA following negotiations (independent variable), the Commission will resort to solution-chasing (dependent variable).
10H2(b): If private/advocacy veto players mobilise against the terms of an FTA following negotiations (independent variable), the Commission will resort to solution-chasing (dependent variable).
Concept operationalisation
11In both H2(a) and H2(b), i) veto-player mobilisation encapsulates the notion of PROs, conceived as the (a) manifested intention to veto an FTA on the part of EU institutional actors (Council, EP, individual Member States); (b) an invitation to reject an agreement by private/advocacy veto players lacking direct ratification powers (NGOs, unions, business organisations); (c) open criticism to an FTA ahead of ratification, even if the intention to resort to a veto is not explicitly voiced. Further, mobilisation can also refer to (d) prolonged contestation of the terms of an FTA in the aftermath of ratification. The dependent variable is kept constant across both H2(a) and H2(b). The concept of ii) solution-chasing is operationalised both as the (a) adoption of additional safeguards or documents complementing an FTA; and as the (b) setting up of institutional fora in charge of carrying out monitoring and review ex post (e.g. DAGs). Both types of instruments are usually implemented in the context of specific FTA chapters.
Design
12The testing of H2 relies on the qualitative comparative analysis (QCA) of post-negotiation debates on specific FTAs. QCA designs look at “configurations of causal factors understood as sufficient to produce an outcome” and display whether the occurrence/absence of one or more of these factors impacts on the realisation of the dependent variable (Gerring 2012: 343; see also Ragin 1987). We focus on the six issue-areas previously examined in H1, this time making reference to a specific FTA in each of them.4 In running our QCA, we define two causal variables expected to interact in determining whether the Commission triggers solution-chasing or not: the i) mobilisation of official-level veto players after signing; the ii) mobilisation of private/advocacy veto players after signing. Each variable is coded dichotomously: 1/0 (i.e. “yes/no”). In building our dataset, we relied on multiple data collection, looking at primary documents, news sources, and personal semi-structured interviews with both Commission officials and non-governmental stakeholders (Buttolph Johnson et al. 2016: 196). Our design is displayed in Table II below.
Table II. Table II displays coding results related to the effect of the mobilisation of veto players (independent variables of H2) on the expected outcome of solution-chasing (dependent variable of H2). In the case of TSD within the EU-Mercosur FTA, we coded solution-chasing as a question mark since the agreement was only concluded in June 2019, and still awaits going through ratification. In the case of “Trade and Labour” (KOREU) the variables mobilisation of official-level veto players and solution-chasing were coded both as 0 and 1 since their change is observable across time. Specifically, when the former variable was absent (0) so was the latter. Likewise, when the former was present (1) so was the latter.
Cases | Independent Variables | Dependent Variable | |
Mobilisation of official-level veto players (H2a) | Mobilisation of private/advocacy veto players (H2b) | Solution-chasing (H2) | |
Investment Protection (CETA) | 1 | 1 | 1 |
TSD (CETA) | 1 | 1 | 1 |
TSD (EU-Mercosur) | 1 | 1 | ? |
Trade and Labour (KOREU) | 0/1 | 1 | 0/1 |
Regulation & NTBs (KOREU) | 1 | 1 | 1 |
SPS Measures (CETA) | 0 | 1 | 0 |
Geographical Indications (CETA) | 0 | 0 | 0 |
13The employment of QCA is motivated by two main analytical and methodological objectives. First, in studying the phenomenon of solution-chasing, we go beyond seeking to establish a causal relationship between a single variable X and Y, rather accounting for the “multiple configurations of factors” concurring with the causal mechanism of interest (Gerring 2012: 333). To that end, QCA can be conceived as a methodological attempt to reconcile the rationale of multivariate analysis with the qualitative observation of small samples of cases. Second, QCA is especially compatible with the comparative approach between our six selected cases. Specifically, assessing which independent variables occur (and do not occur) when solution-chasing is verified can help draw conclusions as to what actors and conditions can effectively push the Commission to revise the terms of a negotiated FTA – and which ones cannot.
Inference
14QCA still shows its limitations when it comes to validating causal inferences. First, QCA lacks random selection, as data are chosen in light of their high representativeness of the causal phenomenon of interest. Second, our QCA relies on a restricted dataset. In general, that implies a small-N problem, by which the number of observations would be insufficient for standard multivariate analysis to statistically corroborate an inference. Yet, QCA can still effectively address small-N issues, by running the maximum number of comparisons between variables across selected cases (Ragin 1987). This implies that, while not testing the impact of multiple variables over one outcome across a large-N, QCA looks at what combinations of these variables account for the outcome of interest. That is crucial to our research, allowing us to refine our understanding of the drives behind solution-chasing. Furthermore, QCA does a better job at controlling for possible alternative explanations than a simple (and potentially spurious) association between X and Y (Trochim 1985: 580). In this light, although a small-N problem remains, the in-depth exploration of patterns of solution-chasing can still prove a helpful starting point for future more extensive and systematic research.
Construct validity and reliability
15With respect to the independent variables, measurements of the mobilisation of official-level and private/advocacy veto players are guided by the documentary dataset relied upon for H1. In using the same dataset, the testing of H2 indirectly benefits from the inter-coder reliability (amounting to 85%) of measurements under H1. This also ensures the consistency of observations across the two hypotheses. Furthermore, the inclusion of additional instances of veto-player mobilisation following FTA ratification (and hence not covered by H1) is grounded on information gathered through semi-structured interviews with both Commission DG Trade officials and EU civil-society advocates. As for the dependent variable, the measurement of solution-chasing is aided by the fact that this concept of interest, and related indicators, consist of policy processes, and of the (non-) occurrence of a number of verifiable facts (e.g. implementation of safeguards, setting-up of advisory groups). This poses fewer measurement issues than in the presence of verbal statements potentially subject to multiple and inconsistent interpretations (cf. H1).
6.3 Hypothesis 3
16H3(a): If a mixed FTA is provisionally enforced by unanimity (independent variable), lock-in dynamics will prevent national parliaments from formally rejecting the agreement (dependent variable).
17H3(b): If the Commission involves private veto players in the provisional implementation of a mixed FTA (independent variable), lock-in dynamics will disincentivise these actors from lobbying for the rejection of the agreement (dependent variable).
Concept operationalisation
18H3(a): With respect to the independent variable, i) unanimous enforcement refers to the unanimous approval by the Council of the provisional enforcement of a mixed FTA. As for the dependent variable, ii) lock-in dynamics are conceived as involving national political majorities which, albeit critical of a provisionally enforced mixed FTA, nevertheless abstain from casting a veto as part of Member-State ratification.
19H3(b): Under the independent variable, i) involvement in provisional implementation is conceived as the inclusion of private/advocacy veto players by the Commission into institutional implementation structures tasked with the monitoring and review of specific FTA chapters being provisionally implemented (e.g. DAGs). As for the dependent variable, ii) lock-in dynamics show multiple empirical configurations. They can take the form of (a) institutional incentives and (b) epistemic incentives binding private veto players to continue participating in these implementation structures. Institutional incentives can be found in how close and compatible the positions of partaking advocates are, which in turn points at how conductive the institution they belong to is towards advocacy coalition-building (Sabatier and Weible 2007). Epistemic incentives are seen as indirectly proportional to the number of trade advocates monitoring FTA implementation on behalf of each member organisation in the DAG, and hence on their degree of dependency on information-sharing with fellow coalition members. The fewer the personnel working on trade in each partaking organisation, the higher its epistemic incentives to remain part of a given implementation structure ceteris paribus. Finally, lock-ins can consist of (c) resource incentives, both in terms of funding and personnel, making implementation structures set up by the Commission cost-efficient in the eyes of partaking veto players – the latter facing limited financial and human resources.5
Design
20The design of H3(a) and H3(b) is grounded on an in-depth analysis of the single case study of CETA’s provisional enforcement. We deem the CETA as theoretically relevant for exploring policy dynamics affecting mixed FTAs. First, the CETA constitutes the only instance of a comprehensive mixed agreement provisionally enforced by the EU and yet to complete ratification by Member States. This FTA went to ratification ahead of ECJ Opinion 2/15, qualifying as shared competence investment-protection and portfolio-investment chapters, which ever since the EU-Singapore FTA have been negotiated separately from main texts (European Court of Justice 2017). Nearly three years into CETA provisional enforcement are hardly sufficient to draw final conclusions as to the lock-in dynamics affecting the implementation of the FTA. Yet, a preliminary assessment can be made, based on existing empirics, as to the dynamics deemed conducive to longer-term path dependency (Pierson 2000). Second, the CETA is particularly representative of the political contestation and ratification obstacles examined by this research (see H1). To a greater extent than other contemporary FTAs, the CETA has been at the centre of intense political debates challenging its ratification, often as a result of spill-overs from the TTIP (Duina 2019).
21H3(a): In accounting for lock-in dynamics in H3(a), we rely on a crucial-case least-likely design, running observations over one unit of analysis a priori unexpected to mirror the theoretical mechanism of interest. If a least-likely case turns out to comport with a theory, it can serve as a relevant empirical validation – to a greater extent than a priori neutral or likely cases (Gerring 2012: 234; Jordan 1999). In light of this, we select the case of Italy as the only country having experienced a significant parliamentary-majority shift since the CETA was provisionally enforced by the Council (European Parliament 2019b). Most crucially, the incumbent parliamentary majority has proven highly critical of the CETA, unlike the previous leadership which gave its consent to provisional enforcement. Thus, Italy is a priori least likely not to reject the CETA as part of national ratification.
22H3(b): To examine lock-in dynamics in H3(b), we selected as our units of analysis private interest and advocacy groups sitting in the CETA DAG. H3(b) is therefore tested through a single-case small-N design. The narrow operationalisation of concepts under H3 has been carried out in view of accounting for multiple “theoretically defined classes of events” within the same case of interest (Levy 2008: 2). Specifically, the distinction between (a) institutional incentives; (b) epistemic incentives; and (c) resource incentives allows us to test the occurrence of the dependent variable through narrowly defined proxies pointing at the outcome of interest.
Inference
23H3(a): As observed by Eckstein (1975), crucial-case least-likely designs are particularly suitable for theory-testing, providing that theoretical predictions are defined carefully. The internal validity of H3(a) benefits from the narrow definition of the theoretical mechanism linking unanimous enforcement (independent variable) to national-level lock-ins (dependent variable). Specifically, unanimity, while constituting the independent variable, also acts as a scope condition narrowing down the focus of H3(a) to unanimous Council decisions on trade. Further, the criterion of FTA “mixedness” also explicitly restricts the population of relevant agreements. Conditions for the selection of the unit of analysis (i.e. individual Member States) are also narrowly defined. In this regard, the case of Italy is selected based on two specific criteria. First, the country is yet to initiate its process of ratification of the CETA. Second, the incumbent parliamentary majority in Rome, at the time of writing, differs from the preceding political leadership which agreed on provisional enforcement in the Council. Thus, the narrow conditions defined above can improve the internal validity of inferences under H3(a), in the face of small-N problems in our design. As Gerring (2007) observes, “[t]he more a theory attains the status of a causal law, the easier it will be to confirm, or disconfirm, with a single case” (117).
24H3(b): Analogously to H3(a), the testing of H3(b) employs a single-case design, exploring lock-in dynamics involving private/advocacy veto players sitting in the CETA DAG. In order to improve the internal validity of our inferences, we tackle the small-N problem faced by this design by running as many empirical observations as possible over the case of the CETA DAG. This is particularly important in single-case designs, where observations across multiple cases cannot be performed, and therefore need to be maximised within a single one. To that end, relying on three different indicators of the concept of lock-in (dependent variable) helps in refining the causal mechanism of interest from multiple perspectives. In that regard, simply looking at veto players’ nominal membership in the DAG would per se tell nothing about the extent to which these actors are dependent on (or locked into) CETA implementation structures. Conversely, examining how lock-ins can take the form of i) institutional incentives, ii) epistemic incentives, and ii) resource incentives offers a more accurate exploration of why veto players are bound to prolongedly contribute to a trade regime they still regard as suboptimal.
Construct validity and reliability
25As regards construct validity, both the testing of H3(a) and H3(b) and their single-case approach are particularly suitable for ensuring measurement accuracy. Specifically, statistical limitations related to small-N problems can be compensated for by closely examining dynamics within a single case. In-depth analysis, in turn, requires careful and detailed operationalisation of theoretical concepts. To that end, indicators for H3 ensure construct validity to the extent that their link to their respective concepts has been theory-driven. For instance, in H3(b), we measure institutional incentives by looking at mandate proximity between DAG advocates. This is dictated by advocacy-coalition theory emphasis on how homogeneous positions among advocates incentivise them to act collectively, allowing them to obviate coordination problems (Sabatier and Weible 2007). Finally, the fact that the concept of lock-in is operationalised through multiple and specific indicators strengthens reliability, allowing single empirical measurements to be less prone to misinterpretations and more easily replicable.
Notes de bas de page
1 Indicators employed for the coding under H1 are exemplified in Annex I.
2 We selected relevant data produced in the context of the KOREU (2010), TTIP (negotiations interrupted in 2016), CETA (2016), EU-Japan EPA (2018), EU-Singapore FTA (2019), EU-Mercosur FTA (2019), EU-Vietnam FTA (2019), and EU-Australia FTA (negotiations ongoing since 2018). Dates refer to the year of signing.
3 Only one document per institution or organisation (if available) was coded under the independent and dependent variable, in order not to inflate the number of textual entries.
4 See Table II. For TSD, empirics were gathered both in regard to the CETA and the EU-Mercosur FTA.
5 See Annex II for the complete dataset pertaining to institutional, epistemic, and resource incentives, relied upon in the examined case study of the CETA DAG (chapter VIII).

Le texte seul est utilisable sous licence Creative Commons - Attribution - Pas d'Utilisation Commerciale - Pas de Modification 4.0 International - CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.
The Development of International Refugee Protection through the Practice of the UN Security Council
Christiane Ahlborn
2010
The SWIFT Affair
Swiss Banking Secrecy and the Fight against Terrorist Financing
Johannes Köppel
2011
The Evolving Patterns of Lebanese Politics in Post-Syria Lebanon
The Perceptions of Hizballah among Members of the Free Patriotic Movement
Fouad Ilias
2010
La justice internationale à l'épreuve du terrorisme
Défis, enjeux et perspectives concernant la Commission d'enquête internationale indépendante (UNIIIC) et le Tribunal spécial pour le Liban
Sébastien Moretti
2009
Aut Dedere, aut Judicare: The Extradite or Prosecute Clause in International Law
Claire Mitchell
2009