1 Introduction and Background

1Modality is a pervasive phenomenon crucial to language understanding, analysis, and automatic processing (Morante and Sporleder, 2012). The creation of modality-annotated data would benefit Natural Language Processing in at least two major aspects: (i) factuality detection, consisting in the automatic distinction between propositions that represent factual events and propositions that represent non factual ones; and (ii) sentiment analysis, which involve the processing of extrapropositional aspects of meaning and the detection of polarised judgements. Additionally, the annotation of modality may also have important repercussions in the field of corpus linguistics, as the techniques developed in the automatic treatment of modality can be used to improve our linguistic knowledge of modality itself.

2As far as the detection of polarised judgments goes, there have been substantial annotation efforts in recent years, exemplified by recurring and increasing sentiment analysis tasks within the context of the Semeval evaluation campaign.¹ Attention has also been given to more specific factuality tasks such as the CoNLL-2010 Shared Task on identifying hedges (Farkas et al., 2010), and factuality annotation in languages other than English, such as Italian (Minard et al., 2014), and Dutch (Schoen et al., 2014). However, these are annotation efforts involving specific phenomena rather than modality in general.

3Indeed, a major bottleneck in the creation of modality-annotated resources is the very notion of modality itself, as encapsulating this phenomenon in one exhaustive but workable definition is far from trivial (Morante and Sporleder, 2012). Building on the function-based proposal advanced in (Nissim et al., 2013) and (Ghia et al., 2016), we have created a comprehensive annotation scheme for epistemic modality and have applied it to multiple languages. Contextually, we have developed and deployed an evaluation strategy which shows that the corpus is annotated reliably.

4Summary of contributions We produced the first multilingual corpus annotated for modality. The annotation scheme is virtually language-independent, and the annotation is evaluated according to a specifically designed methodology which is portable to other tasks where annotators are left with substantial freedom in the selection of the tokens to be marked up. The corpus and the tools for scoring agreement are publicly available (http://modal.msh-vdl.fr/, https://bitbucket.org/lennyklb/modality/).

2 Corpus

5The MODAL Corpus is the first corpus of dialogues in multiple languages annotated for phenomena of (epistemic) modality.

6MODAL consists of three equivalent resources of English, French and Italian dialogues. These were drawn from the Santa Barbara Corpus of Spoken American English (Du Bois et al., 2000) for English, from the ESLO Corpus (Baude and Kanaan, 2014), plus the OTG Corpus and the Accueil UBS Corpus (Antoine et al., 2002) for French, and from the VoLip Corpus (Alfano et al., 2014) for Italian. All data is marked for epistemic modality and amounts to approximately 20.000 words per language for a total of 2824 epistemic constructions (833 for the English Corpus, 1271 for the French Corpus, 720 for the Italian Corpus).

2.1 Approach to annotation

7In the construction of MODAL, we were guided by two main principles: maximum expressivity, and cross-lingual validity. We therefore took an approach to annotation that would simultaneously ensure both.

8Specifically, we did not want to annotate a pre-determined list of epistemic constructions and assign functions to them. Indeed, this would make the scheme very much language-dependent, as specific tokens/constructions would need to be identified for each language. Additionally, it would restrict the annotation to this pre-selection, which could not be exhaustive.

9As an alternative approach, we provided a theoretical meaningful, and operationalisable definition of epistemic modality. On this ground, thus only at a later stage, the annotators identified the linguistic constructions that realise epistemic modality in the three different languages. Thus, rather than going from constructions to functions, we go from functions to constructions.

10While this approach has the advantage of being valid cross-linguistically, and maximising expressivity, it also potentially has a major problem. Letting the annotators choose freely the tokens and the constructions to be annotated without controlling for any pre-selection, incurs the risk of a wide range of choices, and substantially low agreement. We discuss this in the Evaluation section. In the remainder of this section we explain the scheme and the procedure we used to annotate the corpus.

2.2 Procedure and final scheme

11We employed a two-fold procedure: epistemic constructions are first identified, and then annotated with their features.

12Identification of epistemic constructions In order to annotate epistemic modality in dialogues, we subscribed to a communitarian (Stalnaker, 1978), dynamic (Groenendijk and Stokhof, 1991), and interactionist (Ginzburg, 2012) approach to semantics, which led us to refine the traditional definition of epistemic modality. Specifically, we put forward the idea that any construction that explicitly signals the process of shared attribution of a truth value to the propositional tokens that compose a discourse should be considered as an epistemic construction, and thus annotated.

13Consequently, we annotated not only constructions in which a marker is realized by a more grammaticalized element, such as a modal verb (Example 1), but also constructions in which a marker is realized lexically (Example 2) or prosodically (Example 3):

(1) A penguin might lay two eggs and at that point [. . . ]

(2) And I do believe it was thirty days [. . . ]

(3) DON: Oh specifically in the islands?

14Besides, we annotated both monological epistemic constructions in which a marker expresses the evaluation of the truth-value of a scope by a single speaker (Example 4), and dialogical epistemic constructions in which two or more markers are used to negotiate the evaluation of the truth-value of one and the same scope among the participants in a conversation (Example 5):

(4) apparently it was very very muddy it was abnormally warm and it was just a big mudbath out there [. . . ]

(5) ALIC: I don’t think Darren put anything on it .
NICO: Mhm .
ALIC: Right .
ALIC: Okay .

15Annotation of epistemic constructions We represented the epistemic constructions identified in the corpus as triadic constructions consisting of a marker, a scope and a relation between the marker and the scope, as shown in Figure 1.

16We formalised the marker, the scope and the relation between them as three elements each endowed with its own formal and functional properties. Each element is then annotated with syntactic, semantic, and pragmatic features according to the developed annotation scheme.

Figure 1: A construction is conceived as a marker, a scope, and a modal relation between them

17Building on (Nissim et al., 2013; Ghia et al., 2016), we devise a fully-fledged annotation scheme that is functionally motivated and cross-linguistically valid. Annotation features are specified for all three elements of the modalised construction, namely the marker, the scope, and the relation. The features for the markers and the relations are shown in Tables 1 and 2, respectively. For the scope, we use a property syntax and clause/utterance as features.

18Operationalising the annotation task From a theoretical perspective, the scheme is grounded in the Construction Grammar framework (Goldberg, 1995). In practice, the annotators could work with the labels from the annotation schemes, but also with decision trees that guided the process of identification of epistemic constructions as well as feature assignment. The annotation was performed using the Analec annotation tool (Landragin et al., 2012), which produces TEI-compliant XML output. Analec was originally designed for the annotation of anaphoric phenomena and thus lends itself well to the task of annotating a three-way construction, with features for marker, scope, and relation. All data was annotated by three teams of 2 or more annotators (a, b for Italian, a, b, c, d for English, a, e,f for French) and agreement was assessed via a specifically developed evaluation strategy (Section 3).²

3 Evaluation

19The originality of the general approach and of the annotation procedure led us to develop an original technique for testing the inter annotator agreement, essentially based on the percentage of overlap between the spans of text identified as markers or scopes by the annotators (Ghia et al., 2016). In order to assess this, annotations must be aligned. We describe how we align the constructions in practice, how we use alignment information in order to assess agreement, and discuss results.

Figure 2: Distribution of EPISTEMIC TYPES for the Relation annotation in the Italian portion of the corpus (see Table 2).

Figure 3: Distribution of DISCOURSE FUNCTIONS for the Relation annotation in the Italian portion of the corpus (see Table 2).

3.1 Alignment and Agreement

20Annotators can identify any textual element as part of a modalised construction, and each annotator works on their own file. This means that in order to assess agreement, we first need to try to align the constructions marked up in the two files. We do so via anchors. Anchors can be aligned iff:

• they are of the same type (marker or scope)

• they overlap in content by at least a given proportion of lexical material, which we base on character offset. For example, for a required overlap of 50% and a token length of an anchor A of ten tokens, the content of the candidate anchor from the other file needs to have at least five subsequent words in common with A.

21This process results in a collection of pairs of aligned anchors. For example, considering annotator a and annotator b, we would have an aligned pair of marker t_a and marker t_b.

22The final step is to iterate through the relations that judge a introduced and align them with relations that judge b introduced. In order to explain the procedure of further alignment to relations, we take judge a as reference, but in terms of scores it doesn’t make any difference which direction we go, since precision_ab = recall_ba so that eventually fscore_ab = fscore_ba. Relations consist of a marker and one or multiple scope portions. Aligning relations is done by pairing up markers and scopes into relations introduced by judge a and check if the aligned counterparts of these markers and scopes by judge b are part of a relation as well. When this is the case, we deem the two constructions as “the same”.

23Next, we have to assess agreement on the features assigned to relations and markers. While agreement over alignment is measured using precision/recall/f-score as we have to deal with potentially different spans, for the relations’ and markers’ features, we can then use Cohen’s Kappa (Cohen, 1960) over the agreed upon constructions only, as it becomes a plain classification task.

3.2 Results

24Because of freedom in the annotation of the extension of anchors, as mentioned above we evaluated alignment at different percentages of overlap. The scores for the alignment of scopes for all three languages is shown in Table 3. While for Italian we observe that even when evaluating alignment of full strings (i.e. requiring 100% overlap), the agreement stays high, this is not the case for French and English. Indeed, if complete overlap of scopes is required to deem the annotations equivalent, F-scores drop quite a bit. We do not include a table for the scores on the markers as they do not change substantially with varying degrees of overlap. This is due to the fact that markers are often just single words, or very short anyway. F-scores range from 0.91 at 10% and 0.90 at 100% for English, from 0.86 at 10% and 0.85 at 100% for French, and stay stable at 0.94 for Italian. For this reason, we can be lenient with markers’ alignment, which we set at 10% when evaluating relations. Interestingly, though, we can observe that when evaluating agreement on the features, Kappa increases when stricter alignment is required (Table 4). This is likely due to the fact that on fully agreed upon strings, the assigned features are also agreed upon.

25For the Italian annotation of the relation’s features, at overlap 100%, we observe K = 0.86 for the function feature, K = 0.82 for type, and K = 0.72 for polarity.³

26To provide a more detailed view into the disagreements of the type feature, for instance (whose final agreed upon distribution was reported in Figure 2 above), in Figure 4 we show the confusion matrix for Italian. We can observe that the largest number of confusions arise from mixing up the categories indirect_inferential and no_evidence. Indeed, the precise delimitation between these two categories is a long- and hot-debated issue in the literature on epistemic modality (Pietrandrea, 2005, among others).

27Overall, we can see that our annotation, albeit granting the annotators a lot of freedom, is substantially reliable.⁴

Figure 4: Confusion matrix for the annotation of the TYPE feature in Italian.

4 Conclusions

28Modality can be reliably annotated in multiple languages by taking a bottom-up, functional approach paired with a solid annotation scheme, trees to guide the annotators’ decisions, and a rigorous evaluation strategy. With this approach, we have produced the first multilingual corpus annotated for modality, which can be potentially used to train modality detection models as well as to further study modality itself. By making all of the data publicly available, and by sharing our annotation experience, we also hope to provide a blueprint for creating modality-annotated resources in yet more languages.

Acknowledgments

29The research described in this paper was supported by the Maison des Sciences de l’Homme du Val de Loire (grant MoDAL 2015), the IRCOM Consortium, the “Laboratoire Ligérien de Linguistique – UMR7270”, and the Groningen Meaning Bank. The authors are also grateful to the anonymous reviewers for their comments.