1 Introduction and Motivation

1Reasoning about events plays a fundamental role in text understanding. It involves many aspects such as the identification and classification of events, the identification of event participants, the anchoring and ordering of events in time, and their factuality profiling.

2In the context of the 2016 EVALITA evaluation campaign, we organized FactA (Event Factuality Annotation), the first evaluation exercise for factuality profiling of events in Italian. The task is a follow-up of Minard et al. (2015) presented in the track ”Towards EVALITA 2016” at CLiC-it 2015. Factuality profiling is an important component for the interpretation of the events in discourse. Different inferences can be made from events which have not happened (or whose happening is probable) than from those which are described as factual. Many NLP applications such as Question Answering, Summarization, and Textual Entailment, among others, can benefit from the availability of this type of information.

3Factuality emerges through the interaction of linguistic markers and constructions and its annotation represents a challenging task. The notion of factuality is strictly related to other research areas throughly explored in NLP, such as subjectivity, belief, hedging and modality (Wiebe et al., 2004; Prabhakaran et al., 2010; Medlock and Briscoe, 2007; Saurí et al., 2006). In this work, we adopted a notion of factuality which corresponds to the committed belief expressed by relevant sources towards the status of an event (Saurí and Pustejovsky, 2012). In particular, the factuality profile of events is expressed by the intersections of two axes: i.) certainty, which expresses a continuum which ranges from absolutely certain to uncertain; and ii.) polarity, which defines a binary distinction: affirmed (or positive) vs. negated (or negative).

4In recent years, factuality profiling has been the focus of several evaluation exercises and shared tasks, especially for English, both in the newswire domain and in the biomedical domain. To mention the most relevant:

• the BioNLP 2009 Task 3¹ and BioNLP 2011 Shared² Task aimed at recognizing if biomolecular events were affected by speculation or negation;

• the CoNLL 2010 Share Task³ focused on hedge detection, i.e. identify speculated events, in biomedical texts;

• the ACE Event Detection and Recognition tasks⁴ required systems to distinguish between asserted and non-asserted (e.g. hypothetical, desired, and promised) extracted events in news articles;

• the 2012 *SEM Shared Task on Resolving The Scope of Negation⁵ focused one of its substasks on the identification of negated, i.e. counterfactual, events;

• the Event Nugget Detection task at TAC KBP 2015 Event Track⁶ aimed at assessing the performance of systems in identifying events and their factual, or realis, value in news (Mitamura et al., 2015);

• the 2015⁷ and 2016⁸ SemEval Clinical TempEval tasks required systems to assign the factuality value (i.e. attributed modality and polarity) to the extracted events in clinical notes.

5Finally recent work, such as the Richer Event Description annotation initiative⁹, has extended the annotation of factuality on temporal relations between pairs of events or pairs of events and temporal expressions as a specific task, independent from the factuality of the events involved, to represent claims about the certainty of the temporal relations themselves.

6FactA provides the research community with new benchmark datasets and an evaluation environment to assess system performance concerning the assignment of factuality values to events. The evaluation is structured in two tasks: a Main Task, which focuses on the factuality profile of events in the newswire domain, and a Pilot Task, which addresses the factuality profiling of events expressed in tweets. To better evaluate system performance on factuality profiling and avoid the impact of errors from related subtasks, such as event identification, we restricted the task to the assignment of factuality values. Although as many as 13 teams registered for the task, none of those teams actually submitted any output. Nevertheless, we were able to run an evaluation following the evaluation campaign conditions for one system which was developed by one of the organizers, FactPro.

7The remainder of the paper is organized as follows: the evaluation exercise is described in detail in Section 2, while the datasets are presented in Section 3. In Section 4 we describe the evaluation methodology and in Section 5 the results obtained with the FactPro system are illustrated. We conclude the paper in Section 6 with a discussion about the task and future work.

2 Task Description

8Following Tonelli et al. (2014) and Minard et al. (2014), in FactA we represent factuality by means of three attributes associated with events¹⁰, namely certainty, time, and polarity. The FactA task consisted of taking as input a text in which the textual extent of events is given (i.e. gold standard data) and assign to the events the correct values for the three factuality attributes¹¹ according to the relevant source. In FactA, the relevant source is either the utterer (in direct speech, indirect speech or reported speech) or the author of the news (in all other cases). Systems do not have to provide the overall factuality value (FV): this is computed automatically on the basis of the certainty, time, and polarity attributes (see Section 2.2 for details).

2.1 Factuality Attributes

9Certainty. Certainty relates to how sure the relevant source is about the mentioned event and admits the following three values: certain (e.g. ‘rassegnato’ in [1]), non certain (e.g. ‘usciti’ in [2]), and underspecified (e.g. ‘spiegazioni’ in [3]).

1. Smith ha rassegnato ieri le dimissioni; nomineranno il suo successore entro un mese. (“Smith resigned yesterday; they will appoint his replacement within a month.”)

2. Probabilmente i ragazzi sono usciti di casa tra le 20 e le 21. (“The guys went probably out between 8 and 9 p.m.”)

3. L’Unione Europea ha chiesto “spiegazioni” sulla strage di Beslan. (“The European Union has asked for an explanation about the massacre of Beslan.”)

10Time. Time specifies the time when an event is reported to have taken place or is going to take place. Its values are past/present (for non-future events, e.g. ‘capito’ in [4]), future (for events that will take place, e.g. ‘lottare’ in [4]) or ‘nomineranno’ in [1]), and underspecified (e.g. ‘verifica’ in [5]).

4. I russi hanno capito che devono lottare insieme. (“Russians have understood that they must fight together.”)

5. Su 542 aziende si hanno i dati definitivi mentre per le altre 38 si e` tuttora in fase di verifica. (“They have the final data for 542 companies while for the other 38 it is under validation.”)

11Polarity. Polarity captures whether an event is affirmed or negated and, consequently, it can be either positive (e.g. ‘rassegnato’ in [1]) or negative (e.g. ‘nominato’ in [6]); when there is not enough information available to detect the polarity of an event mention, its value is underspecified (e.g. ‘scompone’ in [7]).

6. Non ha nominato un amministratore delegato. (“He did not appoint a CEO.”)

7. Se si scompone il dato sul nero, si vede che il 23% e` dovuto a lavoratori residenti in provincia. (“If we analyze the data about the black market labor, we can see that 23% is due to workers resident in the province.”)

12Event mentions in texts can be used to refer to events that do not correlate with a real situation in the world (e.g. ‘parlare’ in [8]). For these event mentions, participant systems are required to leave the value of all three attributes empty.

8. Guardate, penso che sia prematuro parlare del nuovo preside. (“Well, I think it is too early to talk about the new dean.”)

2.2 Factuality Value

13The combination of the certainty, time, and polarity attributes described above determines the factuality value (FV) of an event with respect to the relevant source.

14As shown in Table 1, the FV can assume five values: i.) factual; ii.) counterfactual; iii.) non-factual; iv.) underspecified; and v.) no factuality (no fact). We illustrate in Table 1 the full set of valid combinations of the attribute values and the corresponding FV.

15A factual value is assigned if an event has the following configuration of attributes:

• certainty: certain

• time: past/present

• polarity: positive

16For instance, the event ‘rassegnato’ [resigned] in [1] will qualify as a factual event. On the other hand, a change in the polarity attribute, i.e. negative, will give rise to a counterfactual FV, like for instance the event ‘nominato’ [appointed] in [6].

17Non-factual events depend on the values of the certainty and time attributes. In particular, a non-factual value is assigned if either of the two cases below occur, namely:

• certainty: non certain; or

• time: future

18This is the case for the event ‘lottare’ [fight] in [4], where time is future, or the event ‘usciti’ [went out] in [2] where certainty is non certain.

19The event FV is underspecified if at least one between certainty and time is underspecified, independently of the polarity value, like for instance in the case of ‘verifica’ [validation] in [5].

20Finally, if the three attributes have no value, FV is no factuality (e.g. ‘parlare’ [discuss] in [8]).

3 Dataset Description

21We made available an updated version of Fact-Ita Bank (Minard et al., 2014) as training data to participants. This consists of 169 documents selected from the Ita-TimeBank (Caselli et al., 2011) and first released for the EVENTI task at EVALITA 2014¹². Fact-Ita Bank contains annotations for 6,958 events (see Table 2 for more details) and is distributed with a CC-BY-NC license¹³.

22As test data for the Main Task we selected the Italian section of the NewsReader MEANTIME corpus (Minard et al., 2016), a corpus of 120 Wikinews articles annotated at multiple levels. The Italian section is called WItaC, the NewsReader Wikinews Italian Corpus (Speranza and Minard, 2015) , and consists of 15,676 tokens (see Table 2). As test data for the Pilot Task we annotated 301 tweets with event factuality, representing a subsection of the test set of the EVALITA 2016 SEN-

23TIPOLC task (Barbieri et al., 2016) (see Table 2). Training and test data, both for the Main and the Pilot Tasks, are in the CAT (Content Annotation Tool) (Bartalesi Lenzi et al., 2012) labelled format. This is an XML-based stand-off format where different annotation layers are stored in separate document sections and are related to each other and to source data through pointers.

4 Evaluation

24Participation in the task consisted of providing only the values for the three factuality attributes (certainty, time, polarity), while the FV score was to be computed through the FactA scorer on top of these values.

25The evaluation is based on the micro-average F1 score of the FVs, which is equivalent to the accuracy in this task as all events should receive a FV (i.e. the total numbers of False Positives and False Negatives over the classes are equal). In addition to this, an evaluation of the performance of the systems on the single attributes (using microaverage F1 score, equivalent to the accuracy) will be provided as well. We consider this type of evaluation to be more informative than the one based on the single FV because it will provide evidence of systems’ ability to identify the motivations for the assignment of a certain factuality value. To clarify this point, consider the case of an event with FV non-factual (certainty non certain, time past/present and polarity positive). A system might correctly identify that the FV of the event is non-factual because certainty is non certain, or erroneously identify that time is future.

5 System Results

26Unfortunately no participants took part in the FactA task. However, we managed to run an evaluation test with a system for event factuality annotation in Italian, FactPro, developed by one of the organizers and respecting the evaluation campaign conditions. The system was evaluated against both gold standard, i.e. the Main and Pilot tasks. In this section we describe this system and the results obtained on the FactA task.

5.1 FactPro module

27FactPro is a module of the TextPro NLP pipeline¹⁵ (Pianta et al., 2008). It has been developed by Anne-Lyse Minard in collaboration with Federico Nanni as part of an internship.

28Event Factuality annotation is performed in FactPro in three steps: (1) detection of the polarity of an event, (2) identification of the certainty of an event and (3) identification of the semantic time. These three steps are based on a machine learning approach, using Support Vector Machines algorithm, and are taken as text chunking tasks in which events have to be classified in different classes. For each step a multi-class classification model is built using the text chunker Yamcha (Kudo and Matsumoto, 2003).

29FactPro requires the following pre-processes: sentence splitting, tokenization, morphological analysis, lemmatization, PoS tagging, chunking, and event detection and classification. As the data provided for FactA consist of texts already split into sentences, tokenized and annotated with events, the steps of sentence splitting, tokenization and event detection and classification are not performed for these experiments.

30Each classifier makes use of different features: lexical, syntactic and semantic. They are described in the remainder of the section. For the detection of polarity and certainty, FactPro makes use of trigger lists which have been built manually using the training corpus.

31Each token is represented using these features as well as some of the features of the previous tokens and of the following ones. We have defined the set of features used by each classifier performing several evaluations on a subsection of the Fact-Ita Bank corpus.

5.2 Results

32Table 3 shows the results of FactPro for the two tasks of FactA against a baseline. The baseline system annotates all events as factual (the predominant class), i.e. being certain, positive and past/present. The performance of FactPro on the Main Task is 0.72 when evaluating the Factuality Value assignment and 0.69 on the combination of the three attributes, and on the Pilot Task 0.56 and 0.51 respectively. On these two tasks FactPro performs better than the baseline. It has to be noted that we ran FactPro on the pilot test set without any adaptation of the different processes.

33In Table 4 we present the F1-score obtained for each value of the three attributes as well for each Factuality Value. We can observe that FactPro does not perform well on the identification of the underspecified values and on the detection of events that do not have a factuality value (no fact).

5.3 Error Analysis of FactPro

34We can observe from Table 3 that FactPro performs better for the detection of polarity and certainty than for the identification of time. One reason is the predominance of one value for the polarity and certainty attributes, and of two values for time. For example, in the training corpus, 94% of the events have a polarity positive and 86% are certain, whereas 71% of the events are past/present and 22% are future.

35An extensive error analysis on the output of the systems for the three attributes was conducted. As for the polarity attribute, the error analysis showed that the system’s failure to detect negated events is not mainly due to a sparseness of negated events in the training data, but it mainly concerns the negation scope, whereas when the system missed a negative event it was mainly due to the incompleteness of the trigger lists (e.g. mancata in dopo la mancata approvazione is a good trigger for polarity negative but it is absent from the trigger list).

36The detection of no certain events works well when the event is preceded by a verb at the conditional and when it is part of an infinitive clause introduced by per. However when the uncertainty of an event is expressed by the semantics of previous words (e.g. salvataggio in il piano di salvataggio) the system makes errors.

37With respect to the annotation of future events, the observations are similar to those for non certain events. Indeed, future events are well recognized by the system when they are part of an infinitive clause introduced by the preposition per as well as when their tense is future.

38Finally, we observed that FactPro makes a lot of errors when the annotation of the factuality of nominal events is concerned. In the Main Task it correctly identified the FV of 81% of the verbal events and only 61% of the nominal events.

6 Conclusion and Future Work

39The lack of participants in the task limits the discussion of the results to the in-house developed system. The main reason for the lack of participation to FactA, according to the outcome of a questionnaire organized by the 2016 EVALITA chairs, was that the participants gave priority to other EVALITA tasks. However, FactA achieves two main results: i.) setting state-of-the-art results for the factuality profiling of events in two text types in Italian, namely news articles and tweets; and ii.) making available to the community a new benchmark corpus and standardized evaluation environment for comparing systems’ performance and facilitating replicability of results.

40The test data used for the Main Task consists of the Italian section of the MEANTIME corpus (Minard et al., 2016). MEANTIME contains the same documents aligned in English, Italian, Spanish and Dutch, thus making available a multilingual environment for cross-language evaluation of the factuality profiling of events. Furthermore, within the NewsReader project, a module for event factuality annotation has been implemented and evaluated against the English section of the MEANTIME corpus (Agerri et al., 2015). The evaluation was performed in a different way than in FactA, in particular no gold events were provided as input to the system, so the evaluation of factuality was done only for the events correctly identified by the event detection module. The system obtained an accuracy of 0.88, 0.86 and 0.59 for polarity, certainty, and time, respectively.

41The Pilot task was aimed at evaluating how well systems built for standard language perform on social media texts, and at making available a set of tweets annotated with event mentions (following TimeML definition of events) and their factuality value. The pilot data are shared between three other tasks of EVALITA 2016 (PoSTWITA, NEEL-IT

42and SENTIPOLC), which contributed to the creation of a richly annotated corpus of tweets to be used for future cross-fertilization tasks. Finally, the annotation of tweets raised new issues for factuality annotation because tweets contain a lot of imperatives and interrogatives that are generally absent from news and for which the factuality status is not obvious (e.g. Ordini solo quello che ti serve).

43The results obtained by FactPro, as reported in Table 3 and Table 4, show that i.) the system is able to predict with pretty high accuracy the FV on events in the news domain and with a lower but good score the factuality of events in tweets; ii.) the difference in performance between the news and tweet text types suggest that specific training set data may be required to address the peculiarities of tweets’ language; iii.) the F1 scores for the certainty, polarity and time attributes clearly indicate areas of improvements and also contribute to a better understanding of the system’s results; iv.) the F1 scores on the FV suggest that extending the training data with tweets could also benefit the identification of values which are not frequent in the news domain, such as no fact.

44Future work will aim at re-running the task from raw text and developing specific modules for the factuality of events according to the text types where they occur. Finally, we will plan to run a cross-fertilization task concerning temporal ordering and anchoring of events and factauality profiling.