Version classiqueVersion mobile

Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

Elisabetta Fersini
Marco Passarotti
Viviana Patti

Contributed Papers: Long Papers

On the Role of Textual Connectives in Sentence Comprehension: A New Dataset for Italian

Giorgia Albertin, Alessio Miaschi et Dominique Brunato


In this paper we present a new evaluation resource for Italian aimed at assessing the role of textual connectives in the comprehension of the meaning of a sentence. The resource is arranged in two sections (acceptability assessment and cloze test), each one corresponding to a distinct challenge task conceived to test how subtle modifications involving connectives in real usage sentences influence the perceived acceptability of the sentence by native speakers and Neural Language Models (NLMs). Although the main focus is the presentation of the dataset, we also provide some preliminary data comparing human judgments and NLMs performance in the two tasks1.

Texte intégral

1. Introduction

1The outstanding performance reached by recent Neural Language Models (NLMs) across a variety of NLP tasks that require extensive linguistic skills has stimulated an increased interest in the theoretical and computational linguistics community towards a better understanding of their inner mechanisms. In particular, the debate is focused on trying to understand what kind of linguistic knowledge these models are able to induce from the raw data they are exposed to and to what extent this knowledge resembles human-like generalization patterns (Linzen and Baroni 2021; Manning 2015). To pursue this investigation, it has become of pivotal importance the availability of challenging test sets, also called ‘diagnostic’ or ‘stress’ tests, built to probe the sensitivity of a model to specific language phenomena.

2So far, most of the efforts have been focused on assessing the syntactic abilities encoded by NLMs by exploiting human curated benchmarks, which are usually proposed in the form of minimal sentence pairs, i.e. minimally different sentences exemplifying a wide array of linguistic contrasts. A well-known one is BLiMP (Benchmark of Linguistic Minimal Pairs) (Warstadt et al. 2020) which contains pairs that contrast in syntactic acceptability and isolating fine-grained phenomena in specific domains of the English grammar, such as subject–verb agreement, island effects, ellipsis and negative polarity items.

3Differently from syntactic well-formedness, less explored is the sensitivity of these models to deeper linguistic dimensions involving semantics and discourse, such as textual cohesion, which are critical to language understanding. With this respect, one of the explicit devices that natural languages use to convey textual cohesion is represented by function words. As observed by Kim et al. (2019), although these words play a key role in compositional meaning as they introduce discourse referents or make explicit relations between them, they are still under-investigated in the literature on representation learning. To this end, the authors released a suite of nine challenge tasks for English aimed to test the NLMs’ understanding of specific types of function word, e.g. coordinating conjunctions, quantifiers, definite articles. Reasoning about conjuncts in conjunctive sentences, Saha et al. (2020), instead, introduced CONJNLI, a challenge stress-test for Natural Language Inference (NLI) over conjunctive sentences, where the premise differs from the hypothesis by conjuncts removed, added, or replaced.

4Taking inspiration from this work, in this paper we focus the attention on the role of textual connectives in the comprehension of a sentence and we introduce a new evaluation resource for Italian which, to our knowledge, is the first one for this language. The resource is articulated into two sections (acceptability assessment and cloze test), each one corresponding to a distinct task aimed at probing, in a different format, to what extent current NLMs are able to properly encode the role of connectives in a sentence. A peculiarity of the dataset is that it contains sentences that were extracted and minimally modified from existing corpora so as to test the comprehension of connectives in the real use of language.

2. Corpus Collection

5This section is divided into two parts. In the first one, we discuss the methodology implemented for the selection of connectives and the extraction of the sentences. Subsequently, we provide an overview of the two tasks defined to test the correct comprehension of connectives.

2.1 Selecting Connectives and Extracting Sentences

6As a first step, we defined the linguistic criteria for the selection of connectives to include in the corpus. By connective we mean specific words that have the function of drawing a relation between two or more clauses (Sanders and Noordman 2000; Graesser and McNamara 2011). To this end, two resources were employed: the INVALSI reading comprehension and language reflection tests designed by the National Institute for the Evaluation of the Education System and the Nuovo Vocabolario di Base of Italian (NVdB) (De Mauro and Chiari 2016). Starting from the collection of the INVALSI tests proposed in the last six years for different grades, we extracted all words which were expressly called ‘connective’ in the tests or were involved in defining a logical relationship between two sentences. We thus obtained a first list of 46 elements, belonging to diverse morpho-syntactic categories (i.e. prepositions, conjunctions, adverbs), which was then integrated with other 19 connectives extracted from the NVdB. We then checked the distribution of the selected items in existing Italian treebanks and extracted the sentences in which these words were unambiguously used as sentence connectives. Three different sections of the Italian Universal Dependency Treebank (IUDT) CITE_ERROR were used: ISDT (Bosco, Montemagni, and Simi 2013), PoSTWITA (Sanguinetti et al. 2018) and TWITTIRò (Cignarella, Bosco, and Rosso 2019)2, the first one representative of standard language and the latter collecting Italian tweets. We employed PML TreeQuery3 to query the treebanks and filter the sentences containing the connectives we were interested in. In particular, to exclude occurrences which do not have the role of phrasal connectives (e.g. the conjunction e joining two nouns), only sentences in which the connective was headed by a verb or a copula were taken into account. We observed that the absolute frequency’s positions of the selected connectives in the three corpora above-mentioned mostly overlap, although their occurrences in PoSTWITA and TWITTIRò (jointly considered as sample of Italian social media language) were lower than in ISDT, also given the different corpora sizes (i.e. 289,343 words in ISDT vs 154,050 words in PoSTWITA and TWITTIRò). Given the partial overlapping of the frequency data and the potential non-standard use of connectives in treebanks representative of social media texts, also due to genre-specific features (e.g. hashtag, emoticons etc.), we decided to consider only the first 21 most frequent connectives occurring in ISDT. As the first Italian corpus for the comprehension on textual connectives, we prefer to focus in sentences as close as possible to standard Italian language. Further considerations on connectives’ distributions led us to the deletion of per, così, ancora, because of their ambiguous behavior as textual connectives (e.g. we noticed that the majority of the occurrences of per involves the presence of an infinite verb, a distribution which is far from the other connectives). The following 18 connectives were finally considered: e, se, quando, come, ma, dove, o, anche, perché, poi, mentre, infatti, prima, però, invece, inoltre, tuttavia, quindi. The distribution of the finally selected connectives from ISDT and from PoSTWITA and TWITTIRò is reported in Appendix A.

7Once established the final list, those sentences which we consider more suitable to be involved in our tasks were manually extracted from ISDT and eventually modified following some patterns, to guarantee sentence comprehension. For example, in some cases two sentences occurring in the treebank in a subsequent order, but that were clearly extracted from the same text, were joined together to form a unique sentence, through the insertion of the appropriate punctuation. This happened e.g. when the connective appeared at the beginning of the second sentence joining this to the first one, which serves as the antecedent to comprehend the logical relationship. We tried to include in the dataset sentences with different degrees of syntactic and lexical complexity, considering the number of subordinate clauses and the variety of the lexicon as related proxies. All the original sentences, later arranged into the acceptability assessment and the cloze test task, are drawn from ISDT.

2.2 Definition of the Tasks

Table 1






L’arte e la scienza sono libere e libero ne è l’insegnamento.


L’arte e la scienza sono libere tuttavia libero ne è l’insegnamento.


Paolo si muove con difficoltà, ma è sempre allegro e di buon umore.


Paolo si muove con difficolt`a, perche´ `e sempre allegro e di buon umore

Cloze test


Che cosa possiamo fare in estate ... vogliamo partire per le vacanze e abbiamo un cane o un gatto? [ se quando perché dove come]


Nelle botteghe artigianali della produzione di piastrelle la smaltatura è ancora tradizionale, ... i forni, come è naturale, oggi funzionano a gas.[mentre invece come dove perchè]

Examples from the dataset. Sentences are indicated with the last part of id, which gives information about the target connective, the position of the sentence in the section and the label in each section (A=‘acceptable’, NA=‘not acceptable’; cl=‘cloze test’). For the cloze task, the target connective is marked in bold and the plausible alternative in italics.

8The collected sentences were grouped in two sections aimed at testing the correct comprehension of connectives in a different format, i.e. through an acceptability assessment task and a cloze test task. Table 1 provides an example of sentences/sentences pairs for each task.

2.2.1 Acceptability Assessment Section

9To design the acceptability assessment task, we selected 15 sentences per connective from the whole dataset. For each sentence, an unacceptable counterpart was created by replacing the original connective with another of the list. The replacement strategy was meant to obtain unacceptable sentences with contradictory or nonsensical meaning but preserving their grammaticality. Indeed those sentences should be the most challenging one for NLMs, which have been shown to be capable of detecting sentence grammaticality (Jawahar, Sagot, and Seddah 2019), but still struggle to track down unacceptable meanings and contradictions. Nevertheless, we were not always able to guarantee this constraint as for some specific contexts none of the available connective could be substituted without affecting the resulting grammaticality. This happened in 98 cases, which we decided to keep in the dataset but we signaled with the label ‘no’ in the field ’grammaticality’, as in:

Nei campi si sopravvive anche intorno tutto muore.

10Although the assessment of grammaticality is not the main focus of this work, given the fact that it was unavoidably violated in the above-reported cases, we feel compelled to provide distinguished analysis for the group of ungrammatical sentences. A few sentences were also deleted due to ambiguity. The final section contains 518 sentence pairs, i.e. 259 acceptable and 259 unacceptable ones.

2.2.2 Cloze Test Section

11The second section was designed as a cloze test task and contains 270 sentences, 15 for connective. For every sentence the original connective was replaced by a blank space and 5 alternatives were proposed for completion: the target, a plausible alternative and three implausible options. For ‘plausible alternative’ we mean another connective of the list that could occupy the same linguistic contest of the target, yielding to an identical meaning or to a different, yet totally plausible, reading. As for the acceptability task, it turns out that for some connectives (e.g. prima) it was very challenging, if not impossible, to propose such a plausible connective. In those cases, that in truth are only a minority, it has been proposed an alternative that at least should guarantee the grammaticality.

3. Corpus Annotation

12The two sections of the dataset were splitted into 9 surveys (5 for the acceptability assessment task and 4 for the cloze task) and submitted to human evaluation by recruiting Italian native speakers of different ages through the Prolific platform4.

13In the acceptability assessment task, participants were asked to judge the acceptability of each sentence on a 5-grade Likert scale (from 1=‘totally unacceptable’ to 5=‘totally acceptable’). Although this makes the dataset more challenging, we assume that acceptability is a gradual rather than binary notion as it is affected by many factors (Sorace and Keller 2005; Sprouse 2007). To disambiguate the interpretation of sentence acceptability and orient annotators in giving their judgments, the survey guidelines encouraged them to think if they found the sentence natural in Italian and if they would have used it in a real conversation or any other communicative context.

14For the cloze test task, participants were required to supply the missing element choosing among the proposed options plus the one “none of the previous options is suitable”.

15Each survey was completed by 20 annotators on average. The number of annotations per sentence in the acceptability task ranges from 16 to 21 and for the cloze task from 18 to 21. To improve data quality, we discarded annotators who took less than 10 minutes to complete the test, considering the average threshold time for each survey. This led us to reject 5 annotators only for the acceptability task.

Table 2

Acceptability label












Average scores assigned by humans (with standard deviation) to the acceptable, unacceptable and unacceptable+ungrammatical sentences.

16Table 2 reports the average human score and standard deviation obtained by the acceptable and unacceptable sentences. For the latter, we separately computed these scores for the subset of sentences which were also labeled as ungrammatical (see Section 2.2.1). As it can be seen, humans perform very well on the task assigning quite higher scores to the acceptable sentences with respect to the unacceptable ones, also with little variability. Within the unacceptable subset, the slightly smaller score received on average by ungrammatical sentences provides further evidence that humans are sensitive to this distinction.

cloze test task

Table 3

Cloze task choice

N. Items





Plausible alt.



Implausible alt.



Target=Plausible alt.



Number and % of sentences for which the majority label was assigned to the target connective, to the plausible alternative, to an implausible alternative or equally balanced between the target and the plausible alternative.

4. Testing the Sensitivity of Neural Language Models to Connectives

17We conclude by presenting some preliminary findings aimed at testing the performance of NLMs in the two tasks. Specifically, we performed two distinct evaluations. For the acceptability assessment task, we computed the perplexity (PPL) score assigned by the GePpeTto model (De Mattei et al. 2020) to all sentences of the corresponding section. We relied on perplexity as it is a standard evaluation measure of the quality of a language model yielding a good approximation of how well a model recognises an unseen piece of text as a plausible one. Accordingly, we assumed that higher PPL scores should be assigned to sentences labeled as unacceptable with respect to their original version. GePpeTto was chosen as it is a traditional unidirectional model built using the GPT-2 architecture (Radford et al. 2019) and, differently from a bidirectional model such as BERT (Devlin et al. 2019), allows computing a well-formed probability distribution over sentences. The sentence-level PPL was calculated using the formula reported in .

18By inspecting the results in Table 4, we observed that the average PPL score assigned to the acceptable sentences is quite lower than the one assigned to the unacceptable ones (i.e. 42.512 vs 78.280).

Table 4

















Average, minimum and maximum perplexity value given by the model to the acceptable, unacceptable and unacceptable+ungrammatical sentences.

19As expected, for the subset of unacceptable sentences, perplexity was on average higher for the ones marked as ungrammatical (98.992), reflecting the model’s capability of encoding syntactic phenomena. Interestingly, among unacceptable sentences, those obtaining lower PPL scores were perfectly well-formed but with an implausible meaning, as in the case of:

Il film ’Le chiavi di casa’ ha partecipato al Festival del Cinema di Venezia di quest’anno, perché non ha vinto nessun premio (PPL=13.892).

20To compare humans and model performance, we also computed the Spearman’s rank correlation (ρ) between the average acceptability score given by annotators and the PPL score assigned by the model to the same sentences. Although limited to this analysis, the resulting very weak correlation (i.e. ρ =-0.120, p-value < 0.01) suggests that connectives differently impact on the ability of humans and models to assess the plausibility of a sentence.

21As for the cloze task test, we relied on the pre-trained Italian version of the BERT model developed by the MDZ Digital Library Team and available trough the Huggingface’s Transformers library (Wolf et al. 2020).5 We extracted the first ten completions provided by the model trough the Masked Language Modeling task (MLM) for each sentence, along with their probabilities. This allowed us to inspect whether and in how many cases either the target connective or the plausible alternative appear in the top-ranked predictions.

Table 5


10 match

1st match






Pl. alt.





Target+Pl. alt.






(Number) and % of BERT’s completions in which only the target, only the plausible alternative, both of them or none of them (Other) occur in the first 10 predictions (10_match). (Number) and % of the completions in which the target and the plausible alternative were predicted with the highest probability are also reported (1st_match).

22As shown in Table 5, for the large majority of cases BERT is able to infer in its first 10 predictions that the sentence should be completed with a correct connective. That happens in 86.29% of the sentences for the target, resulting from the sum of the cases where only the target occurs in the completions (31.48%) with the cases in which both the target and the plausible alternative were predicted (54.81%), and in 59.25% for the plausible alternative (that is 4.44% plus 54.81%). Focusing instead on the first completion for each sentence, we observe that in almost half of the sentences BERT assigns the highest probability to the original connective (41.11%) or to the plausible one (8.52%).

  • 6 Note that in order to assign the acceptability label of each completion we refer to the usage of th (...)
  • 7 the unacceptable completion is marked in bold, the dubious acceptable one is reported in block and (...)

23We are currently performing a more qualitative analysis to better investigate the cases in which the correct connective hasn’t received a high probability score, as well as those in which neither of the two options appeared at all (i.e. Other cases in Table 5), in order to understand whether the other completions can still be considered as plausible ones. Preliminary findings showed that, among the Other cases, about 56 of the completions provided by BERT are unacceptable and 34 of them are dubious acceptable i.e.  not clearly recognizable as acceptable6, as in the case of the following sentence7:

Secondo gli esperti, in Italia i giovani leggono meno i giornali rispetto ai giovani di altri Paesi europei, ... rispetto agli anni passati i giovani tra i 14 e i 19 anni leggono più spesso i giornali. [perché anche però].

24Nevertheless, the majority of Other’s completions can be considered as acceptable ones. In fact, BERT predicted a word leading to the same meaning (or, at least, very similar) to the original sentence in more that 60 cases. Moreover, in most cases (i.e. 92) the completions provided are plausible ones, although in some of them the sentences acquire different meanings.

5. Conclusion

25In the context of studies devoted to assess the linguistic knowledge implicitly encoded by Neural Language Models, we introduced a new evaluation dataset for Italian designed to test the understanding of textual connectives in real-usage sentences. At first, we verified the significance of a set of selected connectives through a frequency analysis on already existing Italian gold corpora. Then, we manually selected only those sentences in which occur a genuine connective. Finally, we grouped the sentences into two different tasks, differing for the format used to elicit sentence comprehension in humans and current state-of-the-art NLMs: acceptability assessment and cloze test tasks. Human evaluation was provided for both the section, to verify the robustness of the dataset, which indeed was confirmed from the judgements collected.

26Preliminary findings on NLMs behaviour on textual connectives showed that in several cases the models are capable of distinguishing between acceptable and unacceptable sentences, thus suggesting their ability to encode sentence meaning within their internal mechanisms. However, it remains unclear to what extent these models rely on semantic acceptability features, since we observed cases in which they fail to recognize implausible meaning of perfectly grammatical sentences.

27We are currently increasing the dataset with the introduction of a new section designed in the form of the traditional Natural Language Inference task, for which the understanding of a given connective will be fundamental to infer the correct entailment relation between a premise and a hypothesis. We also believe that expanding the dataset to further connectives and including sentences representative of non standard italian language usage, i.e. social-media language, would be desirable to improve the robustness of the resource.


Cristina Bosco, Simonetta Montemagni, and Maria Simi. 2013. “Converting Italian Treebanks: Towards an Italian Stanford Dependency Treebank.” In Proceedings of the Acl Linguistic Annotation Workshop & Interoperability with Discourse.

Alessandra Teresa Cignarella, Cristina Bosco, and Paolo Rosso. 2019. “Presenting TWITTIRÒ-UD: An Italian Twitter Treebank in Universal Dependencies.” In Proceedings of the Fifth International Conference on Dependency Linguistics (Depling, Syntaxfest 2019).

Lorenzo De Mattei, Michele Cafagna, Felice Dell’Orletta, Malvina Nissim, and Marco Guerini. 2020. “GePpeTto Carves Italian into a Language Model.” In CLiC-It.

Tullio De Mauro and I Chiari. 2016. “Il Nuovo Vocabolario Di Base Della Lingua Italiana.” Internazionale.[28/11/2020]. Https://Www. Internazionale. It/Opinione/Tullio-de-Mauro/2016/12/23/Il-Nuovo-Vocabolario-Di-Base-Della-Lingua-Italiana.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–86.

Arthur C. Graesser and Danielle S McNamara. 2011. “Computational Analyses of Multilevel Discourse Comprehension.” Topics in Cognitive Science 3 (2): 371–98.

Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. “What Does BERT Learn About the Structure of Language?” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3651–7. Florence, Italy: Association for Computational Linguistics.

Tal Linzen and Marco Baroni. 2021. “Syntactic Structure from Deep Learning.” Annual Review of Linguistics 7: 195–212.

Christopher D. Manning. 2015. “Computational Linguistics and Deep Learning.” Computational Linguistics 41 (4): 701–7.

Alec Radford, Jeff Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. “Language Models Are Unsupervised Multitask Learners.” In.

Ted JM Sanders and Leo GM Noordman. 2000. “The Role of Coherence Relations and Their Linguistic Markers in Text Processing.” Discourse Processes 29 (1): 37–60.

Manuela Sanguinetti, Cristina Bosco, Alberto Lavelli, Alessandro Mazzei, and Fabio Tamburini. 2018. “PoSTWITA-UD: An Italian Twitter Treebank in Universal Dependencies.” In Proceedings of the Eleventh Language Resources and Evaluation Conference (Lrec 2018).

Antonella Sorace and Frank Keller. 2005. “Gradience in Linguistic Data.” Lingua 115 (11): 1497–1524.

Jon Sprouse. 2007. “Continuous Acceptability, Categorical Grammaticality, and Experimental Synta.” Biolinguistics, 1123–1134.

Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. “BLiMP: The Benchmark of Linguistic Minimal Pairs for English.” Transactions of the Association for Computational Linguistics 8: 377–92.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, et al. 2020. “Transformers: State-of-the-Art Natural Language Processing.” In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38–45. Online: Association for Computational Linguistics.


Appendix A

Table 6






























































































(Numbers) and % of the frequency of the 18 finally selected connectives in ISDT corpus and in PoSTWITA+TWITTIRò corpora.


1 The resource is available at:





6 Note that in order to assign the acceptability label of each completion we refer to the usage of the Italian language as standard as possible.

7 the unacceptable completion is marked in bold, the dubious acceptable one is reported in block and the original connective is indicated in italics.

Table des illustrations

Fichier image/jpeg, 17k


University of Padova –


Le texte seul est utilisable sous licence CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.


Open access

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search