Version classiqueVersion mobile

EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

Valerio Basile
Danilo Croce
Maria Maro
et al.

DIACR-Ita: Diachronic Lexical Semantics

UNIMIB @ DIACR-Ita: Aligning Distributional Embeddings with a Compass for Semantic Change Detection in the Italian Language

Federico Belotti, Federico Bianchi et Matteo Palmonari


In this paper, we present our results related to the EVALITA 2020 challenge, DIACR-Ita, for semantic change detection for the Italian language. Our approach is based on measuring the semantic distance across time-specific word vectors generated with Compass-aligned Distributional Embeddings (CADE). We first generate temporal embeddings with CADE, a strategy to align word embeddings that are specific for each time period; the quality of this alignment is the main asset of our proposal. We then measure the semantic shift of each word, combining two different semantic shift measures. Eventually, we classify a word meaning as changed or not changed by defining a threshold over the semantic distance across time.

Texte intégral

1. Introduction

1Semantic change detection is the task of detecting if a word has shifted in meaning between different periods of time (Tahmasebi, Borin, and Jatowt 2018; Kutuzov et al. 2018). The DIACR-Ita (P. Basile et al. 2020) challenge (at EVALITA (V. Basile et al. 2020)) is meant to evaluate approaches for semantic change detection for the Italian Language.

2The task is described as follows: for training, two corpora t1 and t2, consisting of text coming from different periods are given, for testing, a set of unlabeled target words is given, where for each of them a binary scores has to be predicted: 1 identifies lexical change between t1 and t2 while 0 does not.

3In this paper, we present our approach to semantic change detection that is based on two components: 1) an alignment procedure to generate distributional vector spaces that are comparable for t1 and t2 and 2) the use of distance metrics to compute the degree of semantic change for a given word. Our alignment procedure is based on Compass Aligned Distributional Embeddings (CADE) proposed by Bianchi et al. (2020) (note the approach was introduced as Temporal Word Embeddings with a Compass by Di Carlo et al. (2019), but the name was changed to enforce the idea that the embeddings can be used to align more general corpora and not just diachronic ones). Given the aligned embeddings, we use two measures to compute the degree of change based on the similarities of the vectors in the embedded space. Our results show that our methodology for aligning spaces can be useful in detecting lexical semantic change.

2. Description of the System: Semantic Change Detection with Compass Aligned Embeddings

4Our approach is based on measuring the semantic distance across time of time-specific word vectors generated with CADE and on the use of two measures for detecting semantic shifts i.e., the semantic distance between word vectors across time. This distance can be interpreted as a function of the words’ self-similarity across time, where the similarity is measured by a linear combination of cosine and second-order similarity (Hamilton, Leskovec, and Jurafsky 2016b).

5Finally, a threshold over this self-similarity is used to classify a word as changed or not changed.

6This methodology was applied also in the semantic shift detection challenge presented at SemEval2020 (Schlechtweg et al. 2020) (to which we participated after the end of the challenge). The challenge allowed us to explore and understand how the alignment and our self-similarity behaved. In the classification task of the SemEval2020 challenge (the one similar to this task), we eventually achieved 0.703, 0.771, 0.725, 0.742, in accuracy for respectively the English, German, Latin and Swedish languages; these results have been obtained with extensive parameter search given the gold standard available in the post-evaluation.1 In DIACR-Ita, the threshold and few other hyper parameters can be heuristically set to account for the limited number of possible submissions. In the next subsections we provide more details about the alignment methodology and the similarity function; more details about how we set the hyper parameters are provided in Section 3.

2.1 Aligning Embeddings

7Word2vec (Mikolov et al. 2013) is a useful methodology to generate vectors of words allowing us to study word similarity through vector similarity. However, due to the stochasticity of the training procedure, running word2vec on different corpora creates word vectors that are not comparable. Thus, an alignment procedure that puts the temporal word vectors in the same space is needed.

8There are different approaches to generate these aligned embeddings (see for example the work by (Hamilton, Leskovec, and Jurafsky 2016a) and (Yao et al. 2018)). In this paper, we generate aligned embeddings with Compass Aligned Distributional Embeddings (CADE) (Bianchi et al. 2020) (See Figure 1 for a schematic description of the model). CADE is a strategy to align word embeddings that are specific for each time period that extends the word2vec Continuous Bag Of Word (CBOW) model proposed by Mikolov et al. (2013). CADE can be used to generate aligned temporal word embeddings (i.e., time-specific vectors of words, like “amazon1974”) from the different slices.

9Given in input a set of slices of text, where each slice corresponds to text coming from a specific period of time, the alignment procedure is as follows:

10First, the text from all the slices is concatenated and CBOW is run on this corpus in order to obtain a “compass” model, i.e., a model defining the embedding space. The CBOW model uses two matrices to generate the embeddings (U and C in Figure 2), one for the context words and one for the target words. The target word matrix of the compass is then used to initialize the target matrices for each new CBOW model fitted on each of the slices. During training, these new target matrices are frozen, i.e., they are not updated during the training on the slice. This ensures that at the end of the training process, the various temporal embeddings are all aligned in the same embedding space, making them comparable without losing their individual temporal distinctions. We use the publicly available online implementation of CADE.2

Figure 1: An high level overview of the Compass Aligned Distributional Embeddings model

Image 10000201000003B000000133F253595AC30F7926.png

2.2 Computing Semantic Change

11Once the embeddings are aligned, we need measures to evaluate the degree of semantic change. We compute the semantic shift of each word, i.e. the semantic distance between word vectors across time using the combination of two different measures: Local Neighbors (ln), introduced by Hamilton et al. (2016a) and cosine similarity (cos), merging them with a weighted linear combination into a new measure called Move.

Local Neighbors

  • 3 When a neighbor is missing in one time slice, we replace it with the average vector of the space.

ln is based on the similarity between a word and its neighbor words in the two different time periods. Essentially we compute the degree of semantic change of the word w in two slices by first collecting the nearest neighbors (NNs) of Image 10000000000000140000001013039C1C116F98AD.jpgand Image 100000000000002400000010AA2716596449DFDD.jpgin the two respective slices, then given the embeddings at time t the similarities between the vector of Image 10000000000000140000001013039C1C116F98AD.jpgand the vectors of all the neighbors are computed.3 The same process is run for time t+1 with Image 100000000000002400000010AA2716596449DFDD.jpg, eventually giving us two vectors of similarity scores. These two vectors are again compared using cosine similarity. The higher the value of this measure the less the vector has changed with respect to its neighbors and thus the less the word should have shifted in meaning.

Cosine Similarity

12The second measure we use is simply the cosine similarity of the vectors of a word in two different time periods. Similarly as before, the higher the value the less the vector has changed and thus the less the word should have shifted in meaning.

The Move Measure

13We merge these measures together using a weighted linear combination, that is:

Image 10000000000001170000003B3024B7F0EA1735EA.jpg

with Image 1000000000000043000000136B13B8813A22E84D.jpg. In particular λ express the usage strength of the two measures: a high λ will shift Move towards the cosine similarity, while a low one towards the ln measure. As introduced before we classify if the meaning has changed by defining a threshold over s (more details about this are presented in the next Section).

3. Experimental Evaluation

14The dataset provided by the challenge’s organizers (P. Basile et al. 2020) is a collection of documents extracted by newspapers written in the Italian language labeled with temporal information. Participants must train their models only on the data provided, so a pre-processed corpus is given: tab separated, with one token per line, where for each token there are its corresponding part-of-speech (POS) tag and lemma, with sentences separated by empty lines. The corpus is split into two slices, each belonging to a specific period of time, t1 and t2, where t1 < t2.

3.1 Dataset

15For the training data we used the flat version with only the lemmas, obtained by the organizers’ script (P. Basile et al. 2020); in addition we applied a pre-processing step, in which we removed punctuation and non alpha-numeric symbols and we kept only those sentences with at least two tokens.

3.2 Models Considered

16We use the embeddings aligned with CADE and the move measure. The parameters of the moving average we need to consider are: the number of nearest neighbors (NNs) to be collected by ln, λ for the moving average and the threshold for the similarity. We set the threshold to decide if a word is stable or not is set to 0.7, with the decision given by:

Image 10000000000000B500000037F6AD6134C84A1E5B.jpg

Essentially, the less changed are the two vectors of the words (for cos) and the neighbors (for ln) the more the word has been stable between the two time periods. As heuristics we chose Image 100000000000008700000013F1662737D7EFA7F3.jpgto evaluate the relationship between the two measures used to build move, and we set to 22 the number of nearest neighbors to be considered by the ln; this is the general setup that gave the results that have been submitted to the challenge.

17We trained CADE for 10 epochs to learn 100-dimensional vectors, with the window size set to 5, 10 negative examples for every positive one, with the initial learning rate set to 0.025 and decreased linearly during training.

18As other models, in the post evaluation we also considered one that only uses the cos (CADE (cos)) similarity measure and one that uses only the ln metric CADE (ln)) (again with 0.7 as threshold and with the number of NNs for ln set to 22).

19As baselines, the authors propose to use baseline-freq, that is the absolute value of the difference between the words’ frequencies and baseline-colloc, where the Bag-of-Collocations of the two words in the two different periods is built and then cosine similarity is applied. A threshold is used on both metrics to define semantic change (P. Basile et al. 2020). We report also the results of the other participants.

Table 1: Accuracy scores for the binary classification w.r.t. the other participants to the challenge. Image 10000000000000060000000CD82051492560B0F4.jpgidentifies our submitted results












CADE (move) Image 10000000000000060000000CD82051492560B0F4.jpg



























CADE (move) Image 10000000000000060000000CD82051492560B0F4.jpg



CADE (move) Image 10000000000000060000000CD82051492560B0F4.jpg



CADE (cos)



CADE (ln)



3.3 Results

20The evaluation metric used in this challenge is the accuracy, that is, the number of correct predictions over the target data. Table 1 shows the results. Our model was the third most accurate. However, in the post-evaluation we discovered that just using the ln metric and ignoring the use of cos (this is equivalent to using λ=0 in our move measure) improves the performance leading to the second best accuracy score in the leaderboard.

4. Discussion

21Our results show that CADE (Bianchi et al. 2020) is an effective method to generate aligned embeddings for the Italian language. This result, together with those obtained on the SemEval2020 data, suggest that CADE can support models of semantic shift detection in several languages. Indeed, we show that in combination with some simple semantic change measures it is possible to provide a good model for semantic change detection that can be subsequently extended with more features. Appendix A contains some more detailed examples of the words that CADE (ln) and CADE (move), with lambda set to 0.3, could not classify correctly. Also, we show the neighborhood for some of those words to give more context on why we get those errors. A more precise use of pre-processing techniques with the combination of other metrics to compute semantic change might help in reducing these errors.

Table 5: First 10 nearest neighbors by cosine similarity of the word “piovra" from t1 and t2

























Pierpaolo Basile, Annalina Caputo, Tommaso Caselli, Pierluigi Cassotti, and Rossella Varvara. 2020. “DIACR-Ita @ EVALITA2020: Overview of the EVALITA2020 Diachronic Lexical Semantics (DIACR-Ita) Task.” In Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2020), edited by Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. Online:

Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. 2020. “EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian.” In Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (Evalita 2020), edited by Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. Online:

Federico Bianchi, Valerio Di Carlo, Paolo Nicoli, and Matteo Palmonari. 2020. “Compass-Aligned Distributional Embeddings for Studying Semantic Differences Across Corpora.” arXiv Preprint arXiv:2004.06519.

William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016a. “Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change.” In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1489–1501. Berlin, Germany: Association for Computational Linguistics.

William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016b. “Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change.” In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, 2116–21. Austin, Texas: Association for Computational Linguistics.

Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. “Diachronic Word Embeddings and Semantic Shifts: A Survey.” In Proceedings of the 27th International Conference on Computational Linguistics, 1384–97. Santa Fe, New Mexico, USA: Association for Computational Linguistics.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. “Distributed Representations of Words and Phrases and Their Compositionality.” In Advances in Neural Information Processing Systems, 3111–9.

Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. “SemEval-2020 Task 1: Unsupervised Lexical Semantic Change Detection.” arXiv Preprint arXiv:2007.11464.

Nina Tahmasebi, Lars Borin, and Adam Jatowt. 2018. “Survey of Computational Approaches to Lexical Semantic Change.” arXiv Preprint arXiv:1811.06278.

Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018. “Dynamic Word Embeddings for Evolving Semantic Discovery.” In Proceedings of the Eleventh Acm International Conference on Web Search and Data Mining, 673–81.


A CADE Misclassifications

We report in Tables 2 and 3 CADE’s misclassifications with the two best metrics, namely CADE (move) with λ = 0.3 and CADE (ln). Eventually, we also show in Tables 4 and 5 some examples of neighborhood for the target words.

Table 2: Wrong predictions done by CADE (move) with λ = 0.3.






not changed



not changed



not changed

Table 3: Wrong predictions done by CADE (ln)






not changed


not changed


Table 4 shows the top 10 nearest neighbors of the target word “pacchetto" and we think CADE classifies its meaning as changed because during time t1 the meaning is more focused in the economic area, as one can see from neighbors like “azionario", “obbligazione" or “contante" (translated to “stock" as referred to the market, “bond" and “cash" resp.); while at time t2 shifts to a more political sense, as shown by words such as “decreto" or “emendamento" (“decree" and “amendment" resp.).

Table 4: First 10 nearest neighbors by cosine similarity of the word “pacchetto" from t1 and t2























The same it seems to happen for the target word “piovra", as one can see from Table 5, where at time t1 CADE gathers senses from both considering it as the animal, for example from the word “tentacle", or as someone tied to crime in general, given words such as “profittatore" or “ruberia" (“profiteer" and “robbery" resp.); while at time t2 captures a shift towards the Italian crime TV series “La piovra", as emerge from words such as “fiction", “camorra" or “retequattro", which is an Italian television channel.


1 Check the belerico entry in the challenge leaderboard at


3 When a neighbor is missing in one time slice, we replace it with the average vector of the space.


University of Milano-Bicocca Viale Sarca 336, 20126 Milan, Italy –

Bocconi University Via Sarfatti 25, 20136 Milan, Italy –


Open access

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search