Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021
| , ,Contributed Papers: Long Papers
Policycorpus XL: An Italian Corpus for the detection of Hate Speech Against Politics
Résumé
In this paper we present a large corpus of Twitter data in Italian, manually annotated with hate speech in the political domain. Given the rising amount of hate messages in the public debate, we believe that this is a valuable resource for the NLP community. Here we describe the collection of data and test some baseline with classification algorithms.
Texte intégral
The research leading to the results presented in this paper has received funding from the PolicyCLOUD project, supported by the European Union’s Horizon 2020 research and innovation programme under Grant Agreement no 870675.
1. Introduction and Background
1In recent years, computer mediated communication on social media and microblogging websites has become more and more aggressive (Watanabe, Bouazizi, and Ohtsuki 2018). It is well known that people uses social media like Twitter for a variety of purposes like keeping in touch with friends, raising the visibility of their interests, gathering useful information, seeking help and release stress (Zhao and Rosson 2009), but the spread of fake news (Shu et al. 2019) (Alam et al. 2016) has exacerbated a cultural clash between social classes that emerged after the debate about Brexit (Celli et al. 2016). Despite the behavior online is different from the behavior offline (Celli and Polonio 2015), we observe more and more hate speech in social media, to the point where it has become a serious problem for free speech and social cohesion.
2Hate speech is defined as any expression that is abusive, insulting, intimidating, harassing, and/or incites, supports and facilitates violence, hatred, or discrimination. It is directed against people (individuals or groups) on the basis of their race, ethnic origin, religion, gender, age, physical condition, disability, sexual orientation, political conviction, and so forth (Erjavec and Kovačič 2012).
3In response to the growing hate messages, the Natural language Processing (NLP) community focused on the classification of hate speech (Badjatiya et al. 2017) and the analysis of online debates (Celli, Riccardi, and Ghosh 2014). In particular, many worked on systems to detect offensive language against specific vulnerable groups (e.g., immigrants, LGBTQ communities etc.) (Poletto et al. 2017) (Poletto et al. 2021), as well as aggressive language against women (Saha et al. 2018). An under-researched - yet important - area of investigation is anti-policy hate: the hate speech against politicians, policy making and laws at any level (national, regional and local). While anti-policy hate speech has been addressed in Arabic (Guellil et al. 2020) and German (Jaki and De Smedt 2019), most European languages have been under-researched. The bottleneck in this field of research is the availability of data to train good hate speech detection models. In recent years, scientific research contributed to the automatic detection of hate speech from text with datasets annotated with hate labels, aggressiveness, offensiveness, and other related dimensions (Sanguinetti et al. 2018). Scholars have presented systems for the detection of hate speech in social media focused on specific targets, such as immigrants (Del Vigna et al. 2017), and language domains, such as racism (Kwok and Wang 2013), misogyny (Basile et al. 2019) or cyberbullying (Menini et al. 2019). Each type of hate speech has its own vocabulary and its own dynamics, thus the selection of a specific domain is crucial to obtain clean data and to restrict the scope of experiments and learning tasks.
4In this paper we present a new corpus, called Policycorpus XL, for hate speech detection from Twitter in Italian. This corpus is an extension of the Policycorpus (Duzha et al. 2021). We selected Twitter as the source of data and Italian as the target language because Italy has, at least since the elections in 2018, a large audience that pays attention to hyper-partisan sources on Twitter that are prone to produce and retweet messages of hate against policy making (Giglietto et al. 2019).
5The paper is structured as follows: after a literature review (Section 2), we describe how we collected and annotated the data (Section 3), we evaluate some baselines (Section 4), and we pave the way for future work (Section 5).
2. Related Work
6Hate Speech in social media is a complex phenomenon, whose detection has recently gained significant traction in the Natural Language Processing community, as attested by several recent review works (Poletto et al. 2021). High-quality annotated corpora and benchmarks are key resources for hate speech detection and haters profiling in general (Jain et al. 2021), considering the vast number of supervised approaches that have been proposed (MacAvaney et al. 2019). Most datasets in the field of hate speech have been released during competitions and evaluation campaigns. These include SemEval 2019, where it was released a multilingual hate speech corpus against immigrants and women in English and Spanish (Basile et al. 2019), and PAN 2021, that provided a dataset for the detection of hate spreader authors in English and Spanish (Rangel et al. 2021). In Italian there are:
-
the Italian HS corpus (Poletto et al. 2017),
-
HaSpeeDe-tw2018 and HaSpeeDe-tw2020, the datasets released during the EVALITA campaigns (Sanguinetti et al. 2020),
-
the Policycorpus (Duzha et al. 2021), the only dataset in Italian that is annotated with hate speech in the political domain.
7The Italian HS corpus is a collection of more than 5700 tweets manually annotated with hate speech, aggressiveness, irony and other forms of potentially harassing communication. The HaSpeeDe-tw corpora are two collections of 4000 and 8100 tweets respectively, manually annotated with hate speech labels and containing mainly anti-immigration hate (Bosco et al. 2018). The Policycorpus is a collection of 1260 tweets manually annotated with hate speech labels against politics and politicians. We decided to expand it and produce a new dataset.
8Hate speech is hard to annotate and hard to model, with the risk of creating data that is biased and making the models prone to overfitting. In addition to this, literature also reports cases of annotators’ insensitivity to differences in dialect that can lead to racial bias in automatic hate speech detection models, potentially amplifying harm against minority populations. It is the case of African American English (Sap et al. 2019) but it potentially applies to Italian as well, as it is a language full of dialects and regional offenses.
9Hate speech is intrinsically associated to rela-tionships between groups, and also relying in lan-guage nuances. There are many definitions of hate speech from different sources, such as European Union Commission, International minorities asso-ciations (ILGA) and social media policies (For-tuna and Nunes, 2018). In most definitions, hate speech has specific targets based on specific char-acteristics of groups. Hate speech is to incite vio-lence, usually towards a minority. Moreover, hate speech is to attack or diminish. Additionally, hu-mour has a specific status in hate speech, and it makes more difficult to understand the boundaries about what is hate and what is not.
10In the political domain we find all of these aspects, especially messages against a minority (politicians) to attack or diminish. We think that more resources are needed for the classification of hate speech in Italian in the political domain, hence we decided to collect and annotate more data for this task.
11In the next section, we describe how we created the dataset and annotated it with hate speech labels.
3. Data Collection and Annotation
12Starting from the Policycorpus, we expanded it from 1260 to 7000 tweets in Italian, collected using snowball sampling. As initial seeds, we used the same set of hashtags used for the Policycorpus, for instance: #dpcm (decree of the president of the council of ministers), #legge (law) and #leggedibilancio (budget law). We re-moved duplicates, retweets and tweets containing only hashtags and urls. At the end of the sam-pling process, the list of seeds included about 6000 hashtags that co-occurred with the initial ones. We grouped the hashtags into the following categories:
-
Laws, such as #decretorilancio (#relaunchdecree), #leggelettorale (#electorallaw), #decretosicurezza (#securitydecree)
-
Politicians and policy makers, such as #Salvini, #decretoSalvini (#Salvinidecree), #Renzi, #Meloni, #DraghiPremier
-
Political parties, such as #lega (#league), #pd (#Democratic Party)
-
Political tv shows, such as #ottoemezzo, #nonelarena, #noneladurso, #Piazzapulita
-
Topics of the public debate, such as #COVID, #precari (#precariousworkers), #sicurezza (#security), #giustizia (#justice), #ItalExit
-
Hyper-partisan slogans, such as #vergognaConte (#shameonConte), #contedimettiti (#ConteResign) or #noicontrosalvini (#WeareagainstSalvini)
13Examples of collected hashtags are reported in Figure 1.
14Recent shared tasks (Agerri et al. 2021; Cignarella et al. 2020; Aker et al. 2016) promoted the use of contextual information about the tweet and its author for improving the performance of stance detection. Here, with the aim to stimulate the exploration of data augmentation on hate speech detection, we shared additional contextual information based on the post such as: the number of retweets and the number of favours the tweet received, the device used for posting it (e.g. iOS or Android), the posting date and location, and an attribute that states if the post is a tweet, a retweet, a reply, or a quote. Furthermore, we collected contextual information related to the authors of these posts such as: the number of tweets ever posted, the user’s description and location, the number of her/his followers and of her/his friends, the number of public lists that this user is a member of and the date her/his account has been created.
15All these contextual information are respectively part of the “root-level” attributes of the Tweets and Users objects that Twitter returns in JSON format through its APIs. Additionally, we planned to explore the interests of the author collecting the list of her/his following (the users she/he follows) employing the following API endpoint. Moreover, for exploring the author’s social interactions, we used the Academic Full Search API for recovering the list of the users that she/he has retweeted to and replied to in the last two years.
16The enhance Policycorpus have been finally anonymised mapping each tweet_id, users_id, and mention with a randomly generated ID. To produce gold standard labels, we asked two Italian native speakers, experts of communication, to manually label the tweets in the corpus, distinguishing between hate and normal tweets according to the following guidelines: By definition, hate speech is any expression that is abusive, insulting, intimidating, harassing, and/or incites to violence, hatred, or discrimination. It is directed against people on the basis of their race, ethnic origin, religion, gender, age, physical condition, disability, sexual orientation, political conviction, and so forth. Below We provide some examples with translation in English:
- 1 “a clear #NO to #Netherlands that we would like users of the #MES economic resources but in exchang (...)
1) “Un chiaro #NO all #Olanda che ci vorrebbe sì utilizzatori delle risorse economiche del #MES ma in cambio della rinuncia dell Italia alla propria autonomia di bilancio. All Olanda diciamo: grazie e arrivederci NON CI INTERESSA!! ”1
17The first example is normal because it does not contain hate, insults, intimidation, violence or dis-crimination.
- 2 “... There is a weekly catwalk of the #jackal #no #notAtAll! Listening to a Po #clown after a true (...)
2) “...Sta settimanale passerella dello #sciacallo #no #proprioNo! Ascoltare un #pagliaccio padano dopo un vero PATRIOTA un medico di #Bergamo non si può reggere ne vedere ne ascoltare. Giletti dovrebbe smetterla di invitare certi CAZZARIPADANI! #COVID-19 #NonelArena”2
The second example contains hate speech, includ-ing insults like #clown and #jackal.
- 3 “I have my say ... #Draghi is a great economist but we don’t need a #Monti-style economist ... We d (...)
3) “Dico la mia... #Draghi è un grande economista ma a noi non serve un economista stile #Monti... A noi non serve un altro #governo tecnico per ubbidire alla lobby delle banche! A noi serve un leader politico! A noi serve un #ItalExit! A noi serve la #Lira! #No a #DraghiPremier”3
18The last example is a normal case, despite the strong negative sentiment. It might be controversial for the presence of the term lobby, often used in abusive contexts, but in this case, it is not directed against people on the basis of their race, ethnic origin, religion, gender, age, physical condition, disability, sexual orientation or political conviction.
19The Inter-Annotator Agreement was k=0.53.
20Although this score is not high, it is in line with the score reported in the literature for hate speech against immigrants (k=0.54) (Poletto et al. 2017) and indicates that the detection of hate speech is a hard task for humans.
21All the examples in disagreement were discussed and an agreement was reached between the annotators, with the help of a third supervisor. The cases of disagreements occurred more often when the sentiment of the tweet was negative, this was mainly due to:
-
The use of vulgar expressions not explicitly directed against specific people but generically against political choices.
-
The negative interpretation of hyper-partisan hashtags, such as #contedimettiti (#ConteResign) or #noicontrosalvini (#WeareagainstSalvini), in tweets without explicit insults or abusive language.
-
The substitution of explicit insults with derogatory words, such as the word “circus” instead of “clowns”.
22The amount of hate labels in the original Policycorpus was 11% (1124 normal and 140 hate tweets), strongly unbalanced like the it-HS corpus (17% of hate tweets), because it reflects the raw distribution of hate tweets in Twitter. The HaSpeeDe-tw corpus (32% of hate tweets) instead has a distribution that oversamples hate tweets and it is better for training hate speech models. Following the HaSpeeDe-tw example, in Policycorpus XL we collected more tweets of hate to reach 40.6% of hate labels and 59.4% of normal labels, as shown in figure 2.
23In the next section, we report and discuss the results of the experiments.
4. Baselines
24In order to set the baselines for the hate speech classification task on Policycorpus-XL, we tested different classification algorithms. We are using a 70% train and 30% test percentage split, the training set shape is 4900 instances and 300 features, while the test set shape is 2100 instances and 300 features. The 300 features are the normalized frequencies of the 300 most frequent words extracted from tweets without removing the stopwords. Table 1 reports the result of classification.
Table 1: Results of classification with different algorithms
algorithm |
balanced acc |
macro F1 |
majority baseline |
0.500 |
0.37 |
naive bayes |
0.783 |
0.78 |
decision trees |
0.763 |
0.76 |
SVMs |
0.788 |
0.80 |
25We used a majority baseline with a dummy classifier that assigns all the instances to the most frequent class (normal tweets), a naive bayes classifier, a decision tree and Support Vector Machines (SVMs). The best performance for the classification of hate speech has been achieved with the SVM classifier, that has a very high precision (0.94) and poor recall (0.60). The results are in line with the scores obtained by the systems on the HaSpeeDe-tw 2020 dataset at EVALITA, and we believe that there is still great room for improvement with the Policycorpus-XL, as we exploited very simple and limited features.
Figure 3

Wordclouds of the unigrams most associated to the normal and hate classes respectively. It shows a substan-tial overlap among the top unigrams in the two classes.
5. Conclusion and Future Work
26We presented a large corpus of Twitter data in Italian, manually annotated with hate speech labels. The corpus is an extension of a previous one, the first corpus annotated with hate speech in the political domain in Italian.
27Given the rising amount of hate messages online, not just against minorities but more and more against policies and policymakers, it is urgent to understand the phenomenon and train classifiers that could prevent people to disseminate hate in the public debate. This is very important to keep democracies alive and grant a free speech that is respectful of other people’s freedom.
28We plan to distribute the corpus in the next edition of EVALITA for a specific HaSpeeDe-tw task.
Bibliographie
Rodrigo Agerri, Roberto Centeno, María Espinosa, Joseba Fernandez de Landa, and Alvaro Rodrigo. 2021. “VaxxStance@IberLEF 2021: Going Beyond Text in Crosslingual Stance Detection.” In Proceedings of the Iberian Languages Evaluation Forum (Iberlef 2021). CEUR-WS.org.
Ahmet Aker, Fabio Celli, Adam Funk, Emina Kurtic, Mark Hepple, and Rob Gaizauskas. 2016. “Sheffield-Trento System for Sentiment and Argument Structure Enhanced Comment-to-Article Linking in the Online News Domain.” Citeseer.
Firoj Alam, Fabio Celli, Evgeny Stepanov, Arindam Ghosh, and Giuseppe Riccardi. 2016. “The Social Mood of News: Self-Reported Annotations to Design Automatic Mood Detection Systems.” In Proceedings of the Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media (Peoples), 143–52.
Pinkesh Badjatiya, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. “Deep Learning for Hate Speech Detection in Tweets.” In Proceedings of the 26th International Conference on World Wide Web Companion, 759–60.
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Nozza Debora, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, Manuela Sanguinetti, and others. 2019. “Semeval-2019 Task 5: Multilingual Detection of Hate Speech Against Immigrants and Women in Twitter.” In 13th International Workshop on Semantic Evaluation, 54–63. Association for Computational Linguistics.
Cristina Bosco, Felice Dell’Orletta, Fabio Poletto, Manuela Sanguinetti, and Maurizio Tesconi. 2018. “Overview of the Evalita 2018 Hate Speech Detection Task.” In EVALITA 2018-Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian, 2263:1–9. CEUR.
Fabio Celli and Luca Polonio. 2015. “Facebook and the Real World: Correlations Between Online and Offline Conversations.” CLiC It, 82.
Fabio Celli, Giuseppe Riccardi, and Arindam Ghosh. 2014. “CorEA: Italian News Corpus with Emotions and Agreement.” In Proceedings of Clic-It 2014, 98–102.
Fabio Celli, Evgeny A Stepanov, Massimo Poesio, and Giuseppe Riccardi. 2016. “Predicting Brexit: Classifying Agreement Is Better Than Sentiment and Pollsters.” In PEOPLES@ Coling, 110–18.
Alessandra Teresa Cignarella, Mirko Lai, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2020. “SardiStance@EVALITA2020: Overview of the Task on Stance Detection in Italian Tweets.” In Proceedings of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (Evalita 2020). Vol. 2765. CEUR Workshop Proceedings. Aachen, Germany: CEUR-WS.org.
Fabio Del Vigna, Andrea Cimino, Felice Dell’Orletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. “Hate Me, Hate Me Not: Hate Speech Detection on Facebook.” In Proceedings of the First Italian Conference on Cybersecurity (Itasec17), 86–95.
Armend Duzha, Cristiano Casadei, Michael Tosi, and Fabio Celli. 2021. “Hate Versus Politics: Detection of Hate Against Policy Makers in Italian Tweets.” SN Social Sciences 1 (9): 1–15.
Karmen Erjavec and Melita Poler Kovačič. 2012. “‘You Don’t Understand, This Is a New War!’ Analysis of Hate Speech in News Web Sites’ Comments.” Mass Communication and Society 15 (6): 899–920.
Fabio Giglietto, Nicola Righetti, Giada Marino, and Luca Rossi. 2019. “Multi-Party Media Partisanship Attention Score. Estimating Partisan Attention of News Media Sources Using Twitter Data in the Lead-up to 2018 Italian Election.” Comunicazione Politica 20 (1): 85–108.
Imane Guellil, Ahsan Adeel, Faical Azouaou, Sara Chennoufi, Hanene Maafi, and Thinhinane Hamitouche. 2020. “Detecting Hate Speech Against Politicians in Arabic Community on Social Media.” International Journal of Web Information Systems.
Rakshita Jain, Devanshi Goel, Prashant Sahu, Abhinav Kumar, and Jyoti Prakash Singh. 2021. “Profiling Hate Speech Spreaders on Twitter.” In CLEF.
Sylvia Jaki and Tom De Smedt. 2019. “Right-Wing German Hate Speech on Twitter: Analysis and Automatic Detection.” arXiv Preprint arXiv:1910.07518.
Irene Kwok and Yuzhou Wang. 2013. “Locate the Hate: Detecting Tweets Against Blacks.” In Proceedings of the Twenty-Seventh Aaai Conference on Artificial Intelligence, 1621–2.
Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019. “Hate Speech Detection: Challenges and Solutions.” PloS One 14 (8): e0221152.
Stefano Menini, Giovanni Moretti, Michele Corazza, Elena Cabrio, Sara Tonelli, and Serena Villata. 2019. “A System to Monitor Cyberbullying Based on Message Classification and Social Network Analysis.” In Proceedings of the Third Workshop on Abusive Language Online, 105–10.
Fabio Poletto, Valerio Basile, Manuela Sanguinetti, Cristina Bosco, and Viviana Patti. 2021. “Resources and Benchmark Corpora for Hate Speech Detection: A Systematic Review.” Language Resources and Evaluation, 1–47.
Fabio Poletto, Marco Stranisci, Manuela Sanguinetti, Viviana Patti, and Cristina Bosco. 2017. “Hate Speech Annotation: Analysis of an Italian Twitter Corpus.” In 4th Italian Conference on Computational Linguistics, Clic-It 2017, 2006:1–6. CEUR-WS.
Francisco Rangel, GLDLP Sarracén, BERTa Chulvi, Elisabetta Fersini, and Paolo Rosso. 2021. “Profiling Hate Speech Spreaders on Twitter Task at Pan 2021.” In CLEF.
Punyajoy Saha, Binny Mathew, Pawan Goyal, and Animesh Mukherjee. 2018. “Hateminers: Detecting Hate Speech Against Women.” arXiv Preprint arXiv:1812.06700.
Manuela Sanguinetti, Gloria Comandini, Elisa Di Nuovo, Simona Frenda, Marco Stranisci, Cristina Bosco, Tommaso Caselli, Viviana Patti, and Irene Russo. 2020. “Overview of the Evalita 2020 Second Hate Speech Detection Task (Haspeede 2).” In Proceedings of the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian (EVALITA 2020), edited by Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. Online: CEUR.org.
Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Marco Stranisci. 2018. “An Italian Twitter Corpus of Hate Speech Against Immigrants.” In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (Lrec 2018).
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A Smith. 2019. “The Risk of Racial Bias in Hate Speech Detection.” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 1668–78.
Kai Shu, Xinyi Zhou, Suhang Wang, Reza Zafarani, and Huan Liu. 2019. “The Role of User Profiles for Fake News Detection.” In Proceedings of the 2019 Ieee/Acm International Conference on Advances in Social Networks Analysis and Mining, 436–39.
Hajime Watanabe, Mondher Bouazizi, and Tomoaki Ohtsuki. 2018. “Hate Speech on Twitter: A Pragmatic Approach to Collect Hateful and Offensive Expressions and Perform Hate Speech Detection.” IEEE Access 6: 13825–35.
Dejin Zhao and Mary Beth Rosson. 2009. “How and Why People Twitter: The Role That Micro-Blogging Plays in Informal Communication at Work.” In Proceedings of the Acm 2009 International Conference on Supporting Group Work, 243–52. ACM.
Notes
1 “a clear #NO to #Netherlands that we would like users of the #MES economic resources but in exchange for Italy’s renunciation of its budgetary autonomy. To Netherlands we say: thank you and goodbye, WE ARE NOT INTERESTED !!”
2 “... There is a weekly catwalk of the #jackal #no #notAtAll! Listening to a Po #clown after a true PATRIOT a doctor from #Bergamo cannot be held, seen or heard. Giletti should stop inviting certain SLACKERS FROM THE PO VALLEY! #COVID-19 #NonelArena”
3 “I have my say ... #Draghi is a great economist but we don’t need a #Monti-style economist ... We don’t need another technical #government to obey the banking lobby! We need a political leader! We need a #ItalExit! We need the #Lira! #No to #DraghiPremier”
Table des illustrations
![]() | |
---|---|
Titre | Figure 1 |
Légende | Wordclouds of the hashtags collected with fre-quency higher than 2. |
URL | http://books.openedition.org/aaccademia/docannexe/image/10505/img-1.jpg |
Fichier | image/jpeg, 108k |
![]() | |
Titre | Figure 2 |
Légende | Distribution of classes in Policycorpus-XL. 0 is the normal class and 1 is the hate class. |
URL | http://books.openedition.org/aaccademia/docannexe/image/10505/img-2.png |
Fichier | image/png, 77k |
![]() | |
Titre | Figure 3 |
Légende | Wordclouds of the unigrams most associated to the normal and hate classes respectively. It shows a substan-tial overlap among the top unigrams in the two classes. |
URL | http://books.openedition.org/aaccademia/docannexe/image/10505/img-3.jpg |
Fichier | image/jpeg, 74k |
© Accademia University Press, 2022