Version classiqueVersion mobile

EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

 | 
Valerio Basile
, 
Danilo Croce
, 
Maria Maro
, 
et al.

HaSpeeDe: Hate Speech Detection

By1510 @ HaSpeeDe 2: Identification of Hate Speech for Italian Language in Social Media Data

Tao Deng, Yang Bai et Hongbing Dai

Résumé

Hate speech detection has become a crucial mission in many fields. This paper introduces the system of team By1510. In this work, we participate in the HaSpeeDe 2 (Hate Speech Detection) shared task which is organized within Evalita 2020(The Final Workshop of the 7th evaluation campaign). In order to obtain more abundant semantic information, we combine the original output of BERT-Ita and the hidden state outputs of BERT-Ita. We take part in task A. Our model achieves an F1 score of 77.66% (6/27) in the tweets test set and our model achieves an F1 score of 66.38% (14/27) in the news headlines test set.

Texte intégral

1. Introduction

1With the continuous development of computer and networks, social media users have increased year by year, social media has entered people’s daily life and becomes an indispensable part. More and more people use the Internet to express their opinions and ideas on social media platforms. Some offensive, abusive, defamatory contents are easy to spread and incite hatred, and these negative contents can cause some bad effects. The simplest way is that people mark the report and then delete the system warning, which can not be solved efficiently. Therefore, an efficient way is urgently needed to eliminate these negative effects. This paper proposes a hate speech detection system, which can better detect and mark these annoying contents. The HaSpeeDe 2 (Sanguinetti et al. 2020) (Hate Speech Detection) shared task is organized within Evalita 2020 (Basile et al. 2020), the 7th evaluation campaign of Natural Language Processing and Speech tools for Italian, which help to detect whether the Italian language on Twitter contains hate language, with the aim to reduce the spread of hate speeches and online harassment. (Waseem and Hovy 2016)

Figure 1

Image 1000000000000320000001CDCCDEA675EEA335E8.jpg

Our model. L12_H0 is hidden-state of the first token of the sequence(CLS token) at the output of the 12th hidden layer of the BERT-Ita. Similarly, L11_HO and L10_HO are the 11th and 10th hidden layers outputs of BERT-Ita respectively. [32, 768]/[32, 3072] is the output shape (batch size, hidden size)

2In this paper, we take part in task A in the HaSpeeDe 2 task. The BERT model we use is dbmz1 trained on Italian data. In order to obtain more abundant semantic information, we extract the state of hidden layer outputs and we provide a reference for the detection of the hate speech in the Italian language. The rest of the paper is organized as follows. Section 2 briefly shows the related work for the identification of hate speeches. Section 3 elaborates on our approach. It shows the data set officially provided and architecture of our model. Section 4 describes the hyper-parameters and our results. Finally, Section 5 concludes our work.

2. Related Work

3Previously, machine learning (Davidson et al. 2017; MacAvaney et al. 2019a), Bayesian method (Miok et al. 2020; Fauzi and Yuniarti 2018), support vector machine (MacAvaney et al. 2019b; Del Vigna12 et al. 2017), neural network (Badjatiya et al. 2017; Zhang, Robinson, and Tepper 2018) and other methods were proposed for the identification of hate speech. In the Hindi-English mixed language, (Bohra et al. 2018) et al. in parentheses used a supervised classification system to detect the hate speech in the text in the code-mixed language. The classification system used Character N-Grams, Word N-Grams, Punctuations, Negation Words, Lexicon and other feature vectors for classification and training. The accuracy could reach 71.7% with SVM, which proved to be a very effective method for classification tasks. In Danish language, (Sigurbergsson and Derczynski 2019) developed four automatic classification systems to detect and classify hate speech in English and Danish, and proposed a method to automatically detect different types of the hate speech, which achieved good results for the detection of English and Danish hate speeches. In English language, (Aroyehun and Gelbukh 2018) used a linear baseline classifier (nbsvm with n-grams) and improved deep neural network model.

4For the Italian language, (Polignano et al. 2019) proposed an $AlBERTo$ model based on classifier integration, which was verified by cross validation on Facebook and Twitter data sets, and the effect was obvious in offensive words. (Corazza et al. 2018) used recurrent neural network, n-gram neural network and support vector machine to classify Twitter data sets, and its recurrent model had achieved good results. (Bianchini, Ferri, and Giorni 2018) proposed artificial neural network to annotate and classify 3000 message data from Facebook and Twitter, and achieved good results.

3. Methodology

3.1 Data Description

5In this work, we take part in task A, which is a binary classification task aimed at determining the presence or the absence of hateful content in the text towards a given target (among Immigrants, Muslims or Roma people). The organizers provide the training set and test set. For the training set, it is from Twitter. For the test set, the organizers provide in-domain data and out-of-domain data, which come from Twitter and news headlines, respectively. It can be seen from Table 1 that the data set is slightly imbalanced.

Table 1: Distribution of data set in the Task A

Hate speech (HS)

No HS

train data

2766

4071

test data
(tweets)

622

641

test data
(news headlines)

181

319

Table 2: Hyperparameters of the model in our experiments

Hyperparameters

Our Model

output hidden states=True

max sequences length=100

learning rate=1e-5

adam epsilon=1e-8

per gpu train batch size=32

gradient accumulation steps=1

epoch=8

dropout=0.1

3.2 Our approach

6As the train data is very limited we resort to a transfer learning approach. That is, we take an NLP model pre-trained (Peters et al. 2018; Radford et al. 2018; Devlin et al., 2019) on a large corpus of texts and fine-tune it for a specific task at hand. In this work, we used BERT-base-Italian-uncased(BERT-Ita)2 from Transformers library. It is trained on the recent Wikipedia dump and various texts from the OPUS corpora3 collection. The final training corpus has a size of 13GB and 2050 million tokens. For classification tasks, the output of BERT-Ita (pooler output) is obtained by its last layer hidden state of the first token of the sequence (CLS token) further processed by a linear layer and a Tanh activation function. However, the pooler output is usually not a good summary of the semantic information. Therefore, we extract the hidden layer output of BERT-Ita to obtain more abundant semantic information.

(Jawahar, Sagot, and Seddah 2019) pointed that the hidden layer of BERT encodes a rich hierarchy of linguistic information, with surface features at the bottom layer, syntactic features in the middle layer and semantic features at the top layer. Therefore, we get abundant semantic information by extracting the extra semantic features which is the last three hidden layer outputs(Image 10000000000000E200000011E8F9353E62E35080.jpg) of BERT-Ita. We propose the following model which is shown in Figure 1. In the model, we get Image 10000000000000C500000011FAEB5D87A3A60F54.jpgfrom the top hidden layer of BERT-Ita. We concatenate pooler output, Image 10000000000000E200000011E8F9353E62E35080.jpginto the classifier.

4. Experiments and Results

4.1 Preprocessing and Experiments Setup

7In the experiment, we try to preprocess the text but we did not achieve the desired results. We find that after preprocessing the Twitter data, the F1-score of the model decreased on the validation set. We do not preprocess the data and we do not use an extra data set. In this work, the training set is split into the new training set and the validation set by using the Stratified 5-Fold Cross-validation4.The random seed is set 42 in Cross-validation. Due to the imbalance of datasets, the Stratified 5-Fold Cross-validation ensures that the proportion of samples in each category in each fold data set remains unchanged. During the training, the best weight of the model is saved in 8 epochs. Table 2 shows the hyperparameters used in our model.

4.2 Results and analysis

8In the experiment, we find that with the increase of the extra semantic features, the model can obtain more abundant semantic information. Table 3 shows the performance of the model for different semantic features after getting the labels of the test set.5.

Table 3: The performance of the model for these test sets

Task A
test set of tweets(100%)

Task A
test set of news headlines(100%)

Precision/Recall/Macro F1-score

Precision/Recall/Macro F1-score

BERT-Ita+L12_HO

78.61/78.58/78.54

78.69/62.16/61.18

BERT-Ita+L12_HO+L11_HO

75.50/77.27/77.16

78.13/62.23/62.76

BERT-Ita+L12_HO+L11_HO+L10_HO (Our submitted model)

77.80/77.72/77.66

72.07/65.74/66.38

9The confusion matrices (actual values are represented by rows) are shown in Table 4, Table 5, Table 6. These tables show the performance of the model on the test set as the extra semantic features increase. In the tweets test set, we can see from these tables that the ability of the model to detect the hate speech is increasing as the extra semantic features increase. Similarly, in the news headlines test set, the ability of the model to detect the hate speech is also increasing. We think that with the increase of these extra semantic features, the model can learn more semantic information. In addition, we find that our model achieve good results on the tweets test set, but the results of our model are not good on the news headline data set. There are many differences between the syntactic features of tweets and news headlines. For example, there are many irregular expressions in tweets, while news expressions are very standard. Our model is only fine-tuned on the tweets data set, so we think this affects the performance of the model on other types of data.

Table 4: The confusion matrix of BERTIta+L12_HO in test sets

Task A

test set of tweets(100%)

No HS

HS

No HS

489

152

HS

119

503

Task A

test set of news headlines(100%)

No HS

HS

No HS

312

7

Hs

133

48

Table 5: The confusion matrix of BERTIta+L12_HO+L11_HO in test sets

Task A
test set of tweets(100%)

No HS

HS

No HS

463

178

HS

110

512

Task A
test set of news headlines(100%)

No HS

HS

No HS

310

9

HS

128

53

Table 6: The confusion matrix of BERTIta+L12_HO+L11_HOF+L10_HO in test sets

Task A
test set of tweets(100%)

No HS

HS

No HS

478

163

HS

119

503

Task A
test set of news headlines(100%)

No HS

HS

No HS

289

30

Hs

107

74

5. Conclusion

10In this work, this paper introduces the system proposed for HaSpeeDe 2 shared task for identifying and classifying hate speeches on social media. We enriched BERT-Ita with semantic information by extracting the extra semantic features. We find that with the increase of semantic information, the performance of the model for identifying the hate speech is also increasing. Finally, in the official evaluation, our model rank 6th (6/27) in the tweets test set and 14th (14/27) in the news headlines test set. In the future, we will focus on how to make the model learns more semantic information.

Bibliographie

Aroyehun Segun Taofeek, and Alexander Gelbukh. 2018. “Aggression Detection in Social Media: Using Deep Neural Networks, Data Augmentation, and Pseudo Labeling.” In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (Trac-2018), 90–97.

Badjatiya Pinkesh, Shashank Gupta, Manish Gupta, and Vasudeva Varma. 2017. “Deep Learning for Hate Speech Detection in Tweets.” In Proceedings of the 26th International Conference on World Wide Web Companion, 759–60.

Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. 2020. “EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian.” In Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (Evalita 2020), edited by Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. Online: CEUR.org.

Giulio Bianchini, Lore nzo Ferri, and Tommaso Giorni. 2018. “Text Analysis for Hate Speech Detection in Italian Messages on Twitter and Facebook.” EVALITA Evaluation of NLP and Speech Tools for Italian 12: 250.

Aditya Bohra, Deepanshu Vijay, Vinay Singh, Syed Sarfaraz Akhtar, and Manish Shrivastava. 2018. “A Dataset of Hindi-English Code-Mixed Social Media Text for Hate Speech Detection.” In Proceedings of the Second Workshop on Computational Modeling of Peoples Opinions, Personality, and Emotions in Social Media, 36–41.

Michele Corazza, Stefano Menini, Pinar Arslan, Rachele Sprugnoli, Elena Cabrio, Sara Tonelli, and Serena Villata. 2018. “Comparing Different Supervised Approaches to Hate Speech Detection.” In.

Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. “Automated Hate Speech Detection and the Problem of Offensive Language.” arXiv Preprint arXiv:1703.04009.

Fabio Del Vigna12, Andrea Cimino23, Felice DellOrletta, Marinella Petrocchi, and Maurizio Tesconi. 2017. “Hate Me, Hate Me Not: Hate Speech Detection on Facebook.” In Proceedings of the First Italian Conference on Cybersecurity (Itasec17), 86–95.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June. Associ-ation for Computational Linguistics.

M Ali Fauz, and Anny Yuniarti. 2018. “Ensemble Method for Indonesian Twitter Hate Speech Detection.” Indonesian Journal of Electrical Engineering and Computer Science 11 (1): 294–99.

Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. “What Does BERT Learn About the Structure of Language?” In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 3651–7. Florence, Italy: Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1356.

Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019a. “Hate Speech Detection: Challenges and Solutions.” PloS One 14 (8): e0221152.

Sean MacAvaney, Hao-Ren Yao, Eugene Yang, Katina Russell, Nazli Goharian, and Ophir Frieder. 2019b. “Hate Speech Detection: Challenges and Solutions.” PloS One 14 (8): e0221152.

Kristian Miok, Blaz Skrlj, Daniela Zaharie, and Marko Robnik-Sikonja. 2020. “To Ban or Not to Ban: Bayesian Attention Networks for Reliable Hate Speech Detection.” arXiv Preprint arXiv:2007.05304.

Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. “Deep Contextualized Word Representations.” In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2227–37. New Orleans, Louisiana: Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1202.

Marco Polignano, Pierpaolo Basile, Marco de Gemmis, and Giovanni Semeraro. 2019. “Hate Speech Detection Through Alberto Italian Language Understanding Model.” In NL4AI@ Ai* Ia.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. “Improving Language Understanding by Generative Pre-Training.”

Manuela Sanguinetti, Comandini Gloria, Elisa Di Nuovo, Simona Frenda, Marco Stranisci, Cristina Bosco, Tommaso Caselli, Patti Viviana, and Irene Russo. 2020. “HaSpeeDe 2@EVALITA2020: Overview of the EVALITA 2020 Hate Speech Detection Task.” In Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020), edited by Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. Online: CEUR.org.

Gudbjartur Ingi Sigurbergsson, and Leon Derczynski. 2019. “Offensive Language and Hate Speech Detection for Danish.” arXiv Preprint arXiv:1908.04531.

Zeerak Waseem, and Dirk Hovy. 2016. “Hateful Symbols or Hateful People? Predictive Features for Hate Speech Detection on Twitter.” In Proceedings of the Naacl Student Research Workshop, 88–93.

Ziqi Zhang, David Robinson, and Jonathan Tepper. 2018. “Detecting Hate Speech on Twitter Using a Convolution-Gru Based Deep Neural Network.” In European Semantic Web Conference, 745–60. Springer.

Auteurs

School of Information Science and Engineering, Yunnan University, Yunnan, P.R. China – Dtao.top@gmail.com

School of Information Science and Engineering, Yunnan University, Yunnan, P.R. China – baiyang.top@gmail.com

School of Information Science and Engineering, Yunnan University, Yunnan, P.R. China – hbdai_it@126.com

Lire

Open access

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search