Version classiqueVersion mobile

EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

 | 
Valerio Basile
, 
Danilo Croce
, 
Maria Maro
, 
et al.

TAG-it: Topic, Age and Gender Prediction

ItaliaNLP @ TAG-IT: UmBERTo for Author Profiling at TAG-it 2020

Daniela Occhipinti, Andrea Tesei, Maria Iacono, Carlo Aliprandi et Lorenzo De Mattei

Résumé

In this paper we describe the systems we used to participate in the task TAG-it of EVALITA 2020. The first system we developed uses linear Support Vector Machine as learning algorithm. The other two systems are based on the pretrained Italian Language Model UmBERTo: one of them has been developed following the Multi-Task Learning approach, while the other following the Single-Task Learning approach. These systems have been evaluated on TAG-it official test sets and ranked first in all the TAG-it subtasks, demonstrating the validity of the approaches we followed.

Texte intégral

1. Introduction

1Author Profiling (AP) is a known Natural Language Processing task consisting in the extraction or the prediction of information about the authors of some disputed documents. Such information can include the age and the gender of the authors. The AP problem is assuming more and more importance in several fields, such as security, forensics, marketing and sales, and so on. For example, in forensics, detecting the age and the gender of the author of a given document can be very helpful for determining whether a person should be considered as a suspect or not; from the marketing and sales’ perspective, companies can understand what kind of people may or not like their products on the basis of the analysis performed on people’s reviews or blog and social network posts (Rangel et al. 2015).

2In the context of EVALITA 2020 (Basile et al. 2020), the periodic evaluation campaign of Natural Language Processing and speech tools for the Italian language, the task TAG-it (Cimino, Felice, and Malvina 2020) is proposed. TAG-it is an AP task in which the goal is to provide a system capable of predicting the gender and the age of the authors of several blog posts and their topics. This task can be considered as a follow-up of the EVALITA 2018’s GxG task (Dell’Orletta and Nissim 2018) in which the goal was the prediction of the author’s gender for Twitter posts, YouTube comments, Children Essays, Diaries and News; in GXG models were trained and tested cross-genre. These two aspects led to scores lower than ones observed in other campaigns and languages. In order to address this problem and get better performances, in TAG-it only blogs’ genre is considered and longer texts are used, since they provide more evidence than tweets and Youtube comments, which are shorter than blog posts. Moreover, with respect to GxG, TAG-it adds the topic control with the aim of evaluating the interaction of topic and lexically rich models on performances in a more direct way than in GxG, in which this was indirectly done via cross-genre prediction. TAG-it is divided in two subtasks: the goal of the first one (Subtask 1) is to classify gender, age and topic at once, while the goal of the second one is to predict age (Subtask 2a) and gender (Subtask 2b) separately and with topic control.

3De Mattei and Cimino (2018) and Cimino et al. (2018) demonstrated the validity of Multi-Task Learning approach to establish the state of the art for several Italian NLP task, in the context of GxG, developed the best system for this task based on Bidirectional LSTM (Bi-LSTMs) trained using a Multi-Task Learning approach. For TAG-it we replicated the same approach: we developed a baseline system based on SVM, and two neural systems, the first one exploiting a Single-Task Learning approach, the second one a Multi-Task Learning approach. Instead of the Bi-LSTM model used by for TAG-it we exploited a deeper neural pretrained language model: BERT (Devlin et al. 2019).

2. Description of the Systems

4We implemented and tested three different systems. Our early experiments were led on a training set and a test set obtained by shuffling and splitting (80% training - 20% test) the training set provided by the organisers in order to analyse the classifiers’ performances on a labeled dataset. At the end of our experiments, we trained our best classifiers on the whole training set and run them on the TAG-it test sets provided by the organisers.

5For our experiments and runs, as a preprocessing phase, we filtered out all posts less than 20 characters in length and labeled each post of the dataset with the corresponding author’s id, gender, age and topic. In Table 1 we report the distributions of the classes of the TAG-it dataset.

Table 1: TAG-it datasets distributions

Train

Test1

Test2a

Test2b

M

15070

315

344

730

F

3113

96

68

69

0-19

2232

39

76

79

20-29

5412

131

189

230

30-39

3569

95

51

134

40-49

3577

69

48

216

50-100

3393

77

48

140

ANIME

3925

97

0

0

AUTO-MOTO

3648

76

0

0

BIKES

468

12

0

0

CELEBRITIES

1063

22

0

0

ENTERTAINMENT

534

9

0

0

MEDICINE-AESTHETICS

370

16

0

0

METAL-DETECTING

1471

26

0

0

NATURE

481

11

0

0

SMOKE

1574

30

0

0

SPORTS

4593

103

0

0

TECHNOLOGY

56

9

0

0

GAMES

0

0

298

298

ROLE-GAMES

0

0

114

114

CLOCKS

0

0

0

387

6As a first step, our systems make their predictions by classifying the three dimensions post by post. Then they use a voting mechanism according to which the gender, the age and the topic of an author are represented by the most frequent values assigned by the classifiers to his/her posts.

7The first system we implemented uses linear Support Vector Machine as learning algorithm and we used different features for predicting the core dimensions of the dataset, the second system is based on a Single-Task Learning BERT model and the third system is based on a Multi-Task Learning BERT model. In particular, we used UmBERTo1, an Italian pretrained Language Model developed by Musixmatch.

8In the following subsections we will describe these systems in detail.

2.1 Support Vector Machine Classifiers

9As regards the system based on three linear SVM statistical models, we used the scikit-learn2 Python library and we conducted several experiments by testing different configurations for feature extraction. In all the experiments we used the TF-IDF vectorizer, but we changed the tokenizer and the n-grams context window. In particular we tested five different kinds of features: character n-grams, word n-grams, lemma n-grams, Part-Of-Speech n-grams and bleached tokens. As regards the bleached tokens features, they were extracted after performing a bleach tokenization consisting in fading out lexicon in favour of an abstract token representation (Goot et al. 2018). The word n-grams, lemma n-grams and Part-Of-Speech n-grams features were extracted by using the linguistic pipeline for the Italian language provided by spaCy3. For the multi-class classification we applied the One-Vs-Rest method (Rennie and Rifkin 2001). In Table 2 we report the performances in terms of micro-average f-score of the SVM models tested in our experiments.

10These results led us to choose the best SVM classifiers for the official runs on the provided test set; analysing them, we can state that the best SVM classifiers tested in our experiments are the following:

  • Topic Detection: One-Vs-Rest Linear SVM using features extracted through a TF-IDF Vectorizer considering character n-grams;

  • Age Detection: One-Vs-Rest Linear SVM using features extracted through a TF-IDF Vectorizer considering lemma n-grams;

  • Gender Detection: Linear SVM using features extracted through a TF-IDF Vectorizer considering word n-grams.

Table 2: SVM classifiers’ micro-average f1-scores on validation set

Gender

Age

Topic

word n-gram

0.933

0.3873

0.7882

char n-gram

0.9284

0.3739

0.8333

lemma n-gram

0.9265

0.4189

0.7928

pos n-gram

0.9223

0.3063

0.3873

bleached words

0.9223

0.3739

0.4775

2.2 Single-Task BERT-based Classifiers

11Our second system consists of three different BERT models and a classifier on top of each of them. More precisely, we used the UmBERTo language model, which was pretrained on a large Italian Corpus: OSCAR (Ortiz Suárez, Romary, and Sagot 2020).

12This language model have 12-layer, 768-hidden, 12-heads, 110M parameters. On top of the language model we added a ReLU classifier (Nair and Hinton 2010). We applied dropout (Srivastava et al. 2014) to prevent overfitting. As loss function we used the sum of loss functions of the three classifiers. For each classifier, we used Cross Entropy as loss function.

13In Table 3 we report the system’s performances in terms of f1-score obtained on the validation set.

Table 3: Single-Task Learning BERT-based system micro-average f1-scores on validation set

f1-score

Gender

0.86

Age

0.35

Topic

0.66

2.3 Multi-task BERT-based Classifier

14Our last system is based on a unique UmBERTo model and three classifiers on top of it, each one responsible of predicting one of the three core dimensions of the dataset according to the Multi-Task Learning approach used in (Cimino, Mattei, and Dell’Orletta 2018). On top of the model we added three ReLU classifiers, we applied the dropout method and we used the sum of the Cross-Entropy loss functions of the three classifiers as loss function.

15In Table 4 we report the system’s performances in terms of f1-score obtained on the validation set.

Table 4: Multi-Task Learning BERT-based system f1-scores on validation set

f1-score

Gender

0.86

Age

0.39

Topic

0.64

3. Results and Evaluation

16We run all our three systems on the test sets provided by the task organisers. The performances of our systems are reported in Table 5.

17For the Task 1 scoring, TAG-it considers two different rankings. The first ranking is obtained using a partial scoring scheme, giving 0 points if no correct predictions are provided for the three dimensions of the dataset, 1/3 points if one out of three correct answers is given, 2/3 points if two out of three correct answers are given and 1 point if all the answers given by the system are correct. The second ranking assigns 0 points if no correct predictions are provided for the three dimensions of the dataset and 1 point if all the answers given by the system are correct. In both cases, the final score is the sum of the points achieved by the system across all the documents normalized with respect to the number of documents in the test set. For the Task 2, the micro-average f-score is used as scoring function.

Table 5: Systems’ performances evaluation with TAG-it metrics

STL-SVM

MTL-BERT

STL-BERT

Task 1 metric 1

0,6626

0,7178

0,7348

Task 1 metric 2

0,253

0,3090

0,3309

Task 2a

0,8519

0,9247

0,9053

Task 2b

0,3742

0,3667

0,4093

18Analysing the scores in Table 5, we can state that the best system in the TAG-it context is the one based on BERT using the Single-Task Learning (STL-BERT) approach, obtaining the best scores in Task 1 and Task 2b (age prediction). In Task 2a, consisting in gender prediction with topic control, the best system is the Multi-Task Learning BERT-based system (MTL-BERT). Hence, the systems based on deeper neural models outperform the systems based on traditional machine learning techniques, i.e. the SVM (STL-SVM).

Task 1

19In order to compare classifiers’ predictions on Task 1 with regard to each dimension and to understand the correlation between labels, we plotted and analysed some distributions.

Figure 1: Task 1, Distributions of the dimensions’ classes in test set and classifiers’ predictions

Figure 1: Task 1, Distributions of the dimensions’ classes in test set and classifiers’ predictions

20In Figure 1, we reported the distribution of the labels in the test set and in the classifiers’ output. As regards the gender prediction (a), we can note that the STL-SVM classifier overestimates the M class, most likely because the M and F classes are very unbalanced in the training set. STL-BERT and MTL-BERT’s distributions, on the contrary, are closer to the test set’s one: in our setting the neural models appear less affected by the imbalance of a training set.

21Observing the distributions of the Age classes in Figure 1 (b), we can observe that for all the three systems the distributions of the labels are not close to the distribution of the test set. The nearest distribution is the one of MTL-BERT’s output.

22Looking at the Topic classes distributions in Figure 1 (c), we can observe, once again, that the SVM-based system’s one is the less close to the test set in that it has the tendency to overestimate the SPORT, ANIME and AUTO-MOTO classes and it does not recognise the BIKES and TECHNOLOGY classes as they are underrepresented in the training set (respectively the 2.574% and the 0.308% of training set). For the same reason, it has difficulties in recognising the classes ENTERTAINMENT, MEDICINE-AESTHETICS and NATURE (which are respectively the 2.937%, 2.035% and 2.645% of the training set). The two BERT-based systems, on the contrary, are less affected by this imbalance of the training set and their predictions reflect more the reality of the test set, even though, as STL-SVM, also MTL-BERT cannot recognise the BIKES and TECHNOLOGY classes.

Figure 2: Task 1, Distributions of the Topic and Age dimensions in test set and classifiers’ predictions

Figure 2: Task 1, Distributions of the Topic and Age dimensions in test set and classifiers’ predictions

23In Figure 2 we report the distribution of the Age classes with respect to the Topic classes. Figure 2 (b) shows that in the STL-SVM’s output the 0-19 age class is only related to the ANIME topic, the age 20-29 is related more or less with all the detected topics, the 30-39 class is mostly related to SMOKE and MEDICINE-AESTHETICS, the 40-49 class to the METAL-DETECTING, AUTO-MOTO and SMOKE topics and the 50-100 class mostly to AUTO-MOTO, SPORTS and CELEBRITIES. This distribution is quite far from the test set one and it seems that the relation between the class 0-19 and the topics is overestimated. In Figure 2 (c), which refers to MTL-BERT, we can note that authors classified as having age 20-29 are predicted to talk mostly about ANIME, CELEBRITIES, NATURE and SPORTS and are less related to ENTERTAINMENT, MEDICINE-AESTHETICS and NATURE topics than in STL-SVM’s output; the relation between the 30-39 class and ENTERTAINMENT and MEDICINE-AESTHETICS categories on one hand, and 50-100 and AUTO-MOTO, MEDICINE-AESTHETICS, METAL-DETECTING, NATURE and SMOKE on the other is stronger than in STL-SVM’s results. Also this distribution, though, is quite far from the test set’s one, even if ages seem to be more distributed than in STL-SVM’s output. As shown in Figure 2 (d), in STL-BERT’s distribution, the age 0-19 seems mostly related to TECHNOLOGY and ANIME. The class BIKES, which has not been recognised by the other systems, is related to the classes 30-39, 40-49 and, mostly, 50-100. As regards the 20-29 class, its relations are quite similar to the ones found in the STL-SVM’s results, except for the class NATURE, which is related also to the ages 0-19, 40-49 and 50-100. Also this distribution is quite far from the test’s one. All the three distributions differ considerably from the test set because systems do not perform well enough in age prediction.

Figure 3: Task 1, Distributions of the Topic and Gender dimensions in test set and classifiers’ predictions

Figure 3: Task 1, Distributions of the Topic and Gender dimensions in test set and classifiers’ predictions

24The distributions of the topics with respect to gender in the test set and the predictions are reported in Figure 3. As shown in the figure, all the three systems results relate the F class mostly to the ANIME topic, as it is also in the test set. In the STL-SVM’s output, though, this relation seems to be overestimated. Moreover, in STL-SVM the F class, besides ANIME, is only related to a much lesser extent to SMOKE. The relation between M and SMOKE seems to be overestimated too with respect to the test set. As regards the F class in MLT-BERT and STL-BERT outputs, topics are more distributed than in STL-SVM, but the nearest to the test set’s one is STL-BERT: MLT-BERT, in fact, seems to overestimate the relation between F and BIKES and ENTERTAINMENT and to underestimate the relation between F and MEDICINE-AESTHETIC and SPORTS. For what concerns the M class in MLT-BERT and STL-BERT distributions, we can state once again that the distribution which is closer to the test set one is given by STL-BERT: STL-SVM, MLT-BERT overestimates the relation between M and SMOKE and NATURE.

Task 2

25The results reported in Table 5 show that for Task 2a (gender prediction with topic control) the best classifier is MLT-BERT. In this subtask, BERT-based systems outperform in a significant way the system based on SVM.

26As regards the Task 2b, consisting in the age prediction, the best metrics belong to the STL-BERT. In the age prediction the gap between all the systems’ metrics is not very high. In this case, in which only the age dimension must be predicted, the best classifier is the one using a Single-Task Learning approach.

4. Conclusions

27In this paper we reported the performances and the results of the systems we used to participate to the TAG-it task of EVALITA 2020. We compared our systems’ performances and noted that in the case in which the goal is to predict topic, age and gender dimensions at once, and in the case in which only the age must be predicted, the best classifier is the one developed using a Single-Task Learning approach and based on transformers. In the case in which the goal is the gender prediction only a Multi-task Learning approach combined with transformers have slightly better performances. These results prove that the proposed systems based on transformers, are more effective than traditional machine learning techniques in topic, age and gender classification achieving the state of the art for TAG-it shared task. Using deep pretrained language models on this task Multi-Task Learning does not provide any relevant boost of performances.

Bibliographie

Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. 2020. “EVALITA 2020: Overview of the 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian.” In Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (Evalita 2020), edited by Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. Online: CEUR.org.

Andrea Cimino, Dell’Orletta Felice, and Nissim Malvina. 2020. “TAG-It@EVALITA2020: Overview of the Topic, Age, and Gender Prediction Task for Italian.” In Proceedings of Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (Evalita 2020), edited by Valerio Basile, Danilo Croce, Maria Di Maro, and Lucia C. Passaro. Online: CEUR.org.

Andrea Cimino, Lorenzo De Mattei, and Felice Dell’Orletta. 2018. “Multi-Task Learning in Deep Neural Networks at EVALITA 2018.” In Proceedings of the Sixth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2018) Co-Located with the Fifth Italian Conference on Computational Linguistics (Clic-It 2018), Turin, Italy, December 12-13, 2018. Vol. 2263. CEUR Workshop Proceedings. CEUR-WS.org. http://ceur-ws.org/Vol-2263/paper013.pdf.

Lorenzo De Mattei and Andrea Cimino. 2018. Multitask learning in deep neural network for sentiment polarity and irony classification. In Proceedings of the 2nd Workshop on Natural Language for Artifi-cial Intelligence, November.

Felice Dell’Orletta, and Malvina Nissim. 2018. “Overview of the Evalita 2018 Cross-Genre Gender Prediction (Gxg) Task.” In Proceedings of the 6th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA’18).

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” http://arxiv.org/abs/1810.04805.

Rob van der Goot, Nikola Ljubesic, Ian Matroos, Malvina Nissim, and Barbara Plank. 2018. “Bleaching Text: Abstract Features for Cross-Lingual Gender Prediction.” In Iryna Gurevych and Yusuke Miyao, Editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers.

Vinod Nair, and Geoffrey E Hinton. 2010. “Rectified Linear Units Improve Restricted Boltzmann Machines.” In ICML.

Pedro Javier Ortiz Suárez, Laurent Romary, and Benoı̂t Sagot. 2020. “A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages.” In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1703–14. Online: Association for Computational Linguistics. https://www.aclweb.org/anthology/2020.acl-main.156.

Francisco Rangel, Fabio Celli, Paolo Rosso, Martin Potthast, Benno Stein, and Walter Daelemans. 2015. “Overview of the 3rd Author Profiling Task at Pan 2015.” In CLEF (P. 2015). Sn.

Jason D. M. Rennie, and Ryan Rifkin. 2001. “Improving Multiclass Text Classification with the Support Vector Machine.”

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” The Journal of Machine Learning Research 15 (1): 1929–58.

Table des illustrations

Titre Figure 1: Task 1, Distributions of the dimensions’ classes in test set and classifiers’ predictions
URL http://books.openedition.org/aaccademia/docannexe/image/7297/img-1.jpg
Fichier image/jpeg, 204k
Titre Figure 2: Task 1, Distributions of the Topic and Age dimensions in test set and classifiers’ predictions
URL http://books.openedition.org/aaccademia/docannexe/image/7297/img-2.jpg
Fichier image/jpeg, 265k
Titre Figure 3: Task 1, Distributions of the Topic and Gender dimensions in test set and classifiers’ predictions
URL http://books.openedition.org/aaccademia/docannexe/image/7297/img-3.jpg
Fichier image/jpeg, 103k

Auteurs

Aptus.AI / Pisa, Italy – daniela@aptus.ai

Aptus.AI / Pisa, Italy – andrea@aptus.ai

Aptus.AI / Pisa, Italy – maria@aptus.ai

Aptus.AI / Pisa, Italy – carlo@aptus.ai

Aptus.AI / Pisa, Italy – Dipartimento di Informatica, Università di Pisa / Pisa, Italy – Istituto di Linguistica Computazionale “Antonio Zampolli” (ILC–CNR), ItaliaNLP Lab / Pisa, Italy – lorenzo.demattei@di.unipi.it

Lire

Open access

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search