Version classiqueVersion mobile

Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020

 | 
Felice Dell'Orletta
, 
Johanna Monti
, 
Fabio Tamburini

Contributed Papers

Personal-ITY: A Novel YouTube-based Corpus for Personality Prediction in Italian

Elisa Bassignana, Malvina Nissim et Viviana Patti

Résumé

We present a novel corpus for personality prediction in Italian, containing a larger number of authors and a different genre compared to previously available resources. The corpus is built exploiting Distant Supervision, assigning Myers-Briggs Type Indicator (MBTI) labels to YouTube comments, and can lend itself to a variety of experiments. We report on preliminary experiments on Personal-ITY, which can serve as a baseline for future work, showing that some types are easier to predict than others, and discussing the perks of cross-dataset prediction.

Texte intégral

The work of Elisa Bassignana was partially carried out at the University of Groningen within the framework of the Erasmus+ program 2019/20.

1. Introduction

1When faced with the same situation, different humans behave differently. This is, of course, due to different backgrounds, education paths, and life experiences, but according to psychologists there is another important aspect: personality (Snyder 1983; Parks and Guay 2009).

2Human Personality is a psychological construct aimed at explaining the wide variety of human behaviours in terms of a few, stable and measurable individual characteristics (Vinciarelli and Mohammadi 2014).

3Such characteristics are formalised in Trait Models, and there are currently two of these models that are widely adopted: Big Five (John and Srivastava 1999) and Myers-Briggs Type Indicator (MBTI) (Myers and Myers 1995). The first examines five dimensions (Openness to experience, Conscientiousness, Extroversion, Agreeableness and Neuroticism) and for each of them assigns a score in a range. The second one, instead, considers 16 fixed personality types, coming from the combination of the opposite poles of 4 main dimensions (Extravert-Introvert, iNtuitive-Sensing, Feeling-Thinking, Perceiving-Judging). Examples of full personality types are therefore four letter labels such as ENTJ or ISFP.

4The tests used to detect prevalence of traits include human judgements regarding semantic similarity and relations between adjectives that people use to describe themselves and others. This is because language is believed to be a prime carrier of personality traits (Schwartz et al. 2013). This aspect, together with the progressive increase of available user-generated data on social media, has prompted the task of Personality Detection, i.e., the automatic prediction of personality from written texts (Youyou, Kosinski, and Stillwell 2015; Argamon et al. 2009; Litvinova et al. 2016; Whelan and Davies 2006).

5Personality detection can be useful in predicting life outcomes such as substance use, political attitudes and physical health. Other fields of application are marketing, politics and psychological and social assessment.

6As a contribution to personality detection in Italian, we present Personal-ITY, a new corpus of YouTube comments annotated with MBTI personality traits, and some preliminary experiments to highlight its characteristics and test its potential. The corpus is made available to the community1.

2. Related Work

7There exist a few datasets annotated for personality traits. For the shared tasks organised within the Workshop on Computational Personality Recognition (Celli et al. 2013), two datasets annotated with the Big Five traits have been released in 2013 (Essays (Pennebaker and King 2000) and myPersonality2) and two in 2014 (YouTube Personality Dataset (Biel and Gatica-Perez 2013) and Mobile Phones interactions (Staiano et al. 2012)).

8For the 2015 PAN Author Profiling Shared Task (Pardo et al. 2015), personality was added to gender and age in the profiling task, with tweets in English, Spanish, Italian and Dutch. These are also annotated according to the Big Five model.

9Still in the Big Five landscape, collected a dataset of FaceBook comments (700 millions words) written by 136.000 users who shared their status updates. Interesting correlations were observed between word usage and personality traits.

10If looking at data labelled with the MBTI traits, we find a corpus of 1.2M English tweets annotated with personality and gender (Plank and Hovy 2015), and the multilingual TwiSty (Verhoeven, Daelemans, and Plank 2016). The latter is a corpus of data collected from Twitter annotated with MBTI personality labels and gender for six languages (Dutch, German, French, Italian, Portuguese and Spanish) and a total of 18,168 authors. We are interested in the Italian portion of TwiSty.

11Table 1 contains an overview of the available Italian corpora labelled with personality traits. We include our own, which is described in Section 3.

Table 1: Summary of Italian corpora with personality labels. Avg.: average tokens per user

Corpus

Model

# user

Avg.

PAN2015

Big Five

38

1258

TwiSty

MBTI

490

21.343

Personal-ITY

MBTI

1048

10.585

12Regarding detection approaches, tested the usefulness of different sets of textual features making use of mostly SVMs.

13At the PAN 2015 challenge (see above) a variety of algorithms were tested (such as Random Forests, decision trees, logistic regression for classification, and also various regression models), but overall most successful participants used SVMs. Regarding features, participants approached the task with combinations of style-based and content-based features, as well as their combination in n-gram models (Pardo et al. 2015).

14Experiments on TwiSty were performed by the corpus creators themselves Experiments on TwiSty were performed by the corpus creators themselves Experiments on TwiSty were performed by the corpus creators themselves Experiments on TwiSty were performed by the corpus creators themselves Experiments on TwiSty were performed by the corpus creators themselves corpus creators themselves using a LinearSVM with word (1-2) and character (3-4) n-grams. Their results (reported in Table 2 for the Italian portion of the dataset) are obtained through 10-fold cross-validation; the model is compared to a weighted random baseline (WRB) and a majority baseline (MAJ).

Table 2: TwiSty scores from the original paper. Note that all results are reported as micro-average F-score

Trait

WRB

MAJ

f-score

EI

65.54

77.88

77.78

NS

75.60

85.78

79.21

FT

50.31

53.95

52.13

PJ

50.19

53.05

47.01

Avg

60.41

67.67

64.06

3. Personal-ITY

15First, we explain two major choices that we made in creating Personal-ITY, namely the source of the data and the trait model. Second, we describe in detail the procedure we followed to construct the corpus. Lastly, we provide a description of the resulting dataset.

Data

16YouTube is the source of data for our corpus. The decision is grounded on the fact that compared to the more commonly collected tweets, YouTube comments can be longer, so that users are freer to express themselves without limitations. Additionally, there is a substantial amount of available data on the YouTube platform, which is easy to access thanks to the free YouTube APIs.

Trait Model

17Our model of choice is the MBTI. The first benefit of this decision is that this model is easy to use in association with a Distant Supervision approach (just checking if a message contains one of the 16 personality types; see Section 3.1). Another benefit is related to the existence of TwiSty. Since both TwiSty and Personal-ITY implement the MBTI model, analyses and experiments over personality detection can be carried out also in a cross-domain setting.

Ethics Statement

18Personality profiling must be carefully evaluated from an ethical point of view. In particular, often, personality detection involves ethical dilemmas regarding appropriate utilization and interpretations of the prediction outcomes (Weiner and Greene 2017). Concerns have been raised regarding the inappropriate use of these tests with respect to invasion of privacy, cultural bias and confidentiality (Mehta et al. 2019).

19The data included in the Personal-ITY dataset were publicly available on the YouTube platform at the time of the collection. As we will explain in detail in this Section, the information collected are comments published under public videos on the YouTube platform by authors themselves. For a major protection of user identities, in the released corpus only the YouTube usernames of the authors are mentioned which are not unique identifiers. The YouTube IDs of the corresponding channels, which are the real identifiers in the platform, allowing to trace the identity of the authors, are not released. Note also that the corpus was created for academic purposes and is not intended to be used for commercial deployment or applications.

3.1 Corpus Creation

20The fact that users often self-disclose information about themselves on social media makes it possible to adopt Distant Supervision (DS) for the acquisition of training data. DS is a semi-supervised method that has been abundantly and successfully used in affective computing and profiling to assign silver labels to data on the basis of indicative proxies (Go, Bhayani, and Huang 2009; Pool and Nissim 2016; Emmery, Chrupała, and Daelemans 2017).

21Users left comments to some videos on the MBTI theory in which they were stating their own personality type (e.g. Sono ENTJ...chi altro? [en: "I’m ENTJ...anyone else?"]). We exploited such comments to create Personal-ITY with the following procedure.

22First, we searched for as many Italian YouTube videos about MBTI as possible, ending up with a selection of ten with a conspicuous number of comments as the ones above3.

23Second, we retrieved all the comments to these videos using an AJAX request, and built a list of authors and their associated MBTI label. A label was associated to a user if they included an MBTI combination in one of their comments. Table 3 shows some examples of such associations. The association process is an approximation typical of DS approaches. To assess its validity, we manually checked 300 random comments to see whether the mention of an MBTI label was indeed referred to the author’s own personality. We found that in 19 cases (6.3%) our method led to a wrong or unsure classification of the user’s personality (e.g. O tutti gli INTJ del mondo stanno commentando questo video oppure le statistiche sono sbagliate :-)). We can assume that our dataset might therefore contain about 6-7% of noisy labels.

Table 3: Examples of automatic associations user - MBTI personality type

Comment

User - MBTI label

Io sono ENFJ!!!

User1 - ENFJ

Ho sempre saputo di essere connessa con Lady Gaga! ISFP!

User2 - ISFP

24Using the acquired list of authors, we meant to obtain as many comments as possible written by them. The YouTube API, however, does not allow to retrieve all comments by one user on the platform. In order to get around this problem we relied on video similarities, and tried to expand as much as possible our video collection. Therefore, as a third step, we retrieved the list of channels that feature our initial 10 videos, and then all of the videos within those channels.

25Fourth, through a second AJAX request, we downloaded all comments appearing below all videos retrieved through the previous step.

26Lastly, we filtered all comments retaining those written by authors included in our original list. This does not obviously cover all comments by a relevant user, but it provided us with additional data per author.

3.2 Final Corpus Statistics

27For the final dataset, we decided to keep only the authors with a sufficient amount of data. More specifically, we retained only users with at least five comments, each at least five token long.

28Personal-ITY includes 96,815 comments by 1048 users, each annotated with an MBTI label. The average number of comments per user is 92 and each message has on average 115 tokens.

29The amount of the 16 personality types in the corpus is not uniform. Figure 1 shows such distribution and also compares it with the one in TwiSty. The unbalanced distribution can be due to personality types not being uniformly distributed in the population, and to the fact that different personality types can make different choices about their online presence. for example, observed that there is a significant correlation between online–offline choices and the MBTI dimension of Extravert-Introvert: extroverts are more likely to opt for offline modes of communication, while online communication is presumably easier for introverts. In Figure 1, we also see that the four most frequent types are introverts in both datasets. The conclusion is that, despite the different biases, collecting linguistic data in this way has the advantage that it reflects actual language use and allows large-scale analysis (Plank and Hovy 2015).

Figure 1: Distribution of the 16 personality types in the YouTube corpus and in the Italian section of TwiSty

Image 100000000000034500000213E9669AB84EBD5981.jpg

30Figure 2 shows more in detail, trait by trait, the distribution of the opposite poles through the users in Personal-ITY and in TwiSty. As we might have expected, in line with what is observed in Figure 1, the two datasets present very similar trends. Such similarities between Personal-ITY and TwiSty are these similarities are a further confirmation of the reliability of the data we collected.

Figure 2: Comparison of the distributions of the four MBTI traits between Personal-ITY and the Italian part of TWISTY

Image 10000000000001AC000004BBED5AD6EE13873D8E.jpg

4. Preliminary Experiments

31We ran a series of preliminary experiments on Personali-ITY which can also serve as a baseline for future work on this dataset. We pre-processed texts by replacing hashtags, urls, usernames and emojis with four corresponding placeholders. We adopted the sklearn (Pedregosa et al. 2011) implementation of a linear SVM (LinearSVM), with standard parameters. We tested three types of features. At the lexical level, we experimented with word (1-2) and character (3-4) n-grams, both as raw counts as well as tf-idf weighted. Character n-grams were tested also with a word-boundary option. At a more stylistically level, we considered the use of emojis, hashtags, pronouns, punctuation and capitalisation. Lastly, we also experimented with embeddings-based representations, by using, on the one hand, YouTube-specific (Nieuwenhuis and Nissim 2019) pre-trained models, on the other hand, more generic embeddings, such as the Italian version of GloVe (Pennington, Socher, and Manning 2014), which is trained on the Italian Wikipedia4. We looked for all the available embeddings of the words written by each author, and used the average as feature. If a word appeared more than once in the string of comments, we considered it multiple times in the final average.

  • 5 In Tables 4–5, we report the highest scores based on averages of the four traits. Considering the d (...)

32We used 10-fold cross-validation, and assessed the models using macro f-score. Note that the original TwiSty paper uses micro f-score. Thus, for the sake of comparison, we include also micro-F in Table 5 for the MAJ baseline and our lexical n-gram model. Table 4 shows the results of our experiments with different feature types.5 Overall, lexical features (n-grams) perform best. Combining different feature types did not lead to any improvement. Classification was performed with four separate binary classifiers (one per dimension), and with one single classifier predicting four classes, i.e, the whole MBTI labels at once. In the latter case, we observe that the results are quite high considering the increased difficulty of the task. Table 5 reports the scores of our models on TwiSty. As for Personal-ITY, best results were achieved using lexical features (tf-idf n-grams); stylistic features and embeddings are just above the baseline. Our model outperforms the one in (Verhoeven, Daelemans, and Plank 2016) for all traits (micro-F).

Table 4: Results of the experiments on Personal-ITY. FL: prediction of the full MBTI label at once, with a character n-gram model

Trait

MAJ

Lex

Sty

Emb

FL

EI

40.55

51.85

40.46

40.55

51.65

NS

44.34

51.92

44.34

44.34

49.04

FT

35.01

50.67

36.27

35.01

50.86

PJ

29.49

50.53

51.04

47.06

51.03

Avg

37.35

51.24

43.03

41.74

50.65

  • 6 Prediction of the full label at once.

To test compatibility of resources and to assess model portability, we also ran cross-domain experiments on Personal-ITY and TwiSty. In the first setting, we tested the effect of merging the two datasets on the performance of models for personality detection, maintaining the 10-fold cross-validation setting and by using the model performing better on average for YouTube and Twitter data (a character n-grams model). Table 6 contains the result of such experiments6. Scores are almost always lower compared to the in-domain experiments (excepts for NS as regards Twitter scores reported in Table 5: 46.15 Image 10000000000000110000000B019D36C8E5DBEB8E.jpg48.31), but quite increased compared to the majority baseline.

Table 6: Merging Personal-ITY with TwiSty

Trait

MAJ

Lex

EI

41.64

50.57

NS

44.93

48.31

FT

35.04

51.31

PJ

30.66

48.24

Avg

38.07

49.61

  • 7 Better results can be obtained with other specific models.

33In the second setting, instead, we divided both corpora in fixed training and test sets with a proportion of 80/20 and ran the models using lexical features, in order to run a cross-domain experiment. For direct comparison, we run the model in-domain again using this split. Results are shown in Table 7. Cross-domain scores are obtained with the best in-domain model.7 They drop substantially compared to in-domain, but are always above the baseline.

Table 7: Results of the cross-domain experiments. MAJ = baseline on the cross-domain testset

Train

Personal-ITY

TWISTY

Test

IN

CROSS

IN

CROSS

Pers

MAJ

TWI

TWI

MAJ

Pers

EI

58.94

44.94

49.33

55.66

44.59

44.59

NS

52.88

47.87

47.31

47.87

45.31

45.31

FT

49.20

37.58

47.09

65.26

39.13

51.04

PJ

54.43

32.41

32.50

56.87

36.56

38.54

Avg

53.86

40.70

44.06

56.42

41.40

44.87

5. Conclusions

34The experiments show that there is no single best model for personality prediction, as the feature contribution depends on the dimension considered, and on the dataset. Lexical features perform best, but they tend to be strictly related to the context in which the model is trained and so to overfit.

35The inherent difficulty of the task itself is confirmed and deserves further investigations, as assigning a definite personality is an extremely subjective and complex task, even for humans.

36Personal-ITY is made available to further investigate the above and other issues related to personality detection in Italian. The corpus can lend itself to a psychological analysis of the linguistic cues for the MBTI personality traits. On this line, it is interesting to investigate the presence of evidences linking linguistic features with psychological theories about the four considered dimensions (Extravert-Introvert, iNtuitive-Sensing, Feeling-Thinking, Perceiving-Judging). First results in this direction are presented in (Bassignana, Nissim, and Patti 2020).

Bibliographie

Shlomo Argamon, Moshe Koppel, James W. Pennebaker, and Jonathan Schler. 2009. “Automatically Profiling the Author of an Anonymous Text.” Commun. ACM 52 (2): 119–23. https://doi.org/10.1145/1461928.1461959.

Elisa Bassignana, Malvina Nissim, and Viviana Patti. 2020. “Personal-ITY: a YouTube Comments Corpus for Personality Profiling in Italian Social Media.” In Proceedings of the Third Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media, (Peoples@COLING 2020), edited by Viviana Patti, Malvina Nissim, and Barbara Plank. Association for Computational Linguistics.

Joan-Isaac Biel, and Daniel Gatica-Perez. 2013. “The Youtube Lens: Crowdsourced Personality Impressions and Audiovisual Analysis of Vlogs.” Multimedia, IEEE Transactions on 15 (1): 41–55.

Fabio Celli, Fabio Pianesi, David Stillwell, and Michal Kosinski. 2013. “Workshop on Computational Personality Recognition: Shared Task.” In Seventh International Aaai Conference on Weblogs and Social Media.

Chris Emmery, Grzegorz Chrupała, and Walter Daelemans. 2017. “Simple Queries as Distant Labels for Predicting Gender on Twitter.” In Proceedings of the 3rd Workshop on Noisy User-Generated Text, 50–55. Copenhagen, Denmark: Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-4407.

Alec Go, Richa Bhayani, and Lei Huang. 2009. “Twitter Sentiment Classification Using Distant Supervision.” CS224N Project Report, Stanford 1 (12): 2009.

Oliver P. John, and Sanjay Srivastava. 1999. “The Big Five Trait Taxonomy: History, Measurement, and Theoretical Perspectives.” In Handbook of Personality: Theory and Research, edited by L. A. Pervin and O. P. John, 102–38. Guilford Press.

Tatiana Litvinova, P. Seredin, Olga Litvinova, and Olga Zagorovskaya. 2016. “Profiling a Set of Personality Traits of Text Author: What Our Words Reveal About Us.” Research in Language 14 (December). https://doi.org/10.1515/rela-2016-0019.

Yash Mehta, Navonil Majumder, Alexander Gelbukh, and Erik Cambria. 2019. “Recent Trends in Deep Learning Based Personality Detection.” Artificial Intelligence Review, 1–27.

I. B. Myers, and P. B. Myers. 1995. Gifts Differing: Understanding Personality Type. Mobius. https://books.google.it/books?id=D3V-AAAAMAAJ.

Moniek Nieuwenhuis, and Malvina Nissim. 2019. “The Contribution of Embeddings to Sentiment Analysis on YouTube.” In Proceedings of the Sixth Italian Conference on Computational Linguistics, Bari, Italy, November 13-15, 2019. Vol. 2481. CEUR Workshop Proceedings. CEUR-WS.org.

Francisco M. Rangel Pardo, Fabio Celli, Paolo Rosso, Martin Potthast, Benno Stein, and Walter Daelemans. 2015. “Overview of the 3rd Author Profiling Task at PAN 2015.” In Working Notes of CLEF 2015 - Conference and Labs of the Evaluation Forum, Toulouse, France, September 8-11, 2015. Vol. 1391. CEUR Workshop Proceedings. CEUR-WS.org.

Laura Parks, and Russell P Guay. 2009. “Personality, Values, and Motivation.” Personality and Individual Differences 47 (7): 675–84.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, et al. 2011. “Scikit-Learn: Machine Learning in Python.” Journal of Machine Learning Research 12: 2825–30.

James Pennebaker, and Laura King. 2000. “Linguistic Styles: Language Use as an Individual Difference.” Journal of Personality and Social Psychology 77 (January): 1296–1312. https://doi.org/10.1037//0022-3514.77.6.1296.

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. “GloVe: Global Vectors for Word Representation.” In Empirical Methods in Natural Language Processing (Emnlp), 1532–43. http://www.aclweb.org/anthology/D14-1162.

Barbara Plank, and Dirk Hovy. 2015. “Personality Traits on Twitter—or—How to Get 1,500 Personality Tests in a Week.” In Proceedings of the 6th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, 92–98. Lisboa, Portugal: Association for Computational Linguistics. https://doi.org/10.18653/v1/W15-2913.

Chris Pool, and Malvina Nissim. 2016. “Distant Supervision for Emotion Detection Using Facebook Reactions.” In Proceedings of the Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media (PEOPLES), 30–39. Osaka, Japan: The COLING 2016 Organizing Committee. https://www.aclweb.org/anthology/W16-4304.

H. Andrew Schwartz, Johannes C. Eichstaedt, Margaret L. Kern, Lukasz Dziurzynski, Stephanie M. Ramones, Megha Agrawal, Achal Shah, et al. 2013. “Personality, Gender, and Age in the Language of Social Media: The Open-Vocabulary Approach.” PloS One 8 (9): e73791.

Mark Snyder. 1983. “The Influence of Individuals on Situations: Implications for Understanding the Links Between Personality and Social Behavior.” Journal of Personality 51 (3): 497–516.

Jacopo Staiano, Bruno Lepri, Nadav Aharony, Fabio Pianesi, Nicu Sebe, and Alex Pentland. 2012. “Friends Don’t Lie - Inferring Personality Traits from Social Network Structure.” In UbiComp’12 - Proceedings of the 2012 Acm Conference on Ubiquitous Computing, 321–30. https://doi.org/10.1145/2370216.2370266.

Ben Verhoeven, Walter Daelemans, and Barbara Plank. 2016. “TwiSty: A Multilingual Twitter Stylometry Corpus for Gender and Personality Profiling.” In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), 1632–7. Portorož, Slovenia: European Language Resources Association (ELRA). https://www.aclweb.org/anthology/L16-1258.

Alessandro Vinciarelli, and Gelareh Mohammadi. 2014. “A Survey of Personality Computing.” IEEE Transactions on Affective Computing 5 (3): 273–91.

Irving B. Weiner, and Roger L. Greene. 2017. “Ethical Considerations in Personality Assessment.” In Handbook of Personality Assessment, Second Edition, 59–74. Wiley. https://doi.org/10.1002/9781119311263.ch4.

Susan Whelan, and Gary Davies. 2006. “Profiling Consumers of Own Brands and National Brands Using Human Personality.” Journal of Retailing and Consumer Services 13 (6): 393–402.

Wu Youyou, Michal Kosinski, and David Stillwell. 2015. “Computer-Based Personality Judgments Are More Accurate Than Those Made by Humans.” Proceedings of the National Academy of Sciences 112 (4): 1036–40.

© Accademia University Press, 2020

Conditions d’utilisation : http://www.openedition.org/6540

Lire

Open access

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search