Version classiqueVersion mobile

Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020

 | 
Felice Dell'Orletta
, 
Johanna Monti
, 
Fabio Tamburini

Contributed Papers

AriEmozione: Identifying Emotions in Opera Verses

Francesco Fernicola, Shibingfeng Zhang, Federico Garcea, Paolo Bonora et Alberto Barrón-Cedeño

Résumé

We present a new task: the identification of the emotions transmitted in Italian opera arias at the verse level. This is a relevant problem for the organization of the vast repertoire of Italian Opera arias available and to enable further analyses by both musicologists and the lay public.

We shape the task as a multi-class supervised problem, considering six emotions: love, joy, admiration, anger, sadness, and fear. In order to address it, we manually-annotated an opera corpus with 2.5k verses —which we release to the research community— and experimented with different classification models and representations. Our best-performing models reach macro-averaged F1 measures of ~0.45, always considering character 3-grams representations. Such performance reflects the difficulty of the task at hand, partially caused by the size and nature of the corpus, which consists of relatively short verses written in 18th-century Italian.1

Texte intégral

This research is carried out in the framework of CRICC: Centro di Ricerca per l’interazione con le Industrie Culturali e Creative dell’Università di Bologna; a POR-FESR 2014-2020 Regione Emilia-Romagna project (https://site.unibo.it/cricc).
We thank Ilaria Gozzi and Marco Schillaci, students at Università di Bologna, for their support in the manual annotation of the AriEmozione 1.0 corpus.

1. Introduction

1Opera lyrics have the function of expressing the emotional state of the singing character. In 17th- and 18th-century operas, characters brought on stage passions induced in their souls by the succession of events in the drama. Musicological studies use these affects as one of the interpretative keys of the work as a whole (Zoppelli 2001; McClary 2012). Being able to automatically identify the emotions expressed by the different arias of each work would provide scholars with a useful tool for a systematic study of the repertoire. The technology to identify the emotion(s) expressed by an aria represents an effective tool to study the vast repertoire of arias and characters of this period for musicologists and the lay public alike. As an aria may express more than one emotion, we go one granularity level lower —at the verse level. The task is defined as follows:

Identify the emotion expressed in a verse, in the context of an aria.

2In order to do that we created the AriEmozione 1.0 corpus: a collection of 678 operas with 2.5k verses, each of which has been manually annotated with respect to emotion. We experimented with different supervised models (e.g., SVMs, neural networks) and text (e.g., character n-grams and distributed representations).

3Our experiments show that, regardless of the model, character 3-grams outperform all other representations, reaching weighted macro-averaged F1 measures of ~0.45. Under-represented classes (e.g., fear) are the hardest to identify. Others, such as anger and sadness, being both negative, are often confused between each other.

4The rest of the contribution is distributed as follows. Section 2 describes the AriEmozione 1.0 corpus. Section 3 describe the explored models and representations. Section 4 discusses the experiments and obtained results. Section 5 overviews some related work. Section 6 closes with conclusions and proposals for future work.

2. The AriEmozione 1.0 Corpus

  • 2 CORAGO is the Repertoire and archive of Italian opera librettos. It constitutes the first implement (...)

5The corpus AriEmozione 1.0 is a subset of the materials collected by project CORAGO.2 AriEmozione 1.0 contains a selection of 678 operas composed between 1655 and 1765. We consider the lyrical text in the arias only. A. Zeno and P. Metastasio are among the most represented librettists in the corpus (~30% of the operas); they are two of the most representative and prolific librettists of the 18th century. All texts are written in the 18th century Italian and articulated in verses and stanzas.

Figure 1: Instructions given to the annotators of the emotions in the AriEmozione 1.0 corpus

First of all, thank you for helping with this work. We are a group of researchers from the D. of Classical Philology and Italian Studies and the D. of Interpreting and Translation, both at UniBO. Your work will help us to produce artificial intelligence models to analyse the lyrics in music.

At this stage we are focused on opera. You will annotate arie in Italian from diverse periods, looking for the emotions that they express. Your work consists of identifying the emotion expressed in each of the verses composing an aria. You can choose among six emotions (or none of them), which are defined next: […]

Each row is divided in six columns:

id A unique id, tied to the verse. Do not modify it.

verse A verse, inside of an aria. This is the text that you are going to analyse.

emotion Here you can select the expressed emotion (or none of them)

emotion sec. This is available to choose a secondary emotion, in case it is really difficult to choose just one

confidence Not being 100% sure is ok. If that is the case, please let us know by choosing the right confidence level (default: “I am sure”).

comments Feel free to tell us something about this instance, if you feel like.

6We labeled the emotions transmitted by every single verse, as we observed that this is the right granularity to obtain full text snippets expressing one single emotion. René Descartes wrote in 1649 “Les passions de l’âme”, a sort of compendium of all possible emotions and their possible causes (Garavaglia 2018). For the sake of concreteness, we leveraged Parrott’s  tree of emotions classification. The first level of such tree includes six primary emotions: love, joy, surprise, anger, sadness, and fear. Based on the nature of the material under review, we substitute surprise with admiration, ending with the following six classes:

  • Amore (love) incl. affection, lust, longing.

  • Gioia (joy) incl. cheerfulness, zest, contentment, pride, optimism, enthrallment, relief.

  • Ammirazione (admiration) admiration or adoration of someone’s talent, skill, or other physical or mental qualities.

  • Rabbia (anger) incl. irritability, exasperation, rage, disgust, envy, torment.

  • Tristezza (sadness) incl. suffering, disappointment, shame, neglect, sympathy.

  • Paura (fear) incl. horror and nervousness.

7An extra class nessuna (none) applies mostly to verses with non-actionable words only, neglected in the current experiments.

Table 1: AriEmozione 1.0 corpus statistics

amore

gioia

Ammirazione

rabbia

tristezza

paura

nessuna

total

train

289

274

289

414

503

166

38

1,973

dev

  36

  31

  23

  84

  61

  12

  3

    250

test

  37

  39

  30

  64

  54

  15

 11

    250

overall

362

344

342

562

618

193

52

2,473

8Two native speakers of Italian annotated all 2,473 instances independently considering the instructions displayed in Figure 1. They were asked to include (i) the emotion transmitted by the verse, (ii) an optional secondary label (in case they perceived a second emotion), and (iii) their level of confidence: total confidence, partial confidence, or very doubtful.

Table 2: Instances from the AriEmozione 1.0 corpus, including unique identifier, verse in Italian and English translation, and class. We include free (unofficial) translations for clarity

id

verse

class

ZAP1593570_03

Non ho più lagrime; non ho più voce; non posso piangere; non so parlar

Tristezza

I have no more tears; I have no more voice; I cannot cry; I don’t know how to speak

ZAP1596431_00

Barbaro! Oh dio mi vedi divisa dal mio ben; barbaro, e non concedi ch’io ne dimandi almen

Rabbia

Barbarian! Oh Lord, you see me separated from my own good; barbarian, you don’t even allow me but one demand

ZAP1593766_01

Guardami e tutto obblio e a vendicarti io volo; di quello sguardo solo io mi ricorderò

Amore

Look at me, all else is forgotten and I haste to avenge you; only I shall remember that gaze

ZAP1594229_00

Su la pendice alpina dura la quercia antica e la stagion nemica per lei fatal non è;

Ammirazione

Up on the slope of the mountain the ancient oak tree still lives on, and the adverse season poses no fatal threat

ZAP1596807_00

In questa selva oscura entrai poc’anzi ardito; or nel cammin smarrito timido errando io vo

Paura

I entered this dark forest not too long ago, boldly; having now lost the path I wander around, shyly

ZAP1599979_01

Vede alfin l’amate sponde, vede il porto, e conforto prende allor di riposar

Gioia

Finally, the beloved shores, the harbor, are all in sight and with them come solace and sleep

3

9Table 2 shows examples of verses in the corpus, including one of each of the six emotions.

3. Models and Representations

10The nature of the corpus —a small amount of short verses written in 18th-century Italian— led us to select a humble set of models and representation alternatives. The baseline is a k–Nearest Neighbors algorithm (kNN), considered thanks to its success in classification tasks (Zhang and Zhou 2007). We also experiment with multi-class SVMs, logistic regression, and neural networks. Regarding the latter, we experiment with a number of architectures with two and three hidden layers. Finally, we experiment with a FastText classifier (Joulin et al. 2017). Table 2 summarizes the explored configurations.4

Table 3: Experimental settings overview

Model

Settings

k-NN

.

SVM

and .

Log Reg

Multinomial Logistic Regression with Newton-CG solver.

NN

[32, 64, 96, 128, 256] ( [8, 16, 32, 64, 96]); 20% dropout; ReLu for input/hidden layers; softmax for output layer; categorical cross-entropy loss function; Adam; epochs [1,\ldots15]

FastText

[0.3,0.6,1]; epochs [1,3,5,10,…, 100]

11As for the text representations, we consider TF–IDF vectors of both character 3-grams and word 1-grams (no higher n values are considered due to the corpus dimensions). For pre-processing, we employ the spacy Italian tokenizer5 and casefold the texts. We also explore with dense representations, derived from the TF–IDF vectors, by means of both LDA (Hoffman, Bach, and Blei 2010) and LSA (Halko, Martinsson, and Tropp 2011). In both cases, we target reductions to 16, 32, and 64 dimensions. As for embeddings, we adopted the pre-trained 300-dimensional Italian vectors of FastText (Joulin et al. 2017), and tried with character 3-grams and words.

4. Experiments

12We conducted several experiments to find the best combination of parameters and representations. Given the amount of instances available, we merged the training and development partitions and performed 10-fold cross validation. As standard, the test partition was left aside and only one prediction was carried out on it, after identifying the best configurations.

Table 4: F1 and accuracy on cross-validation held-out test for some of the model/representation combinations.

model

10-fold CV

test

    representation

F1

Acc

F1

Acc

kNN

    char 3-grams

0.38

38.51

0.35

35.15

    words

0.36

36.08

0.35

34.73

    LDA char

0.30

29.97

0.31

30.54

SVM–RBF

    char 3-grams

0.44

43.70

0.43

43.00

    words

0.42

42.00

0.44

44.00

    LDA char

0.28

28.00

0.30

30.00

Log reg

    char 3-grams

0.44

45.57

0.42

43.10

    words

0.41

43.20

0.41

43.10

    LDA char

0.28

30.63

0.29

30.96

2-layers NN

    char 3-grams

0.42

43.61

0.47

46.86

    words

0.42

42.91

0.43

43.10

    LDA char

0.27

29.56

0.27

31.80

3-layers NN

    char 3-grams

0.49

41.86

0.40

41.84

    words

0.47

42.60

0.40

41.84

    LDA char

0.26

31.41

0.30

31.80

FastText

    char 3-grams

0.43

45.00

0.41

42.37

    pre-trained chars

0.43

47.00

0.41

41.00

    words

0.42

42.56

0.39

44.07

    pre-trained words

0.38

41.00

0.40

42.00

13We evaluate our models on the basis of accuracy and weighted macro-averaged F1 measure to account for the class imbalance. Table 4 shows the results obtained with some interesting configurations and representations both for the cross-validation and on the test set.6 Character and word n-grams TF-IDF, LSA, and LDA were tested with all models except for FastText, on which we test with and without pre-trained embeddings. Notice that we are not interested in combining features, but in observing their performance in isolation.

14The most promising representation on cross-validation appears to be the simple character 3-grams, with which we obtained the best results across all models; although it also features the highest variability across folds. Among all 3-gram derived representations, LDA consistently obtained the worst results across all models. Still, it is more stable across folds than the sparse 3-gram representation. As for fastText, with the same epoch number and learning rate, the character 3-gram vectors always achieved much higher accuracy than the word vectors.

15Similar patterns are observed when projecting to the unseen test set. The character 3-grams in general hold the best performance, while the 3-gram LDA tends to remain the worst in spite of the model used. This behavior does not hold in all cases. For instance, the logistic regression model achieves F1=0.44 on cross-validation, but drops to 0.42 on test. This might be the result of over-fitting.

16It is worth noting that all models tend to confuse rabbia and tristezza. Table 5 shows the confusion matrix for the best model on test. These two emotions get confused between each other on an average of 18% of the cases. The classifiers tend to confuse ammirazione for gioia as well, which is understandable given their semantic closeness.

Table 5: Confusion matrix for the 2-layers neural network with TF-IDF character 3-grams

ammirazione

amore

gioia

paura

rabbia

tristezza

ammirazione

0.37

0.03

0.18

0.07

0.11

0.06

amore

0.03

0.43

0.13

0.00

0.09

0.17

gioia

0.27

0.16

0.31

0.20

0.09

0.07

paura

0.10

0.03

0.00

0.40

0.02

0.07

rabbia

0.20

0.14

0.03

0.13

0.64

0.17

tristezza

0.17

0.14

0.13

0.07

0.19

0.48

5. Related Work

17Building on the numerous pre-existing studies focusing on sentiment analysis (Ain et al. 2017; Shi et al. 2019), some researchers have been seeking to dig deeper, towards multi-class emotion analysis. Most of the work thus far has focused on social media (e.g. Twitter). built a classifier for seven emotions: happiness, sadness, anger, love, hate, sarcasm and neutral; i.e. an overlap of five classes with respect to the ones in ariEmozione. In contrast to our experiments, they focused on exploiting the polarity of the words from each instance to be fed to a random forest classifier.

18tried to distinguish among happy, sad, anger, disgust, fear and surprise using WordNet Affect (Valitutti, Strapparava, and Stock 2004). Given that no Word-net-Affect is currently available for Italian, such an approach is unfeasible.

19Promising work has been carried out on news articles (Ye, Xu, and Xu 2012), news headlines (Strapparava and Mihalcea 2007) and children’s narrative (Alm, Roth, and Sproat 2005). While a lexical-based approach is the most frequent to determine the binary positive vs negative classification, combined a high-dimensional word space produced from word TF-IDF vectors with a set of seed words to predict the valence of a text exploiting the syntagmatic relations between words. A bottom-up semantic approach has also been proposed (Seal, Roy, and Basak 2020).

20To the best of our knowledge, no work in the field of either emotion or sentiment analysis has been performed on operas.

6. Conclusions and Future Work

21We addressed the novel problem of emotion classification of opera arias at the verse level. The task is interesting because of the lack of automated tools for the analysis of operas and challenging due to both the language used in 17th- and 18th-century lyrics and the complexity to produce the necessary amount of quality supervised data.

22We explored with various classification models and representations. A neural network with two hidden layers fed with a simple TF-IDF character 3-gram representation is among the most promising approaches to the problem. Among the six possible emotions, the most difficult to identify are rabbia and tristezza, which tend to be confused with each other, followed by ammirazione, which is often confused by gioia. In order to foster the research on this topic, we release the AriEmozione 1.0 corpus to the community (cf. footnote 2).

23As for the future work, we intend to increase the size of the AriEmozione 1.0 corpus by means of active learning (Yang et al. 2009). Once a larger data volume is produced, we plan to explore with models to identify the emotion at the aria rather than at the verse level. Following the theory of emotion proposed by , we could identify the emotion of a whole aria by combining the emotions at the verse level, and then conduct experiments to verify which granularity is more adequate as a single emotion unit. In order to address the issue of emotional polysemy and ambiguity of aria verses, we aim at producing explainable models by highlighting the specific fragments expressing the emotion.

24Another interesting alternative is the one highlighted by Zhao and Ma (2019), who adopted an efficient meta-learning approach to augment the learning ability of emotion distribution; i.e. the intensity values of a set of emotions within a single sentence, when the training dataset is small, as in the AriEmozione 1.0 corpus.

Bibliographie

Ain, Qurat Tul, Mubashir Ali, Amna Riaz, Amna Noureen, Muhammad Kamran, Babar Hayat, and A Rehman. 2017. “Sentiment Analysis Using Deep Learning Techniques: A Review.” Int J Adv Comput Sci Appl 8 (6): 424.

Alm, Cecilia O., Dan Roth, and Richard Sproat. 2005. “Emotions from Text: Machine Learning for Text-Based Emotion Prediction.” In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (Hlt–Emnlp), 579–86.

Fleiss, Joseph L., Jacob Cohen, and B.S. Everitt. 1969. “Large Sample Standard Errors of Kappa and Weighted Kappa.” Psychological Bulletin 72 (5): 323–27.

Garavaglia, Andrea. 2018. “Funzioni Espressive Dell’aria a Metà Seicento Secondo Il "Giasone" Di Cicognini E Cavalli.” Il Saggiatore Musicale Anno XXV (1): 5–31.

Halko, Nathan, Per-Gunnar Martinsson, and Joel A. Tropp. 2011. “Finding Structure with Randomness:Probabilistic Algorithms for Constructing Approximatematrix Decompositions.” SIAM Review 53 (2): 217–88.

Hoffman, Matthew, Francis R. Bach, and David M. Blei. 2010. “Online Learning for Latent Dirichlet Allocation.” In Advances in Neural Information Processing Systems 23, edited by J. D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta, 856–64. Curran Associates, Inc. http://papers.nips.cc/paper/3902-online-learning-for-latent-dirichlet-allocation.pdf.

Joulin, Armand, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. “Bag of Tricks for Efficient Text Classification.” In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, 427–31. ACL.

McClary, Susan. 2012. Desire and Pleasure in Seventeenth-Century Music. 1st ed. Berkeley, CA: University of California Press. http://www.jstor.org/stable/10.1525/j.ctt1ppsj5.

Pompilio, Angelo, Lorenzo Bianconi, Fabio Regazzi, and Paolo Bonora. 2005. “RADAMES: A New Management Approach to Opera: Repertory, Archives and Related Documents.” In Proceedings - First International Conference on Automated Production of Cross Media Content for Multi-Channel Distribution. IEEE. https://doi.org/10.1109/AXMEDIS.2005.42.

Seal, Dibyendu, Uttam K. Roy, and Rohini Basak. 2020. “Sentence-Level Emotion Detection from Text Based on Semantic Rules.” In Information and Communication Technology for Sustainable Development, edited by Milan Tuba, Shyam Akashe, and Amit Joshi, 423–30. Singapore: Springer Singapore.

Shi, Yong, Luyao Zhu, Wei Li, Kun Guo, and Yuanchun Zheng. 2019. “Survey on Classic and Latest Textual Sentiment Analysis Articles and Techniques.” International Journal of Information Technology & Decision Making 18 (04): 1243–87.

Strapparava, Carlo, and Rada Mihalcea. 2007. “SemEval-2007 Task 14: Affective Text.” In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), 70–74. Prague, Czech Republic: Association for Computational Linguistics. https://www.aclweb.org/anthology/S07-1013.

Valitutti, Alessandro, Carlo Strapparava, and Oliviero Stock. 2004. “Developing Affective Lexical Resources.” PsychNology Journal 2 (1).

Yang, Bishan, Jian-Tao Sun, Tengjiao Wang, and Zheng Chen. 2009. “Effective Multi-Label Active Learning for Text Classification.” In Proceedings of the 15th Acm Sigkdd International Conference on Knowledge Discovery and Data Mining, 917–25. KDD ’09. New York, NY: Association for Computing Machinery. https://doi.org/10.1145/1557019.1557119.

Ye, Lu, Rui-Feng Xu, and Jun Xu. 2012. “Emotion Prediction of News Articles from Reader’s Perspective Based on Multi-Label Classification.” In 2012 International Conference on Machine Learning and Cybernetics, 5:2019–24. IEEE.

Zhang, Min-Ling, and Zhi-Hua Zhou. 2007. “ML-Knn: A Lazy Learning Approach to Multi-Label Learning.” Pattern Recognition 40 (7): 2038–48.

Zoppelli, Luca. 2001. “Il Teatro Dell’umane Passioni: Note Sull’antropologia Dell’aria Secentesca.” In I Luoghi Dell’immaginario Barocco. Napoli, Italia: Liguori.

Notes

1 Copyright © 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

2 CORAGO is the Repertoire and archive of Italian opera librettos. It constitutes the first implementation of the RADAMES prototype (Repertoriazione e Archiviazione di Documenti Attinenti al Melodramma E allo Spettacolo) (Pompilio et al. 2005); http://corago.unibo.it.

3 [note1]The corpus is available at https://zenodo.org/record/4022318.

4 The code is available at https://github.com/TinfFoil/AriEmozione. We used Sklearn for the kNN, SVM, and logistic regression models; Keras for the neural networks, and the Facebook-provided library for FastText (cf. https://scikit-learn.org, https://keras.io/, and https://github.com/facebookresearch/fastText).

5 https://spacy.io/models/it

6 The full batch of results is available at https://docs.google.com/spreadsheets/d/1Ztjry2mJs6ufCZM1O5CQRyZ8pA5YDnToN0h0NGX1nW0/edit?usp=sharing

Auteurs

Department of Interpreting and Translation, Università di Bologna, Forlì, Italy – francesco.fernicola@studio.unibo.it

Department of Interpreting and Translation, Università di Bologna, Forlì, Italy – zhang.shibingfeng@studio.unibo.it

Department of Interpreting and Translation, Università di Bologna, Forlì, Italy – federico.garcea2@unibo.it

Department of Classical Philology and Italian Studies, Università di Bologna, Bologna, Italy – paolo.bonora@unibo.it

© Accademia University Press, 2020

Conditions d’utilisation : http://www.openedition.org/6540

Lire

Open access

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search