• Contenu principal
  • Menu
OpenEdition Books
  • Accueil
  • Catalogue de 15900 livres
  • Éditeurs
  • Auteurs
  • Facebook
  • X
  • Partager
    • Facebook

    • X

    • Accueil
    • Catalogue de 15900 livres
    • Éditeurs
    • Auteurs
  • Ressources numériques en sciences humaines et sociales

    • OpenEdition
  • Nos plateformes

    • OpenEdition Books
    • OpenEdition Journals
    • Hypothèses
    • Calenda
  • Bibliothèques

    • OpenEdition Freemium
  • Suivez-nous

  • Lettre d’information
OpenEdition Search

Redirection vers OpenEdition Search.

À quel endroit ?
  • Accademia University Press
  • ›
  • Collana dell'Associazione Italiana di Li...
  • ›
  • Proceedings of the Eighth Italian Confer...
  • ›
  • Contributed Papers: Long Papers
  • ›
  • Quale testo è scritto meglio? A Study on...
  • Accademia University Press
  • Accademia University Press
    Accademia University Press
    Informations sur la couverture
    Table des matières
    Liens vers le livre
    Informations sur la couverture
    Table des matières
    Formats de lecture

    Plan

    Plan détaillé Texte intégral 1. Introduction 2. Corpus Collection 3. Analysis of Human Judgments 4. Data Analysis 5. Conclusions Bibliographie Notes de bas de page Auteurs

    Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

    Ce livre est recensé par

    Précédent Suivant
    Table des matières

    Quale testo è scritto meglio? A Study on Italian Native Speakers’ Perception of Writing Quality

    Aldo Cerulli, Dominique Brunato et Felice Dell’Orletta

    p. 63-69

    Résumé

    This paper presents a pilot study focused on Italian native speakers’ perception of writing quality. A group of native speakers expressed their preferences on 100 pairs of essays extracted from an Italian corpus of compositions written by L1 students of lower secondary school. Analysing their answers, it was possible to identify a set of linguistic features characterizing essays perceived as well written and to assess the impact of students errors on the perception of text quality. The paper describes the crowdsourcing technique to collect data as well as the linguistic analysis and results.1

    Texte intégral Bibliographie Notes de bas de page Auteurs

    Texte intégral

    1. Introduction

    1The institution of distance learning paradigms, which has become crucial during the Covid-19 pandemic, showed the need to provide schools and universities with Natural Language Processing (NLP)-based tools to assist students, teachers and professors. Nowadays, language technologies are more and more exploited to develop educational applications, such as Intelligent Computer-Assisted Language Learning (ICALL) systems (Granger 2003) and tools for automated essay scoring (Attali and Burstein 2006) or automatic error detection and correction (Ng et al. 2013). A fundamental requirement for developing this kind of applications is the availability of electronically accessible corpora of learners’ productions. Corpora created so far differ in many respects. For instance, considering the types of examined learners, they can gather productions written by L2 students or by native speakers: the former have been built for many languages (e.g. English, Arabic, German, Hungarian, Basque, Czech, Italian), while the latter are mainly available for English. In both cases, a peculiarity of existing corpora is that they are cross-sectional rather than longitudinal. A notable exception in the context of Italian as L1 – which is the focus of our contribution – is represented by CItA (Corpus Italiano di Apprendenti L1), which was jointly developed by the Institute for Computational Linguistics of the Italian National Research Council (CNR) of Pisa and the Department of Social and Developmental Psychology at Sapienza University of Rome (Barbagli et al. 2016): it is the first digitalized collection of essays written by the same group of Italian L1 learners in the first two years of the lower secondary school2.

    2The diachronic and longitudinal nature of CItA makes it particularly suitable to study the evolution of L1 writing competence over the two years, assuming that many remarkable changes in writing skills occur in this period. For instance, in their recent work, Miaschi, Brunato, and Dell’Orletta (2021) showed that it is possible to automatically learn the writing development curve of students: they extracted a wide set of linguistic features from the essays and used them to train a binary classification algorithm able to predict the chronological order of two productions written by the same pupil at different times.

    3The present study ranks among research based on CItA, but chooses a different approach from the one just mentioned: instead of tracking the development of students’ writing competence, we focused on the perception of writing quality by Italian L1 speakers with the aim of understanding whether it is possible to find the linguistic features that are crucially involved in the distinction between ‘better’ and ‘worse’ essays according to our target reader.

    Contributions

    4To the best of our knowledge, this is the first paper that (i) introduces a dataset of evaluated essays in terms of perceived writing quality by means of a crowdsourcing task, (ii) deals with the correlation between linguistic features and perceived quality of writing and (iii) assesses the impact of students errors on quality perception.

    2. Corpus Collection

    5As previously mentioned, the starting point of our study was the CItA corpus. It comprises 1,352 essays, written by 156 pupils of seven lower secondary schools in Rome (three in the historical center and four in the suburbs) during the school years 2012-2013 and 2013-2014. The productions respond to 124 writing prompts that pertain to five textual typologies: reflexive, narrative, descriptive, expository and argumentative. An additional ‘common prompt’ was presented at the end of each school year, in which students were asked to write a letter to advise a younger friend how to compose better essays. The common prompts were aimed at understanding how learners internalize the different writing instructions given by teachers.

    6Each essay contained in CItA is also provided by a set of metadata tracking students’ biographical, sociocultural and sociolinguistic information. Beyond the longitudinal nature, the most significant novelty introduced by CItA regards error annotation, which was manually performed by a middle school teacher according to a new three-level schema including: the macro-class of error (i.e. grammatical, orthographic and lexical); the class of error (i.e. verbs, prepositions, monosyllabes); and the corresponding type of modification required to correct it. More details about the CItA collection are reported in Barbagli et al. (2016).

    2.1 Essay Selection

    7For the purpose of our investigation, we selected 200 essays from CItA to be submitted to human evaluation. The essays ranged from a minimum of 141 tokens to a maximum of 1153 tokens and their average length was 359.4 tokens. Then, to gather judgments on writing quality, we created ten questionnaires, each one consisting of ten pairs of essays of the same grade, and distribute them to native speakers of all ages and cultural background.

    8Table 1 reports the criteria we adopted to select the pairs of essays. As it can be seen, Survey 1 allows the comparison between essays responding to the common prompts written by students attending the first or the second grades. In surveys 2-8, we chosen essays pertaining to the same textual typology – assuming that their similarity with regard to the content could let the annotator focus on stylistic issue to orient their judgment – and paired them according to the school year in which they were written. Instead, essays in questionnaires 9 and 10 were paired according to their number of errors: for each year, we divided the range between the minimum amount of errors (0) and the maximum one (49 for the first year, 43 for the second one) into ten error bins and designed the two surveys choosing a couple of productions for each bin. Surveys comparing essays with a similar amount of errors were meant to understand which categories of errors have a greater impact on human judgment.

    Table 1: Criteria used for pairing the essays and number of essays for each survey

    Survey

    Selection criteria

    Number of pairs

    I year

    II year

    1

    Common prompts

    5

    5

    2

    Narrative

    10

    0

    3

    Narrative

    0

    10

    4

    Reflexive

    10

    0

    5

    Reflexive

    0

    10

    6

    Descriptive

    8

    2

    7

    Expository

    3

    7

    8

    Argumentative

    3

    7

    9

    Error bins

    10

    0

    10

    Error bins

    0

    10

    2.2 Human Evaluation

    9After designing the surveys, we moved on to their implementation using the QuestBase platform3. We defined a three-section structure including the filling-in instructions, the personal data entry form and the essays evaluation pages.

    Figure 1: Comparison of a pair of essays extracted from one of the ten surveys

    Image 1000020100000362000001D8D9062BC56F809D45.png

    10Filling-in instructions. The first section reported the following submission guidelines:

    Ciao! Il presente sondaggio è rivolto a partecipanti di madrelingua italiana. La sua compilazione richiede circa 20 minuti. Pima di proseguire, dando il consenso alla partecipazione, ti spieghiamo in cosa consiste. Nelle pagine che seguono leggerai dieci coppie di temi scritti da studenti del primo e del secondo anno di scuola media. I testi possono contenere un certo numero di errori. Per ciascuna coppia ti chiediamo di indicare quale dei due temi ritieni sia scritto meglio. Non esistono risposte giuste o sbagliate: conta semplicemente quello che pensi! Tieni presente che i temi di una stessa coppia possono trattare argomenti diversi, ma questo non deve influire sul tuo giudizio. La tua partecipazione al sondaggio è completamente libera. Se in qualsiasi momento dovessi cambiare idea e volessi interrompere il test, potrai farlo liberamente. Un’ultima cosa: prima di iniziare il sondaggio, ti chiediamo di darci alcune tue informazioni anagrafiche, che serviranno solo a fini statistici. I dati rimarranno completamente anonimi e in nessun modo le risposte verranno associate alla tua persona. Se hai dubbi, curiosità o proposte di miglioramento, scrivimi all’indirizzo: a.cerulli1@studenti.unipi.it. Buona lettura!

    11For the sake of completeness, we also report an English translation of the same guidelines:

    Hello! This survey is addressed to Italian native speakers. Its submission requires about 20 minutes. By completing it, you give your consent to participation. Before going on, we explain to you what it consists of. In the following pages you will read ten pairs of essays written by Italian L1 learners during the first two years of lower secondary school. The essays may contain linguistic errors. For each pair, you are asked to choose the best written of the two essays. No answers are right or wrong: you only have to express your opinion! Bear in mind that the essays of a pair can concern different topics, but this must not affect your judgment. Your participation to the survey is completely free. You may withdraw from it at any time. Before starting the survey, we ask you to provide some personal information that will be used for statistical purposes. Data will remain completely anonymous and will not be connected to you in any way. If you have doubts, curiosities or improvement proposals, please write me to the address: a.cerulli1@studenti.unipi.it. Have a good read!

    12Personal data entry form. The surveys were obviously anonymous. However, as we mentioned before, we asked the annotators to entry some personal information (age, sex, education) for statistical purposes.

    13Essays evaluation. The third section comprised ten pages, each occupied by two side by side essays and a field to give the answer (Figure 1). The user had to choose the label ‘1’ if they had preferred the first essay, ‘2’ otherwise.

    14After carrying out a pilot study to test the adequacy of the structure as well as the completeness and clearness of the instructions, we started collecting evaluations. Using Linktree4 we added the ten questionnaires links to a single web page and shared its link through WhatsApp, Facebook and Instagram: clicking on it, users were redirected to the page and could access every survey.

    3. Analysis of Human Judgments

    15We collected 223 annotations distributed quite homogeneously among the ten surveys, except for the first one, submitted 28 times. It is worth to focus on the heterogeneous composition of the readers sample. Concerning sex, the large majority of answers (183 units, equal to 82.1%) were given by women, against the 38 (17%) by men; just two people preferred not to specify their gender.

    16Regarding age, we divided the group into six bins (Figure 2). The most frequent class (97 units) was ‘20-24 years’, followed by ‘25-29 years’ (64 units). This means that most readers (72.5%) ranged from 20 to 29 years of age. 35 evaluations (15.8%) were made by natives between 30 and 39 years of age. People belonging to the remaining bins contributed to the task for an overall 11.7%.

    Figure 2: Distribution of annotations with respect to readers’ age bins

    Image 100000000000026D000001BCBDA11E60C18CEAF4.png

    17Finally, Figure 3 shows the distribution of submissions with respect to readers’ education: 91.9% of annotations were given by people holding an academic degree (118 units, equal to 53.2%) or a high school diploma (86 units, equal to 38.7%). 12 annotators (5.4%) had a middle school certificate; 4 (1,8%) held a doctoral degree; the last two indicated a non-specific ‘Other’.

    Figure 3: Distribution of annotations with respect to readers’ education

    Image 10000000000001F5000001666276FA925A2408A2.png

    3.1 Inter-Annotator Agreement

    18At this point, we defined a selection function to discard inaccurate annotations and obtain the same number of coherent annotations for each survey. Thus, we firstly built the average vector of every survey as the set of ten values ‘1’ or ‘2’ chosen according to the most assigned label to each pair of essays; then, we calculated the distance between each survey average vector and all its annotations. We implemented the euclidean metric generalized to the n-dimensional space that computes the distance between two vectors as the square root of the sum of their sizes squared difference:

    19Image 10000000000000720000003C688BE272595F6F1A.jpg                (1)

    20To give relevance to the deviating degree of answers differing from the average, we assigned every pair a weight (wk) equal to the number of times in which the ‘winning’ essay was chosen; then, we computed the weighted distance between annotations and average vectors.

    21Image 10000000000000880000003CCC4300DF01CBED3E.jpg                 (2)

    22Finally, we ranked weighted and unweighted distance values of each survey in ascending order and calculated the Inter-annotator agreement (IAA) of the first 15 and 20 annotations. We implemented Krippendorff’s alpha (α), a coefficient that expresses IAA in terms of observed (Do) and casual (De) disagreement (Krippendorff 2011):

    23Image 10000000000000880000003CCC4300DF01CBED3E.jpg              (3)

    24We noticed that IAA values of the first 15 submissions ordered by their increasing weighted distance were the highest. Thus, we took them into account (150 total annotations) for the analysis and discarded the remaining 735. It is noteworthy that the selection led us to an average IAA of 0.26, that is a much higher value than the initial 0.12. Relying on the selected annotations, we established the ‘winning’ and ‘loser’ essay of each pair.

    4. Data Analysis

    25We carried out two evaluations: a first one was meant to identify which linguistic features impact more on the human assessment of the writing quality; a second one focused on the impact of students errors on annotators’ judgments. In what follows we describe the approach underlying the two perspectives and discuss our most interesting findings.

    4.1 Linguistic Profiling and Stylistic Analysis

    26The first analysis relies on linguistic profiling, a NLP-based methodology in which a large set of linguistically-motivated features automatically extracted from annotated texts are used to obtain a vector-based representation of it. Such representations can be then compared across texts representative of different textual genres and varieties to identify the peculiarities of each (Montemagni 2013; Halteren, n.d.). To perform the analysis, we relied on Profiling-UD6, a recently introduced tool that allows the extraction of a wide set of lexical, morpho-syntactic and syntactic features from texts linguistically annotated according to the Universal Dependencies (UD)7 formalism. These features, described in details in Brunato et al. (2020), have been shown to be involved in many tasks, all related to modeling the form rather than the content of a text, such as the assessment of text readability and linguistic complexity and the identification of stylistic traits of an author or groups of authors.

    27We thus split our annotated corpus into two sections: one comprised all ‘winning’ essays and the other all ‘loser’ ones. Using Profiling-UD, we extracted for each text of the two subsets a feature-based vector representation. For each considered feature we calculated the average value, the standard deviation and the coefficient of variation (\frac{SD}{Avg}) in the two subsets and we assessed whether the variation between mean values was significant using the Wilcoxon rank sum test.

    Table 2: Linguistic features whose average varies significantly between the two subsets

    Feature

    ‘Winning’

    ‘Losers’

    Avg.

    SD

    Avg.

    SD

    n_tokens

    374.9

    127.4

    342.7

    116.3

    ttr_form_chunks_100

    0.72

    0.06

    0.70

    0.06

    upos_dist_NOUN

    16.31

    2.49

    16.98

    2.63

    verbs_tense_dist_Fut

    2.75

    4.37

    2.47

    6.90

    verbs_form_dist_Ger

    3.13

    3.52

    2.32

    3.25

    aux_mood_dist_Sub

    4.41

    7.22

    2.48

    4.51

    n_prepositional_chains

    10.70

    6.28

    9.50

    5.98

    28Table 2 shows the seven linguistic features whose variation turned out to be statistically significant (p-value<0.05), ordered by increasing p-values. It emerges that ‘winning’ essays are on average longer (32.2 tokens more) than the ‘losers’ (n_tokens), a finding that may suggest that longer compositions are evaluated as more reasoned, structured and content-rich. Interestingly, this also reflects the students’ perception of school writing: Barbagli et al. (2015) showed that two of the most frequent suggestions contained in essays that respond to ‘common prompts’ are Leggi/scrivi molto (“Read/write a lot") and Lavora sodo, fai vedere che ti impegni (“Work hard, show your dedication"). Thus, pupils possibly write more so as to show their dedication and get higher grades. Secondly, we noticed that a richer vocabulary (ttr_form_chunks_100) plays a crucial role in native’s judgment. This is in line with another advice of the just mentioned ranking, Usa un vocabolario ricco ed espressivo (“Use a rich and expressive vocabulary"), that reflects teachers’ encouragement to use synonyms in order to write clearer and more readable compositions. Values related to the third feature (upos_dist_NOUN) reveal that ‘loser’ essays present a slightly higher distribution of nouns. A predominant use of nouns is typical of highly informative texts (e.g. newspaper articles, laws), while genres closer to speech contain more verbs (Montemagni 2013). Belonging to the second category, a school essay with fewer nouns is probably perceived as more coherent with its genre. Concerning verbal inflection, ‘better’ productions include, on average, 0.28% more future verbs (verbs_tense_dist_Fut), 0.81% more gerund verbs (verbs_form_dist_Ger) and 1.93% more subjunctive auxiliary verbs (aux_mood_dist_Sub). Verbal tenses differing from present and moods differing from indicative require elevated linguistic skills, which positively influence annotators’ choices. The last feature significantly varying between the two groups is the number of prepositional chains (n_prepositional_chains): ‘winning’ compositions have, on average, 1.2 more of them.

    Table 3: The 10 features that, maximally varying in ‘loser’ essays, are more uniform in the ‘winning’ ones

    Feature

    ‘Winning’

    ‘Losers’

    Avg.

    SD

    Avg.

    SD

    verbs_tense_dist_Fut

    2.75

    4.37

    2.47

    6.90

    dep_dist_cop

    1.85

    0.98

    1.93

    1.24

    dep_dist_flat:foreign

    0.03

    0.14

    0.02

    0.17

    dep_dist_flat:name

    0.31

    0.52

    0.32

    0.79

    dep_dist_det:predet

    0.27

    0.26

    0.24

    0.30

    dep_dist_parataxis

    0.13

    0.21

    0.15

    0.31

    obj_pre

    31.35

    13.02

    30.02

    15.87

    verb_edges_dist_0

    1.23

    1.62

    1.06

    1.74

    verb_edges_dist_1

    13.45

    5.44

    12.48

    6.30

    upos_dist_CCONJ

    4.17

    1.28

    4.51

    1.61

    29A further study was focused on the variability degree of linguistic features in the two essay groups. For each subset, we ordered the features by their increasingly coefficients of variation; then, we calculated the difference between the two rankings in order to identify the features that were maximally uniformly distributed in ‘better’ essays as compared to the ‘worse’ ones (Table 3). It can be noticed that future verbs (verbs_tense_dist_Fut) are very uniformly distributed among ‘better’ essays. We have previously commented that their frequency is higher in the ‘winners’; it proves again that natives interpret the use of complex verbal forms as an indicator of higher skills. Also parataxis distribution (dep_dist_parataxis) is quite uniform in ‘winning’ essays; however, its average value is higher in the ‘loser’ ones. It can be deduced that annotators prefer hypotaxis but this is not surprising: hypotactic periods are more structured and elegant and require refined abilities to be built. The same evidence is given on the morphosyntactic level (upos_dist_CCONJ), since ‘better’ compositions include 0.34% less coordinating conjunctions. Curiously, ‘better’ essays have, on average, 0.1% more foreign terms (dep_dist_flat:foreign); this may suggest that annotators appreciate these expressions. Finally, it is worth highlighting a higher and more uniform percentage of verbs with few modifiers in the ‘winning’ essays (verb_edges_dist_0, verb_edges_dist_1).

    4.2 Students Errors Impact

    30The last analysis was aimed at assessing whether and in what measure students errors impact on human judgments. We counted the pairs of essays whose ‘winning’ composition had a lower number of errors, those in which the ‘loser’ one had more mistakes and those with an equal number of errors. We noticed that essays with fewer errors had won in 56% cases, reaching the 79% if including pairs with the same number of errors. This procedure gave a first empirical answer to our starting question: errors substantially affect human assessment.

    31At this point, we focused on error categories to identify which ones affect more the perception of writing quality. For each category, we calculated the average number of errors and their standard deviation in both subsets; then, relying on Wilcoxon rank sum test, we found out that grammatical and orthographic mistakes vary significantly between the two groups (Table 4). As expected, ‘loser’ essays have, on average, 1.29 more grammatical errors and 0.85 more orthographic errors. It is worth to add that orthographic mistakes variation (p-value=0.007) is more significant than the other (p-value=0.029). This could mean that natives judge deviations in orthography worse than those in grammar. Once again, our findings are in line with Barbagli et al. (2015): Usa una corretta ortografia (“Use correct orthography") is the of the most frequent suggestions given in the second year; moreover, Errori di ortografia (“Orthography errors") occupies the and the position among the most salient terms respectively of the first and the second year. The non-significant variations of lexical (p-value=0.581) and punctuation errors (p-value=0.617) are probably due to their scarce amount in the analysed essays.

    Table 1: Error categories whose average varies significantly between the two subsets

    Category

    ‘Winning’ Essays

    ‘Loser’ Essays

    Avg.

    SD

    Avg.

    SD

    Grammar

    3.28

    5.516

    4.57

    6.126

    Orthography

    3.18

    4.517

    4.03

    4.826

    5. Conclusions

    32We presented a pilot study towards the identification of the linguistic features that are own of well written perceived essays. We collected Italian natives’ preferences on 100 pairs of essays written by L1 students, that we analysed in terms of linguistic profiling and errors distribution. Our results reveal an interesting correspondence between annotators’ judging criteria and writing instructions that L1 learners receive by teachers. Our findings could be interpreted as an indicator of the reliability of our data and, more in general, could suggest the effectiveness of crowdsourcing methods to quickly build large and reliable datasets. Considering the lack of Italian corpora of graded essays, such datasets could be valuable resources for the development of Computer-Assisted Learning Systems.

    33The limited size of our dataset certainly reduced the amount of results. Thus, we have to expand it (i) by collecting more annotations for the already existing surveys and (ii) by creating and distributing new surveys in order to gather judgments on new pairs of essays. Analysis on the enlarged dataset could provide more features that are own of good essays. Following the model of Miaschi, Brunato, and Dell’Orletta (2021), we could use the results to train a classifier that, given a pair of essays, recognizes the best written one.

    34The tool would not presume to replace teachers, but it could be a valuable teaching aid. Students could use it to get an immediate and preliminary self-assessment on their written productions so as to better understand their mistakes and hopefully avoid repeating them. Such tools can be very useful if integrated into educational processes based on distance learning paradigms, which need adequate technological infrastructures to be really efficient.

    Bibliographie

    Des DOI sont automatiquement ajoutés aux références bibliographiques par Bilbo, l’outil d’annotation bibliographique d’OpenEdition. Ces références bibliographiques peuvent être téléchargées dans les formats APA, Chicago et MLA.

    Format

    • APA
    • Chicago
    • MLA
    Granger, S. (2013). Error-tagged Learner Corpora and CALL. Equinox Publishing. https://doi.org/10.1558/cj.v20i3.465-480
    Granger, Sylviane. “Error-Tagged Learner Corpora and CALL”. CALICO Journal. Equinox Publishing, January 14, 2013. doi:10.1558/cj.v20i3.465-480.
    Granger, Sylviane. “Error-Tagged Learner Corpora and CALL”. CALICO Journal, vols. 20, nos. 3, Equinox Publishing, 14 Jan. 2013, pp. 465-80. Crossref, https://doi.org/10.1558/cj.v20i3.465-480.

    Cette bibliographie a été enrichie de toutes les références bibliographiques automatiquement générées par Bilbo en utilisant Crossref.

    Yigal Attali, and Jill Burstein. 2006. “Automated Essay Scoring With E-RaterV. 2.” The Journal of Technology, Learning, and Assessment 4 (3). https://ejournals.bc.edu/index.php/jtla/article/view/1650/1492.

    Alessia Barbagli, Pietro Lucisano, Felice Dell’Orletta, and Giulia Venturi. 2015. “Il Ruolo Delle Tecnologie Del Linguaggio Nel Monitoraggio Dell’evoluzione Delle Abilità Di Scrittura: Primi Risultati.” Italian Journal of Computational Linguistics (IJCoL) 1 (1): 99–117. http://www.italianlp.it/wp-content/uploads/2016/03/IJCL_competenze_linguistiche.pdf.

    Alessia Barbagli, Lucisano Pietro, Felice Dell’Orletta, Simonetta Montemagni, and Giulia Venturi. 2016. “CItA: An L1 Italian Learners Corpus to Study the Development of Writing Competence.” In Proceedings of the Tenth International Conference on Language Resources and Evaluation (Lrec’16), 88–95. Portorož, Slovenia: European Language Resources Association (ELRA). https://aclanthology.org/L16-1014.pdf.

    Dominique Brunato, Andrea Cimino, Felice Dell’Orletta, Simonetta Montemagni, and Giulia Venturi. 2020. “Profiling-UD: A Tool for Linguistic Profiling of Texts.” In Proceedings of the 12th Conference of Language Resources and Evaluation (Lrec 2020), 7145–51. Marseille, France: European Language Resources Association (ELRA). https://aclanthology.org/2020.lrec-1.883.pdf.

    Sylviane Granger. 2003. “Error-Tagged Learner Corpora and CALL: A Promising Synergy.” CALICO Journal 20 (3): 465–80. http://www.sfs.uni-tuebingen.de/~dm/papers/Granger-03.pdf.

    10.1558/cj.v20i3.465-480 :

    Hans van Halteren. n.d. “Linguistic Profiling for Author Recognition and Verification.” In Proceedings of the Association for Computational Linguistics, 200–207.

    Klaus Krippendorff. 2011. “Computing Krippendorff’s Alpha-Reliability.” University of Pennsylvania. https://repository.upenn.edu/cgi/viewcontent.cgi?article=1043&context=asc_papers.

    Alessio Miaschi, Dominique Brunato, and Felice Dell’Orletta. 2021. “A NLP-Based Stylometric Approach for Tracking the Evolution of L1 Written Language Competence.” Journal of Writing Research (JoWR) 13 (1): 71–105. https://www.jowr.org/articles/vol13_1/JoWR_2021_vol13_nr1_Miaschi_et_al.pdf.

    Simonetta Montemagni. 2013. “Tecnologie Linguistico-Computazionali E Monitoraggio Della Lingua Italiana.” Studi Italiani Di Linguistica Teorica E Applicata (SILTA), 145–72. http://www.italianlp.it/wp-content/uploads/2014/04/montemagni_silta_submission_rif.pdf.

    Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, and Joel Tetreault. 2013. “The CoNLL-2013 Shared Task on Grammatical Error Correction.” In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, 1–12. Sofia, Bulgaria: Association for Computational Linguistics. https://aclanthology.org/W13-3601.pdf.

    Notes de bas de page

    1 Copyright  2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

    2 The corpus is freely available for research goals at http://www.italianlp.it/resources/cita-corpus-italiano-di-apprendenti-l1/

    3 https://story.questbase.com/

    4 https://linktr.ee/

    5 The corpus of evaluated essays is available at http://www.italianlp.it/EvaluatedEssays.zip

    6 http://linguistic-profiling.italianlp.it/

    7 https://universaldependencies.org/

    Auteurs

    • Aldo Cerulli

      University of Pisa – a.cerulli1@studenti.unipi.it

    • Dominique Brunato

      Istituto di Linguistica Computazionale “Antonio Zampolli” (ILC–CNR) – ItaliaNLP Lab - www.italianlp.it – dominique.brunato@ilc.cnr.it

    • Felice Dell’Orletta

      Istituto di Linguistica Computazionale “Antonio Zampolli” (ILC–CNR) – ItaliaNLP Lab - www.italianlp.it – felice.dellorletta@ilc.cnr.it

    Précédent Suivant
    Table des matières

    Creative Commons - Attribution - Pas d'Utilisation Commerciale - Pas de Modification 4.0 International - CC BY-NC-ND 4.0

    Le texte seul est utilisable sous licence Creative Commons - Attribution - Pas d'Utilisation Commerciale - Pas de Modification 4.0 International - CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.

    Voir plus de livres
    Proceedings of the Second Italian Conference on Computational Linguistics CLiC-it 2015

    Proceedings of the Second Italian Conference on Computational Linguistics CLiC-it 2015

    3-4 December 2015, Trento

    Cristina Bosco, Sara Tonelli et Fabio Massimo Zanzotto (dir.)

    2015

    Proceedings of the Third Italian Conference on Computational Linguistics CLiC-it 2016

    Proceedings of the Third Italian Conference on Computational Linguistics CLiC-it 2016

    5-6 December 2016, Napoli

    Anna Corazza, Simonetta Montemagni et Giovanni Semeraro (dir.)

    2016

    EVALITA. Evaluation of NLP and Speech Tools for Italian

    EVALITA. Evaluation of NLP and Speech Tools for Italian

    Proceedings of the Final Workshop 7 December 2016, Naples

    Pierpaolo Basile, Franco Cutugno, Malvina Nissim et al. (dir.)

    2016

    Proceedings of the Fourth Italian Conference on Computational Linguistics CLiC-it 2017

    Proceedings of the Fourth Italian Conference on Computational Linguistics CLiC-it 2017

    11-12 December 2017, Rome

    Roberto Basili, Malvina Nissim et Giorgio Satta (dir.)

    2017

    Proceedings of the Fifth Italian Conference on Computational Linguistics CLiC-it 2018

    Proceedings of the Fifth Italian Conference on Computational Linguistics CLiC-it 2018

    10-12 December 2018, Torino

    Elena Cabrio, Alessandro Mazzei et Fabio Tamburini (dir.)

    2018

    EVALITA Evaluation of NLP and Speech Tools for Italian

    EVALITA Evaluation of NLP and Speech Tools for Italian

    Proceedings of the Final Workshop 12-13 December 2018, Naples

    Tommaso Caselli, Nicole Novielli, Viviana Patti et al. (dir.)

    2018

    EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

    EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

    Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian Final Workshop

    Valerio Basile, Danilo Croce, Maria Maro et al. (dir.)

    2020

    Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020

    Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020

    Bologna, Italy, March 1-3, 2021

    Felice Dell'Orletta, Johanna Monti et Fabio Tamburini (dir.)

    2020

    Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

    Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

    Milan, Italy, 26-28 January, 2022

    Elisabetta Fersini, Marco Passarotti et Viviana Patti (dir.)

    2022

    Voir plus de livres
    1 / 9
    Proceedings of the Second Italian Conference on Computational Linguistics CLiC-it 2015

    Proceedings of the Second Italian Conference on Computational Linguistics CLiC-it 2015

    3-4 December 2015, Trento

    Cristina Bosco, Sara Tonelli et Fabio Massimo Zanzotto (dir.)

    2015

    Proceedings of the Third Italian Conference on Computational Linguistics CLiC-it 2016

    Proceedings of the Third Italian Conference on Computational Linguistics CLiC-it 2016

    5-6 December 2016, Napoli

    Anna Corazza, Simonetta Montemagni et Giovanni Semeraro (dir.)

    2016

    EVALITA. Evaluation of NLP and Speech Tools for Italian

    EVALITA. Evaluation of NLP and Speech Tools for Italian

    Proceedings of the Final Workshop 7 December 2016, Naples

    Pierpaolo Basile, Franco Cutugno, Malvina Nissim et al. (dir.)

    2016

    Proceedings of the Fourth Italian Conference on Computational Linguistics CLiC-it 2017

    Proceedings of the Fourth Italian Conference on Computational Linguistics CLiC-it 2017

    11-12 December 2017, Rome

    Roberto Basili, Malvina Nissim et Giorgio Satta (dir.)

    2017

    Proceedings of the Fifth Italian Conference on Computational Linguistics CLiC-it 2018

    Proceedings of the Fifth Italian Conference on Computational Linguistics CLiC-it 2018

    10-12 December 2018, Torino

    Elena Cabrio, Alessandro Mazzei et Fabio Tamburini (dir.)

    2018

    EVALITA Evaluation of NLP and Speech Tools for Italian

    EVALITA Evaluation of NLP and Speech Tools for Italian

    Proceedings of the Final Workshop 12-13 December 2018, Naples

    Tommaso Caselli, Nicole Novielli, Viviana Patti et al. (dir.)

    2018

    EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

    EVALITA Evaluation of NLP and Speech Tools for Italian - December 17th, 2020

    Proceedings of the Seventh Evaluation Campaign of Natural Language Processing and Speech Tools for Italian Final Workshop

    Valerio Basile, Danilo Croce, Maria Maro et al. (dir.)

    2020

    Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020

    Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020

    Bologna, Italy, March 1-3, 2021

    Felice Dell'Orletta, Johanna Monti et Fabio Tamburini (dir.)

    2020

    Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

    Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

    Milan, Italy, 26-28 January, 2022

    Elisabetta Fersini, Marco Passarotti et Viviana Patti (dir.)

    2022

    Voir plus de chapitres

    ISACCO: a corpus for investigating spoken and written language evelopment in Italian school–age children

    Dominique Brunato et Felice Dell’Orletta

    Overview of the EVALITA 2018 Cross-Genre Gender Prediction (GxG) Task

    Felice Dell’Orletta et Malvina Nissim

    DARC-IT: a DAtaset for Reading Comprehension in Italian

    Dominique Brunato, Martina Valeriani et Felice Dell’Orletta

    AcCompl-it @ EVALITA2020: Overview of the Acceptability & Complexity Evaluation Task for Italian

    Dominique Brunato, Cristiano Chesi, Felice Dell’Orletta et al.

    Sentences and Documents in Native Language Identification

    Andrea Cimino, Felice Dell’Orletta, Dominique Brunato et al.

    The CHROME Manifesto: integrating multimodal data into Cultural Heritage Resources

    Francesco Cutugno, Felice Dell’Orletta, Isabella Poggi et al.

    Italian in the Trenches: Linguistic Annotation and Analysis of Texts of the Great War

    Irene De Felice, Felice Dell’Orletta, Giulia Venturi et al.

    Word Embeddings in Sentiment Analysis

    Ruggero Petrolito et Felice Dell’Orletta

    Tandem LSTM-SVM Approach for Sentiment Analysis

    Andrea Cimino et Felice Dell’Orletta

    Overview of the EVALITA 2018 Hate Speech Detection Task

    Cristina Bosco, Felice Dell’Orletta, Fabio Poletto et al.

    TAG-it @ EVALITA2020: Overview of the Topic, Age, and Gender Prediction Task for Italian

    Andrea Cimino, Felice Dell’Orletta et Malvina Nissim

    Quantitative Linguistic Investigations across Universal Dependencies Treebanks

    Chiara Alzetta, Felice Dell’Orletta, Simonetta Montemagni et al.

    Voir plus de chapitres
    1 / 12

    ISACCO: a corpus for investigating spoken and written language evelopment in Italian school–age children

    Dominique Brunato et Felice Dell’Orletta

    Overview of the EVALITA 2018 Cross-Genre Gender Prediction (GxG) Task

    Felice Dell’Orletta et Malvina Nissim

    DARC-IT: a DAtaset for Reading Comprehension in Italian

    Dominique Brunato, Martina Valeriani et Felice Dell’Orletta

    AcCompl-it @ EVALITA2020: Overview of the Acceptability & Complexity Evaluation Task for Italian

    Dominique Brunato, Cristiano Chesi, Felice Dell’Orletta et al.

    Sentences and Documents in Native Language Identification

    Andrea Cimino, Felice Dell’Orletta, Dominique Brunato et al.

    The CHROME Manifesto: integrating multimodal data into Cultural Heritage Resources

    Francesco Cutugno, Felice Dell’Orletta, Isabella Poggi et al.

    Italian in the Trenches: Linguistic Annotation and Analysis of Texts of the Great War

    Irene De Felice, Felice Dell’Orletta, Giulia Venturi et al.

    Word Embeddings in Sentiment Analysis

    Ruggero Petrolito et Felice Dell’Orletta

    Tandem LSTM-SVM Approach for Sentiment Analysis

    Andrea Cimino et Felice Dell’Orletta

    Overview of the EVALITA 2018 Hate Speech Detection Task

    Cristina Bosco, Felice Dell’Orletta, Fabio Poletto et al.

    TAG-it @ EVALITA2020: Overview of the Topic, Age, and Gender Prediction Task for Italian

    Andrea Cimino, Felice Dell’Orletta et Malvina Nissim

    Quantitative Linguistic Investigations across Universal Dependencies Treebanks

    Chiara Alzetta, Felice Dell’Orletta, Simonetta Montemagni et al.

    Accès ouvert

    Accès ouvert

    ePub

    PDF

    PDF du chapitre

    1 Copyright  2021 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

    2 The corpus is freely available for research goals at http://www.italianlp.it/resources/cita-corpus-italiano-di-apprendenti-l1/

    3 https://story.questbase.com/

    4 https://linktr.ee/

    5 The corpus of evaluated essays is available at http://www.italianlp.it/EvaluatedEssays.zip

    6 http://linguistic-profiling.italianlp.it/

    7 https://universaldependencies.org/

    Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

    X Facebook Email

    Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

    Vous pouvez vous connecter à votre bibliothèque à l’adresse suivante : https://freemium.openedition.org/oebooks

    Suggérer l’acquisition à votre bibliothèque

    Si vous avez des questions, vous pouvez nous écrire à access[at]openedition.org

    Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

    Vérifiez si votre bibliothèque a déjà acquis ce livre : authentifiez-vous à OpenEdition Freemium for Books.

    Vous pouvez suggérer à votre bibliothèque d’acquérir un ou plusieurs livres publiés sur OpenEdition Books. N’hésitez pas à lui indiquer nos coordonnées : access[at]openedition.org

    Vous pouvez également nous indiquer, à l’aide du formulaire suivant, les coordonnées de votre bibliothèque afin que nous la contactions pour lui suggérer l’achat de ce livre. Les champs suivis de (*) sont obligatoires.

    Veuillez, s’il vous plaît, remplir tous les champs.

    La syntaxe de l’email est incorrecte.

    Référence numérique du chapitre

    Format

    Cerulli, A., Brunato, D., & Dell’Orletta, F. (2022). Quale testo è scritto meglio? A Study on Italian Native Speakers’ Perception of Writing Quality. In E. Fersini, M. Passarotti, & V. Patti (éds.), Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021. Torino: Accademia University Press. https://doi.org/10.4000/books.aaccademia.10512
    Cerulli, Aldo, Dominique Brunato, et Felice Dell’Orletta. « Quale Testo è Scritto Meglio? A Study on Italian Native Speakers’ Perception of Writing Quality ». In Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-It 2021, édité par Elisabetta Fersini, Marco Passarotti, et Viviana Patti. Torino: Accademia University Press, 2022. doi:10.4000/books.aaccademia.10512.
    Cerulli, Aldo, et al. « Quale Testo è Scritto Meglio? A Study on Italian Native Speakers’ Perception of Writing Quality ». Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-It 2021, édité par Elisabetta Fersini et al., Accademia University Press, 2022, https://doi.org/10.4000/books.aaccademia.10512.

    Référence numérique du livre

    Format

    Fersini, E., Passarotti, M., & Patti, V. (éds.). (2022). Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021. Torino: Accademia University Press. https://doi.org/10.4000/books.aaccademia.10417
    Fersini, Elisabetta, Marco Passarotti, et Viviana Patti, éd. Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-It 2021. Torino: Accademia University Press, 2022. doi:10.4000/books.aaccademia.10417.
    Fersini, Elisabetta, et al., éditeurs. Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-It 2021. Accademia University Press, 2022, https://doi.org/10.4000/books.aaccademia.10417.
    Compatible avec Zotero Zotero

    1 / 3

    Accademia University Press

    Accademia University Press

    • Plan du site
    • Se connecter

    Suivez-nous

    • Facebook
    • Flux RSS

    URL : http://www.aaccademia.it/

    Email : info@aaccademia.it

    Adresse :

    Accademia University Press

    Via Carlo Alberto 55

    I‐10123

    Torino

    Italia

    OpenEdition
    • Candidater à OpenEdition Books
    • Connaître le programme OpenEdition Freemium
    • Commander des livres
    • S’abonner à la lettre d’OpenEdition
    • CGU d’OpenEdition Books
    • Accessibilité : partiellement conforme
    • Données personnelles
    • Gestion des cookies
    • Système de signalement