The computational results presented have been achieved in part using the Vienna Scientific Cluster (VSC).
1 Introduction
1Part-of-speech (PoS) tagging is an essential processing stage for virtually all NLP applications. Subsequent tasks, like parsing, namedentity recognition, event detection, and machine translation, often utilise PoS tags, and benefit (directly or indirectly) from accurate tag sequences.
2Actual work on PoS tagging, meanwhile, mainly concentrated on standardized texts for many years, and frequent phenomena in computermediated communication (CMC) and Web corpora such as emoticons, acronyms, interaction words, iteration of letters, graphostylistics, shortenings, addressing terms, spelling variations, and boilerplate (Androutsopoulos, 2007; Bernardini et al., 2008; Beißwenger, 2013) still deteriorate the performance of PoS-taggers (Giesbrecht and Evert, 2009; Baldwin et al., 2013).
3On the other hand, the interest in automatic evaluation of social media texts, in particular for microblogging texts such as tweets, has been growing considerably, and specialised tools for Twitter data have become available for different languages. But Italian completely lacks such resources, both regarding annotated corpora and specific PoS-tagging tools1. To this end, the POS tagging for Italian Social Media Texts (PoSTWITA) task was proposed for EVALITA 2016 concerning the domain adaptation of PoS-taggers to Twitter texts.
4Our system combined word2vec (w2v) word embeddings (WEs) with a single-layer Long Short Term Memory (LSTM) recurrent neural network (RNN) architecture. The sequence of unlabelled w2v representations of words is accompanied by the sequence of n-grams of the word beginnings and endings, and is fed into the RNN which in turn predicts PoS labels.
5The paper is organised as follows: We present our system design in Section 2, the implementation in Section 3, and its evaluation in Section 4. Section 5 concludes with an outlook on possible implementation improvements.
2 Design
6Overall, our design takes inspiration from as far back as Benello et al. (1989) who used four preceding words and one following word in a feedforward neural network with backpropagation for PoS tagging, builds upon the strong foundation laid down by Collobert et al. (2011) for a neural network (NN) architecture and learning algorithm that can be applied to various natural language processing tasks, and ultimately is a variation of Nogueira dos Santos and Zadrozny (2014) who trained a NN for PoS tagging, with characterlevel and WE representations of words.
7Also note that an earlier version of the system was used in Stemle (2016) to participate in the EmpiriST 2015 shared task on automatic linguistic annotation of computer-mediated communication / social media (Beißwenger et al., 2016).
2.1 Word Embeddings
8Recently, state-of-the-art results on various linguistic tasks were accomplished by architectures using neural-network based WEs. Baroni et al. (2014) conducted a set of experiments comparing the popular w2v (Mikolov et al., 2013a; Mikolov et al., 2013b) implementation for creating WEs to other distributional methods with state-of-the-art results across various (semantic) tasks. These results suggest that the word embeddings substantially outperform the other architectures on semantic similarity and analogy detection tasks. Subsequently, Levy et al. (2015) conducted a comprehensive set of experiments and comparisons that suggest that much of the improved results are due to the system design and parameter optimizations, rather than the selected method. They conclude that ”there does not seem to be a consistent significant advantage to one approach over the other”.
9Word embeddings provide high-quality low dimensional vector representations of words from large corpora of unlabelled data, and the representations, typically computed using NNs, encode many linguistic regularities and patterns (Mikolov et al., 2013b).
2.2 Character-Level Sub-Word Information
10The morphology of a word is opaque to WEs, and the relatedness of the meaning of a lemma’s different word forms, i.e. its different string representations, is not systematically encoded. This means that in morphologically rich languages with longtailed frequency distributions, even some WE representations for word forms of common lemmata may become very poor (Kim et al., 2015).
11We agree with Nogueira dos Santos and Zadrozny (2014) and Kim et al. (2015) that subword information is very important for PoS tagging, and therefore we augment the WE representations with character-level representations of the word beginnings and endings; thereby, we also stay language agnostic—at least, as much as possible—by avoiding the need for, often language specific, morphological pre-processing.
2.3 Recurrent Neural Network Layer
12Language Models are a central part of NLP. They are used to place distributions over word sequences that encode systematic structural properties of the sample of linguistic content they are built from, and can then be used on novel content, e.g. to rank it or predict some feature on it. For a detailed overview on language modelling research see Mikolov (2012).
- 2 For an overview see, e.g. Turian et al. (2010).
13A straight-forward approach to incorporate WEs into feature-based language models is to use the embeddings’ vector representations as features2. Having said that, WEs are also used in NN architectures, where they constitute (part of) the input to the network.
14Neural networks consist of a large number of simple, highly interconnected processing nodes in an architecture loosely inspired by the structure of the cerebral cortex of the brain (O’Reilly and Munakata, 2000). The nodes receive weighted inputs through these connections and fire according to their individual thresholds of their shared activation function. A firing node passes on an activation to all successive connected nodes. During learning the input is propagated through the network and the output is compared to the desired output. Then, the weights of the connections (and the thresholds) are adjusted step-wise so as to more closely resemble a configuration that would produce the desired output. After all input cases have been presented, the process typically starts over again, and the output values will usually be closer to the correct values.
15RNNs are NNs where the connections between the elements are directed cycles, i.e. the networks have loops, and this enables them to model sequential dependencies of the input. However, regular RNNs have fundamental difficulties learning long-term dependencies, and special kinds of RNNs need to be used (Hochreiter, 1991); a very popular kind is the so called long short-term memory (LSTM) network proposed by Hochreiter and Schmidhuber (1997).
16Overall, with this design we not only benefit from available labelled data but also from available general or domain-specific unlabelled data.
3 Implementation
17We maintain the implementation in a source code repository at https://github.com/bot-zen/. The version tagged as 1.1 comprises the version that was used to generate the results submitted to the shared task (ST).
18Our system feeds WEs and character-level subword information into a single-layer RNN with a LSTM architecture.
3.1 Word Embeddings
19When computing WEs we take into consideration Levy et al. (2015): they observed that one specific configuration of w2v, namely the skip-gram model with negative sampling (SGNS) ”is a robust baseline. While it might not be the best method for every task, it does not significantly underperform in any scenario. Moreover, SGNS is the fastest method to train, and cheapest (by far) in terms of disk space and memory consumption”. Coincidentally, Mikolov et al. (2013b) also suggest to use SGNS. We incorporate w2v’s original C implementation for learning WEs3 in an independent pre-processing step, i.e. we pre-compute the WEs. Then, we use gensim4, a Python tool for unsupervised semantic modelling from plain text, to load the pre-computed data, and to compute the vector representations of input words for our NN.
3.2 Character-Level Sub-Word Information
20Our implementation uses a one-hot encoding with a few additional features for representing subword information. The one-hot encoding transforms a categorical feature into a vector where the categories are represented by equally many dimensions with binary values. We convert a letter to lower-case and use the sets of ASCII characters, digits, and punctuation marks as categories for the encoding. Then, we add dimensions to represent more binary features like ’uppercase’ (was uppercase prior to conversion), ’digit’ (is digit), ’punctuation’ (is punctuation mark), whitespace (is white space, except the new line character; note that this category is usually empty, because we expect our tokens to not include white space characters), and unknown (other characters, e.g. diacritics). This results in vectors with more than a single one-hot dimension.
3.3 Recurrent Neural Network Layer
21Our implementation uses Keras, a high-level NNs library, written in Python and capable of running on top of either TensorFlow or Theano (Chollet, 2015). In our case it runs on top of Theano, a Python library that allows to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently (The Theano Development Team et al., 2016).
22The input to our network are sequences of the same length as the sentences we process. During training, we group sentences of the same length into batches and process the batches according to sentence length in increasing order. Each single word in the sequence is represented by its subword information and two WEs that come from two sources (see Section 4). For unknown words, i.e. words without a pre-computed WE, we first try to find the most similar WE considering 10 surrounding words. If this fails, the unknown word is mapped to a randomly generated vector representation. In Total, each word is represented by 2, 280 features: two times 500 (WEs), and sixteen times 80 for two 8-grams (word beginning and ending). If words are shorter than 8 characters their 8-grams are zero-padded.
23This sequential input is fed into a LSTM layer that, in turn, projects to a fully connected output layer with softmax activation function. During training we use dropout for the projection into the output layer, i.e. we set a fraction (0.5) of the input units to 0 at each update, which helps prevent overfitting (Srivastava et al., 2014). We use categorical cross-entropy as loss function and backpropagation in conjunction with the RMSprop optimization for learning. At the time of writing, this was the Keras default—or the explicitly documented option to be used—for our type of architecture.
4 Results
24We used our slightly modified implementation to participate in the POS tagging for Italian Social Media Texts (PoSTWITA) shared task (ST) of the 5th periodic evaluation campaign of Natural Language Processing (NLP) and speech tools for the Italian language EVALITA 2016. First, we describe the corpora used for training, and then the specific system configuration(s) for the ST.
4.1 Training Data for w2v and PoS Tagging
4.1.1 DiDi-IT (PoS, w2v)
25didi-it (Frey et al., 2016) (version September 2016) is the Italian sub-part of the DiDi corpus, a corpus of South Tyrolean German and Italian from Facebook (FB) users’ wall posts, comments on wall posts and private messages.
26The Italian part consists of around 100,000 tokens collected from 20 profiles of Facebook users residing in South Tyrol. This version has about 20,000 PoS tags semi-automatically corrected by a single annotator.
27The anonymised corpus is freely available for research purposes.
4.1.2 Italian UD (PoS, w2v)
28Universal Dependencies (UD) is a project that is developing cross-linguistically consistent treebank annotation for many languages5.
29italian-UD6 (version from January 2015) corpus was originally obtained by conversion from ISDT (Italian Stanford Dependency Treebank) and released for the dependency parsing ST of EVALITA 2014 (Bosco et al., 2014). The corpus has semi-automatically converted PoS tags from the original two Italian treebanks, differing both in corpus composition and adopted annotation schemes.
- 7 Creative Commons Attribution-NonCommercialShareAlike 3.0 Unported, i.e. the data can be copied and (...)
30The corpus contains around 317,000 tokens in around 13,000 sentences from different sources and genres. It is available under the CC BY-NCSA 3.07 license.
4.1.3 PoSTWITA (PoS and w2v)
31postwita is the Twitter data made available by the organizers of the ST. It contains Twitter tweets from the EVALITA2014 SENTIPLOC corpus: the development and test set and additional tweets from the same period of time were manually annotated for a global amount of 6438 tweets (114,967 tokens) and were distributed as the development set. The data is PoS tagged according to UD but with the additional insertion of seven Twitterspecific tags. All the annotations were carried out by three different annotators. The data was only distributed to the task participants.
4.1.4 C4Corpus (w2v)
32c4corpus8 is a full documents Italian Web corpus that has been extracted from CommonCrawl, the largest publicly available general Web crawl to date. See Habernal (2016) for details about the corpus construction pipeline, and other information about the corpus.
33The corpus contains about 670m tokens in 22m sentences. The data is available under the CreativeCommons license family.
4.1.5 PAISÀ (w2v)
34paisa (Lyding et al., 2014) is a corpus of authentic contemporary Italian texts from the web (harvested in September/October 2010). It was created in the context of the project PAISÀ (Piattaforma per l’Apprendimento dell’Italiano Su corpora Annotati) with the aim to provide a large resource of freely available Italian texts for language learning by studying authentic text materials.
35The corpus contains about 270m tokens in about 8m sentences. The data is available under the CC BY-NC-SA 3.09 license.
4.2 PoSTWITA shared task
36For the ST we used one overall configuration for the system but three different corpus configurations for training. However, only one corpus configuration was entered into the ST: we used PoS tags from didi-it + postwita (run 1), from italian- UD (run 2), and from both (run 3). For w2v we trained a 500-dimensional skip-gram model on didi-it + italian-UD + postwita that ignored all words with less than 2 occurrences within a window size of 10; it was trained with negative sampling (value 15). We also trained a 500-dimensional skip-gram model on c4corpus + paisa that ignored all words with less than 33 occurrences within a window size of 10; it was trained with negative sampling (value 15).
- 10 -sample 1e-3 -iter 5 -alpha 0.025
37The other w2v parameters were left at their default settings10.
38The evaluation of the systems was done by the organisers on unlabelled but pre-tokenised data (4759 tokens in 301 tweets), and was based on a token-by-token comparison. The considered metric was accuracy, i.e. the number of correctly assigned PoS tags divided by the total number of tokens.
Table 1: Official result(s) of our PoS tagger for the three runs on the PoSTWITA ST data.
(1) didi-it + postwita
|
76.00
|
(2) italian-UD
|
80.54
|
(3) didi-it + postwita + italian-UD
|
81.61
|
Winning Team
|
93.19
|
39We believe, the unexpectedly little performance gain from utilizing the much larger italian-UD data over the rather small didi-it + postwita data may be rooted in the insertion of Twitter-specific tags into the data (see 4.1.3), something we did not account for, i.e. 18, 213 of 289, 416 and more importantly 7, 778 of 12, 677 sentences had imperfect information during training.
5 Conclusion & Outlook
40We presented our submission to the PoSTWITA task of EVALITA 2016, where we participated with moderate results. In the future, we will try to rerun the experiment with training data that takes into consideration the Twitter-specific tags of the task.