1 Introduction

1Sentiment analysis is a fundamental problem aiming to give a machine the ability to understand the emotions and opinions expressed in a written text. This is an extremely challenging task due to the complexity and variety of human language.

2The sentiment polarity classification task of EvalItalia-20161 (sentipolc) consists of three subtasks which cover different aspects of sentiment detection: T1 : Subjectivity detection: is the tweet subjective or objective? T2 : Polarity detection: is the sentiment of the tweet neutral, positive, negative or mixed? T3 : Irony detection: is the tweet ironic?

3The classic approaches to sentiment analysis usually consist of manual feature engineering and applying some sort of classifier on these features (Liu, 2015). Deep neural networks have shown great promises at capturing salient features for these complex tasks (Mikolov et al., 2013b; Severyn and Moschitti, 2015a). Particularly successful for sentiment classification were Convolutional Neural Networks (CNN) (Kim, 2014; Kalchbrenner et al., 2014; Severyn and Moschitti, 2015b; Johnson and Zhang, 2015), on which our work builds upon. These networks typically have a large number of parameters and are especially effective when trained on large amounts of data.

4In this work, we use a distant supervision approach to leverage large amounts of data in order to train a 2-layer CNN2. More specifically, we train a neural network using the following threephase procedure: P1 : creation of word embeddings for the initialization of the first layer based on an unsupervised corpus of 300M Italian tweets; P2 : distant supervised phase, where the network is pre-trained on a weakly labelled dataset of 40M tweets where the network weights and word embeddings are trained to capture aspects related to sentiment; and P3 : supervised phase, where the network is trained on the provided supervised training data consisting of 7410 manually labelled tweets.

5As the three tasks of EvalItalia-2016 are closely related we apply a form of multitask training as proposed by (Collobert et al., 2011), i.e. we train one CNN for all the tasks simultaneously. This has two advantages: i) we need to train only one model instead of three models, and ii) the CNN has access to more information which benefits the score. The experiments indicate that the multi-task CNN performs better than the single-task CNN. After a small bugfix regarding the data-preprocessing our system outperforms all the other systems in the sentiment polarity task.

2 Convolutional Neural Networks

6We train a 2-layer CNN using 9-fold cross-validation and combine the outputs of the 9 resulting classifiers to increase robustness. The 9 classifiers differ in the data used for the supervised phase since cross-validation creates 9 different training and validation sets.

7The architecture of the CNN is shown in Figure 1 and described in detail below.

8Sentence model. Each word in the input data is associated to a vector representation, which consists in a d-dimensional vector. A sentence (or tweet) is represented by the concatenation of the representations of its n constituent words. This yields a matrix S ∈ , which is used as input yto the convolutional neural network.

9The first layer of the network consists of a lookup table where the word embeddings are represented as a matrix X ∈ , where V is the vocabulary. Thus the i-th column of X represents the i-th word in the vocabulary V .

10Convolutional layer. In this layer, a set of m filters is applied to a sliding window of length h over each sentence. Let S_[i:i+h] denote the concatenation of word vectors s_i to s_i+h. A feature c_i is generated for a given filter F by:

11A concatenation of all vectors in a sentence produces a feature vector c ∈ . The vectors c are then aggregated over all m filters into a feature map matrix C ∈ . The filters are learned during the training phase of the neural network using a procedure detailed in the next section.

12Max pooling. The output of the convolutional layer is passed through a non-linear activation function, before entering a pooling layer. The latter aggregates vector elements by taking the maximum over a fixed set of non-overlapping intervals. The resulting pooled feature map matrix has the form: C_pooled , where s is the length of each interval. In the case of overlapping intervals with a stride value s_t, the pooled feature map matrix has the form C_pooled ∈ . Depending on whether the borders are included or not, the result of the fraction is rounded up or down respectively.

13Hidden layer. A fully connected hidden layer computes the transformation α(W ∗ x+b), where W ∈ is the weight matrix, b ∈ the bias, and α the rectified linear (relu) activation function (Nair and Hinton, 2010). The output vector of this layer, x ∈ , corresponds to the sentence embeddings for each tweet.

14Softmax. Finally, the outputs of the hidden layer x ∈ are fully connected to a soft-max regression layer, which returns the class yˆ [1, K] with largest probability,

where w_j denotes the weights vector of class j and a_j the bias of class j.

15Network parameters. Training the neural network consists in learning the set of parameters Θ = X, F₁, b₁, F₂, b₂, W, a , where X is the embedding matrix, with each row containing the d-dimensional embedding vector for a specific word; F_i, b_i(i = 1, 2 ) the filter weights and biases of the first and second convolutional layers; W the concatenation of the weights w_j for every output class in the soft-max layer; and a the bias of the soft-max layer.

16Hyperparameters For both convolutional layers we set the length of the sliding window h to 5, the size of the pooling interval s is set to 3 in both layers, where we use a striding of 2 in the first layer, and the number of filters m is set to 200 in both convolutional layers.

17Dropout Dropout is an alternative technique used to reduce overfitting (Srivastava et al., 2014). In each training stage individual nodes are dropped with probability p, the reduced neural net is updated and then the dropped nodes are reinserted. We apply Dropout to the hidden layer and to the input layer using p = 0.2 in both cases.

18Optimization The network parameters are learned using AdaDelta (Zeiler, 2012), which adapts the learning rate for each dimension using only first order information. We used the default hyper-parameters.

3 Training

19We train the parameters of the CNN using the three-phase procedure as described in the introduction. Figure 2 depicts the general flow of this procedure.

Figure 2: The overall architecture of our 3-phase approach.

4 Conclusion

30In this work we presented a deep-learning approach for sentiment analysis. We described the three-phase training approach to guarantee a high quality initialization of the CNN and showed the effects of using a multi-task training approach. To increase the robustness of our system we applied a meta-classifier on top of the CNN. The system was evaluated in the EvalItalia-2016 competition where it achieved 1^st place in the polarity task and high positions on the other two subtasks.