1. Introduction

1In a process with data that change very quickly and the need to resort to complete training in the shortest possible time, transfer learning techniques have made possible a fast fine-tuning of BERT models. The distillation of a model made it possible to decrease the load and the times without significantly losing accuracy. These models, therefore, require, at least, constant fine-tuning training. In addition, a BERT model specially designed for the Italian language and with a vocabulary containing technical terms increases its effectiveness.

2Constant and multi-disciplinary training requires specific skills and tailor-made services. In this research, we will see an effective way to make both things possible. The idea is to use a way to exchange AI models between library and frameworks, the ONNX project, and a runtime, the ONNX runtime project, to optimize inference for many platforms, languages and hardware. The ONNX runtime is still working to optimize the training directly in the ONNX format.

3The second goal is to find a viable alternative with acceptable performance at the start of a new project while waiting for a trained BERT model.

4The research was carried out for the ATE ABSITA (de Mattei et al., 2020) task in the EVALITA 2020 (Basile et al., 2020), using all 3 available sub-tasks.

2. Description of the system

5To start using a sentiment analysis system, we need several elements. Certainly, a starting dataset with the related labels. In the tasks of the challenge, we have the reviews of 23 different products. Each review has a corresponding rating assigned by the end-user. For each review, it was required to extract the aspects contained in it. By aspect, we mean every opinion word that expresses a sentiment polarity. Finally, each aspect was classified as a pair of values: positive or negative, for 4 possible states.

6Imagine a system that receives an unspecified number of reviews in real-time with new products and different categories. We find ourselves in the situation of always having to fine-tune our models.

7The complexity of BERT makes training time difficult for constant alignment. Being able to reduce the training time, or being able to put in place an alternative in the meantime, new perspectives open up, such as:

• Made inference calls before a full trained model is completed.

• Training of the new model.

• Running the BERT model.

• (optional) reclassify recent product reviews after the model update.

8In this perspective, in order to validate my hypotheses, I used the AlBERTo (Polignano at al., 2019) model, used in the baseline, and Ktrain³, a wrapper for TensorFlow⁴, with the autofit option.

9The first submission, called ghostwriter19_a, was obtained training all the models with the Ktrain framework.

10The results for the three tasks for the second submission, called ghostwriter19_b, were obtained in two different way:

• for the first two tasks, I used the model of the first submission but exported on ONNX and ran with the ONNX runtime.

• for the third task, I trained the model with TensorFlow using a Zero-Shot learner [ZSL] (Brown et al, 2020).

11To test the models, I used two different machines with Ubuntu 20.04 LTS:

• 6 vCPU on Intel Xeon E5-2690 v4 - 112GB with P100 (GPU)

• 14 cores on Intel Xeon E5-2690 v4 - 32GB (CPU)

2.1 Task 1 – ATE: Aspect Term Extraction

12To identify an aspect, the dataset contains a label for every single word with three possible values:

• B for Begin of an aspect.

• I for Inside an aspect.

• Or for Outside, not in an aspect.

13For example, the review “La borraccia termica svolge egregiamente il proprio compito di mantenere la temperatura, calda o fretta che sia. La costruzione è ottimale e ben rifinita. Acquisto straconsigliato!” is labeled as:

Figure 1

14The model will be evaluated with the F1-score. The score results from the full matched aspects, the partial matched ones, and the missed ones.

15The preliminary results with the Ktrain model were encouraging (table 1).

16At this point, the model has been exported with ONNX in maximum compatibility mode. The model ran with the ONNX runtime optimized for CPU.

17The performances have remained unchanged, but the speed of inference has significantly improved (table 2).

18The improvement is 17x for the CPU version and 1.75x for the GPU version.

2.2 Task 2 – ABSA: Aspect-based Sentiment Analysis

19For this task, the aspects identified in Task 1 have been used. This implies that an error in Task 1 will have a decisive impact on Task number 2.

20The aspect can be classified as:

• positive (POS:true,NEG:false)

• negative (POS:false,NEG:true)

• mixed polarity(POS:true, NEG:true)

• neutral polarity (POS:false, NEG:false)

21As showed to the image from the challenge website⁵:

Figure 1

22The results on the DEV test outperform the baseline (table 3).

23Also, for this task, the performance is improved with the use of ONNX runtime (table 4).

24The improvement is 9.5x for the CPU version and 1.59x for the GPU version.

2.3 Task 3 – SA: Sentiment Analysis

25Task 3 is a classification problem. However, fully understanding the score is not easy. The evaluation operation is carried out by different people and with different styles. A product with a similar review is rated according to the expectations and judgment of other users differently.

26Furthermore, in order to obviate the long training time that a constant updating requires, compared with systems used by the previous version of EVALITA, such as an ensemble system with Tree Random Forest and Bi-LSTM (Bennici and Portocarrero, 2018) or with an SVM system (Barbieri et al., 2016), I used a Zero-Shot Learner [ZSL] (Pushp & Srivastava, 2017). A ZSL is a way to make predictions without prior training (Petroni, 2019). ZSL will refer to the embedding of a previous matrix, AlBERTo in this case, and of the proposed labels as a possible result (Schick and Schütze, 2020).

27The proposed labels were the possible numbers for evaluation, then the numbers from 1 to 5.

28The proposed prediction value is a weighted average of the two values with the highest probability, if and only if the gap between the two values is less than $10^{- 3}$. Otherwise, only the value with the highest probability will be considered valid.

29For this task, I omitted the ONNX runtime test because a stable converter for the ZSL version is not available.

30The score for this task is the Root Mean Squared Error between the polarity predicted and the polarity assigned by the user.

31The loss in performance is 18%, but the entire previous training phase is skipped (table 5).

3. Results

32The results obtained with the DEV dataset are very positive both in terms of accuracy and performance. ZSL has proven to be an incredible technology to invest in. The Ktrain seems to suffer a heavy overfit.

33The research aims not to have a relevant model but to prove that a model could be production-ready with fewer resources and time.

34However, in all three tasks, the models outperformed the baseline with a significant gap in terms of accuracy/RMSE.

3.1 Results for Task 1

35The final results with the TEST dataset are:

36The results are about 12% lower than those obtained in the research phase (table 6).

37It will be interesting to continue experimenting with different ONNX options to find a better combination of compatibility and performance.

3.2 Results for Task 2

38The final results with the TEST dataset are:

39The loss from DEV to TEST is about 17% (table 7). However, the percentage of the difference between the results of Tasks 1 and 2 have been maintained with the DEV and TEST datasets.

40This is in line with expectations, worse model performance in Task 1 impacted Task 2 proportionally. In return, working on a better model will improve both tasks.

3.3 Results for Task 3

41For Task 3 we have:

42The difference between the DEV and TEST datasets is marked here only for trained model, 14% (table 8). The untrained one performed slightly better, 2%, with the TEST dataset.

43This result confirms that an underperforming model has the same performance of a model that use ZSL, as assumed.

44The price to pay, however, is that the average inference time for the ZSL is 157x higher than the pure TensorFlow model obtained with Ktrain.

4. Conclusion

45The results demonstrated that it is possible to create hybrid systems for training and inference to make the power of BERT more accessible.

46In the time it takes to train a new and optimized model, an untrained ZSL model can make up for it in the meantime.

47Optimizing, and in future training, our models to be intrinsically optimized for the platform and framework we have chosen to use does not affect performance and future use.

48The improvements obtained in the use of ONNX runtime for these Italian tasks are in line with what Microsoft demonstrated, for the English language, at the beginning of 2020 (Ning at al., 2020).

49The next step is to make the ONNX export work with a Zero-Shot learner [ZSL] in order to compensate, at least in part, for the more significant resources that this inevitably introduces.