Version classiqueVersion mobile

Proceedings of the Eighth Italian Conference on Computational Linguistics CliC-it 2021

Elisabetta Fersini
Marco Passarotti
Viviana Patti

Contributed Papers: Long Papers

On the Development of Customized Neural Machine Translation Models

Mauro Cettolo, Roldano Cattoni et Marco Turchi


Recent advances in neural modeling boosted performance of many machine learning applications. Training neural net-works requires large amounts of clean data, which are rarely available; many methods have been designed and inves-tigated by researchers to tackle this is-sue. As a partner of a project, we were asked to build translation engines for the weather forecast domain, relying on few, noisy data. Step by step, we developed neural translation models, which outper-form by far Google Translate. This paper details our approach, that - we think - is paradigmatic for a broader category of applications of machine learning, and as such could be of widespread utility.

Texte intégral

1. Introduction

1The field of machine translation (MT) has experienced significant advances in recent years thanks to improvements in neural modeling. On the one hand, this represents a great opportunity for industrial MT, on the other it also poses the great challenge of collecting large amounts of clean data, needed to train neural networks. MT training data are parallel corpora, that is collections of sentence pairs where a sentence in the source language is paired with the corresponding translation in the target language. Parallel corpora are typically gathered from any available source, in most cases the web, without much guarantees about quality nor domain homogeneity.

2Over the years, the scientific community has accumulated a lot of knowledge on ways to address the problem of the quantitative and qualitative inadequacy of parallel data necessary to develop translation models. Among others, deeply investigated methods are: corpus filtering (Koehn et al., 2020), data augmentation such as data selection (Moore and Lewis, 2010; Axelrod et al., 2011) and back-translation (Bertoldi and Federico, 2009; Sennrich et al., 2016), model adaptation (Luong and Manning, 2015; Chu and Wang, 2018). They should be the working tools of anyone who has to develop neural MT models for specific language pairs and domains.

3This paper reports on the development of neural MT models for translating forecast bulletins from German into English and Italian, and from Italian into English and German. We were provided with in-domain parallel corpora for each language pair but not in sufficient quantity to train a neural model from scratch. Moreover, from the preliminary analysis of data, the English side resulted noisy (e.g. missing or partial translations, misaligned sentences, etc.), affecting the quality of any pair involving that language. For this very reason, we focus on one of the pairs involving English we had to cover, namely Italian-English.

4An overview of the in-domain data and the description of their analysis are given in Section 2, highlighting the issues that emerged. Section 3 describes the previously listed methods together with their employment in our specific use-case. Developed neural translation models are itemized in Section 4, where their performance are compared and discussed; our best models outperform by far Google Translate and some examples will give a grasp of the actual translation quality.

5We think that our approach to the specific problem we had to face is paradigmatic for a broader category of machine learning applications, and we hope that it will be useful to the whole NLP scientific community.

2. Data

6We were provided with two csv files of weather forecast bulletins, issued by two different forecast services that from here on are identified with the acronyms BB and TT. Each row of the BB csv contains, among other things, the text of the original bulletin written in German and, possibly, its translation into Italian and/or English; in the TT csv rows, the Italian bulletin is paired with its translation into German and/or English.

2.1 Statistics

7BB Bulletins were extracted from the BB csv file and paired for any possible combination of languages. Each bulletin is stored on a single line but split in a few dozen fields; the average length of each field (about 18 German words) is appro­priate for MT systems, which process long sentences with difficulty. Table 1 shows statistics of the training and test sets for the it-en language pair.

Table 1: Statistics of the BB it-en benchmark. The label nsy will be clear after reading Section 3.2





#src w

#trg w















8TT Bulletins were extracted from the TT csv file and paired for each language combination. Differently than the BB case, each TT bulletin was stored on a single line without any field split; since bulletins are quite long for automatic processing (on average 30 Italian words) and are the concatenation of rather heterogeneous sentences, we decided to segment them by splitting on strong punctuation. This requires a re-alignment of source/target segments because in general they differ in number. The re-alignment was performed by means of the hunalign sentence aligner1 (Varga et al., 2005). Table 2 shows statistics of the training and test sets for the it-en language pair.

Table 2: Statistics of the TT it-en benchmark





#src w

#trg w















9As a good practice before starting the creation of MT models, data have been inspected and analyzed looking for potential problems. Several critical issues emerged, which are described in the following paragraphs.

  • 2 3-gram LMs with modified shift beta smoothing were es­timated using the IRSTLM toolkit (Federico et (...)
  • 3 In order to isolate the genuine PP of the text, the dictionary upperbound to compute OOV word penal (...)

10Non-homogeneity of data - Since data originated from two distinct weather forecast services (BB and TT), first of all it must be established whether they are linguistically similar and, if so, to what extent. For this purpose, focusing on the languages of the it-en benchmarks, we measured the perplexity of the BB and TT test sets on n-gram language models (LMs) estimated on the BB and TT training sets:2 the closer the perplexity values of a given text on the two LMs, the greater the linguistic similarity of BB and TT training sets. Table 3 reports values of perplexity (PP) and out-of-vocabulary rates (%OOV) for all test sets vs. LMs combinations.3

Table 3: Cross comparison of BB and TT texts

LM trained on

BB trn

TT trn






BB tst





TT tst






BB tst





TT tst





11Overall, we can notice that the PP of the two test sets significantly varies when computed on in- and out-of-domain data. The PP of any given test set is 4 (42.4 vs. 10.8) to 9 (92.0 vs. 10.3) times higher when measured on the LM estimated on the text of the other provider than on the text of the same provider. These results highlight the remarkable linguistic difference between the bulletins issued by the two forecast services.

12In-domain data scarcity - Current state-of-the-art MT neural networks (Section 4.1) have dozens to hundreds million parameters that have to be estimated from data. Unfortunately, the amount of provided data does not allow an effective estimation from scratch of such a huge number of parameters, as we will empirically prove in Section 4.3.

13BB English side - BB data have a major problem on the English side. In fact, looking at csv file, we realized that many German bulletins were not translated at all into English. Moreover, in the English side there are 20% fewer words than in the corresponding German or Italian sides, a difference that is not justified by the morpho-syntactic variations between languages. In fact, it happens that entire portions of the original German bulletins are not translated into English, or that a definitely more compact form is used, as in:

de: Der Hochdruckeinfluss hält bis auf weiteres an.
en: High pressure conditions.

14This critical issue affects both training and test sets, as highlighted by figures in Table 1; as such, it negatively impacts both the quality of the translation models, if trained/adapted on such noisy data, and the reliability of evaluations, if run on such distorted data. A careful corpus filtering is therefore needed, as discussed in Section 3.2.

3. Methods

3.1 MT Model Adaptation

15A standard method for facing the in-domain data scarcity issue mentioned in Section 2.2 is the so-called fine-tuning: given a neural MT model trained on a large amount of data in one domain, its parameters are tuned by continuing the training using a small amount of data from another domain (Luong and Manning, 2015; Chu and Wang, 2018). Though effective on the new in-domain data supplied for model adaptation, fine-tuning typically suffers from performance drops on unseen data (test set), unless proper regularization techniques are adopted (Miceli Barone et al., 2017). We avoid overfitting by fine-tuning our MT models with dropout (set to 0.3) (Srivastava et al., 2014) and performing only a limited number of epochs (5) (Miceli Barone et al., 2017).

3.2 Corpus Filtering

16Machine learning typically requires large sets of clean data. Since rarely large data sets are also clean, researchers devoted much effort to data cleaning, the automatic process to identify and remove errors from data. The MT community is no exception. Even, WMT - the conference on machine translation - in 2018, 2019 and 2020 editions organized a Shared Task on Parallel Corpus Filtering. Koehn et al. (2020) provide details on the task proposed in the more recent edition, on participants, their methods and results. For reference purposes, organizers set up a competitive baseline based on LASER (Language-Agnostic SEntence Representations)4 (Schwenk and Douze, 2017) multilingual sentence embeddings. The underlying idea is to use the cosine distance between the embeddings of the source and the target sentences to measure their parallelism. In a similar way we cleaned the BB noisy benchmark, filtering with a threshold of 0.9; statistics of the resulting bi-text are given in Table 4.

Table 4: Stats of the filtered BB it-en benchmark





#src w

#trg w















17The filtered bi-text does not suffer anymore from the imbalance number of words but it is 20 times smaller than the original one.

3.3 Data Augmentation

18Since the corpus filtering discussed in the previous section removes most of the original data, further exacerbating the problem of data scarcity, we tried to overcome this unwanted side effect by means of data augmentation methods.

3.3.1 Data Selection

19A widespreadly adopted data augmentation method is data selection. Data selection assumes the availability of a large general domain corpus and a small in-domain corpus; in MT, the aim is to extract parallel sentences from the large bilingual corpus that are most relevant to the target domain as defined by the small corpus.

  • 5

20On the basis of the bilingual cross-entropy difference (Axelrod et al., 2011), we sorted the sen­tence pairs of the OPUS collection,5 used as general domain large dataset, according to their relevance to the domain determined by the concate­nation of the BB and TT training sets. To establish the optimal size of the selection, we trained LMs - created in the same setup described in non-homogeneity of data paragraph of Section 2.2 - on increasing amounts of selected data and computed the PP of BB and TT test sets, separately for each side. Figure 1 plots the curves; the straight lines on the bottom correspond to the PP of the same test sets on LMs built on the in-domain training sets.

Figure 1

Figure 1

Perplexity of test sets on LMs estimated on increasing amounts of selected data.

21The form of curves is convex, as usual in data selection. In our case, the best trade-off between the pertinence of data and its amount occur when something more than a million words is selected; therefore, we decided to mine from OPUS the bilingual text whose size is given in row DS of Table 5. Anyway, note that the lowest PP for selections is at least one order of magnitude greater than on LMs trained on in-domain training sets.

Table 5: Stats of selected and back translated data. 3.3.2 Back Translation




#src w

#trg w










22Another well-known data-augmentation method, which somehow also represents an alternative way to corpus filtering for dealing with the BB English side issue, is back-translation. Back-translation (Bertoldi and Federico, 2009; Sennrich et al., 2016; Edunov et al., 2018) assumes the availability of an MT system from the target language to the source language and of target monolingual data. The MT system is used to translate the target monolingual data into the source language. The result is a parallel corpus where the source side is the synthetic MT output while the target is human text. The synthetic parallel corpus is then used to train or adapt a source-to-target MT system. Although simple, this method has been shown to be very effective. We used back-translation to generate a synthetic, but hopefully cleaner, version of the BB training set. The translation into Italian of the 31k English segments of the training set (Table 1) was performed by an in-house generic en-it MT engine (details in Ap­pendix A.1 of (Bentivogli et al., 2021)). Row BT of Table 5 shows the statistics of this artifi­cial bilingual corpus; similarly to what happened with the filtering process, the numbers of Italian and English words are much more compatible than they are in the original version of the corpus.

Table 6: Stats of the parallel generic training sets


#src w

#trg w





4 Experimental Results

4.1 MT Engine

23The MT engine is built on the ModernMT framework6 which implements the Transformer (Vaswani et al., 2017) architecture. The original generic model is Big sized, as defined in (Vaswani et al., 2017) by more than 200 million parameters. For training, bi-texts were downloaded from the OPUS repository6 and then filtered through the already mentioned data selection method (Axelrod et al., 2011) using a general-domain seed. Statistics of the resulting corpus are provided in Table 6. Training was performed in the setup detailed in (Bentivogli et al., 2021).

24The same Big model and its smaller variants, the Base with 50 million parameters and the Tiny with 20 million parameters, were also trained on in-domain data only for the sake of comparison.

4.2 MT Models

  • 7 BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a, TER+tok.tercom-nonorm-punct-noasian-uncased, chrF2+nu (...)

25We empirically compared the quality of translations generated by various MT models: two generic, three genuine in-domain of different size and several variants of our generic model adapted (Section 3.1) on in-domain data resulting from the presented methods: filtering (Section 3.2), data selection (Section 3.3.1) and back-translation (Section 3.3.2). Performance was measured on the BB and TT test sets in terms of BLEU (Papineni et al., 2002), TER (Snover et al., 2006) and CHRF (Popovió, 2015) scores computed by means of SacreBLEU (v1.4.14) (Post, 2018), with default signatures.7

Table 7: BLEU/TER/CHRF scores of MT models on it-en test sets. 1, 2 and 3 indicate the “podium position” among the adapted models of each column. (⋆) Google Translate, as it was on 14 Sep 2021.

MT model



noisy test set

clean test set

test set










Generic models:











FBK (Transformer big)










Pure in-domain models trained on BBtrn-nsy+TTtrn:

Transformer tiny










Transformer base










Transformer big










FBK model adapted on:



























































































4.3 Results and Comments

26Scores are collected in Table 7. First, as expected (in-domain data scarcity paragraph of Sec­tion 2.2), it is not feasible to properly train a huge number of parameters with few data; in fact, the best performing pure in-domain model is the smallest one (Transformer tiny). Instead, the naive application of the MT state-of-the-art would have led to simply train a Transformer big model on the original in-domain data. This model would not have been competitive with GT on TT data (28.01 vs. 32.20 BLEU); it would have been on BB data if we had only considered the noisy test set (20.45 vs. 11.45) resulting in an important misinterpretation of the actual quality of the two systems; conversely, our preliminary analysis allowed us to discover the need of cleaning BB data, which guarantees a reliable assessment (24.73 vs. 32.59).

27Data augmentation methods (DS, BT) are both effective in making available additional useful bitexts; for example, the BLEU score of the model BBtrn-cln+TTtrn increases by 3 absolute points

28The fine-tuning of a Transformer big generic model to the weather forecast domain turned out to be more effective than any training from scratch using original in-domain data only: the top performing model - BBtrn-cln+TTtrn+DS+BT - definitely improves the Transformer tiny with respect to all metrics on the BB clean test set (40.09/45.28/.6617 vs 35.80/61.05/.5808), and to two metrics out of three on the TT test set (TER: 51.01 vs. 51.79, CHRF: .6803 vs. .6488). Moreover, all its scores are a lot better than those of Google Translate.

4.4 Examples

29To give a grasp of the actual quality of automatic translations, Table 8 collects the English text gen­erated by some of the tested MT models fed with a rather complex Italian source sentence. The man­ual translations observed in BB data are shown as well: their number, their variety, some question­able/wrong lexical choices in them (“high” instead of “upper-level currents”, “South-western” instead of “Southwesterly”) and one totally wrong (“Weak high pressure conditions.”) prove the difficulty of learning from such data and the need to pay particular attention to the evaluation phase. Concern­ing translations, GT is able to keep most of the meaning of the source text but the translation is too literal to result in fluent English. FBK only partially transfers the meaning from the source and generates a rather bad English text. Trans­former tiny provides a very good translation both from a semantic and a syntactic point of view, losing only the negligible detail that the “air masses” are “more humid”, not simply “humid”. Finally, BBtrn-cln+TTtrn+DS+BT, the model that on the basis of our evaluations is the best one, on this specific example works very well at the semantic level but rather poorly on the grammatical level.

Table 8: Examples of manual and automatic translations

Italian source sentence:

Le correnti in quota si disporranno da sudovest avvicinando masse d’aria pi`u umida alle Alpi.

Manual English translations found in BB bulletins:

Weak high pressure conditions.

The high currents will turn to south-west and humid air mass will reach the Alps. Southwesterly currents will bring humid air masses to South Tyrol.

South-western currents will bring humid air masses to the Alps.

South-westerly upper level flow will bring humid air masses towards our region. More humid air masses will reach the Alps.

Humid air reaches the Alps with South-westerly winds.

Automatic English translations generated by some MT models:


The currents at high altitudes will arrange themselves from the southwest, bringing more humid air masses closer to the Alps.


Currents in altitude will be deployed from the southwest, bringing wet air masses closer to the Alps.

Transformer tiny

South-westerly upper level flow will bring humid air masses towards the Alps.


The upper level flow will be arranged from the southwest approaching more humid air masses to the Alps.

30This example shows that pure in-domain models, as expected, are “more in-domain” than generic models, though adapted, showing greater adherence to domain-specific language. On the other hand, according to scores in Table 7, adapted models should be better in generalization. Only subjective evaluations involving meteorologists can settle the question of which model is the best.

5 Conclusions

31In this paper we described the development process that led us to build competitive customized translation models. Given the provided in-domain data, we started by analyzing them under several perspectives and discovered that they are few, noisy and heterogeneous. We faced these issues by exploiting a number of methods which represent established knowledge of the scientific community: adaptation of neural models, corpus filtering and data augmentation techniques such as data selection and back-translation. In particular, corpus filtering allowed us to avoid the misleading results observed on the original noisy data, while adaptation and data augmentation proved useful in effectively taking advantage of out-of-domain resources.


Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain Adaptation via Pseudo In-Domain Data Selection. In Proc. of EMNLP, pages 355–362, Edinburgh, Scotland, UK.

Luisa Bentivogli, Mauro Cettolo, Marco Gaido, Alina Karakanta, Alberto Martinelli, Matteo Negri, and Marco Turchi. 2021. Cascade versus Direct Speech Translation: Do the Differences Still Make a Difference? In Proc. of ACL/IJCNLP (Volume 1: Long Papers), pages 2873–2887, Bangkok, Thailand.

Nicola Bertoldi and Marcello Federico. 2009. Domain Adaptation for Statistical Machine Translation with Monolingual Resources. In Proc. of WMT, pages 182–189, Athens, Greece.

Chenhui Chu and Rui Wang. 2018. A Survey of Domain Adaptation for Neural Machine Translation. In Proc. of COLING, pages 1304–1319, Santa Fe, US-NM.

Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding Back-Translation at Scale. In Proc. of EMNLP, pages 489–500, Brussels, Belgium.

Marcello Federico, Nicola Bertoldi, and Mauro Cettolo. 2008. IRSTLM: An Open Source Toolkit for Handling Large Scale Language Models. In Proc. of Interspeech, pages 1618–1621, Brisbane, Australia.

Philipp Koehn, Vishrav Chaudhary, Ahmed El-Kishky, Naman Goyal, Peng-Jen Chen, and Francisco Guzmán. 2020. Findings of the WMT 2020 Shared Task on Parallel Corpus Filtering and Alignment. In Proc. of WMT, pages 726–742, Online.

Minh-Thang Luong and Christopher D. Manning. 2015. Stanford Neural Machine Translation Systems for Spoken Language Domains. In Proc. of IWSLT, pages 76–79, Da Nang, Vietnam.

Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. 2017. Regularization Techniques for Fine-tuning in Neural Machine Translation. In Proc. of EMNLP, pages 1489–1494, Copenhagen, Denmark.

Robert C. Moore and William Lewis. 2010. Intelligent Selection of Language Model Training Data. In Proc. of ACL (Short Papers), pages 220–224, Uppsala, Sweden.

Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proc. of ACL, pages 311–318, Philadelphia, US-PA.

Maja Popovió. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proc. of WMT, pages 392–395, Lisbon, Portugal.

Matt Post. 2018. A Call for Clarity in Reporting BLEU Scores. In Proc. of WMT, pages 186–191, Belgium, Brussels.

Holger Schwenk and Matthijs Douze. 2017. Learning Joint Multilingual Sentence Representations with Neural Machine Translation. In Proc. of RepL4NLP, pages 157–167, Vancouver, Canada.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving Neural Machine Translation Models with Monolingual Data. In Proc. of ACL (Volume 1: Long Papers), pages 86–96, Berlin, Germany.

Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A Study of Translation Edit Rate with Targeted Human Annotation. In Proc. of AMTA, pages 223–231, Cambridge, US-MA.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56):1929–1958.

Dániel Varga, Péter Halácsy, András Kornai, Nagy Viktor, Nagy Laszlo, N. László, and Tron Viktor. 2005. Parallel Corpora for Medium Density Languages. In Proc. of RANLP, pages 590–596, Borovets, Bulgaria.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In Proc. of NIPS, pages 5998–6008, Long Beach, US-CA.


1 2.2 Analysis and Issues

2 3-gram LMs with modified shift beta smoothing were es­timated using the IRSTLM toolkit (Federico et al., 2008).

3 In order to isolate the genuine PP of the text, the dictionary upperbound to compute OOV word penalty was set to 0; the OOV rates are shown for this very reason.




7 BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a, TER+tok.tercom-nonorm-punct-noasian-uncased, chrF2+numchars.6+space.false (from 37.02 to 40.09) when DS and BT data are added to the adaptation corpus.

Table des illustrations

Titre Figure 1
Légende Perplexity of test sets on LMs estimated on increasing amounts of selected data.
Fichier image/jpeg, 70k


Le texte seul est utilisable sous licence CC BY-NC-ND 4.0. Les autres éléments (illustrations, fichiers annexes importés) sont « Tous droits réservés », sauf mention contraire.


Open access

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search