1. Introduction

1In task-oriented dialogue systems, a Spoken Language Understanding (SLU) component typically involves intent classification (IC) and slot filling (SF)  (Tur and De Mori 2011) tasks. For example, in “Show me the fares for Delta flights from Dallas to San Francisco“, the intent is asking an airfare and its corresponding slots are Delta (airline-name), Dallas (city-origin), and San Francisco (city-destination). Scaling SLU models to other languages is still challenging, especially when there is limited or no labeled data available in the target language (Louvan and Magnini 2020).

2To approach this problem, previous work studies IC and SF tasks in a zero-shot cross-lingual setting (Schuster et al. 2019; Upadhyay et al. 2018; Xu, Haider, and Mansour 2020), where it is assumed that a labeled dataset is only available for a high resource language (e.g., English). With the rise of pretrained multilingual language models (LMs) (Devlin et al. 2019; Lample and Conneau 2019) the most common approach is by fine-tuning the pretrained multilingual model on the English labeled data, and then evaluate the model directly on the target language data that are not seen during fine-tuning.

3While direct fine-tuning serves as a strong baseline, pretrained LMs are not necessarily universal and they may need domain-specific adaptation. Recent works have shown that adding a second pretraining stage (or continued pretraining) before fine-tuning can give positive impact on the model performance (Beltagy, Lo, and Cohan 2019; Lee et al. 2020; Gururangan et al. 2020). During continued pretraining, we continue training the pretrained language model using a domain-specific or task-specific unlabeled dataset, with the same masked language model objective. This stage is useful to alleviate the domain mismatch between the original pretraining and the target task data. By continued pretraining on domain specific unlabeled data, the model acquires prior knowledge which is expected to be helpful in the fine-tuning stage. This approach has shown promising results on text classification, typically on English. However, it remains unclear whether it is applicable in the context of zero-shot cross-lingual SLU.

Figure 1

4In contrast to previous work which has mostly focused on English classification tasks, we investigate the effectiveness of continued pretraining for zero-shot cross-lingual SLU tasks on eight target languages. Our study reveals that the existing continued pretraining method (Gururangan et al. 2020), that is successful in English text classification tasks, does not always generalize to the context of zero-shot cross-lingual SLU. We focus on the following research questions:

(Q1) Is continued pretraining effective for zero-shot cross-lingual SLU tasks?

5Our experiments on the MultiATIS++ dataset (Xu, Haider, and Mansour 2020) reveal that incorporating continued pretraining on intermediate English data can improve performance over direct fine-tuning for all languages, on zero-shot SLU. The performance gain is especially evident for languages with Latin script writing system. The benefit of continued pretraining diminishes as we inject cross-lingual supervision in the fine-tuning stage, even with simple data augmentation through code-switching.

(Q2) What are the factors that influence the effectiveness of the continued pretraining stage?

6Using the target language for continued pretraining before fine-tuning on English can be detrimental to the overall performance. However, this can be largely alleviated by code-switching the fine-tuning data. We also observe that performance improvement are not obtained by merely adding more continued pretraining data; higher domain similarity between the continued pretraining data and the fine-tuning data is indeed more important.

2. Continued Pretraining in Zero-Shot SLU

7Figure 1 shows a comparison between the standard direct fine-tuning approach with the continued pretraining approach. The main difference is the additional intermediate pretraining stage (second block in Figure 1), in which we continue training the model on an intermediate unlabeled data using the same masked language modeling objective. As the original pretraining data is relatively far from the task-oriented dialogues used in SLU, we hypothesize that continued pretraining can alleviate the domain mismatch and ingest a better prior knowledge that will be useful during fine-tuning.

Intermediate Data for Continued Pretraining

8We define several criteria for the intermediate pretraining data for the continued pretraining stage. First, their domain should be relatively close to the target dataset. We interpret the term domain as a multidimensional variety space (Ramponi and Plank 2020; Plank 2016): a domain comprises multiple aspects (style, topic, and genre (Wees et al. 2015)) that contribute to the text variation. Using this perspective and considering the target domain of a task-oriented dialogue system, we require that the intermediate data comprises text that presents a spoken language dialog style and covers a broad range of topics. Second, the dataset should be several magnitudes larger in size than the target task dataset. Finally, it must be available in many languages to support our study of continued pretraining with the target language.

3. Experimental Setup

9In this section, we describe the experimental settings related to models, evaluation metrics, and datasets.

3.1 Models

10For all of our experiments, we use a transformer-based model (Vaswani et al. 2017), namely multilingual BERT (mBERT) (Devlin et al. 2019), as the pretrained model. This model was pretrained on Wikipedia articles covering 104 languages, and we use the bert-base-multilingual-cased version.

Continued Pretraining

11For the continued pretraining stage, we further train mBERT with unlabeled intermediate data using only the Masked Language Modeling (MLM) objective for 12.5K steps, and mostly adopt the hyperparameters in . We compare the following configurations: (i) DAPT_Tgt  a continued domain adaptive pretraining (DAPT) of mBERT on intermediate unlabeled data on the target language. (ii) DAPT_En  a continued DAPT of mBERT on intermediate unlabeled data on English.

Fine-Tuning

12As the baseline model, without any adaptation (No DAPT), we use the joint IC and SF model architecture (Chen, Zhuo, and Wang 2019). This model is the state-of-the-art for IC and SF (Louvan and Magnini 2020), and it is often used as one of the baselines in recent zero-shot cross-lingual SLU studies (Xu, Haider, and Mansour 2020; Li et al. 2021). The model is trained on the English dataset; as the setup is zero-shot cross-lingual and we use the model’s last epoch for zero-shot evaluation following . We evaluate the effectiveness of each of the DAPT configurations when applied to the following fine-tuning scenarios:

• Fine-tuning on English (Finetune-En). This is the standard fine-tuning scenario, where we take mBERT either with DAPT or no DAPT, fine-tune it on the English IC and SF data, and then perform zero-shot prediction to all target language data.

• Fine-tuning on the English code-switched data (Finetune-CS). In this scenario, we perform data augmentation on the English fine-tuning dataset via code-switching. We follow the approach from Qin et al. (2020), where we replace the English words with their translation in the target language using the Panlex bilingual dictionary (Kamholz et al., 2014). Given a training batch, we randomly sample sentences and tokens to replace. We use the same hyperparameter used by Qin et al. (2020), that defines both sentence and word ratio to control the word replacement. We include FineTune-CS because we want to study the benefits of DAPT when adding stronger cross-lingual supervision in the fine-tuning stage.

13We did not experiment with more complex models models as our main goal is to investigate the effect of the the continued pretraining stage, rather than achieving the state of the art performance per se.

Implementation & Model Evaluation metric

14For the intent and SF models, we adapt the implementation from in which they make it publicly available (https://github.com/kodenii/CoSDA-ML). The sentence and token ratio replacement for code-switching is set to 1.0 and 0.9 respectively. For training, the learning rate is set to 10^-5, batch size is set to 32, number of epoch is set to 20. We did not do extensive hyperparameter tuning, as this is a zero-shot cross lingual case where the target dataset is not available, we use the same hyperparameters as Xu et al. (2020). For the continued pretraining we use the language modeling script from Huggingface (Wolf et al. 2019). We use the bert-base-multilingual-cased, hidden state size is 768, we apply dropout probability of 0.1. The number of training steps is 12,500 following , the batch size is set to 16.

3.2 Dataset

SF and IC Dataset

15We use the MultiATIS++ (Xu, Haider, and Mansour 2020) dataset, which contains nine languages (Table 1). The dataset is derived from the original ATIS English dataset (Hemphill, Godfrey, and Doddington 1990), widely used as a benchmark for IC and SF for task-oriented dialogue systems. Utterances are related to conversations of a user asking for flight information to a system.

Continued Pretraining Dataset

16We use the OpenSubtitle (OpenSub) (Lison and Tiedemann 2016) (Table 2) dataset for the continued pretraining stage for several reasons. First, the dataset is constructed from movies and TV series containing spoken language in dialogue settings covering a broad range of topics. Second, OpenSubtitle covers all the languages that we use on the downstream tasks, which enables us to evaluate not only DAPT_En but also DAPT_Tgt. Third, the dataset is large in size, thus ideal for continued pretraining. Typically, the dataset used for continued pretraining is larger than that used for fine-tuning. For our experiments we randomly sampled 100K sentences for each language in the OpenSub dataset, resulting in a dataset around 20 times larger than the downstream task dataset.

4. Results

17The main goal of our experiment is to answer research question (Q1). Table 3 compares the zero-shot performance for SF and IC across languages. In terms of language (by column in Table 3), we observe that all languages improve over No-DAPT in at least one DAPT setting, suggesting that DAPT is effective across languages. Observing the results per task, SF benefits from either DAPT_En or DAPT_Tgt for German, Spanish, French, Portuguese, and Turkish, which all are languages with Latin scripts writing system. For these languages, the margin obtained from DAPT when fine-tuning on English (FineTune-En) is higher than when we apply DAPT on code-switched data (FineTune-CS). The margin of DAPT when applied on FineTune-CS diminishes because FineTune-CS uses a stronger supervision signal in the fine-tuning stage, thus providing a higher baseline. For languages with non-Latin script writing system, continued pretraining is less useful; we only observe marginal improvement on Japanese when applying DAPT_En and FineTune-En. Similar to , we believe that performance is also affected by typological language proximity such as the subject, verb, and object ordering, phonology features or other aspect related to the original size of the pre-training data of mBERT. We leave this for future work.

Table 3

18DAPT is less effective for IC than for SF. The only language that consistently benefits from continued pretraining in both fine-tuning scenarios is Turkish. We found that it is harder to improve the model performance of languages with Latin script through DAPT because the baseline is relatively high; a stronger supervision signal would thus be needed. The performance gain is small even for those languages that do benefit from DAPT. We also observe that using a different language between continued pretraining and fine-tuning stages, DAPT_Tgt and FineTune-En, may hamper performance.

5. Analysis and Discussion

19To answer the research question (Q2), we analyze our results focusing on the performance variation when using different languages in DAPT and fine-tuning (5.1) and the effect of domain distribution in different sources for DAPT_En (5.2).

5.1 Performance Variation when Applying DAPT

20As we have noticed in Section 4, there are cases where performance drop when we use DAPT_Tgt and FineTune-En, especially for IC. This behaviour holds even for languages relatively close to English, such as German and French. One possible reason for the drop in accuracy is that the language difference introduces instability in fine-tuning. Our post-hoc analysis shows that the target language performance during training on the dev set has a large deviation and continues fluctuating even after the English dev performance has stabilized. This observation resonates with a previous study from , which shows that, for zero-shot text classification, English dev performance often does not correlate with those of the target language. Using DAPT_Tgt and FineTune-En pronounces the disagreement of performance between the English and the target dev set. Figure 2 shows the comparison of the IC performance during training across continued pretraining strategies when fine-tuning on English for French. However, for the SF task, we do not observe a large performance variation even with a language mismatch: this might indicate that text classification is more susceptible to instability than sequence tagging. The variability caused by DAPT_Tgt is largely alleviated when we use DAPT_En. For the FineTune-CS scenario, the system is relatively stable even when combined with DAPT_Tgt or DAPT_En.

Figure 2

5.2 Domain Relevance for DAPT_En

21We aim at investigating whether the improvement from the continued pretraining comes from the domain relevance of the intermediate data. For this purpose, we selected a few written text datasets instead of spoken language, which are focused on a specific topic. Specifically, we use the European Medicines Agency (EMEA) and European Central Bank corpus (ECB) from Tiedemann (2012). EMEA contains articles about human, veterinary, or herbal medicines extracted from the EMEA website. ECB contains financial documents that are extracted from the website and documentation of the European Central Bank. In order to check that EMEA and ECB are more distant in terms of domain from MultiATIS than OpenSub, we compute the Jensen Shannon Divergence (JSD) measure of the term distribution (Dai et al., 2020; Ruder and Plank, 2017). We compute the JSD between the MultiATIS English dataset that is used for fine-tuning and each English intermediate dataset. Based on the JSD measure, EMEA and ECB are more distant to MultiATIS than OpenSub (Table 4).

22For each intermediate dataset, we randomly sample 100K sentences and use them for continued pretraining. We compare the SF performance of DAPT_En with FineTune-En on OpenSub, EMEA, and ECB in Table 5. We focus on languages that belongs to Indo-European family which mostly obtain benefit from DAPT on SF (Table 3) Overall, we see that DAPT using OpenSub obtains improvements over No-DAPTin all cases. The DAPT performance using EMEA and ECB are lower than OpenSub in most cases. Even for DE and PT languages, DAPT with ECB obtains substantially lower performance than No-DAPT. However, there are cases when EMEA or ECB match or even perform better than OpenSub i.e., for Spanish. These cases indicate that performing data selection before continued pretraining could be beneficial to construct more optimal DAPT dataset. It would be interesting also to observe how continued pre-training would work using smaller unlabeled pre-training data but more task relevant. We leave this possibility for future work.

6. Related Work

Zero-Shot Cross-Lingual SLU

23Before the advent of the pre-trained multilingual transformer models, most approaches relied on pre-trained cross-lingual embeddings to perform zero-shot SLU. Upadhyay et al. (2018) uses cross-lingual embedding (Bojanowski et al. 2017) to perform zero-shot SLU while Schuster et al. (2019) uses multilingual embedding (Cove) from pre-trained multilingual bi-LSTM encoder used in Neural Machine Translation (NMT). Liu et al. (2019) leverages transferable latent variables to improve the sentence representation across languages. More recently, as pre-trained multilingual transformer models show potential in zero-shot settings, most approaches focus on improving their multilingual representation through augmentation and alignment methods. Qin et al. (2020) proposes multilingual code-switching using a bi-lingual dictionary to improve mBERT’s multilingual representation. Xu et al. (2020) introduces soft alignment of slots between English and the target language produced by a machine translation system that eliminates the need for an annotation projection pipeline. Kul-shreshtha et al. (2020) study the effect of various cross-lingual alignment methods to improve mBERT representation.

Continued Pre-training

24Domain adaptation is a long-studied problem in the NLP community (Daumé III 2007; ???), in which we assume data in the target domain might be hard to obtain while being abundant in source domains. Continued pre-training – where the model is trained on relevant data using the same pre-training objective – is used for mitigating the distribution mismatch between the pre-training and the fine-tuning data in terms of domain (Logeswaran et al., 2019; Han and Eisenstein, 2019; Gururangan et al., 2020; Beltagy et al., 2019), task (Gururangan et al. 2020), and language (Pfeiffer et al., 2020). A complementary approach performs a first fine-tuning on related auxiliary tasks (for which training data are easy to obtain) before the final fine-tuning on the downstream task (Arase and Tsujii, 2019; Garg et al., 2020; Khashabi et al., 2020). Our work is in line with Gururangan et al. (2020) where we investigate further the effectiveness of continued pre-training in the context of zero-shot cross-lingual SLU.

7. Conclusion

25We systematically study the effectiveness of continued pre-training of a multilingual model on intermediate English unlabeled spoken language data for zero-shot cross-lingual tasks, namely intent classification and slot filling, on 8 languages. Our results show that the domain knowledge learned in English is transferable to other languages. The gain from continued pre-training diminishes as we inject cross-lingual supervision in the fine-tuning stage. There are several factors that influence the effectiveness of the continued pre-training: (i) Using different language between pre-training and fine-tuning can hamper performance and introduce instability in the model training, which can be alleviated with code switching. (ii) Domain similarity is important. The more similar – in terms of data distribution – the intermediate data to the target dataset yields better performance.