1. Introduction

1This paper is mainly motivated by the interest of exploring, and improving, the capacity of current data-driven task-oriented conversational systems to address shifts of domain and changes of language. Our starting point is the dialogue domain adaptation (DDA) approach proposed by (Labruna and Magnini 2021), which allows to adapt training dialogues collected for a source domain knowledge (e.g., restaurants in Cambridge) to a domain where certain changes (e.g., a new restaurant opens, a restaurants changes its food , etc.) have occurred. The idea behind DDA is, that, rather then trying to improve the model robustness, it is worth to generate new training dialogues that are consistent with the domain changes. In this paper we extend and experiment DDA, so that also changes of language are included, particularly moving from English to Italian.

2A further motivation for our work is related to the scarcity of annotated data for task-oriented dialogues systems for the Italian language. Among the dialogic resources originally collected in Italian, we mention the recent JILDA dataset (Sucameli et al. 2020) (Sucameli et al., n.d.), a corpus of dialogues in the domain of job offers, semantically annotated. NESPOLE (Mana et al. 2004) is a corpus of human-human dialogues, which is not annotated for training purposes. The LUNA dataset (Dinarelli et al. 2010), although annotated with frame information, is not publicly available. Finally, the SUGAR dataset was presented at Evalita 2018 (Di Maro, Origlia, and Cutugno 2018) allowing to train systems able to interpret spoken commands in Italian for the preparation of recipes.

Figure 1: Example of dialogue domain and language adaptation. Words in bold indicate slot-values that have been adapted

3A second type of data is derived through automatic translation from English. This is the case of the Italian version of the SNIPS dataset (Bellomaria et al. 2019), which provides annotations for spoken language understanding and can be used for slot filling and intent detection. Similarly, an Italian version of the SQuAD question-answer pair corpus is available (Croce, Zelenanska, and Basili 2019) for Italian through translation. It has to be remarked that the Italian translations of both SNIPS and SQuAD do not rely on task-oriented dialogues, rather just on question-answer pairs. In addition, they are produced through simple translation from English, i.e., preserving domain information (e.g., names of places, restaurants, hotels, etc. reflect the English domain). We also notice that, unfortunately, the recent MultiATIS++ dataset (Xu, Haider, and Mansour 2020), does not include Italian among the languages.

4We are particularly interested in data-driven approach of dialogue state tracking (DST) (Balaraman and Magnini 2021) for the Italian language. DST captures the capacity of a model to predict the correct dialogue state at each turn in a dialogue, representing both the communicative goals (dialogue acts) of the user and the portion of domain knowledge involved in such goals (slot-value pairs). To the best of our knowledge, the only dataset in Italian that can be used to model dialogue state tracking is JILDA (Sucameli et al., n.d.), where dialogue state annotations were carried on following the MultiWOZ style. However, being concluded very recently, still there are no available DST baselines for JILDA, and, for this reason, we have developed an Italian version of the MultiWOZ dataset (Han et al. 2020).

5Starting from MultiWOZ 2.3, a popular dataset in English developed for booking traveling facilities (e.g., restaurants, hotels, trains, attractions) in the area of Cambridge, we incrementally operated both language and domain shifts. We provide three experimental configurations: (i) a translation of the Cambridge data set into Italian; (ii) a domain shift from Cambridge to Pisa, maintaining English as language; and, finally, (3) a configuration where both the initial domain and the language are changed. As a relevant side effect, the datasets for the three configurations are now available for further research on dialogue state tracking for Italian1.

6In the paper we first introduce the relavant background in dialogue domain adaptation (Section 2), then we explain how dialogue domain adaptation is concretely applied to domain changes, and finally we report the experiments we have conducted (Section 4 and 5).

2. Dialogue Domain Adaptation

7In the Dialogue Domain Adaptation setting (Labruna and Magnini 2021), we assume an initial conversational domain, represented in a KB-source, and corresponding annotated training dialogues D-source. Then, as in real application scenarios, we assume that a number of changes occur in KB-source, such that a new conversational domain KB-target needs to be considered. Dialogue domain adaptation consists in the capacity to automatically produce new annotated dialogues D-target, such that they maintain both the linguistic structure and the linguistic variability of the initial D-source dialogues, while, at the same time, being consistent with the new KB-target.

8Figure 1 shows an example of dialogue adaptation. On the left side we have a user-system dialogue in English grounded on the Cambridge domain, while on the right side we have the same dialogue translated into Italian and adapted to the Pisa domain. In this paper we show how to generate such adapted dialogues (i.e. D-target), which differ from the original dialogues (D-source) both in language and domain. The goal is then to train a dialogue state tracking model either on D-source or D-target, and to investigate the impact of such adaptations on the model performance.

2.1 Slot-Value Substitution

9Following (Labruna and Magnini 2021), we focus on domain changes due to different slot-values, while assuming the same slot-names for both the source and target domains. As for language shift, it is based on translating all the utterances in a dialogue with the exclusion of the slot-values.

10Given a slot-value occurring in a source dialogue D-source, the dialogue domain adaptation process consists of choosing the best slot-value in KB-target to substitute the slot-value in the D-source utterance. The first step is to check whether the slot-value is known in KB-source. If it is known, we look for a correspondence in KB-target, otherwise we directly keep it in D-target (or, in case of different languages, translate it into target language). In order to decide if the slot-value is in the KB-target, we use a similarity function based on a variation of the Gestalt Pattern Matching algorithm (Black 2004). We select the most similar value in the KB-target and we compare it to a empirically estimated threshold. Once we found a specific slot-value in KB-source and we ensured it exceeds the threshold, the corresponding slot-value to be selected from the KB-target depends on the adaptation strategy we choose to adopt.

11For the experiments of this paper we have used Frequency-KB, an adaptation strategy based that obtained the best performance in (Labruna and Magnini 2021). Given a slot-value in KB-source, Frequency-KB basically consists of selecting the slot-value in KB-target that has the most similar frequency distribution in the KB.

3. Method

12We broke down the problem of adapting a conversational dataset to a new language and a new domain into three different steps: first we performed delexicalization by inserting some placeholders in the place of the slot values; then we automatically translated the dataset, leaving the placeholders unchanged; finally, we substituted the placeholders with the new domain slot-values. Each one of these steps is discussed in the following sub-sections.

3.1 Delexicalization

13The setting that we are presenting involves the annotations being specifically slot-name slot-value pairs. Both the slot-values contained in the utterances, and those in the annotations, can not be translated the same way as the rest of the text, but need to undergo a Domain Adaptation process (e.g., we don’t want I need a taxi to The Old Castle to be translated into Ho bisogno di un taxi per Il Vecchio Castello).

14For this reason, the first step is to delexicalize a D-source dialogue, i.e., substituting all the slot-values in the utterances with placeholders. The example above shows this placeholder insertion, for moving from the following original sentence:

“I need a restaurant in the north that has Caribbean food and a moderate price range please .”

to the utterance:

“I need a restaurant in <#0#> that has <#1#> food and a <#2#> price range please .”

3.2 Translation

15The second step is to perform the translation from the source language to the target language without considering the placeholders. According to our example, we will produce the following Italian utterance:

“Ho bisogno di un ristorante a <#0#> che abbia <#1#> cibo e un <#2#> fascia di prezzo per favore .”

3.3 Slot-Value Substitution

16As a third step, the placeholders need to be substituted back with slot-values of the target domain KB-target. Which slot-values to substitute depends on the Dialogue Domain Adaptation strategy and will be discussed later.

17Finally, all the slot-values - both from utterances and annotations - that could not be substituted through DDA, need to be automatically translated, which will result in the following:

“Ho bisogno di un ristorante a est che abbia caraibico cibo e un economico fascia di prezzo per favore .”

18As can be noted, a downside of using placeholders is that this method does not consider the subject-verb agreement, nor the order of the words to be different between the original and the translated text. It should also be observed that in the cases of north and moderate, the slot substitution selects different values from the KB, while in the case of Caribbean it could not find a correspondence in the KB, hence it got translated directly from the original.

4. Experimental Setting

19We started from the public available dataset MultiWOZ 2.3 (Han et al. 2020), which consists of a collection of more than ten thousand annotated dialogues (with dialogue states) spanning over seven domains related to traveling in Cambridge (e.g., restaurants, hotels, attractions, trains).

Pisa KB-target

20We manually created a KB-target for Pisa, mirroring the instance distribution of the KB-source for Cambridge. For every entity instance of the Cambridge KB, a corresponding Pisa instance was created, keeping the slot-names as they were in the original, and changing only the slot-values. The specific instances were chosen by analysing the frequency distribution in the Cambridge KB and finding a similar correlation in the Pisa domain. For example, all the Cambridge restaurants with indian food type, which is the most common in Cambridge, were substituted with Pisa restaurants with italian food type, which is the most common in Pisa. All the Pisa instances were taken from publicly available datasets containing real information on Pisa entities.2

Automatic translation

21As for translation from English to Italian, we used the automatic translator available at FBK.3 The MT engine is built on the ModernMT framework4 which features neural machine translation implementing the Transformer architecture (Vaswani et al. 2017). A big model (more than 200 million parameters) is trained on generic domain data, taken from the OPUS repository5.

22Test data used in the experiments were manually checked, correcting a number of translation issues, including, for instance, wrong prepositions used for time expressions (from di 13:00 to delle 13:00), and wrong agreements (from prezzi medio to prezzi medi). Training data were not corrected.

Datasets

23We run experiments over the following four datasets:

• Cam-ENG. This is the original MultiWOZ 2.3 dataset, with Cambridge as domain and English as language. It is used as referent for the other experiments.

• Cam-ITA. This is the translation to Italian of the original MultiWOZ 2.3 dataset, with Cambridge as domain.

• Pisa-ENG. This is the original MultiWOZ 2.3 dataset adapted to the new Pisa knowledge base, using dialogue domain adaptation, as described in Section 3.

• Pisa-ITA. This is the MultiWOZ 2.3 dataset, first translated into Italian and then domain adapted to the Pisa knowledge base.

24For all the datasets we kept the same training/test split of dialogues as in the original MultiWOZ 2.3. In addition, we have experimented the following combinations:

• Cam-ITA + Cam-ENG. This combination provides all the available data for the Cambridge domain, mixing the two languages.

• Pisa-ENG + Cam-ENG. This combination provides all the available data for English, mixing the two domains.

• Cam-ITA + Pisa-ITA. This combination provides all the available data for Italian, mixing the two domains.

• Pisa-ITA + Pisa-ENG. This combination provides all the available data for the Pisa domain, mixing the two languages.

Dialogue State Tracking Model

25The goal of the experiments is to assess the robustness of a dialogue state tracking model when domain and language are changed. As for DST model, we have used TRADE (Wu et al. 2019), an algorithm optimized for being used on multi-domain dialogues such MultiWOZ.

5. Results

26Results of the experiments are presented in Table 1. The first column indicates which dataset the model was trained on; the second column reports the dataset used for testing the model; the last four columns report measures on the model performance. Training Accuracy refers to the Joint Accuracy obtained at training time; Turn Accuracy indicates how many single predictions were actually correct; the Joint F1 score reflects the accuracy of the model, considering both precision and recall; finally, the Joint Accuracy, measures the percentage of correct predictions of dialogue states for every dialogue turn, where a prediction is considered correct if all the slot values in the dialogue turn are correctly predicted. Results are reported into four groups depending on the dataset that has been used for testing. For every group we have three configurations: the first experiment reports the performance with the initial dataset, the second considers the extension of the initial dataset with the second language, and finally, the third experiment considers the extension of the initial dataset with the second domain.

6. Discussion

27Results reveal several interesting aspects. First, we register a decrease in performance between the datasets in English and those automatically translated to Italian. This can be due to the process of placeholder insertion and subsequent substitution of slot-values, along with the translation itself, which can be source of errors. On the other side, the domain adaptation from Cam-ENG to Pisa-ENG and from Cam-ITA to Pisa-ITA did not show the same decrease of performance, rather it resulted even in a small increase for the first case.

28The central part of our work, however, focused on generating adapted dialogues and investigating the performance variations derived from them. With regards to this aspect, it should be noted that the addition of a second language resulted in a significant improvement almost in all cases, with an increase of 5% for Cam-ITA, 4% for Pisa-ENG and 13.5% for Cam-ITA. On the other side, the addition of the second domain does not bring much advantage, resulting in zero change for Cam-ENG and Pisa-ITA, a small decrease for Cam-ITA and a small increase for Pisa-ENG.

6.1 Overlaps Between Datasets

29In order to better understand the factors that affect the variation of Joint Accuracy performances between the datasets of each group, we have analysed the overlaps among the training datasets. We estimated such overlap as the proportion of slot-values in two datasets for every domain that are exactly the same .

30We have observed that in most of the cases adding a dataset with high overlap for a certain domain produces an improvement in DST performance for that domain. As an example, the domain with highest overlap between the Cam-ITA dataset and the Pisa-ITA dataset is Taxi (86.11% of overlap). On the other side, the domain with lowest overlap between the same datasets is Attraction (44.45% of overlap). These overlaps have strong correlation with the DST performances on the two domains: the Cam-ITA + Cam-ENG dataset produces an improvement of 1.5 points with respect to the Cam-ITA dataset on the Taxi domain, and shows a decrease of 1 point on the Attraction domain.

31This correlation can also be verified if we look at a slot-name level. Table 2 shows some examples of slot-names with corresponding overlaps between dialogues and slot-name prediction accuracy, taken from the Cam-ITA setting with domain and language additions. As it can be noted, when the slot-name overlap between the aggregated dialogue and Cam-ITA is higher, the respective prediction accuracy also tends to be higher.

7. Conclusion

32We have investigated domain and language shift for data-driven task-oriented dialogue systems. We have extended recent work on dialogue domain adaptation to a cross-language setting, where both the domain and the language are changed. We showed that: (i) state-of-the-art models are very sensible to language shift obtained through automatic translation; (ii) combining training data for the two languages is always beneficial; on the contrary, combining data of different domains does not produce any improvement in all of our settings. Finally, as a relevant side effect of our work, we present a new dataset for dialogue state tracking available for Italian, derived from MultiWOZ 2.3. All the data are made available for further research on dialogue domain adaptation.