1. Introduction

1Topic models are statistical methods that aim to extract the hidden topics underlying a collection of documents (Blei, Ng, and Jordan 2003; Blei 2012; Boyd-Graber, Hu, and Mimno 2017). Topics are often represented by sets of words that make sense together, e.g. the words “cat, animal, dog, mouse” may represent a topic about animals. Topic models’ evaluations are usually limited to the comparison of models whose hyper-parameters are held fixed (Doan and Hoang 2021; Terragni, Fersini, and Messina 2020; Terragni, Nozza, et al. 2020). However, hyper-parameters can have an impressive impact on the models’ performance and therefore fixing the hyper-parameters prevents the researchers from discovering the best topic model on the selected dataset.

2Recently, OCTIS (Terragni, Fersini, et al. 2021 Optimizing and Comparing Topic Models is Simple) has been released: a comprehensive and open-source framework for training, analyzing, and comparing topic models, over several datasets and evaluation metrics. OCTIS determines the optimal hyper-parameter configuration according to a Bayesian Optimization (BO) strategy (Archetti and Candelieri 2019; Snoek, Larochelle, and Adams 2012; Galuzzi et al. 2020). The framework already provides several features and resources, among which at least 8 topic models, 4 categories of evaluation metrics, and 4 pre-processed datasets. However, the framework uses a single-objective Bayesian optimization approach, disregarding that a user may want to simultaneously optimize more than one objective (Terragni and Fersini 2021). For example, a user may be interested in obtaining topics that are coherent but also diverse and separated from each other.

Contributions

3In this paper, we propose OCTIS 2.0, an extension of the existing framework that integrates both a single-objective and multi-objective hyper-parameter optimization strategy, using Bayesian optimization. Moreover, we also pre-process and include two novel datasets in Italian. We will then briefly show the potentiality of the extended framework by comparing different topic models on the new released Italian datasets. We believe these resources can be useful for the topic modeling and NLP communities, since they can be used as benchmarks for the Italian language.

2. OCTIS: Optimizing and Comparing Topic Models is Simple!

2.1 OCTIS 1.0

4OCTIS (Terragni, Fersini, et al. 2021 Optimizing and Comparing is Simple!) is an open-source evaluation framework for the comparison of topic models, that allows a user to optimize the models’ hyper-parameters for a fair experimental comparison. The evaluation framework is composed of different modules that interact with each other: (1) dataset and pre-processing tools, (2) topic modeling, (3) hyper-parameter optimization, (4) evaluation metrics. OCTIS can be used both as a python library and through a web dashboard. It also provides a set of pre-processed datasets, state-of-the-art topic models and several evaluation metrics.

5We will now briefly describe the two components that we will extend in this work: the pre-processed datasets and the hyper-parameter optimization module.

Pre-processing and Datasets

6OCTIS currently provides functionalities for pre-processing the texts, which include the lemmatization of the text, the removal of punctuation, numbers and stop-words, and the removal of words based on their frequency. Moreover, the framework already provides 4 pre-processed datasets, that are ready to use for topic modeling. These datasets are 20 NewsGroups,1 M10 (Lim and Buntine 2014), DBLP,2 and BBC News (Greene and Cunningham 2006). All the datasets are split into three partitions: training, testing and validation.

7All the currently provided datasets are in English. OCTIS already provides language-specific pre-processing tools (e.g. lemmatizers for multiple languages), but it does not present datasets in other languages. Creating benchmark datasets for other languages is useful for investigating the peculiarities of different topic modeling methods.

Single-Objective Hyper-parameter Optimization

8OCTIS uses single-objective Bayesian Optimization (Snoek, Larochelle, and Adams 2012; Shahriari et al. 2015) to tune the topic models’ hyper-parameters with respect to a selected evaluation metric. In particular, the user specifies the search space for the hyper-parameters and an objective metric. Then, BO sequentially explores the search space to determine the optimal hyper-parameter configuration. Since the models are usually probabilistic and can give different results with the same hyper-parameter configuration, the objective function is computed as the median of a given number of model runs (i.e., topic models run with the same hyper-parameter configuration) computed for the selected evaluation metric. OCTIS uses the Scikit-Optimize library (Head et al. 2018) for the implementation of the single-objective hyper-parameter Bayesian optimization.

9The use of a single-objective approach is however limited. In fact, this strategy disregards other objectives. For example, a user may require to optimize the coherence of the topics and their diversity at the same time.

2.2 OCTIS 2.0

New dataset resources for the Italian language

10Since OCTIS provides only English datasets, we extend the set of datasets by including two new datasets in Italian. We build the two datasets from the Italian version of the Europarl dataset3 and from the Italian abstracts of DBPedia.4 In particular, we randomly sample 5000 documents from Europarl and we randomly sample 1000 Italian abstracts for 5 DBpedia types (event, organization, place, person, work), for a total of 5000 abstracts.

11We preprocess the datasets using the following strategy: we lemmatize the text, we remove the punctuation, numbers and Italian stop-words, we filter out the words with a document frequency higher than the 50% and less than the 0.1% for Europarl and 0.2% for DBPedia and we also remove the documents with less than 5 words. These values have been chosen by manually inspecting the resulting pre-processed datasets.

12We report the most relevant statistics of the novel Italian datasets in Table1. Following the original paper, we split the datasets in three partitions: training (75%), validation (15%), and testing (15%).

From Single-objective to Multi-objective Hyper-parameter Bayesian Optimization

13Given the limitations of the single-objective hyperparameter optimization approach, we extend OCTIS by including a multi-objective approach (Kandasamy et al. 2020; Paria, Kandasamy, and Póczos 2019). Single-objective BO can be in fact generalized to multiple objective functions, where the final aim is to recover the Pareto frontier of the objective functions, i.e. the set of Pareto optimal points. A point is Pareto optimal if it cannot be improved in any of the objectives without degrading some other objective. Using a multi-objective hyper-parameter optimization approach thus allows us not only to identify the best performing model, but also to empirically discover competing objectives.

14Since the original Scikit-Optimize library does not provide multi-objective optimization tools, we use the dragonfly library5 (Paria, Kandasamy, and Póczos 2019). Like the single-objective optimization, the user must specify the hyper-parameter search space. But in addition, they also need to specify which functions they want to optimize. We report a simple coding example below:

# loading of a pre-processed dataset
dataset = Dataset()
dataset.fetch_dataset("DBPedia_IT")

#model instantiation
lda = LDA(num_topics=25)

#definition of the metrics to optimize
td = TopicDiversity()
coh = Coherence()
metrics = [td, coh]

#definition of the search space
config_file = "path/to/search/space/file"

#define and launch optimization
mmm = MOOptimizer(
       dataset=dataset, model=model,
      config_file=config_file,
       metrics=metrics, maximize=True)
mmm.optimize()

15The snippet will run a multi-objective optimization experiment that will return the Pareto front of the diversity and coherence metrics on the Italian dataset DBPedia by optimizing the hyper-parameters (defined in a configuration file) of LDA with 25 topics.

16In keeping with the spirit of the first version of OCTIS, the framework extension is open-source and easily accessible, in order to guarantee researchers and practitioners a fairer, accessible and reproducible comparison between the models (Bianchi and Hovy 2021). OCTIS 2.0 is available as extension of the original library, at the following link: https://github.com/mind-Lab/octis.

3. Experimental Setting

17In the following, we will show the capabilities of the extended framework on the new datasets by carrying out a simple experimental campaign.

18We assume an experimental setting in which a topic modeling practitioner is interested in discovering the main thematic information of the two novel datasets in Italian. However, the user does not have prior knowledge on the datasets, therefore does not know which topic model is the most appropriate. Moreover, the user aims to get topics which are coherent and make sense together but which are also diverse and separated from the others. Let us notice that a user could consider a different set of metrics to optimize, by selecting one of the already defined metrics available in OCTIS or by defining novel metrics.

3.1 Evaluation Metrics

19We briefly describe the two evaluation metrics (one of topic coherence and one of topic diversity) that we will target as the two objectives of the multi-objective Bayesian optimization. Both metrics need to be maximized.

IRBO

20 (Bianchi, Terragni, and Hovy 2021; Terragni, Fersini, and Messina 2021) is a measure of topic diversity (0 for identical topics and 1 for completely different topics). It is based on the Ranked-Biased Overlap measure (Webber, Moffat, and Zobel 2010). Topics with common words at different rankings are penalized less than topics sharing the same words at the highest ranks.

NPMI

21 (Lau, Newman, and Baldwin 2014) measures Normalized Pointwise Mutual Information of each pair of words (w_i, w_j) in the 10-top words of each topic. It is a topic coherence measure, that evaluates how much the words in a topic are related to each other.

3.2 Topic Models and Hyper-Parameter Setting

22We focus our experiments on four well-known topic models that OCTIS already provides, two of them are considered classical topic models and the others are neural models. In particular, we trained Latent Dirichlet Allocation (Blei, Ng, and Jordan 2003 LDA), Non-negative Matrix Factorization (Lee and Seung 2000 NMF), Embedded Topic Model (Dieng, Ruiz, and Blei 2020 ETM), Contextualized Topic Models (Bianchi, Terragni, and Hovy 2021; Bianchi et al. 2021 CTM).

23We summarize the models’ hyper-parameters and their corresponding ranges in Table 2. For each model, we optimize the number of topics, ranging from 5 to 100 topics. We select the ranges of the hyper-parameters similarly to previous work (Terragni and Fersini 2021).

24Regarding LDA, we also optimize the hyper-parameters α and β priors that the sparsity of the topics in the documents and sparsity of the words in the topic distributions respectively. These hyper-parameters are set to range between 10^-3 and 10^-1 on a logarithmic scale.

25The hyper-parameters of NMF are mainly related to the regularization applied to the factorized matrices. The regularization hyper-parameter controls="true" if the regularization is applied only to the matrix V, or to the matrix H, or both. The regularization factor denotes the constant that multiplies the regularization terms. It ranges between 0 and 0.5 (0 means no regularization). L1-L2 ratio controls="true" the ratio between L1 and L2-regularization. It ranges between 0 and 1, where 0 corresponds to L2 regularization only, 1 corresponds to L1 regularization only, otherwise it is a combination of the two types. We also optimize the initialization method for the two matrices W and H.

26Since ETM and CTM are neural models, their hyper-parameters are mainly related to the network architecture. We optimize the number of neurons (ranging from 100 to 1000, with a step of 100). For simplicity, each layer has the same number of neurons. We also consider different variants of activation functions and optimizers. We set the dropout to range between 0 and 0.9 and the learning rate, that to range between 10^-3 and 10^-1, on a logarithm scale. We fix the batch size to 200 and we adopted an early stopping criterion for determining the convergence of each model.

27Moreover, only for CTM we also optimized the momentum, ranging between 0 and 0.9, and the number of layers (ranging from 1 to 5). Following (Bianchi et al. 2021), we use the contextualized document representations derived from SentenceBERT (Reimers and Gurevych 2019). In particular, we use the pre-trained multilingual Universal Sentence Encoder.6

28For all the models, we set the remaining parameters to their default values. Finally, we train each model 30 times and consider the median of the 30 evaluations as the evaluation of the function to be optimized. We sample the n initial configurations using the Latin Hypercube Sampling, with n equal to the number of hyperparameters to optimize plus 2 to provide enough configurations for the initial surrogate model to fit. The total number of BO iterations for each model is 125. We use Gaussian Process as the probabilistic surrogate model and the Upper Confidence Bound (UCB) as the acquisition function.

4. Results

29In the following, we report the results of the comparative analysis between the considered models on the Italian datasets.

4.1 Quantitative Results

30We jointly consider the results of both objectives by plotting the Pareto frontier of the results of topic diversity and topic coherence. Figure 1 shows the frontier of each model for the pair of metrics (NPMI, IRBO). We can notice that the topic models have similar frontiers in each dataset. The most competitive models are NMF and CTM. In particular, NMF outperforms the others for the topic coherence but gets a lower coherence as the diversity increases. Therefore, CTM is the model to prefer if a user wants to get totally separated topics but good coherence. Instead, LDA and ETM have lower performance than the others. We also noticed from our experiments that the performance of ETM is affected when the documents are shorter (on the Europarl dataset), often originating the phenomenon of mode collapsing, i.e. obtaining all the topics equal to the others.

Figure 1

4.3 Qualitative Results

31In Table 3 we report an example of topics discovered by the models. We selected the best hyper-parameter configuration discovered by the models with 5 topics and randomly sampled a model run among the 30 runs. Let us notice that, for the sake of simplicity, we have to fix the number of topics here and select a run among the total of 30 runs. Therefore, the qualitative results reported in Table 3 may not reflect the overall results.

32We can notice that NMF obtains more coherent and stable topics. CTM and LDA obtain topics that have a higher variance: in particular, CTM discovers a topic (the fourth one, NPMI=-0.51) that lowers the average coherence, while LDA discovers a topic (the second one, NPMI=0.48) that effectively increases the average coherence. On the other hand, the topics discovered by ETM are more stable but have a lower coherence on average. As already observed in previous work (AlSumait et al. 2009; Doogan and Buntine 2021), obtaining junk or mixed topics is common in topic models and this problem can be addressed by filtering out the topics that are less relevant.

5. Conclusion

33In this paper, we presented OCTIS 2.0, the extension of the evaluation framework OCTIS for topic modeling. This tool can now address the problem of estimating the optimal hyper-parameter configurations of different topic models using a multi-objective Bayesian optimization approach. Moreover, we also released two novel datasets in Italian which can be used as benchmark datasets for the Italian topic modeling and NLP communities.

34We conducted a simple experimental campaign to show to potentiality of the extended framework. We have seen that using a multi-objective hyperparameter optimization approach allows us not only to identify the best performing model over the others, thus guaranteeing a fairer comparison among different models, but also to empirically discover the relationships between different objectives.

35As future work, we aim to extend the framework by considering additional datasets in different and possibly low-resource languages, which require different pre-processing strategies and would allow researchers to investigate the peculiarities of different topic modeling methods.