Introduction: The motivation for the production of aligned resources

1Compilation of Serbian parallel corpora at the Faculty of Mathematics (Belgrade) began with the participation in the TELRI project, and production of CD "East meets West - A Compendium of Multilingual Resources", which contains, among other resources Plato’s Republic aligned in 17 languages and Orwell’s 1984 aligned in 8 languages, in both cases including Serbian. We have continued to collect texts, mainly for French-Serbian parallel corpus where French is source and Serbian target language. Corpus consists predominantly of literary and newspaper texts. Aligned literary texts include among others Voltaire’s Candide, J. Vern’s Le tour du monde en quatre-vingt jours, G. Flaubert’s Bouvard et Pécuchet, and P. Louys’s La Femme et son pantin. In some cases two transaltion in Serian were obtained. These texts have been tagged to the sentence level, and then aligned using either Vanilla (Danielsson, 1997) or MLAlign alignment program (Romary, 1995). The obtained output had to be in both cases hand proven.

2Certain number of contemporary texts translated to Serbian is being prepared for the alignment as well. Those are mainly texts related to philosophy, sociology, ethnology, and sciences. However, for contemporary texts, contrary to literary classics, it was usually difficult to obtain both the original and the translation in digital form.

3The newspaper corpus consists mainly of the French monthly "Le Monde Diplomatique" and its translation to Serbo-Croatian. The acquisition of texts has started in March 2001, when the publication of Serbian translation began, and collecting of both source text and target text is done regularly since: the source texts are downloaded from the "Le Monde Diplomatique" site while the translation is obtained directly either from the publisher or translator. Part of these articles has been aligned, the other part is in the phase of preprocessing in order to be aligned.

4The motivation for the construction of this kind of multilingual resource is multifold:

1. It can be used as a powerful linguistic resource, for instance for language teaching.

2. In bilingual or multilingual lexicography aligned texts can be a source of reliable data. For the production of traditional bilingual dictionaries aligned corpora can provide evidence of translation equivalents. Also, for the construction of semantic networks, such as BalkaNet that is being constructed using WordNet methodology (Miller, 1990) they can be used to check semantic relations.

3. More specifically, this kind of resource can help to solve the problem of structural derivation by redefining a Serbian entry in dictionaries of the DELAS/DELAF type or by constructing appropriate finite automata. In this case French is a kind of meta-language with which one can try to encompass the differences in translation. For instance, the analysis of Candide’s aligned texts showed that four entries in Serbian DELAS, namely <baron> (Engl. baron), <baronica> (Engl. baroness), <baronov> (Engl. baron’s), <baroničin> (Engl. baroness’s) correspond to one entry in French DELAS <baron> (Engl. baron), (Vitas 2002).

1 – Software for text alignment

5Two basic approaches are used for automatic text alignment: statistical and structural. A well known program using statistical approach is Vanilla (Danielsson, 1997). This program works with texts segmented in two levels at most. These two levels are usually interpreted as paragraphs and sentences, but can actually be any other tags. The algorithm is based on the presumption that two units that correspond to each other have approximately the same number of characters. As a consequence, it is not possible to align a structurally tagged text with a rough text. Structural tags are not taken into consideration during the alignment process. The program is simple to use and can be obtained as an open code. However, in some cases many interventions in aligned texts have to be done by hand. A serious drawback of the algorithm is the prerequest that both texts have to have the same number of higher level units. The program is not supported by a concordancer, but the latter can be developed independently. One example of aligned units from already mentioned Vern’s novel is:

*** Link: 1 - 2 ***
<seg id=’VernFr.1.1.7.4’>En tout cas, il n’était prodigue de rien, mais non avare, car partout où il manquait un appoint pour une chose noble, utile ou généreuse, il l’apportait silencieusement et même anonymement.</seg> . EOS
<seg id=’VernSr.1.1.7.4’>U svakom slucyaju nije bio rasipnik, ali ni tvrdica.</seg> <seg id=’VernSr.1.1.7.5’>Gde god je nesxto trebalo za neku plemenitu, korisnu ili velikodusxnu stvar, on je davao cxutecxi i neopazxeno.</seg> . EOS

6XML tag <seg> is used in both target and source text for tagging sentence elements. In this example, one source sentence is aligned wirh two target sentences, as is notified by Link: 1 – 2 sign.

7An example of a structural approach is MLAlign program by Laurent Romary and Patrice Bonhomme. For the use of this program the logical layout of texts has to be XML tagged. The program maps the logical layout of the source text to the logical structure of the target text. A concordancer has been developed that supports it. The problem is that not all of the available resources are XML tagged. Our experience shows that human XML tagging is time consuming and biased by human tagger, while, the automatical tagging is error prone.

2 – The use of lexical resources for alignment

8Various ideas have already been exploited in order to improve the alignment process (for instance, (Chan, 1993)) but none of them concentrates on a selection of a subset of textual units that are, as a rule, literally translated. The idea to use the lexical resources in alignment arose during our first experiments with the exploitation of aligned texts of Candide. This experience showed that the use of Intex (Silberztein, 1993) with each of the texts in turn was more useful than the use of sentence aligned texts. The reason for this is simple: usually the concordances of both the original and the translation are consulted in search for the translation equivalents, and lexical resources incorporated in Intex enabled a precise specification of search requirements for both languages.

9The experience with aligned texts also suggested that there are certain text units that are more often than not translated literally. In order to check this presumption one simple experiment has been undertaken using texts from the issue of "Le Monde diplomatique" of May 2001 where the main topic are advertisements. The initial presumtion was proven right for certain simple text units, such as dates and currencies. These elements can be described by graphs that are similar in both source and target languages. The distribution of some other lexical units for which simple graphs can be constructed for both languages has shown similar behaviour on the same text samples. Such units are, for instance, toponyms and proper names. Besides these self-evident sets of literally translated text units another set of this kind emerged: it contains those units that Intex identifies as "unknown words" where one usually finds trademarks, different acronyms, etc. In order to identify their occurrences in both original and translated text it is sufficient to construct a graph that recognizes units that belong to the intersection of sets of unknown words in both original and its translation. Those units are not being translated, they are rather transferred to the translated text. For instance:

10The excerpt from “unknown words” in the French version of “Le Monde Diplomatique”

11The corresponding excerpt "unknown words" from the Serbian version

12The density of some of these literaly translated lexical units for the original text is illustrated in Figure 1.

13The density of lexical elements that are literally translated shows that they can be reliable ‘anchors’ for the corresponding segments in the original text and its translation. In principle, using the elements recognized by Intex, it is possible to pre-edit the text for the aligners of Vanilla type (using the paragraph tags, if they exist, and {S} tags incorporated by Intex for sentences), as well as for MLAlign (XML/SGML output from the FST that recognizes the literally translated lexical units).

14On the other hand, the alignment process can be seen as a generalization of the bootstrapping method proposed in (Gross, M. 2000) where local grammars are developed step by step in order to cover specific meanings of keywords in concordances. Here, the bootstrapping method is applied in order to incorporate as many corresponding tags in source and target text as necessary to cover it with anchors with appropriate density.

15Finally, this process of lexical recognition enables XML-tagging of both the original and the translation by tags of the form:

<tag attribute = "recognized unit">textual unit</tag>.

16For instance, the same tag is inserted in the French text (<temp value= "33C."> 33 degrée Celsius </temp>) and in the Serbian text <temp value= "33C.">Trideset i tri stepena Celziusa </temp>, where the attribute value represents the "canonic" value of the recognized sequences obtained as the output of the corresponding transducer.

17The information obtained form such tags is twofold: the tag name represents the type of the recognized lexical unit while the attribute value enables the comparison of tags in the source and target text. Only the tags with the same name and attribute value can be potential anchors.

One more example

18The existence of literal translations is less obvious in literary texts. The next experiment with lexically driven alignment was undertaken using the text of Flaubert’s novel Bouvard et Pécuchet.

19The first step was to automatically identify the stable "points", or anchors, that connect the original and the translation in a way suitable for alignment. In Flaubert’s novel those "points" were, in the first place, names of the main heroes. However, while in French text only the form Bouvard occurred, in Serbian text the name of the same hero appeared in several inflective forms of the noun <Buvar> and its corresponding possessive adjective <Buvarov>:

Buvar; N: Buvar+Buvara+Buvaru+Buvarom+Buvare
Buvarov; AdjPoss : Buvarov+Buvarova+Buvarovoj+Buvarovom+

Buvarovog+Buvarovu+Buvarovih+Buvarovi+Buvarovim+Buv arove+Buvarovo

20A similar situation occurred with the keyword Pécuchet, as one French form – Pécuchet – corresponded to several forms of the noun Pekisxe and possessive adjective Pekisxeov in Serbian text:

Pekisxe; N: Pekisxe+Pekisxea+Pekisxeu+Pekisxeom
Pekisxeov; AdjPoss : Pekisxeov+Pekisxeova+Pekisxeove+
Pekisxeovi+Pekisxeovo+Pekisxeovu+Pekisxeovim+Pekisxeov og+
Pekisxeovoj+Pekisxeovom

21However, when these correspondences were taken into consideration the frequencies of occurrences of names Bouvard et Pécuchet were exactly the same in the original and translated text. Their distribution in the original and translation showed the expected similarity, as can be seen in figures 2 and 3 respectively.

22The diagram shows that both in the original and its translation the names of the main characters covered just above 1 % of whole text. Excerpts from the concordances of the original and the translated text are given in Appendix I.

23These concordances were produced by using the following regular expression in ‘Locate Pattern’ option

((<Bouvard>+< Pécuchet>) (: < verbs of announcement >))*

24and then by sorting the produced lines according to ‘Text Order’ (Silberztein, 2002).

25This regular expression specifies the refinedsubset of the third subset represented in Figure 3.

3 – The basic idea of the algorithm

26In order to produce the aligned concordances with "literally" translated equivalents as keywords the information in the corresponding concord.ind files can be used that gives the starting position of every such tag with respect to the beginning of the file. The aligned unit now is not a sentence but rather a text excerpt that begins with one keyword and ends with another. The further development of the program was done in two steps:

271. The user can combine files with indices of the original and the translation in several different ways. For instance, one choice is to align using only one keyword, that is pairs of identical tags such as {ime: buvar}, or to use various keywords that user can choose from a list. The prerequisite is, of course, that the text has been indexed with the chosen keyword (s).

282. The program that produces concordances on the basis of intervals that approximately represent the equivalent sequences in both texts functions as follows:

All the occurrences that correspond to a certain pattern in both original and translated text are identified using Locate Pattern facility. In the course of the concordance production in one language, an independent application associates to the context of current text (original or translation) the corresponding interval from the aligned text, under the assumption that the order of literally translated units in source and target texts are the same.

29The input data for the program, as well as the program results are given in Appendix II. In aligned concordances two consecutive tags bound the lines, with approximately 20 characters preceding the starting tag and 20 characters following the ending tag. This expansion outside tag boundaries seems necessary, as translation equivalents tend to occur in the vicinity of anchors, that is literally translated units. Along with concordances the program produces row statistics: number of characters between tags in source and original text and their ratio.

30The first obtained results seem to be promising, as the explicit tagging of original and translation becomes obsolete. Moreover, it is possible to generate aligned concordances according to various keywords, and even to align according to sentence boundary (Intex {S} tag) using some additional keyword that would enable alignment of text units shorter than a sentence.