Version classiqueVersion mobile

Formaliser les langues avec l’ordinateur : de INTEX à Nooj

 | 
Svetla Koeva
, 
Denis Maurel
, 
Max Silberztein

Premiere partie. Formalisation de ressources linguistiques

8. Hungarian Morphology in INTEX/Nooj

Tamás Varadi, Péter Vajda et Viktor Nagy

Texte intégral

Introduction

1The Hungarian language is a member of the Finno-Ugric language family, whose members share the property, among others, that they are agglutinative. Agglutinativity often entails morphological richness i.e. high number of inflectional categories and derivational operations as is the case for Hungarian. This complexity of morphology stands as a challenge to a wide number of linguistic applications which are optimized for Western Languages.

2The complexity of Hungarian inflectional morphology comes from two factors : the large number of distinct forms in a paradigm and the large number of distinct paradigmatic types.

3Let us consider the nominal inflectional paradigm. Hungarian nouns have 17 cases, which can be combined with a plural marking morpheme or with a singular or plural anaphoric possessive morpheme (so 17 gets multiplied by 2 × 3, we have already 85 slots in the paradigm). Nouns can additionally bear possessive markings in three persons and two numbers, together with the lack of possessor there are 7 cases.17 ×2 ×3 × 7 = 714 (we did not take familiar plural into account as its usage is very limited for semantic reasons). Adjectives are also gradable, thus the adjectival paradigms contains three times more forms (base, comparative, superlative) (2 142). Without entering into the details, the size of verbal paradigms in Hungarian can be stated as 163 but the participles derived from verbs have the same complexity as the nouns.

4The large number of inflectional types is caused by the high variability of phonological and orthographical properties across the lexemes. The Dictionary of Hungarian Inflections [Elekfi 1994] enumerates ca.1 200 nominal and 500 verbal inflectional types. (See [Abondolo 1988] and [Kornai 1994] for more about Hungarian morphology.)

5The finite state technology is well suited to store large morphological databases in a compact format and to apply them fast, but the compilation of minimal and deterministic automata might face high memory costs. Defining the appropriate Hungarian morphological transducer is also not a trivial task. For example the simple brute-force method that lists all the word forms would do the job for English or French but is certainly not viable for Hungarian : a lexicon containing 10 000 nouns, 10 000 adjectives and 10 000 verbs would generate more than 33 million Inflectional forms and we did not even count the derived forms. This paper presents the various approaches we tried and the experiences we gained in our attempt to implement Hungarian morphology in INTEX/NooJ finite state framework.

I. The Hungarian Morphological Database

6The starting point of our work was the Morphological Database of Hungarian Academy of Sciences, which is the electronic version of the Dictionary of Hungarian Inflections (DHI). Contrary to its name, the dictionary covers not only the inflection but several productive derivational categories, as well. All the lexemes in the dictionary are assigned to a paradigm class. The paradigm class describes what affix morphs the stem of the lexeme gets and how the stem changes in the paradigm. If two lexemes’ affixation differ in only one position of the paradigm, they are ordered to two distinct paradigms. This classification shows a very high degree of redundancy. Figure 1 illustrates how the paradigms are defined in the database.

Table 1 : Definition of nominal paradigm 1A1 (sample stem : kocsi “carriage”) and 7B2 (sample stem : meder/medr “river-bed”)

Class :

1A1

7B2

nominative

˷ø

kocsi

˷ø

meder

illative

˷ba

kocsiba

˷be

mederbe

accusative

˷t

kocsit

@et/˷t

medret/ medert

plural

˷k

kocsik

@ek

medrek

anaphoric possessive

˷é

kocsié

˷é

mederé

˷ means the basic stem, @ refers to the altered stem.

7The database can be used for synthesis in a convenient way, however a more effective format is required for analysis. At the transition from the morphological database to the INTEX / NooJ grammar we followed two solutions. The first approach kept the paradigm classification intact and left the efficiency up to INTEX / NooJ. We present two methods for this approach in Sections 3 and 4. The second approach uses a feature based classification, which was derived from our database, so it suppresses the redundancy and allows a much smaller description of affixation. This method is presented in Section 5. In Section 6 we show how the simple brute-force method can be used as a fallback solution.

II. Paradigm based method

8Our first experiment was to use the information of the DHI as it is, which means that for each paradigm we automatically created a subgraph with all the possible suffixes of that paradigm. We also created a main graph which would recognize the lemma and the relevant subgraph. Thus the main graph would contain some 2 000 subgraphs, and each of them contains between 700 and 1 000 suffixes. A small part of such a subgraph can be seen in Figure 1.

Figure 1 : Subgraph for one nominal paradigm class (detail)

Figure 1 : Subgraph for one nominal paradigm class (detail)

9The dictionary contains the lemma and stem-variants and the paradigm information for each entry, altogether 50 000 entries for nominals. This huge amount of information was too big for INTEX to handle, the graphs could not be transformed into FSA’s. We tried to compile the graph with fewer paradigms, and the biggest part INTEX could handle was 55 subgraphs, which compiled in about 4 hours. Applying the lexical resources to a text of 5164 words took 40 seconds (see Table3). The resulting analysis is precise (as the graphs constructed contain all the morphological information), but it would be misleading to compare recall to that of other methods, as this set-up contains only one third of the nominal paradigm, and no verbal paradigms were included in the dictionaries.

10A somewhat similar approach would be to use inflectional grammars in NooJ for describing Hungarian morphology, as with these grammars all the paradigms could be described. Furthermore stem-variants can be calculated by using NooJ’s”Generic commands” such as <B> (Backspace) or <R> (Go right one character). As stem variants follow a well-defined pattern inside a paradigm, they are easy to define as shown in the following example.

bokor, bokor, N+NOM
bokor+nak, bokor, N+DAT
bokr+ot, bokor, N+ACC

11The inflectional description of the forms in the above examples would be as follows :

bokor = <E>/NOM + nak/DAT + <L><B><R>ot/ACC ;

12The corresponding entry in the dictionary is :

bokor, , N+FLX=bokor

13This information is coded in the DHI (see Table 1), so we could automatically create the inflectional description. The compilation times of the associated dictionaries are shown in Table3. The inflectional descriptions contain about 1 000 rules (suffixes) per paradigm (as above with the sub-graph method) and the compilation was only possible with few hundred word dictionaries because of memory limitations. Once the dictionary is compiled into a .nod form, the Lexical Analysis is faster than in INTEX : it analyzes20 000 words in about 2 seconds on a 1.7GHz machine.

III. Suffix-sequence dictionary

14To reduce the number of paths in the graph, we experimented with enlisting all the possible suffix-sequences in one graph for the nominal paradigm (There are almost 6 000 of them). Attaching these to a stem without any further constraints, gives us a quick but not very accurate method. Accuracy is low because of the great amount of homonymy in Hungarian, both for lemmas, and suffixes alike. For example the singular 3rd person possessive form of the word nemzet (“nation”) is nemzet-ei, but as the suffix -ei has an allomorph - i, and nemzet-i is also a well-formed word (adjective derived from the noun above) this latter would be badly analyzed as singular 3rd person possessive, although the correct analysis would appear as well. This is only one of many examples, which makes this method impractical, as recall is high, but the precision is very low.

15To improve this method we introduced constraints, which would, by the use of paradigm numbers, allow only the appropriate form to attach to a certain stem. We used only NooJ dictionaries for this method. There is one dictionary for the stems, which includes e.g. :

nemzet, nemzet, N+C3B+stem=1+NW

16There is another dictionary which contains all the sequences of suffixes for each of the paradigms (There are about 960 000 of them). For the above example the relevant entries would be :

...
(*) ei, <$1=N+C3B+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+ NOM}
(*) ei, <$1=N+C3B2+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+ NOM}
(*) ei, <$1=N+C3B3+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+ NOM}
(*) ei, <$1=N+C3B4+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+ NOM}

...
(*) i, <$1=N+C1B+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+N OM}
(*) i, <$1=N+C1B1+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+ NOM}
(*) i, <$1=N+C1B2+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+ NOM}
(*) i, <$1=N+C1B3+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+ NOM}
(*) i, <$1=N+C1B4+stem=1>{$0,$1L, N$1S+PS+ps_sg+ps_3+i+ NOM}
...

17Here, the paradigm numbers (e.g. +C3B) act as lexical constraints, and prevent the form nemzet-i to be analyzed as PS+ps_sg +ps_3+i+NOM. Additionally the +stem feature chooses the basic stem allomorph.

18The zero affixed forms are stored separately in a third dictionary for efficiency reasons, e.g. :

nemzet, nemzet, N+NOM

19This method proved successful as regard the dictionary compilation times (See Table3). Its drawback is that it can not be easily extended, new derivational morphemes are difficult to add to. Moreover the number of lexical constraint lookups makes this method quite slow, as a suffix can be present in a few hundred paradigms on average.

IV. Feature based method

20The main disadvantage of the DHI’s classification is that it is difficult to observe the systematic differences and similarities among the paradigms. It uses only a two dimensional classification, one dimension is the vowel harmony, the other comprises complex features of the stems, such as stem-types, the presence of linking vowel, etc. Furthermore it creates a new paradigm even for the most minute differences as we saw before in Section 2. This is why it has some 1 700 different paradigms. In reality there are fewer differences, especially if we take into account some shared properties of the different paradigms. One such property is, that there are only about six different stem-variant types in terms of both how the variants are calculated from the lexical form, and how the different suffixes attach to the lexical and the variant forms. Another property is the presence or absence of “j” for some of the possessive suffixes, or the presence or absence of a linking vowel before the accusative suffix which are independent of the”stem-variant property”. The DHI has different paradigms for all the possible combinations of these properties, whereas the independent ones could be handled with the assignment of morphophonological features to the stems, and lexical constraints to the suffixes. Creating a grammar with this method enables us to reduce the number of states in the graphs, and gives us a linguistically adequate solution.

21The finite state transducers of INTEX / NooJ allow us the use of lexical features and constraints by checking, during the lexical analysis, if the features of the stem in question, and the constraints of the suffixes match each other. To create the grammar, which includes the suffixes, we had to describe the possible ordering of the morphemes, and we had to assign the relevant features to the stems. The assignment could be done automatically, as from the Morphological Database we could get all the suffixes that help to decide whether the stem in question has to have a certain feature or not. In Table 2 we can see that according to the morphemes possessive 3rd person singular and accusative case there are four different cases, and these words do belong into four different paradigms in the DHI, and had to be stored in four different sub-graphs in the previous solution. By using features we have to include the suffixes in the grammar only once together with their appropriate constraints.

Table 2 : Assigning features according to allomorphs

stem

PS. SG3

ACC

features

gloss

hajó

- ja

- t

+j, +accnolink

ship

ház

- a

- at

+nonj, +acclink

house

hold

- ja

- at

+j, +acclink

moon

gonosz

- a

- t

+nonj, +accnolink

evil

22In Figure 2 we can see how this information is coded in simplified NooJ morphological (sub) graphs. The lexical constraints j, nonj, acclink, accnolink, select between the allomorphs and they are checked against the dictionary, which includes :

Figure 2 : Subgraphs for Table 2

Figure 2 : Subgraphs for Table 2

hajó, , N+j+accnolink+...
ház, , N+nonj+acclink+...
hold, , N+j+acclink+...
gonosz, , N+nonj+accnolink+...

23We first used this method in INTEX. We encountered some difficulties, e.g. : INTEX only allows one-letter codes for both features and morphological annotation. Furthermore owing to memory problems INTEX could not handle our grammar and dictionary, i.e.. it did not analyze all the forms covered by the grammar. Developing this method in NooJ, performance was much better, and we arrived at a solution where the recall figures became acceptable (see Table3).

Table 3 : Evaluation of methods. Recall and analysis time were measured on a 5 000 word newswire text

Experiment

Compilation time

Analysis time

Recall

Sub-graphs (INTEX)

4h

40s

15 %

Suffix-sequences with lexical constraints (NooJ)

1m:10s

1m:32s

85,4 %

INTEX feature-based

1m:12s

12s

48,5 %

NooJ feature-based

4s per .dic

3m:12s

89 %

V. Listing of word forms

24We showed above that none of the available INTEX methods could produce a reliable morphological module for Hungarian. However, since lexicon and morphology are essential for parsing at a higher level, we had to compromise.

25We collected all the word forms with their analysis from the Hungarian National Corpus [Váradi 2002] occurring more than once. As we do not have a robust morphological analyzer based on our morphological database yet, the analysis was produced by the HUMOR [Prószéky and Tihanyi 1993] analyzer. Words that were unknown to the analyzer were left out. The word form collection contains about 2.5 million entries, which had to be split into smaller DELAF dictionaries, because we did not have enough resources to compress the whole lexicon into one transducer. In a 1.5GHz machine with 512MB memory the compression of 600 thousands entries took33 seconds, the compression of 900 thousands entries took 62 seconds with INTEX.

26The parts of the corpus-based lexicon can be applied together with INTEX and the parsing runs fast. The analysis achieved 93 % recall on a newswire text collection, which is not part of the Hungarian National Corpus. For texts coming from a more specific domain (e.g. psychology) the recall can be more lower, but we can create a specialized dictionary with the same collect-and-analyze method if there is a large enough corpus available in that domain.

27We tried to create the same dictionary resources with NooJ, but it performed weaker : compression of300 thousands entries took 80 seconds on the same computer, and 600 thousands entries demanded more memory than it was available physically.

Bibliographie

Bibliography

ABONDOLO Daniel M. (1998), Hungarian Inflectional Morphology Akadémiai Kiadó, Budapest.

ELEKFI László (1994), Magyar ragozási szótár – Dictionary of Hungarian Inflections, MTA Nyelvtudományi Intézete, Budapest.

KORNAI András (1994), On Hungarian Morphology, MTA Nyelvtudományi Intézete, Budapest.

PRÓSZÉKY Gábor, TIHANYI László (1993), “Humor : High-Speed Unification Morphology and Its Applications for Agglutinative Languages”, La tribune des industries de la langue, n° 10, OFIL, Paris, p. 28-29.

VÁRADI Tamás (2002), “The Hungarian National Corpus”, in Proceedings of the Third International Conference on Language Resources and Evaluation, vol. II, Las Palmas.

SILBERZTEIN Max, The INTEX Manual, http://intex.univfcomte.fr/downloads/Manual.pdf.

Table des illustrations

Titre Figure 1 : Subgraph for one nominal paradigm class (detail)
URL http://books.openedition.org/pufc/docannexe/image/27039/img-1.jpg
Fichier image/jpeg, 67k
Titre Figure 2 : Subgraphs for Table 2
URL http://books.openedition.org/pufc/docannexe/image/27039/img-2.jpg
Fichier image/jpeg, 7,3k
Légende hajó, , N+j+accnolink+...ház, , N+nonj+acclink+...hold, , N+j+acclink+...gonosz, , N+nonj+accnolink+...
URL http://books.openedition.org/pufc/docannexe/image/27039/img-3.jpg
Fichier image/jpeg, 8,1k

Auteurs

Research Institute for Linguistics (Budapest, Hungary)
E-mail : varadi@nytud.hu.

Research Institute for Linguistics (Budapest, Hungary)
E-mail : vajda@nytud.hu.

Research Institute for Linguistics (Budapest, Hungary)
E-mail : nagvy@nytud.hu.

© Presses universitaires de Franche-Comté, 2007

Conditions d’utilisation : http://www.openedition.org/6540

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search