Version classiqueVersion mobile

Formaliser les langues avec l’ordinateur : de INTEX à Nooj

 | 
Svetla Koeva
, 
Denis Maurel
, 
Max Silberztein

Introduction

2. NooJ’s Linguistic Annotation Engine

Max Silberztein

Texte intégral

Introduction

  • 1 Cf. (Silberztein 1999a) for a description of the INTEX toolbox, and (Silberztein 1999b) for a descr (...)

1NooJ is a new linguistic development environment, similar to the INTEX software1 and designed to replace it in the near future. NooJ allows users to process corpora made of large numbers of texts in real time. Users can build, accumulate and manage sophisticated indices from dictionaries and morpho-syntactic grammars.

I. A new technology

2NooJ is based on the . NET “Component programming” technology, which goes a step beyond the Object-Oriented approach (Löwy 2003, Silberztein 2004). Beside being an excellent programming development environment, . NET gives NooJ several key advantages, including :

  • It allows NooJ to process any document that can be managed on the user’s computer. For instance, on a typical MS-Windows computer, NooJ can process corpora in 100 + file formats, including all variants of ASCII, ISO and Unicode, HTML, RTF, XML, MS-WORD, etc.

  • It allows other . NET applications to access all NooJ’s public methods via its software component library. For instance, a programmer can easily run a NooJ method to extract sequences of texts that match a NooJ grammar from a document that is opened in the current application (e.g. MS-WORD). Beside its command-line programs that are roughly equivalent in functionalities to INTEX’s, NooJ’s linguistic engine can be accessed via a Object-Oriented dynamic library (DLL) to create any number of independent instances of NooJ’s linguistic engine. Developpers can then build applications that could process mutilingual documents, or multiple documents, or multiple linguistic phenomena at the same time. NooJ’s architecture makes it adequate to be used in an Multi-Agent System similar to Talisman’s, see (Stefanini 1995).

  • NooJ’s linguistic engine is based on the . NET class : System. Globalization. CultureInfo which gives it access to the properties of any of the 200+ language/culture pairs supported by . NET. For instance, . NET processes 13 cultures for English (en-AU, en-GB, en-US, etc.), 6 cultures for French, (fr-BE, fr-CA, fr-FR, fr-CH, etc.), 19 for Spanish, etc. NooJ knows the alphabet of each language, its alphabetical order (which depends on each culture and might be quite complex), its elision and hyphenation algorithms, the date and currency formats, etc. Of course, NooJ allows users to add support for other culture or languages not supported by . NET, as for instance Latin, Chiac (a Canadian dialect), etc.

II. A new linguistic engine

3NooJ’s most important characteristic, of course, is its linguistic engine, which is based on an annotation system. An annotation is a pair (position, information) that states that at a certain position in the text, a sequence is associated with a certain piece of information. NooJ annotates texts, and annotations are stored in each Text’s Annotation Structure.

1. Text Annotation Structure

4NooJ provides several linguistic tools to annotate texts :

5– NooJ’s morphological parser is capable of analyzing complex word forms, such as Hungarian words and Germanic compounds, as well as tokenizing Asian languages. When analyzing complex word forms, the morphological parser usually produces sequences of annotations. For instance, figure 1 shows an extract of the morphological grammar (here, a simple Finite-State Transducer) that analyzes contracted word forms such as “du” : notice that the annotation (the resulting output of the graph) consists of a sequence of two annotations : {de, PREP} followed by {le, DET} :

Figure 1 : annotating contracted words

Figure 1 : annotating contracted words

6– NooJ’s lexical parser processes the inflection and the derivation of large dictionaries for simple and compound words. As of now, NooJ contains dictionaries for Arabic, Armenian, Chinese, Danish, English, French, Hungarian, Italian and Spanish. In general, running NooJ’s lexical parser results in adding multiple lexical annotations to a text. The annotation system can represent all types of morphological, lexical, syntactic and semantic ambiguities, such as between compounds and sequences of simple words (e.g. “round table”), overlapping or embedded compounds (e.g. “round table mat”), etc. Figure 2 displays the annotation structure produced by the lexical parser.

Figure 2 : lexical annotations

Figure 2 : lexical annotations

7– NooJ’s syntactic grammars are Recursive Transition Net-works ; they allow users to recognize and annotate certain sequences of words in texts. NooJ’s graphical editor provides a dozen tools to edit, test and debug local grammars, to organize them in libraries, and to apply them to texts, either as queries, or to add (or filter out) annotations to the Text Annotation Structure. For instance, the grammar in figure 3 can be used to add the annotation <DATE> to the text :

Figure 3 : a syntactic grammar produces a <DATE> annotation

Figure 3 : a syntactic grammar produces a <DATE> annotation

8Note that one controls exactly where annotations start and end (not obligatorily the entire matching sequence). Like the morphological parser, the syntactic parser can produce complex sequences of embedded annotations. It is also possible to compute annotations’ properties incrementally, according to each path in the graph, e.g. <DATE+Past>, <N+Hum+Politics+USA>, etc.

9Figure 4 shows a syntactic grammar that filters out (rather than adds) annotations : all text annotations that are not compatible with the output are to be deleted from the Text Annotation Structure. This feature makes it possible for NooJ to simulate teh behavior of INTEX’s disambiguation grammars. Both operations, i.e. add or remove annotations, can be performed at the same time by a single grammar.

10Syntactic grammars can be applied to a text either manually (the result is a concordance that might be edited), or automatically, at the time when the text is parsed (the grammar has to be added to the Syntactic Analysis module, see Info > Preferences). In that latter case, grammars can be ordered, and then applied in cascade ; grammars that share the same order number will be applied together (NooJ computes their union).

Figure 4 : a disambiguation grammar

Figure 4 : a disambiguation grammar

11NooJ’s query system and parsers can access any previously inserted annotation. For instance, the following query includes references to word forms (e.g. “mind”) as well as to two annotations (noted between angle brackets) :

(the+these) <N+Hum> <lose> their (mind+temper)

12<N+Hum> matches all sequences of the text that are associated with an “N” annotation with property “Hum” ; these annotations might have been added by NooJ’s lexical parser (e.g. for the word “director”), by NooJ’s syntactic parser when applying a grammar to recognize Human proper names (e.g. for the sequence “Mr. John M. Smith, head of the XYZ inc.”), or by NooJ’s importation of an XML tag from an XML document. Similarly, <lose> matches all sequences of the text that are associated with an annotation whose lemma is “lose” ; these annotations might have been added by the lexical parser (for all conjugated forms of “to lose”, e.g. “lost”), or by a syntactic grammar that recognizes compound tenses, e.g. “have not yet lost”. When all resulting matching sequences have been indexed (e.g. “These men have not yet lost their mind”) they can be annotated in turn, and the new annotations are then instantly available for other queries or for further syntactic grammars.

2. NooJ TAS and XML

13Text information represented as tags in an XML document can be imported to NooJ ; for instance, importing the following XML text :

<N Hum> Mr. John Smith </N>

14produces an annotated text in which the sequence “Mr. John Smith” is annotated with the NooJ tag “N+Hum” (category “N” ; property “Hum”). This annotation in turn can be used in any NooJ grammar or query, such as “<N+Hum> <V>” to locate human nouns followed by a verb. Note that XML tags become transparent, allowing users to run queries such as “Smith <V>” (in that case, NooJ simply ignores the ending XML tag after “Smith”).

15NooJ can also export its Text Annotation Structures to XML documents. In that case, all relevant NooJ’s information will be mapped as XML tags. This feature allows NooJ to be integrated with, and / or to communicate to another NLP application.

16Because of the possible ambiguities in NooJ’s TAS, it is not always possible to export NooJ’s annotations to an XML document. For instance, in the following text :

... Fair Trade Coffee machine...

17NooJ’s lexical parser will recognize and annotate free terms : “Fair Trade”, “Fair Trade Coffee” and “Coffee machine”. However, the following text :

<N1><N2>Fair Trade</N2><N3>Coffee</N1> machine</N3>

18would not constitute a valid XML text. In that case, NooJ uses a set of priorities (“longest match”, “left to right”) to ignore some of the annotations, in order to produce a valid XML document. In the previous example, NooJ would produce the following XML text :

<N1>Fair Trade Coffee</N1> machine

19For this reason, it is best to reduce the number of import / export processes to a strict minimum.

3. INTEX dictionaries

20NooJ is compatible with the INTEX application, the LADL methodology and its dictionaries. However, NooJ innovates in the way dictionaries are built and used.

21The INTEX linguistic engine (Silberztein 1993) used two sets of dictionaries that were designed at the LADL laboratory : on the one hand, DELA-type dictionaries describe the morphology of simple and compound words (Courtois, Silberztein 1990) ; on the other hand, lexicon-grammars describe syntactic properties of predicates (verbs or predicative nouns) as well as frozen expressions (Gross M. 1993).

22A number of problems with INTEX have already been discussed in (Silberztein 2004). The most important are :

  • The separation of DELAS-F (simple word forms) and DE-LAC-F (compound word forms) dictionaries is artificial, and has a number of unpleasant consequences. For instance, we need to process “priori” (from the adverb “a priori”) as a lexical entry of a DELAS ; there are hundreds of such artifacts. A large number of terms have parts of their orthographical variants stored in DELAS dictionaries, and others in DELACs ; e.g. “pianoforte” is stored in a DELAS, whereas “piano forte” is stored in a DELAC. INTEX is not capable of processing compounds’ inflection automatically, etc.

    • 2 (Roche 1993) does describe the formalization of one lexicon-grammar table with meta-graphs that are (...)

    Lexicon-grammar tables are awkward to use. They need to be associated with meta-graphs, that cannot be re-used with other INTEX syntactic grammars. These meta-graphs become more and more complex as semi-frozen expressions accept aspectual variants and different sets of syntactic structures, to the point that they would be totally impractical for use in large numbers (2).

  • DELAS / DELAC dictionaries are not adapted to either multilingual applications or semantic applications in which lexical entries are associated with synonymous terms or expressions, hyperonyms, etc (Koeva S., Mihov S. 2005). Not to mention that INTEX needs five different types of dictionaries, each one with its own formalism...

4. NooJ’s dictionaries

23In NooJ, these five dictionaries are represented in one unique format. In order to unify the formalism, we needed to (1) merge simple and compound words (2) eliminate DELAFs and DELACFs and (3) add information to DELA-type dictionaries so that they can be processed in the same way as lexicon-grammar tables.

a. Merging simple and compound words

24NooJ dictionaries contain indistinctly simple or compound words thanks to a new inflectional and derivational engine that can process both simple and compound words’ morphology in a unified way. For instance, the two following lexical entries :

academic program, N+FLEX=APPLE
window, N+FLEX=APPLE

25inflect the same way (they take an “s” in the plural). Therefore, they are both associated with the same inflectional class : APPLE. The class APPLE is defined by the following expression :

APPLE = <E>/singular + s/plural ;

26that states that if one adds nothing to the lexical entry (“<E> singular”) ; if one adds an “s” to the end of the lexical entry, one gets the plural form (“plural”).

  • 3 NooJ contains a dozen labs aimed at demonstrating its functionalities. The “inflection” lab is an i (...)

27NooJ’s inflectional and derivational engine uses a dozen default commands that operate on the suffix of lexical entries, for instance <B> (for “Backspace”, <D> for “duplicate”, etc.) ; linguists can override these commands, and add their own : for instance, we have added operators such as <A> (“remove accent”) and <À> (“add a grave accent”) to the inflectional engine of certain Romance languages, and the operator <F> (“Finalize letter”) for Hebrew (3).

28In contrast with INTEX, NooJ is capable of inflecting compounds. For instance, the class “ACTOFGOD” is defined by the following expression :

ACTOFGOD = <E>/singular + <PW>s/plural ;

29The operator <PW> stands for : “go to the end of the first component of the lexical entry”. Note that the following three entries are associated with this class, even though their length is different :

bag of tricks, N+FLX=ACTOFGOD
balance of payment deficit, N+FLX=ACTOFGOD
member of the opposite sex, N+FLX=ACTOFGOD

30In the same manner, even though the lexical entries blank piece of paper, last line of defense, family history of cancer, sexual harassment in the work place, etc. have different lengths, they are associated with a unique morphological class, because their inflection is carried by the same component (the second one).

31Agreements between components of a compound can be described as well. For instance, the following inflectional expression formalizes the agreement between the two components of compounds such as journeyman carpenter :

<E>/singular + s<P><B2>en/plural

32To get the plural form, add an “s” to the end of the compound, then go back to the previous (<P>) component, delete the two last characters (<B2>), and add the suffix “en”.

33In conclusion, NooJ can process simple and compound words’ inflections completely automatically. This has allowed us to unify descriptions of simple and compound words.

b. No more DELAFs

  • 4 For compatibility reasons, users can still import their own DELAF-DELACF into NooJ. However, if the (...)

34NooJ’s lexical parser requires no DELAF/DELACF type dictionaries (4).

35NooJ dictionaries, which are effectively similar to DELAS (or DELAC) dictionaries, are compiled into a Finite-State Transducer that is more powerful than previous DELAF transducers, and faster in some cases.

36This characteristic is more important than a mere increase in comfort for linguists and users, or even an increase in computational efficiency : encoding the inflectional information inside NooJ dictionaries gives NooJ the ability to perform morphological operations during parse time. Whereas INTEX could link an inflected form only to its lemma (by a lookup of the DELAF transducer), NooJ can link an inflected form to any other inflected form. Thus, NooJ can perform complex transformations within texts. For instance, it is now possible to replace a specific conjugated verb with its past participle form, and vice-versa, within a particular text :

John eats the apple <=> the apple is eaten by John
be (command : “_PR+3+s”) => is
eats (command : “_PP”) => eaten

37Using this new functionality will enhance several current NLP applications, such as automatic translation and information retrieval applications.

c. Unifying Lexicon-Grammar tables and DELA dictionaries

38NooJ dictionaries combine the advantages of both lexicon-grammar tables and DELA-type dictionaries : they can be displayed either in list form (i.e. as DELA) or in table form (as lexicon-grammar tables). This has lead us to adapt both the notation of lexicon-grammar tables and DELA-type dictionaries.

  • 5 Lexicon-grammar tables need to be slightly adapted to be used by NooJ. For instance, the “-” charac (...)

39Importing a lexicon-grammar table in NooJ is straightforward (5), but requires rendering all properties common to the table entries explicit. Each lexicon-grammar table is indeed defined by a set of syntactic properties ; see (Leclère 2002). For instance, the table “4” which is described in (Gross 1975), is associated with the transitive structure verbs N0 V N1, in which N0 (the subject) is an abstract noun or a sentence, and N1 (the object) is a Human noun, e.g. :

This problem upsets John
That it rained all day amuses Mary

40Therefore, in NooJ all table 4 entries must be associated with the explicit properties : +N0VN1 (to state that the verb enters in the transitive syntactic structure), +N0= : Abst and +N1= : Hum.

41When a lexicon-grammar table has been adapted to NooJ, displaying it in a list view is straightforward : each row of a table corresponds to a line in the list ; non-negative cells are prefixed with a character “+”, e.g. “+N0VN1” ; textual cells are transcribed into a pair Property=Value, e.g. “+PREP=from” ; negative cells are simply ignored.

  • 6 Marie-Hélène Stefanini’s team at the Laboratoire d’Informatique Fondamentale de Marseille has forma (...)

42The next figure displays a sample of the table 4 of the lexicon-grammar6 :

Figure 5 : NooJ’s compiled version of a lexicon-grammar table

Figure 5 : NooJ’s compiled version of a lexicon-grammar table

43Importing a traditional DELA dictionary is more complex : it requires that features of the dictionary (which correspond to property values of a lexicon-grammar) be typed, so that all the values of a common property can be regrouped in one same column. We also need to state the number of relevant properties for each category of word (e.g. Tense for Verbs, Number for Nouns, etc.), in order to distinguish absent default values from irrelevant ones. This is done via a Property Definition file that contains rules such as :

N_Distribution = Hum + Conc + Abst ;
N_Gender = m + f ;
N_Number = s + p ;
...

V_Tense = Present + Futur +... ;
V_Pers =1 + 2 + 3 ;
V_Number = s + p ;
...

44A given property may be associated with more than one category (e.g. Number is relevant both for nouns and verbs). But NooJ checks that one feature (e.g. “+p”) does not correspond to more than one property (e.g. “Gender” and “Tense”).

45Associating NooJ lexical features with typed properties opens up possibilities for implementing unification mechanisms in the future.

46By breaking the INTEX association of lexicon-grammar tables and their specific meta-graphs, we have been able to process any type of lexicon-grammar table, even the ones that describe free constructs. Meta-graphs, which have been effectively replaced with “regular” NooJ grammars, can now be reused and accumulated.

Conclusion

47NooJ’s technology, linguistic engine, and annotation and dictionary systems are a significant enhancement of INTEX’s.

48NooJ’s annotation system is more powerful than INTEX’s text transducer : it does not need to be contiguous, and it can import information from (and export to) XML documents.

49The ability to manage both DELA-type dictionaries and lexicon-grammar tables in a unified way ends the artificial dichotomy that has existed for years between two types of activities : users can now display their dictionaries as they wish, in table or list form, and can access both morphological and syntactic properties at the same time.

50Although NooJ has just come out and its technology is quite new, it is already being used by several research teams in a variety of projects. For more information on NooJ, see NooJ’s WEB site : http://www.nooj4nlp.net.

Bibliographie

Bibliographie

COURTOIS B., SILBERZTEIN, M. Eds (1990), Dictionnaires électroniques du français, Langue française n° 87, Paris, Larousse.

COURTOIS B. (1990), Le dictionnaire DELAS, in Dictionnaires électroniques du français, Langue française n° 87, Paris, Larousse, Paris, p. 11-22.

GROSS M. (1975), Méthodes en Syntaxe, Paris, Hermann.

GROSS M. (1993), Les phrases figées en français, in L’information grammaticale, Paris, p. 36-41.

LECLERE C. (2002), Organization of the Lexicon-Grammar of French verbs, in Lingvisticæ Investigationes 25 :1, Amsterdam / Philadelphia, John Benjamins Publishing Company, p. 29-48.

KOEVA S., MIHOV S. (forthcoming 2005). Semantic Relations, in INTEX in INTEX pour la Linguistique et le traitement automatique des langues (2), Les Cahiers de la MSH Ledoux, Besançon, Presses Universitaires de Franche-Comté.

LÖWY J., Programming . NET components, USA, John Osborne Ed. O’Reilly & Associates.

ROCHE E. (1993), Une représentation par automate fini des textes et des propriétés transformationnelles des verbes, in Lingvisticae Investigationes XVII :1, Amsterdam / Philadelphia, John Benjamins Publishing Company, p. 189-222.

SILBERZTEIN M. (1990), Le dictionnaire DELAC, in Dictionnaires électroniques du français, Langue française n° 87, Paris, Larousse, p. 73-83.

SILBERZTEIN M. (1993), Dictionnaires électroniques et analyse automatique de textes : le système INTEX, Paris, Masson.

SILBERZTEIN M. (1999), Traitement des expressions figées avec INTEX, in Analyse lexicale et syntaxique : le système INTEX, Lingvisticae Investigationes XXII, Amsterdam / Philadelphia, John Benjamins Publishing Company, p. 425-449.

SILBERZTEIN M. (2004), NooJ : an oriented object approach, in INTEX pour la Linguistique et le traitement automatique des langues, Les Cahiers de la MSH Ledoux, Besançon, Presses Universitaires de Franche-Comté.

STEFANINI M.-H, DEMAZEAU Y (1995), TALISMAN, A Multi-Agent System for Natural Language Processing, SBIA 1995, p. 312- 322.

VIETRI S. (2004), Lemmatization of Idioms in Italian, in INTEX pour la Linguistique et le traitement automatique des langues, Les Cahiers de la MSH Ledoux, Besançon, Presses Universitaires de Franche-Comté. SILBERZTEIN Max (1999), INTEX : a finite-state transducer toolbox, in Theoretical Computer Science #233 :1, p. 33-46.

Notes

1 Cf. (Silberztein 1999a) for a description of the INTEX toolbox, and (Silberztein 1999b) for a description of its application as a corpus processing system. See various INTEX WEB sites for references and information on its applications, workshops and communities : http://intex.univ-fcomte.fr and the NooJ WEB site for a description of NooJ : http://www.nooj4nlp.net.

2 (Roche 1993) does describe the formalization of one lexicon-grammar table with meta-graphs that are similar to INTEX’s, but the size of the resulting grammar and the number of grammars that would have to be implemented (there are a hundred tables) makes this approach extremely cumbersome to implement for an actual large corpus processing application.

3 NooJ contains a dozen labs aimed at demonstrating its functionalities. The “inflection” lab is an interactive window in which students can play with these operators and see via a graphical simulator, how the inflection engine works.

4 For compatibility reasons, users can still import their own DELAF-DELACF into NooJ. However, if the inflectional system is not integrated in NooJ, NooJ will not be able to perform morphological operations during parse time.

5 Lexicon-grammar tables need to be slightly adapted to be used by NooJ. For instance, the “-” character (binary “false”) had to be distinguished from the “<E>” symbol (empty string), words that could be inflected (between angles) had to be distinguished from constant word forms, etc.

6 Marie-Hélène Stefanini’s team at the Laboratoire d’Informatique Fondamentale de Marseille has formalized the lexicon-grammar tables of 8,000+ French.

Table des illustrations

Titre Figure 1 : annotating contracted words
URL http://books.openedition.org/pufc/docannexe/image/26954/img-1.jpg
Fichier image/jpeg, 35k
Titre Figure 2 : lexical annotations
URL http://books.openedition.org/pufc/docannexe/image/26954/img-2.jpg
Fichier image/jpeg, 67k
Titre Figure 3 : a syntactic grammar produces a <DATE> annotation
URL http://books.openedition.org/pufc/docannexe/image/26954/img-3.jpg
Fichier image/jpeg, 35k
Titre Figure 4 : a disambiguation grammar
URL http://books.openedition.org/pufc/docannexe/image/26954/img-4.jpg
Fichier image/jpeg, 26k
Titre Figure 5 : NooJ’s compiled version of a lexicon-grammar table
URL http://books.openedition.org/pufc/docannexe/image/26954/img-5.jpg
Fichier image/jpeg, 30k

© Presses universitaires de Franche-Comté, 2007

Conditions d’utilisation : http://www.openedition.org/6540

Rechercher dans OpenEdition Search

Vous allez être redirigé vers OpenEdition Search