Version classiqueVersion mobile
OpenEdition Books

What is the Text Encoding Initiative?

Lou Burnard

The TEI cornucopia, part two

Texte intégral

Editorial matters

1When digitization of primary source materials is undertaken, the first priority is usually to produce good quality digital images of the original source documents. Transcribing and encoding those images is an expensive and difficult operation which cannot readily be automated, if at all, but it is an important next step in the creation of a true ‘digital edition’. For the skilled human reader, the digital image of its pages may be all that is needed to make the source useful. For a wider audience however, and certainly for any kind of automated analysis or searching, an encoded transcription will be an essential complement to the image. The TEI provides a number of mechanisms to facilitate the production of digital editions of every kind.

Image and transcription

2 We mentioned above the <facsimile> element which can be used to group and organize digital images in terms of the physical surfaces they represent (using the <surface> element) and the logical zones of interest within those surfaces (using the <zone> element). A <facsimile> may be all that a TEI document contains, or this may be complemented by a transcription.

3Where both facsimile and transcription are available, a global attribute @facs may be used to link them, or parts of them, as appropriate. Suppose for example that we have a single leaf, with images of its two surfaces in two files called page1r.png and page1v.png respectively. We transcribe the two surfaces within a <text> element as usual, using <pb> to mark the points in the transcription that correspond with the start of each new page in the source. The resulting document might look like this:

	<surface xml:id="s1">
		<graphic url="page1r.png"/>
	<surface xml:id="s2">
		<graphic url="page1v.png"/>
<pb facs="#s1"/>
<!-- text transcribed from page 1 recto here -->
<pb facs="#s2"/> <!-- text transcribed from page 1 verso here -->

4Note that associating the transcription with the surface rather than with the image file directly offers us the flexibility of adding further image files (of higher or lower resolution, for example) without perturbing the structure.

5The @facs attribute used above is global, and can therefore be used to associate any part of a transcription with any part of a <facsimile>. We could for example link a few words in the transcription with that part of the image where they appear.

Editing a transcription

6When transcribing a source, particularly an ancient one, there is always a tension between the desire to represent faithfully the actual state of the document, and the desire to make that document comprehensible to a modern or untrained reader. The process of transcription necessarily involves a kind of normalization, in which the different written symbols (the glyphs) of a manuscript or an early print document are mapped on to the unambiguous codes of a modern character set. An encoding which tries to represent the words of a document produced according to norms different from our own, or to correct ‘self evident’ errors in the copy text is thus almost inevitable. The TEI provides ways of marking stretches of text as having been normalized or corrected, by the encoder, whether or not we consider this to be ‘editing’ in the traditional sense.

7We mentioned above that the TEI consciously places itself within the long western philological tradition concerned with the identification and recovery of text from scattered documentary witnesses. It is scarcely surprising therefore to find that it also provides facilities for the representation of such phenomena as scribal correction or alteration where these are manifest in a particular document, or for the representation of a collation of variant readings. Where pre-digital textual editors were necessarily concerned with the production of a single carefully considered and substantiated print version of a text, the economics of digital production have greatly facilitated the production of ‘diplomatic’ digital or documentary editions, which claim to present objectively the facts of a given collection of documents in such a way that the reader may determine the plausibility of any accompanying ‘edited’ version, or even derive such an edition for themselves.

8When transcribing a source text, the <corr> element can be used to enclose a part that has been corrected and the <reg> element a part that has been normalised. For the opposite case, the <sic> element may be used where something should have been corrected but has not; and the <orig> element for cases where the original spelling has been retained even though it looks unfamiliar. For example, a modernized version of the Shakespeare sonnet shown above would probably begin

<l>My Mistress' eyes are nothing like the sun,</l>
<l>Coral is far more red than her lips red,</l>

9Even in such a modernized version, a conscientious encoder wishing to show where they have departed from the original source may simply wrap the words Mistress’, sun etc. with the <reg> element. More usually however an encoder will wish to retain the original orthography along with the modernized version, so that either is available for display. The <choice> element addresses this requirement:

	</choice> eyes are nothing like the
	</choice> is far more red
	</choice> her lips red,

10 The <choice> element may be used wherever several different possible encodings are possible but only one of them may be chosen for a particular application. It is less appropriate in the case where there are many different possible witnesses, providing multiple readings all of which need to be taken into consideration. The TEI provides a more powerful and complex set of tags to mark up a traditional critical apparatus of this kind. The element <app> (apparatus) is used to mark the point of variation being considered; within this element, the encoder can supply a series of <rdg> (reading) elements, one for each variant reading identified. If editorial policy permits, one of these readings may be considered as the ‘primary’ or ‘correct’ one, in which case it will be tagged using the <lem> element rather than as just another <rdg>. The <rdg> element has a @wit attribute which can be used to identify the witnesses in which the reading in question is attested. For example, the first line of Chaucer’s Wife of Bath’s Tale begins Experience though noon Auctoritee..., but its first word is spelled quite differently in three of the surviving manuscripts. Taken as a whole, such spelling variations provide useful evidence for the reconstruction of the history of the manuscript tradition in question, and our encoding therefore wishes to preserve them. We might do so as follows:

<l n="1">
	<rdg wit="#El #Hg">Experience</rdg>
	<rdg wit="#La">Experiment</rdg>
	<rdg wit="#Ra2">Eryment</rdg>
</app> though noon Auctoritee...

11Here the values of the @wit attribute identify the manuscripts in question: as the hash-sign indicates, they are actually links to items in a list of the manuscripts which were collated to produce this apparatus.

Encoding the writing process

12Our last example of the facilities offered by the TEI for working on the detailed transcription of primary source materials introduces a few of the elements needed to record the actual writing process, for example to record passages which have been deleted, added, corrected etc. whether by the author of a literary text or by a scribe copying out a manuscript. An analysis of such documentary modifications may be essential before a reading text can be presented, and is clearly of importance in the editorial process.

13The example is taken from the surviving authorial manuscript of a poem by the English writer Wilfred Owen, a part of which is shown here:

Figure 6

Figure 6

Detail from Dulce et decorum est autograph manuscript in the English Faculty Library, Oxford University.

14Owen first wrote Helping the worst amongst us, but then deleted it, adding Dragging the worst amongt us over the top. In the same way, he revised the phrase half-blind by deleting the half- and adding all above it. In the last line, he started a word beginning fif before deleting it and writing the word five-nines. We can encode all of this as follows:

<l>And towards our distant rest began to trudge,</l>
	<del>Helping the worst amongst us</del>   
	<add>Dragging the worst amongt us</add>
	</subst>, who’d no boots
<l>But limped on, blood-shod. All went lame;
	<del status="shortEnd">half-</del>
	</subst> blind;
<l>Drunk with fatigue ; deaf even to the hoots</l>
<l>Of tired, outstripped
	<del>fif</del> five-nines that dropped behind.

15The tags <add> and <del> elements are used to enclose passages added or deleted respectively. Additional attributes are available such as @resp to indicate responsibility for the modification, or @place to indicate where in the text (for example, above or below the line) the modification has been made. Where the encoder wishes to assert that the addition and deletion make up a single editorial act of substitution, these elements can be combined within a <subst> element as shown above.

16A very careful examination of Owen’s second modification shows that he really did write amongt rather than amongst, presumably in error. An equally careful editor wishing to restore the missing s might use the <supplied> element to indicate that they have done so:

<add>Dragging the worst among<supplied resp="#ED">s</supplied>t us

17Here the @resp attribute has been used to indicate that the S was not supplied by Owen but by someone else, specifically the person documented elsewhere by an element with the identifier ED.

Transcribed speech

18The digital medium stores and represents sound or video just as easily and effectively as it does images. Transcriptions of sound or video recordings may be used to complement them in the same way as they are used to complement page images. Whether automatically or manually produced, transcriptions of speech are of particular interest in many branches of linguistics; for example, in language acquisition studies, in studies of language variation, or of the relationship between speech and writing in both literate and pre-literate societies. The study of transcribed speech is also an essential component of sociological studies such as oral history.

19 However, the units in which such analysis is carried out — in TEI terms, the component elements of a <text> which contains transcribed speech — are not universally agreed on, and may even be controversial. Another complicating factor is that the process of transcribing speech is technically difficult and often carried out with the aid of special-purpose software; this is a complication because most such software systems have a preference for using their own internal formats rather than an externally defined standard.

20Nevertheless, the TEI does propose a set of elements which may be useful for the detailed encoding of speech transcriptions, and which a recent article by Thomas Schmidt shows could be used as a kind of interlingua for existing alternative encoding methods.

21The TEI model is concerned with speech as a communicative system: it therefore proposes the explicit labelling of any part of a recorded sound which has a communicative function or effect. Events such as a bell ringing or an airplane flying overhead are therefore explicitly noted using the <incident> element. Spoken communication is effected using the voice, but not all voice effects are regarded as lexical, and not all communication is vocal: the TEI therefore provides an element <vocal> for non-lexical phenomena such as coughs or laughter as well as an element <kinesic> for non-lexical and non-vocalic phenomena such as gestures or facial expression. These elements are complemented with a small number of others concerned with what might be called the rendition of speech : notably <pause> and <shift> ; the latter being used to indicate changes in voice quality (for example in loudness, pitch, or speed). These function in a way very similar to the milestone elements introduced earlier.

22One of the more controversial notions in speech research is that of turn or utterance: the TEI proposes an element <u> which can be used to delimit a stretch of speech from a single speaker, without making any claim as to the status or discourse function of such a stretch. The element carries a @who attribute which can be used to link it to another element providing information about the person speaking, in the same way as we have already seen for the @ref attribute.

23For many kinds of linguistic analysis, the <s> element mentioned above may be a more useful way of organizing a transcription since it permits the encoder to categorize each segment independently of the utterance containing it, particularly since utterances may be much longer (or shorter) than the usual units of analysis.

24Here is a simplified example taken from the British National Corpus (BNC) in which both <s> and <u> have been used to segment the transcribed speech signal:

<u who="#PS6P0">
 <s n="3">There ain’t</s>
<u who="#PS6NY">
 <s n="4">
  <shift new="laughing"/>There is<shift/>
<u who="#PS6P0">
 <s n="5">Oh shut up</s>
<u who="#PS6NY">
 <s n="6">Ach you do get corners on boats</s>
<u who="#PS6P0">
 <s n="7">No</s>
 <s n="8">Boats are shaped like a bloody rugby ball shape type</s>
<u who="#PS6NY">
 <s n="9">No they ain’t</s>
 <s n="10">One end is and the other one ain’t <pause/> and it was a yacht <pause/> and
   a yacht they got little rooms in or something cos <unclear/> innit</s>
<u who="#PS6P0">
 <s n="11">Oh that<pause/> th the rooms are shaped ni <pause/> like to the size of the
   boat<pause/> you nonce <vocal desc="tut"/>
<u who="#PS6NY">
 <s n="12">Let’s ask your Mum if there’s <pause/> if there’s any corners on a boat</s>
<u who="#PS6P0">
 <s n="13">Of course there ain’t</s>
<u who="#PS6NY">
 <s n="14">Just ask your Mum that</s>
<u who="#PS6P0">
 <s n="15">Yeah</s>
<u who="#PS6NY">
<u who="#PS6P0">
 <s n="16">I bet she’ll probably side with you</s>

25When transcribing speech, the transcriber must also make decisions about how to represent the non-standard or only semi-lexicalised sounds that speakers make, such as hmmm, or the ach or tsk tsk in the above example (in this case the decision has been to treat the former as a word and the latter as a <vocal>). Even when it is clear that the sound is lexical, it is not always clear how it should be spelled: for example, the sound represented above as yeah might equally well be spelled yeh or yuh or even yes.

26In language corpora of this kind it is also common practice to markup each word explicitly, often using the <w> element. Deciding how this tokenisation should be done is not always self-evident: in the BNC, for example, elided forms like ain’t and there’s are tagged as two <w> elements and additional tags are used to distinguish false starts or truncated forms like th and ni in the above example. The main use of such tagging is however to support additional information such as part of speech or morphosyntactic analyses, as described in the section on Linguistic Annotation below.

27There is one further important feature of the TEI model for transcribing speech: alignment. Whereas in an image the transcription has to be aligned within a two-dimensional co-ordinate space, within a time-based medium such as audio, the various parts of a transcription must be located within time. Speakers typically interrupt one another, or speak at the same time; understanding a dialogue is impossible without information about the sequence of events transcribed. The TEI proposes a way of explicitly recording a kind of temporal scale, using the element <timeline>, which contains definitions of points in time, represented as <when> elements. Events in a transcription taking place at a given point in time can then be aligned with the appropriate <when> elements using attributes such as @sync (synchronous). The advantage of this method is that it permits a transcription to represent the minimum of information needed to align its components, without necessitating an accurate timestamp for each of them; such timestamping is however generally provided by an automatic transcription system and it may therefore be simpler to use that.


28The TEI provides elements for the detailed encoding of only a few distinct kinds of document, preferring generally to define elements specific to particular kinds of analysis or approach. However, it does provide a set of tags for the markup of dictionaries and lexica, perhaps because historically these were amongst the first types of major reference work to be demonstrably better managed in digital form than in print.

29Dictionaries, particularly in print form, are unlike many kinds of document in that they are not composed of running text, but rather of discrete entries, within which individual and semantically significant subcomponents can readily be recognised, even though they may not always be presented as consistently as a database designer might wish. A dictionary entry always has a headword, followed typically by some linguistic information such as its part of speech, and by a series of sense definitions, often arranged hierarchically. Etymological information, examples of usage, references to homonyms, etc. may also be present.

30In a TEI-style encoding, all of these and other parts of an entry will be distinguished as clearly as possible, whether the purpose is to produce a new dictionary (in which these things must all be rendered in a consistent way), or to represent an existing one (in which distinguishing these things will improve the range of intelligent searching).

31As a first simple example, consider the following:

Figure 7

Figure 7

‘Apple’ from Johnson’s Dictionary of the English Language (1755)

32A minimal TEI encoding for this entry would distinguish the word form used as lemma, its spelling and grammatical coding, the etymology provided, and the two senses:

  <form type="lemma">
    <orth rend="ALLCAPS">A’PPLE</orth>. <gramGrp>
    <pos norm="noun">n.s.</pos> </gramGrp>
  <etym>[<foreign xml:lang="ang">æppel</foreign>, <lang>Saxon</lang>.] </etym>
  <sense n="1">
    <def>The fruit of the apple tree.</def>
      <quote type="verse">
        <l>Tall thriving trees confess’d the fruitful mold;</l>
        <l>The red’ning <hi>apple</hi> ripens here to gold.</l>
      <bibl>Pope’s <title>Odyssey</title>.</bibl>
  <sense n="2"> <def>The pupil of the eye.</def> <cit> <quote> He instructed him; he kept him as the apple of his eye.</quote> <bibl>Deut. xxxii. 10.</bibl>   </cit>


33Texts, particularly old ones, are often accompanied by extensive annotation and commentary, either supplied by the original writer, or more usually by an editor or later commentator. All kinds of academic writing are also typified by footnotes, sidenotes, glosses, bibliographic references, etc. For all of these features, the most natural kind of tagging is to use a <note> element to contain the annotation itself, and to embed it within the text being annotated at the ‘point of attachment’, that is, the place where a footnote reference, or other mark typically appears in a printed text. Attributes on the <note> element can be used to categorize it in some way, to indicate its approximate placement, and to indicate who is responsible for it.

34In modern printing practice, explanatory annotations are usually moved off to the back of the book. For example, in a discussion of contemporary attitudes to linguists published in 1995, we find the following comment:

Figure 8

Figure 8

Extract from Deborah Cameron’s Verbal Hygiene (p 229)

35The raised figure 6 here is an indication that the author has provided a further comment, which appears duly collected with others some 18 pages later on:

Figure 9

Figure 9

Extract from Deborah Cameron’s Verbal Hygiene (p 247)

36One simple way of encoding this would be as follows:

	<head>Beyond "anything goes"</head>
	<p> Why does the language-maven in the street 
	(or the senior common-room, or the bar at the Groucho
	Club <note>An establishment patronized by media folk
	in London (provided the club will have them as
	members).</note>) have such a low opinion of 
	linguists? Because...

37Here the note is encoded inline even though the printed source text has separated the notes out from the text. This makes it easier to present the note in a variety of ways (for example, as a pop up in an online version) and also simplifies its encoding. However, it is also perfectly feasible to treat the notes as a separate section of the document, and to place explicit links in the text at the point of attachment, using the <ptr> element:

	<head>Beyond "anything goes"</head>
	<p> Why does the language-maven in the street
	(or the senior common-room, or the bar at the Groucho
	Club <ptr target="#note6"/>) have such a low opinion of
	linguists? Because...</p>
	<!-- other notes here -->
	<note xml:id="note6">An establishment patronized
	by media folk in London (provided the club will have 
	them as members).
	</note> <!-- and here -->

38To represent the link between the text and the note, we first give the <note> element a unique identifier (note6) and we then place a <ptr> element at the point in the text from which we wish to refer to it (the “point of attachment”). This pointer element both represents the supralinear 6 in our original source and indicates that its function is to point to the note whose identifier is supplied by its @target attribute. Alternatively, if we care to record exactly the way the point of attachment appears in the text, we could use the <ref> element:

 ...the Groucho Club
            <ref target="#note6">6</ref>) have ...

Linguistic annotation

39 In the course of preparing of an encoded text, there is a natural tendency for the encoder to wish to add their own annotations. In a sense, of course, every piece of markup added to a text represents the result of an analysis, whether human or automatic, and so it is natural to think of representing such annotations incrementally by means of markup to a digital text. The <note> element may be used for this purpose. However, for many people, linguistically motivated analyses (such as this is a verb, or this sequence of words has this syntactic function, or even this is a metaphor) are of a rather different kind from the kinds of annotation (this is a title, this is a personal name, etc.) discussed elsewhere in this book. In particular, there is less consensus about the categories involved: most people will readily agree that ‘place name’ is a useful and definable concept, but the concepts behind linguistic analysis are much more various. The TEI does not, therefore, propose tags such as <noun> or <adjunct> or <metaphor> , but instead it proposes general purpose elements for segmenting texts into smaller units, any of which can carry labels defined by the project.

40This method is particularly useful when a digital text is automatically analysed by a computer program, (for example a morphosyntactic analyser, or part-of-speech tagger) since the format used for the outputs from such automated systems differ considerably. The TEI offers a simple way of explicitly segmenting running text into word-like units, and associating linguistic codes with each such segment. For example, like many other language corpora, the BNC provides codes identifying the part of speech (noun, adjective, verb etc.) of each word as well as its base or uninflected form. We might therefore represent the sentence quoted above in TEI as follows:

<u who="PS6NY">  <s n="12">
	<w ana="#VM0" lemma="lets">Lets</w>
	<w ana="#VVI" lemma="ask">ask </w>
	<w ana="#DPS" lemma="you">your </w>
	<w ana="#NN1" lemma="mum">Mum </w>
	<w ana="#CJS" lemma="if">if </w>
	<w ana="#EX0" lemma="there">there</w>
	<w ana="#VBZ" lemma="be">'s</w>
	<w ana="#DT0" lemma="any">any </w>
	<w ana="#NN2" lemma="corner">corners </w>
	<w ana="#PRP" lemma="on">on </w>
	<w ana="#AT0" lemma="a">a </w>
	<w ana="#NN1" lemma="boat">boat</w> <pc>. </pc>
</s> </u>

41In this encoding (production of which is largely automated, of course) the <w> element has been used to delimit each lexical item. For each such unit, the attribute @ana is used to point to a part of speech definition, and the attribute @lemma to supply a base form.

42Linguistic analyses of this kind have been in use for many decades and are essential to most current systems used in Natural Language Processing (NLP). Consequently, both the techniques used and the categories represented are in the process of being standardised, at least for major Western European languages. The TEI approach allows the encoder to choose how much or how little they wish to make explicit the meaning of the analysis pointed to by the @ana attribute: that is, how to define what VBZ or NN1 actually means.

43One method is to provide a further definition for the category, typically as part of the encoding description in the TEI Header, using a <taxonomy> element. Such a definition will typically be meaningful only to a human being, but it can still be very useful to provide one.

44At the other extreme, categories may be formally defined using the joint TEI/ISO recommendations for representing linguistic analyses in terms of feature structures (see further Chapter 18 of the TEI Guidelines). Such analyses are now used quite widely in the NLP community, for example as a means of mapping information between different computational lexica.

45Or, intermediate between the two, linguistic categorizations may be made with reference to an existing standard, such as that provided by the ISO Data Category Registry (see further Ide and Romary 2009). This approach has much in common with many other data harmonization efforts on the Web: it makes it simpler for systems to interoperate, in the same way as TEI documents in general can be interoperable.

46It is also worth noting that the same mechanisms are available for any kind of analysis, not necessarily a linguistic one; stylistic analyses (which typically characterize stretches of text larger than a single word) can also be encoded in much the same way, by attaching the @ana attribute to whatever element is chosen to enclose the stretch of text concerned, for example a <p>, <div>, <s>, etc.

47Of course the units of text being analysed do not necessarily coincide with the units of text directly represented within the text. We mentioned one way of dealing with this common ‘overlap problem’ in the ‘Milestones’ section of the preceding chapter. Another is to use ‘stand off’ markup, in which the units analysed are represented not by XML elements but by pointers. Chapter 20 of the TEI Guidelines discusses these and other techniques in more detail.

Table des illustrations

Titre Figure 6
Légende Detail from Dulce et decorum est autograph manuscript in the English Faculty Library, Oxford University.
Fichier image/png, 107k
Titre Figure 7
Légende ‘Apple’ from Johnson’s Dictionary of the English Language (1755)
Fichier image/png, 74k
Titre Figure 8
Légende Extract from Deborah Cameron’s Verbal Hygiene (p 229)
Fichier image/png, 266k
Titre Figure 9
Légende Extract from Deborah Cameron’s Verbal Hygiene (p 247)
Fichier image/png, 658k


Volume papier