Version classiqueVersion mobile
OpenEdition Books

Text and Genre in Reconstruction

 | 
Willard McCarty

1. Never Say Always Again: Reflections on the Numbers Game1

John Burrows

Texte intégral

  • 1 This paper was originally given at King’s College, London, 25 October 2006, as the third biennial (...)

For Harold Love: Vale.

1In order to ’place’ my argument, I declare myself, first and last, a student of English literature. I took up computing because it seemed likely to answer some of my questions. Perhaps it has, but only by opening up a thousand more. My object in this paper is to look back at the quarter-century I have spent in computational stylistics and to consider what such work entails. I must speak, accordingly, with some generality: yet I am neither a theorist nor a philosopher. Our heuristic procedures have much in common with those of experimental science: yet I am no scientist and have never claimed that work like mine lay in that domain. What I have to say is not offered as an apologia: yet I would like, if I can, to allay the doubts that such work arouses in many of our colleagues and to face some of the questions that they raise.

1. Three Case Studies

2I shall begin with an account of three cases where my tests seem to indicate that a particular text had two identifiable authors. Here, as also with parody and pastiche, the problem of attribution can be more delicate than usual. But if problems like these can be successfully resolved, we are better placed to handle the suggestion that a text is ’contaminated’ whether that suggestion arises in reality or is introduced as a skeptical gambit.

3In the first two cases, my results concur with well-founded documentary evidence. The third case is more contentious because the work in question has always been regarded as having a single known author. While these cases will occupy us for a little while, they are offered as preamble for a broader question. It was put to my friend and colleague, Hugh Craig, in a letter from an eminent Shakespeare scholar: How much confidence should I place in a result like this?

4The question quite transcends its immediate occasion and goes to the heart of our whole endeavour. It should, I suggest, evoke a sense of scholarly obligation on both sides. It is always incumbent upon those who introduce new methods to proceed cautiously, to work with rigour, and to present their findings as plainly as they can. As we advance, it is increasingly incumbent upon other scholars in the humanities to put aside any instinctive aversion for unfamiliar forms of inquiry. Both parties can unite in an awareness that new methods are not always appropriate and that, like old ones, they can be misused.

5The first of my three case-studies concerns St. Ives. When Robert Louis Stevenson died in 1894, his unfinished manuscripts included thirty chapters – over a hundred thousand words – of a romance about a French prisoner of war in Napoleon’s day. After escaping from Edinburgh Castle, the eponymous hero goes into England to claim his inheritance from a dying uncle, a wealthy grandee of the ancien regime. Our hero then quixotically returns to Scotland to clear his name and claim the hand of his beloved. Back in Edinburgh after many an astonishing escapade, he is stranded by the author’s untimely death. He is rescued by Arthur Quiller Couch (1863-1944), who added thirty thousand words and saw the whole work published in 1897. While it is a lively specimen of an adventure story, both authors give it an extravagantly mock-heroic air. Whenever St Ives blunders, his enemies are at the ready. Whenever he is most in need, the very man he needs is waiting round the corner. Not so much a subversion of the genre as a mildly ’camped-up’ version of it.

  • 2 http://www.gutenberg.org/wiki/Main_Page [accessed 10/2/10].

6As a framework of comparison for determining whether it is possible to show who wrote which part of St. Ives, I began with a 72,000-word sample from the novels and stories of each target-author. All of this material was downloaded from the Project Gutenberg.2 It was supported by 12,000-word samples from the work of thirty other authors born within forty years before and forty after Stevenson’s own date of birth. In each case, the selection drew on at least three pieces by a given author. These thirty selections came from our own archive, entered by keyboard over the years. Apart from all this were two samples of narrative by Stevenson and Quiller Couch selected from works not used above. They, too, came from the Gutenberg Project and incorporated about thirty thousand words by each author. Like St. Ives, all the samples chosen were of retrospective first-person narrative. All told then, the main set runs to half a million words by thirty-two authors. The two independent samples run to another sixty thousand and St. Ives itself to one hundred and thirty-five thousand.

7Using standardized word-frequencies in order to allow the other thirty authors equal play with the target-authors, I constructed a ranked list of the three hundred most frequent words of all. The top two hundred included few lexical words and were put to most use in my tests. Neither St. Ives itself nor the two independent samples participated in the formation of the word-list. Contrary to my former practice, I did not introduce tags to distinguish homographic forms from each other.

  • 3 The calculation and use of Delta scores and Delta z-scores have been shown elsewhere and their hig (...)

8Using the software package Excel, a work-sheet for the statistical test I have called Delta3 was then set up to compare St. Ives with the thirty-two members of the main set. It suffices for the moment to recall that, as used here, Delta scores register differences between a target-text and the several members of a group of genuine or notional candidates for its authorship. The lowest score in a set marks the least difference. When (as in the present case) the set is large enough, the scores can be converted into z-scores. These have a mean of zero and diverge from it in units of standard deviation. It follows that the strongest negative z-score in a set marks the least of all the differences. Scores ranging out below about – 1.5 usually repay attention.

Figure 1. St. Ives and thirty-two authors.
Delta z-scores for 6000-word rolling segments, with 1000-word increments.

9The two hundred most frequent words made the final set of variables. The target-text, St. Ives, was broken into successive ’rolling segments’ of six thousand words, with increments of a thousand words. This allows us to model a reader’s progress through the text while testing authorship at each step. It is as if we were to move through the calendar year, registering rainfall for successive six-month periods – January through June, February through July, March through August, and so on. In its 135,000 words, St. Ives offers 129 segments of this kind.

10Figure 1 offers Delta z-scores for the differences, in each of 129 rolling segments of St. Ives, between that text and our thirty-two authors. The entries for Stevenson and Quiller Couch run across the foot of the chart, outside the cloud of other entries. One of the two leads the whole field in every one of the 129 results and his rival usually lies second. (Harriet Beecher Stowe and an Australian, Louis Becke, trail the field.) The only challenger worth naming is Anne Bronte, but even she barely impinges on the scores for the main pair. The strength of the result is emphasised by the fact that, at each step, the lowest z-score almost always lies below – 2.0. Save for a lapse at Segment 34, Stevenson easily leads the whole field for the first ninety-eight segments. After that, the lead passes to Quiller Couch, who holds it without challenge. This pattern exactly matches what we know – that Stevenson wrote the first thirty chapters and Quiller Couch the rest.

11The scores for Quiller Couch in both phases are worth more thought. Since he has no role whatever in the first thirty chapters of St. Ives, the fact that he often ranks next after Stevenson shows that his usual fictional style is much like that of St. Ives. And yet it is even more like our main sample of Stevenson, zigging where that zigs and zagging where it zags. St. Ives is least different from both writers in its passages of vigorous action. It differs less from others when dialogue supervenes. Segment 34, where it differs more from Stevenson than from Quiller Couch, is a long passage of reflection and description as the hero is travelling alone down the Great North Road. Much the same note is struck, as we shall see, in some of Stevenson’s sombre, introspective tales. It is also to be heard now and then in his romances, as when Jim Hawkins first goes ashore on Treasure Island and when David Balfour thinks himself marooned. But the more characteristic passages in St. Ives, those where the Delta z-scores approximate to –3.00, are the prisoners’ duel in Edinburgh Castle and some of the more perilous moments of the hero’s travels. All this suggests that Quiller Couch was always a most appropriate author to carry on where Stevenson broke off. He certainly persisted in a mock-Stevensonian vein in later work like Poison Island (1912), where a sybaritic recluse lures treasure-hunters to their deaths on his private Caribbean island.

12In the second phase of the novel, where Quiller Couch takes over, his Delta z-scores approach –3.00, receding slightly at the end. Those for Stevenson recede considerably but remain comparatively strong. Quiller Couch’s St. Ives, in other words, remains very much his own. Yet it is a good enough imitation to stand nearer to Stevenson overall than to any of our other thirty writers.

13The salient feature of Figure 1 is the sharp transition from Segment 98 on.

14Our second case-study shows quite another pattern. Its subject is a novel called The Boy in the Bush (1924), which has an unusual history. The main outcome of D. H. Lawrence’s visit to Australia in the early twenties was Kangaroo (1923), a powerful but turgid novel about a right-wing political movement of the day. But when he and his wife Frieda arrived, they met Mollie Skinner, a nurse in a convalescent home near Perth. A part-time novelist, she asked him to read a manuscript of hers. This was later to appear as Black Swans (1925). After reading it, he suggested that she try a new, less sentimental direction and agreed to help her bring the resulting work to publication. When her typescript reached him in America the next year, he not only began to tinker with the text but continued so vigorously that the novel is included in the Cambridge edition of his works as being by ’D. H. Lawrence and M. L. Skinner’. Its editor, Paul Eggert, shows that the decision is well founded. The question for us is not whether the work is collaborative but whether we can trace a more complex collaborative pattern than that presented by St. Ives. Nothing of Mollie Skinner’s typescript is known to exist. But Eggert uses the evidence of the letters they exchanged, of Lawrence’s autograph copy of the whole work, of two revised proof-copies, and of Lawrence’s known habits of revision to claim it as ’a Lawrence novel’(Eggert 1990: liii). Eggert shows that, besides introducing an important change of direction towards the end, Lawrence added two closing chapters of which Mollie Skinner disapproved. Beyond this, he suggests that Lawrence made extensive changes throughout. The overall effect is to give a sort of colonial Bildungsroman a strong infusion of the idiosyncratic desert mysticism preoccupying Lawrence at that time.

15Apart from the change of target-text, the only underlying difference between Figure 2 and Figure 1 is that long extracts from Lawrence’s Kangaroo (61,125 words) and Skinner’s Black Swans (51,014 words) take the place of Stevenson and Quiller Couch. The other thirty authors stand unchanged and the same word-list is used. Since both Lawrence and Mollie Skinner are of an age with the later-born members of the main set, the set fits them a little less well than it fitted Stevenson. But the advantage of close replication prevailed.

16Partly, perhaps, for that reason the lowest range of Delta z -scores in Figure 2 runs out beyond –3.00. But Lawrence is always more idiosyncratic than most. In this work, moreover, there is much of the strong vernacular that colours Black Swans and Kangaroo and so draws them closer to it. This interpretation is supported by the fact that, of the other thirty authors, Emily Bronte is nearest to The Boy in the Bush while Edgar Allan Poe’s high old-fashioned style stands at the opposite extreme.

17Of the thirty-two authors studied, one or other of the two collaborators is least far from The Boy in the Bush in all 130 segments. That is as it should be. But, in a pattern quite different from Figure 1, the scores for the two of them keep criss-crossing and are often sharply opposed. Mollie Skinner has much of the running in the middle of the novel. Lawrence has the ascendancy not only in those later passages where he is known to have taken the initiative but also in the opening chapters. All this is in line with Paul Eggert’s evidence and in close keeping with his conclusions.

Figure 2. The Boy in the Bush and thirty-two authors.
Delta z-scores for 6000-word rollimg segments, with 1000-word increments.

18There are nevertheless some segments where the collaborators stand close to each other. This is the product, I suggest, of a particular level of revision where the changes are such that neither author’s frequency-pattern is able to prevail over the other. At such moments, moreover, as around Segments 73, 95, and 119, the difference between our two known authors and Emily Brontë is at its least. If one composes artificial models, hybrid frequency-profiles, to which each collaborator contributes in designed proportions, the effect is to overlay the distinctive frequency-patterns of both authors. As the proportions are gradually altered to favour either author, the favoured one eventually prevails. But there can sometimes be transitional stages in which the two authors are not easy to distinguish from each other. At just such stages, quite other writers, with no claim at all to the authorship of the target-text, sometimes outscore them both.

19The argument so far can be summed up in general terms. In ordinary single author works of sufficient length, the frequency patterns fluctuate from episode to episode or segment to segment as the text goes forward. Such fluctuations are governed by shifts of style as from action to description or from narrative to dialogue. Stevenson’s part of St. Ives, as represented in Figure 1, is a fair specimen. It converges on and diverges from our main set of Stevenson’s fiction but almost always stands nearer to Stevenson than to any of the other thirty-one authorial sets.

Plates 1 and 2 (opposite). Henry James, The American (1877).
Specimen of light revision (plate 1) and heavy revisions (plate 2) for the New York edition (1907).

20Texts of a second kind, where one author seeks to imitate another can also be expected to fluctuate in the same fashion. But whether or not a close resemblance to the original is achieved, there is no inherent reason for any high level of inconsistency as the text goes forward. The imitator’s rendering will reproduce or exaggerate some features of the original, overlook some others, and preserve many of his or her own idiosyncrasies. Quiller Couch’s section of St. Ives shows a closer affinity for his main text than for Stevenson’s. Some other imitator might come closer to the target. But, either way, an essentially uniform rendering of the original is likely to pervade the whole.

21Texts of a third kind, where an existing work is extensively revised, are likely to fluctuate more violently than either of the other kinds. Figure 2 reflects a typically inconsistent pattern of events in which Lawrence, like other revisers, makes few changes here and more there, sometimes leaving a passage almost untouched, sometimes inserting a passage of his own. Whereas many revisers weary of the task and emend more lightly as they proceed, Lawrence remains assiduous, making some of his greatest and best attested changes towards the end. Yet there are segments where he seems to have let Mollie Skinner’s version stand with little or no alteration.

22Most authors are obliged to accept the pervasive but comparatively slight changes introduced by publishers’ and printers’ house-styles. Few are willing to accept wholesale changes by another author. But something comparable occurs when Henry James, in his mid-sixties, revises novels he had written up to thirty years before. The Scolar Press facsimile of The American is a fine specimen, in which both versions can be seen at once. The facsimile is of a large workbook, now held in the Houghton Library, into which James pasted each page of the 1877 edition of the novel. He had returned to it thirty years after it appeared in order to revise it for the great New York edition of his fiction. By this time, as is well known, his style had altered greatly – so greatly, in fact, that, on tests like mine, his late novels could almost be the work of a different author. These are matters on which David Hoover is currently working. For our immediate purpose, it is enough to offer samples of the different levels of intervention that occurred as James worked through his old text in order to make it new. Plates 1 and 2 show the difference between lighter and heavier revision. In other places, pages of the original text are completely replaced by passages of manuscript or typescript. The varying levels of change observe no logic but that of the reviser’s preference.

23Taking the preceding discussion as a framework, we can turn to our third case-study. The text in question is The History of Ophelia (1760), a novel that has always been attributed to Sarah Fielding, younger sister of Henry Fielding. The case for her authorship is not supported by the title-page, where ’by the author of David Simple’, her usual phrase, gives way to ’published by the author of David Simple’. In her Advertisement, moreover, she actually disclaims authorship, saying that she found it in ’an old Buroe’ and published it. The phrase used on the title-page has been treated as an inconsequential variation upon the expressions of anonymity expected of eighteenth-century women authors. The disclaimer in the Advertisement has been seen as an early example of a practice that was to become widespread, especially in ghost stories and other fictional extravagances. In both subject and treatment, there is much in the novel that anyone who knows her writings would accept as Sarah Fielding’s. And yet, in the course of adding parts of this text to our set of Sarah Fielding’s work, we quite unexpectedly came upon resemblances to the work of her famous brother. A series of different tests, some treating of very frequent words, some of much less frequent ones, have all supported the notion that Henry Fielding had a part in the composition of this work. If that is so, the only likely rationale is that Sarah Fielding drew upon some of his surviving papers but that her own contribution is far greater than she claimed.

24Figure 3 employs the same procedures as those used above. The main set of texts has been replaced. The substitute, comprising almost 540,000 words, is made up of samples of first-person narrative by twenty novelists of the early and middle eighteenth century. The samples of the Fieldings’ work comprise eight such narratives apiece, amounting to over 75,000 words each. All told, The History of Ophelia itself runs to almost 102,000 words.

25The strongest z-scores in Figure 3 are in the positive register, where the bluff vernacular of Defoe and the breathless muddle of Mary (’Perdita’) Robinson carry them far from all the rest. The effect of such strong ’outliers’ is to drive other entries together. (The real differences in distance among a range of London suburbs are similarly made to seem smaller if Canterbury and Cambridge are shown on the same map.) It is evident, even so, that a majority of the strongest negative z-scores are Sarah Fielding’s and that her brother is her principal rival. His ascendancy is strongest at the beginning but he also leads the field around Segments 40 and 70. Nowhere else in Sarah Fielding’s writings, save for an acknowledged contribution to her Familiar Letters, does Henry Fielding score so strongly on tests like these. A range of other statistical tests, some treating of these frequent words, and some of the more unusual words, yield similar, often stronger, results. The most likely explanation is that, long after her brother’s death, Sarah Fielding took up some of his manuscript fragments and wove them into a novel whose authorship she never claimed. The chapters dealing with the heroine’s early life, her imprisonment in a country house, and her later visits to Bedlam and Tunbridge Wells are those where his hand is clearest.

Figure 3. The History of Ophelia and twenty authors.
Delta z-scores for 6000-word rollimg segments, with 1000-word increments.

26Whenever the z-scores for the Fieldings diverge sharply from each other, the stronger of them leads the field of twenty authors. Whenever they converge, three other authors take a turn or two. In prose style (to speak simply as a reader), both Eliza Haywood and Charlotte Smith often show some resemblance to Sarah Fielding while Charlotte Lennox’s more pointed style is not unlike that of Henry Fielding. Nowhere but in The History of Ophelia, however, has it proved difficult to distinguish any of them from either of the Fieldings. It appears likely that, as in The Boy in the Bush, some levels of revision blur the frequency-profiles and admit interlopers. On external evidence, none of those mentioned is a genuine candidate for the authorship of The History of Ophelia.

27There is no need, however, to resort to external evidence. If each of these unlikely candidates is tested, in turn, against each of the Fieldings, one or other Fielding is the victor. The simpler discrimination of a head to head contest allows a much sharper focus.

28One further possibility should be entertained. Sarah Fielding is known to have collaborated with a friend, Jane Collier, on The Cry (1757). Since Jane Collier’s only independent work, An Essay on the Art of Ingeniously Tormenting (1753), is quite different in kind, it is misleading to include her in a direct comparison (where she scores badly). But a helpful inference is possible. Tests on The Cry show that it differs more from Henry Fielding than does Sarah Fielding’s other work. A putative contribution by Jane Collier, therefore, is unlikely to account for those parts of The History of Ophelia where Henry prevails.

29Together with Anthony Hassall, a Fielding scholar, I have written a more detailed study of this matter in which he takes up the historical and literary questions that arise (Burrows and Hassall 2006). After encountering some resistance from students of Sarah Fielding, this piece found an editor willing to publish a mildly controversial finding so that scholars can form their own opinions. It should be acknowledged that Peter Sabor, the most recent editor of The History of Ophelia, rejects our case and proposes that the stylistic anomalies of the novel may stem from its being written by Sarah Fielding intermittently over a period of years (Sabor 2004: 14-15). That, however, is not a difficult question to test. Sarah Fielding’s style does indeed change over time. Nowhere else does it show so close a resemblance to her brother’s work as in parts of this novel.

30But my interest is not in arguing for victory. When the evidence is put before the scholarly community, our case will stand or fall on its merits. I raise it here because it is a suitable point of departure for a more general argument. Like any of us, literary scholars find it easy to preserve their equanimity in the face of evidence that favours their opinions. It is usually when the evidence runs the other way that we meet the pointed question: How much confidence should I place in a result like this? It is, as people say, a good question – a very good question indeed. The literary scholars who maintained their doubts in the matter of the Elegy by W. S. were ultimately vindicated. Such doubts as theirs are often held but they are not always justified by the evidence.

2. Questions of Confidence

2.1 Has Statistical Analysis a Place in Literary Studies?

31On the face of it, literary texts offer an ideal arena for statistical analysis. They make up a vast range of specimens whose provenance and characteristics are well known and a further range where challenging questions arise on every side. By way of measurable variables, they incorporate large populations of phenomena, some frequent in occurrence, others extremely rare. Some behave according to straightforward rules, others are less predictable. Computers now make it possible to count, sort, and classify such phenomena with unprecedented speed and accuracy. As Paul Fortier, still much missed by many of us, would put it, the very nature of evidence in literary scholarship has changed. Whereas we would gather examples to illustrate a proposition and test it on our fellows, we can now gather all the relevant examples and put the case more firmly.

32Among the many sorts of phenomena available, let us address ourselves to the study of words. Problems of definition declare themselves at once. Should I’ll be taken as it stands or resolved into its constituents? In the latter case, should it be I shall or I will? Should word-types like so and that, which embrace several homographic forms, be left alone or given grammatical tags? Tagging brings an advantage in accuracy but, besides being laborious, it makes it too hard for others to replicate one’s work. Much to Paul Fortier’s satisfaction, I have come to believe that the text is usually best left as it stands.

33Far subtler difficulties arise with truly polysemous words like blue where numerous literal meanings shade off into all sorts of metaphorical senses. Is it feasible to treat such diverse tokens as instances of a single word-type? In general, I believe, a given word-form can be seen as having one or more cores of meaning, populated by many almost synonymous instances and surrounded by a penumbra of increasingly unlike specimens. Especially in their outer, more sparsely populated reaches, different penumbra will sometimes overlap. At the very core, Shakespeare’s let can mean either hinder or permit. But there will be few occasions on which blue means red or chalk means cheese or romantic, classical. There are several reasons for turning I into we, as in Margaret Thatcher’s ’We are a grandmother’. But is it only in Australia that redheads used often to be addressed as ’Blue’? The virtue of a statistical approach is that a few exceptions are readily absorbed in any large population. This line of reasoning transcends the visible cases of difficulty and answers also to the more pervasive objection that every word-token is unique – that, as Robert Louis Stevenson put it, there are no true synonyms in English. But, I repeat, if each word-type is a cluster of near synonyms, it is usually a distinguishable cluster. At this point, the argument that words lose meaning when they are taken from their contexts can be seen as a version of a general objection to counting and classification. The idea that such procedures distort the very selfhood of unique living creatures (especially moi) can be either a rational complaint about unsuitable measures or a superstitious fancy. In either case, the questions arising are not peculiar to literary studies but bear on the ubiquitous presence of statistics in our lives.

34In The World We Have Lost, Peter Laslett cites a survey of 1688 in which the population was divided almost equally into those ’Increasing the Wealth of the Kingdom’ and those ’Decreasing the Wealth of the Kingdom’ (Laslett 1971: 36). Economic rationalists may still be content to leave it at that. But Ian Hacking (1990) shows how such simple demographic records have broadened their range and increased their sophistication over the last two hundred years. His title, The Taming of Chance, is apt. Three main forms of statistics, I suggest, now pervade our lives. There are confidential surveys conducted, chiefly for exploitative purposes, by people like politicians and commercial entrepreneurs. Then there are scientific studies conducted, chiefly for the public good, by epidemiologists and others. And there is the unconscious or almost unconscious kind of statistical work that enables us all to live our daily lives. The first of these is of great influence but very little intellectual interest. Its principal effect is upon what we are allowed to buy, in the shops or at the ballot-box, and how it is packaged. The second can be a powerful force for the public good as, for example, when epidemiology led to a proper understanding of the malign effects of tobacco. It can also go a little awry when the statistics are incomplete or the results are misunderstood. Many people of my age benefited from being persuaded to turn from cigarettes to pipes. We now know that lung cancer was only part of the story and that pipe-smoking also did great harm. In another affair, one of the most effective drugs for osteoarthritis was withdrawn because it had some dangerous side-effects. It has emerged that the sideeffects came from overdosage and that the media had misinterpreted and exaggerated the statistical evidence. But the drug remains unavailable. The third great branch of current statistics comprises those that we all use more or less unconsciously all the time. It embraces everything from planning the future or investing money to crossing busy roads, navigating roundabouts, and carrying raincoats – everything in which we assess risks and rely, often quite intuitively, on judgments of probability. In all three of these large areas, some measures best fit the case. Even the poignant cry, ’That traffic-light is always against me’ can be stilled, if one cares to, by keeping an honest tally. The impetuous young motorist and the timid elderly one make different risk-assessments. Though both cause accidents, they are outmatched by the novice. No statistician he, he errs at random.

35Like it or not, we cannot escape the use of numbers and the calculations of probability that beset us. We need to know all we can of the statistics imposed upon us. For those we ourselves employ, our best course is to seek appropriate measures, whether formal or merely rule of thumb, and to apply them with such rigour as seems due. All this considered, it is hard to see how literary studies could be exempt. On the positive side, let one immediate instance stand for many. The study of D. H. Lawrence must be advanced by fresh evidence bearing on the belief that he contributed extensively to The Boy in the Bush and identifying further passages as likely (or unlikely) to be his.

2.2 What Sorts of Measures Best Suit Literary Studies?4

  • 4 For more information about the sorts of statistical method referred to in this section but not act (...)

36When engineers use principal component analysis, I am told, they usually choose factors (such as velocity and mass) on which any group of specimens will record different scores. The chosen factors allow the scores to be arranged in ranked sequences (or vectors) showing how performance varies in the group. In literary applications of the method, we allow chosen specimens to distribute themselves according to their own dictates and then try to infer meanings for the main vectors that emerge. The top-down approach might bear, for example, on defining suitable limits of tolerance as specimens diverge from declared norms. Our bottom-up approach, however, has chiefly to do with classification – with the possibility that

37specimens of known proclivities may form interesting classes, with the relative importance of such classes, and with the affiliations displayed by specimens of doubtful or unknown provenance.

38This simple difference is a watershed. On the one side lie categorical methods, like discriminant analysis and artificial neural networks, whose outcome is either Yes or No, either Accept or Reject. In cases of doubtful authorship, for example, these methods come into their own when only two candidates have any claim. On the other side lie more empirical methods, like cluster analysis and the ’bottom up’ version of principal component analysis, whose outcome is more complex. The Delta procedure is of this more open kind. These are all better fitted for exploratory work in areas where that is requisite.

39Authors themselves and the works they produce can be classified in many ways. Computational stylistics uses patterns of phenomena like word-frequency as part of that endeavour. We have mounting evidence that work by different authors, work in different genres, work of different eras, work in different national forms of English can all comprise statistically distinguishable groups. A further contrast, distinguishing between male and female authors, is coloured by differences of education in former times. But even the basic validity of such evidence is still contested. Partly because we are dealing with complex entities and partly because computational work in the humanities is still in its infancy, we do well to cultivate exploratory methods of analysis. For we have yet to find our Linnaeus: we have no agreed hierarchy of classes and no settled opinion as to whether our classes have strict boundaries or, as I suppose, more shadowy borderlands where monsters sometimes dwell.

40Cluster analysis and principal component analysis both highlight whatever patterns of resemblance and difference emerge when a number of specimens are assessed in terms of their scores on a set of variables. When these methods are used in an exploratory way, any meaning attaching to a given pattern is a matter of inference. But, while they provide the genesis of many fruitful discoveries, such inferences must be tested sternly and persistently.

41The last decade has seen an increasing attention to the need for more tests than one. More attention has been given to the use of replication and of suitable ’controls’. But we have been slower than we might to appreciate the need to make our tests as independent of each other as possible. The behavior of the language at different levels of word-frequency is so subtly interlocked that even the move from frequent words to rare ones does not yield complete independence.

42In order to try a different tack, let us take a case where our results accord with the known truth. In the 129 rolling segments of St. Ives, all except one of the results are correct. By setting aside our outside knowledge, we may suppose that the test results need verification. The Delta test suggests, as we have seen, that Stevenson wrote most of St. Ives and that Quiller Couch contributed the closing chapters. But, taken alone, Figure 1 does not quite justify that conclusion because Delta is not an absolute test of authorship. It treats, in the present case, relative differences between successive segments of St. Ives, and thirty-two authorial samples of prose fiction. Since those by Stevenson and Quiller Couch are least unlike St. Ives, authorial inferences are obviously admissible. And yet, improbable as it may be, another author altogether might prove even less unlike St. Ives than any of these thirty-two. And, again, it is conceivable – though barely so – that our large samples of Stevenson and Quiller Couch stand apart for some non-authorial reason.

43Student’s t-test and the Mann-Whitney non-parametric test are used to determine which of many variables serve to differentiate two sets of specimens. To provide an adequate number of specimens, our main samples of Stevenson and Quiller Couch were each broken into thirty-six segments of two thousand words. The three hundred most frequent words of our thirty-two author corpus were taken as variables. Eighty-three of the three hundred words satisfied both tests at the 5% level of statistical significance or better. (The degrees of freedom usually exceeded sixty, and allowance was made for the ’two-tailed’ character of the data.) A yield of eighty-three significant results out of three hundred, at a level where chance would admit a mere fifteen, shows that the two main sets of data come from genuinely distinct populations.

44When these new data are employed on the original range of specimens, the outcome, not surprisingly, is even more clear-cut. At this point, however, the secondary sets of thirty thousand words apiece by Stevenson and Quiller Couch come into their own. These selections from their tales and stories are entirely independent of the previous sets. They were each broken into five successive segments – not rolling segments – of six thousand words. Stevenson’s part of St. Ives was broken into seventeen segments of that size, of which five were chosen at random. Quiller Couch’s part of St. Ives yielded five more. Word-counts for the eightythree ’significant’ variables were derived from each of these twenty segments. None of the twenty segments had any part in establishing the main list of three hundred words or in selecting eighty-three of them as differentiae.

Figure 4. Cluster analysis of twenty 6000-word segments.
Analysis based on eighty-three Stevenson-Quiller Couch Differentiae.

  • 5 Cluster analysis compares the members of a set of specimens, each with every other, on the basis o (...)

45Figure 4 is the outcome of a cluster analysis of these twenty specimens.5 The division into two authorial families verifies our previous finding, justifies the belief that it was indeed authorial and helps to close the door on any putative interloper. The detail of Figure 4 also repays attention. On Stevenson’s side, only one entry from the Tales intrudes into the sub-group of entries for St. Ives. Like most long single works, St. Ives develops a ’note’ of its own, as Henry James would say, and stands a little apart from a mixed sample of its author’s writings. On Quiller Couch’s side, the first three entries for St. Ives stand together. The last two lie in the sub-group of Quiller Couch’s stories. Like most imitators, it may be, he gradually losessight of his object and drifts towards his own style.

46A sterner form of replication yields a supportive, though less powerful, result. If the two new samples by Stevenson and Quiller Couch are put in place of the original pair and no other change of any kind is made, a new version of Figure 1 can be established. When that is done, Quiller Couch still easily resists any challenge to his part of St. Ives. In his part, Stevenson still prevails on average but no longer at almost every point. The next step is to identify sets of ’discriminating words’ in the same manner as the eighty-three described above. When that is done, the various challengers all fail in head to head contests with Stevenson. But because this further task must be undertaken iteratively, with each challenger in turn, the spectre of the unknown interloper can never quite be put to rest. A return to the texts explains why the new result is less clear-cut than the original. The original sample of Stevenson comprised selections from his romances. The new one draws on sombre, introspective tales like ’The Body Snatchers’. Both sets fall well within his stylistic repertoire but the former is more in keeping with St. Ives.

2.3 But is All this Really Scientific?

47While this question deserves attention, I begin with reservations. I do not accept that scientists have a monopoly on serious heuristic methods. To call computational stylistics a science does not hallow it and may even misrepresent it. Nor, I believe, are scholars in our field much given to calling it a science. The suggestion is usually made by others, not always with a favourable intent. These reservations of mine are the product of long experience: but I know, only too well, that I am now moving on to ground where I am less at home.

48Encouraged by Willard McCarty, I have done a little fresh reading about ’scientific method’. Putting it beside my shabby, old furniture is like putting a new bathroom into an old French hotel – one admires the facilities but wonders about the plumbing. At any rate, I take two examples. Sir Peter Medawar (1982) focuses on the central phase of experimental procedure. Claude Bernard (1865), whom Medawar quotes approvingly, sets such procedures in a much broader heuristic framework.

49Despite some differences of terminology, they are at one in an emphasis on the central experimental role of hypotheses as propositions susceptible of testing against fresh facts. Both of them regard unfalsifiable propositions as alien to the spirit of inquiry. They draw attention, albeit in different ways, to the asymmetry of proof: falsification is more conclusive than corroboration. In that light, they stress the need to persist unflaggingly in the task of disproof and they maintain that verification can never be more than dubitative or probationary. I can see nothing here that scholars in any field of rational inquiry would dispute.

50Medawar’s account of the genesis of hypotheses is narrower and, for me, less congenial than Bernard’s. Perhaps because he is convinced that there is no such thing as an ’innocent eye’, Medawar regards anything that precedes the formulation of a hypothesis as of little interest. It is inexplicable, an expression of mere ’idle wonder’. One may grant that the inquiring mind is no tabula rasa, but still share Bernard’s belief that we often perceive important anomalies when our minds are not on duty. The response ’That’s odd’ rests upon an experience of what is normal but remains close to innocence. It is only when ’I wonder why’ gives way to the truly pre-hypothetical ’I wonder if’ that an experiment is in the making. But the ability to say ’That’s odd’ is fundamental. If the discovery of penicillin is the classic case, the example can readily be multiplied.

51The ability to say ’That’s odd’ may be of particular value in fields like ours where theory does not – or does not yet – prevail. The many meanings attached to the word ’theory’ cloud the question of whether empirical work like that of computational stylistics lacks a theoretic basis. (McCarty and Love write strongly on this topic from opposed standpoints. See McCarty, 2005, 139-55 and Love, 2002, ch. viii). From mere hobby-horses, through hypotheses (usually of some breadth), the word reaches out to include such general laws as Newton’s and Einstein’s. Let us cherish our hobby-horses but keep them in their stables. Let us welcome hypotheses and put them to the test. And let us stand firm against the suggestion that we can do nothing until we are ready, like Newton, Einstein, or Chomsky himself, to embrace the universe.

52It is possible, nevertheless, to aspire to a modest generality. If language can be seen as a manifestation of cognitive behavior, it is only to be expected that the performance of any individual in this field, as in many others, should not only resemble but also diverge from the performances of other people. Ferdinand de Saussure brings this to a point in his distinction between langue and parole, between the set of linguistic resources available to us and the subset of choices we each make. Through a simple corollary, one may suppose that parole has a frequentative as well as a selective aspect. What else could give it an enduring shape of its own? One may also hope that a better understanding of computational classification will help to fill the ground between de Saussure and the various experiments we undertake. As matters stand at present, I have yet to see or hear of a text of any length whose patterns of word-frequency could not be distinguished from others of its kind or which failed to show meaningful affinities with others written by its author. To the extent that this tests his theory, de Saussure still holds his ground and serves us well.

53In discussing what comes after the experimental phase of a scientific inquiry, Medawar and Bernard concur. Where an hypothesis is upheld, it still remains open to further inquiry. Failed hypotheses are to be rejected. (Medawar adds ’or modified’ but says no more). Neither of them, unfortunately, sheds much light on the meaning of failure or on its sequel.

54What are we to suppose? The strict position these eminent scientists take is a necessary response to human ingenuity. Most sets of data are susceptible of several hypothetical explanations. Any anomalies can usually be rationalized away. Why pretend to test hypotheses at all if one is determined to ’save’ them when they falter? It is in this spirit, I suppose, that I have been reproached for explaining such anomalous specimens as Segment 34 of Figure 1. The question here is whether it is legitimate to try to diagnose the behavior of a single anomalous specimen in a field of 129. I do not believe that the objection to ’saving the hypothesis’ should be carried so far. For if a single aberrant specimen spells failure, there is no place in science for the study of human behavior and no role for statistical analysis. Since that is patently absurd, we must think again.

55An error in observation or calculation has no bearing on a hypothesis. When one is found, it should be rectified without ado. Such trivialities apart, it seems that scientists must distinguish between crucial, casual, and fruitful anomalies and between acceptable and unacceptable levels of tolerance. These will differ greatly in different fields of inquiry. But, in any field where ’zero tolerance’ is not required, the value of statistical measures of significance and error is immense. It seems clear that we, too, must work along such lines and that we must persuade our traditionalist colleagues that it is right to do so.

56And after a hypothesis is seen to fail? Dead end or new point of departure? That will depend upon the residual merits, if any, of the original hypothesis. Willard McCarty (2005: 286, s. v. ’failure’) writes compellingly on the fruitfulness of failure. After a failure, to put it in my terms, there is likely to be a period of disappointment. But then ’I wonder why’ comes back into its own, ’I wonder if’ soon follows, and the game begins again. A new hypothesis emerges to transcend the old and is tested in its turn. If that is unscientific behavior, we must gladly draw our skirts aside and leave the scientists to their own ways.

3. How Much Confidence Should I Place in a ResultLike This?

57The question underlying this whole discussion sometimes reflects a desire for certainty. But successful scientific hypotheses remain dubitative. Statistical analysis deals only in levels of probability. Scholarship in the humanities is seldom regarded as definitive. Life itself is, in many ways, a game of chance. Should literary scholars ask us, as some do, for certainty? In its usual form, the suggestion is that, unless computational methods can offer certainty, they are more trouble than they are worth. A subtler version is latent in Joseph Rudman’s demand (1998, 2000) for ’verifiably unique’ authorial signatures. Many of his structures have been of great value. But here, as with his notion that it is time to settle upon a single agreed method of analysis, he, too verges on naivety.

58We cannot offer certainty. But where there is an adequate body of material, computational stylistics usually offers strong evidence and is getting better at it. (For a somewhat more conservative assessment than mine, see Craig, 2004.) In my view, head to head contests are rarely difficult to resolve. In more open cases, the likely candidates can be identified. We cannot proceed with equal confidence in cases where an unknown interloper may be in the game. Even there, however, a weak set of results for known candidates may indicate that it is desirable to look elsewhere.

59Our findings usually concur with the scholarly consensus because both parties are usually right. When there is no such consensus, our findings can be especially serviceable. And when our findings are seriously at odds with the consensus, both parties need to think again. The statistical evidence may not be valid and, when it is, it can still give rise to mistaken inferences. And even when, on the other side, a scholarly opinion is widely held and of long standing, it is open to correction.

Notes

1 This paper was originally given at King’s College, London, 25 October 2006, as the third biennial Wisbey Lecture and the Inaugural Lecture for the London Seminar in Digital Text and Scholarship, Institute of English Studies, University of London. The biennial Wisbey Lecture was initiated by Harold Short to honour the pioneering work of Roy Wisbey in the field of humanities computing. It is a pleasure to bring their names together and to acknowledge their successive achievements in and far beyond King’s College.

2 http://www.gutenberg.org/wiki/Main_Page [accessed 10/2/10].

3 The calculation and use of Delta scores and Delta z-scores have been shown elsewhere and their high but not unfailing reliability has been assessed. See Burrows (2002, 2003) and Hoover (2004). For a part-precursor, see Forsyth, Holmes, and Tse (1999, 393).

4 For more information about the sorts of statistical method referred to in this section but not actually employed here, see Holmes (1994).

5 Cluster analysis compares the members of a set of specimens, each with every other, on the basis of their relative scores on a given set of variables. (In the present case, the specimens are texts, the variables are a range of words, and the scores are standardized word-counts) For each specimen in turn, its differences from every other are calculated and then squared to eliminate the negatives. The sum of these squared differences is extracted. The two specimens showing the smallest such difference are then united. The next smallest such difference either unites two other specimens or adds a third member to the first pair. The next smallest difference is introduced in the same way and the process continues until all specimens have been embraced. The outcome is then plotted in a dendrogram, an inverted treestructure. As seen from the foot of the page, the y-axis shows a progressive departure from 100 % similarity. The specimens are arrayed on the x-axis. The twigs of the inverted tree show which specimens united first, as being least different from each other. These pairs and trios unite in branchlets, lesser branches, greater branches, and so on until all are united in one trunk (usually at a low or even negative level of resemblance). Where specimens neighbouring each other on the x-axis do not unite, their proximity to each other is usually a reflection of the configuration of other unions.

The version of cluster analysis used here is part of the MINITAB package. The main options chosen embrace ’standardized variables,’ ’Ward’s linkages,’ and ’squared Euclidean distances.’ For an explanation of these terms, the MINITAB Help file refers the reader to Lance and Williams (1967).

Table des illustrations

Légende Figure 1. St. Ives and thirty-two authors.Delta z-scores for 6000-word rolling segments, with 1000-word increments.
URL http://books.openedition.org/obp/docannexe/image/646/img-1.jpg
Fichier image/, 122k
Légende Figure 2. The Boy in the Bush and thirty-two authors.Delta z-scores for 6000-word rollimg segments, with 1000-word increments.
URL http://books.openedition.org/obp/docannexe/image/646/img-2.jpg
Fichier image/, 130k
Légende Plates 1 and 2 (opposite). Henry James, The American (1877).Specimen of light revision (plate 1) and heavy revisions (plate 2) for the New York edition (1907).
URL http://books.openedition.org/obp/docannexe/image/646/img-3.jpg
Fichier image/, 34k
URL http://books.openedition.org/obp/docannexe/image/646/img-4.jpg
Fichier image/, 80k
Légende Figure 3. The History of Ophelia and twenty authors.Delta z-scores for 6000-word rollimg segments, with 1000-word increments.
URL http://books.openedition.org/obp/docannexe/image/646/img-5.jpg
Fichier image/, 120k
Légende Figure 4. Cluster analysis of twenty 6000-word segments.Analysis based on eighty-three Stevenson-Quiller Couch Differentiae.
URL http://books.openedition.org/obp/docannexe/image/646/img-6.jpg
Fichier image/, 19k

Auteur

John Burrows is Emeritus Professor of English at the University of Newcastle, NSW Australia, Director of the Centre for Literary and Linguistic Computing there from 1989-2001 and winner of the 2001 Roberto Busa Award for his work in literary computing. He is the pre-eminent scholar in computational stylistics, with numerous publications on questions of authorship in 17th through 19th-century English literature.