1This chapter does not directly address the central subject of this volume – the confusion of public with private records (although, as will be seen, such confusion continues and, in certain respects, is actually increasing). Rather it is concerned with the related problems of preservation and, above all, the presentation of public records. How are the records of central government selected for preservation in the United Kingdom? How are their contents made public? More specifically, what is the contribution of the government’s Official History programme to both these challenges?

2It is conventional to refer to the “three drafts” of history. The first is written immediately by investigative journalists, working from “informed” sources.1 The second is typically composed within five to ten years by social scientists, largely from published material or oral testimony. The third is written by professional historians after 30 years (when official records are fully released) or more usually after some 50 years (when an even fuller range of public and private evidence is available and, so it is felt, a proper perspective can be attained). One group of historians, however, has a head start because they enjoy privileged access to all relevant government records both within the thirty year exclusion zone and before the final sélection for permanent preservation has been made. Accordingly they can aspire to defy convention by writing an early “third draft”. These are the historians working within government, and especially those within the Official History programme.

3What, then, is the Official History programme? What are its historians contracted to do? And given the ethical and methodological issues raised by any commissioned history, ought they to be doing it? These are the questions which this chapter will address. First, however, to provide an essential context, it is necessary to look at the continuing confusion between public and private records and also at the impact of the Freedom of Information (FOI) Act. This Act became operable on 1 January 2005 and permits public access to some official records less than thirty years old. Like the Official History programme, therefore, it too has the potential to upset the conventional distinction between the “three drafts” of history.

Continuing confusion and the impact of FOI

4In the second half of the twentieth century, there have been – or remain – three principal sets of records which, although generated at taxpayers’ expense, have either not been made available at TNA or not been deemed public records. In the first category there are the papers of the “private offices” of ministers and senior officials, and those of special advisers; and in the second, the records of agencies in civil society which – as either a contractor or part of a policy network – play an increasingly important role in the formulation and delivery of policy.2

5This continuing confusion is in defiance of attempts in the first half of the twentieth century to consolidate and formalize central government in Britain (and thereby, incidentally, to establish a clear distinction between public and private records). First, a Cabinet Secretary was appointed in 1916 (to regularize collective decision-making by ministers); and then, in 1919, the head of the Treasury was formally recognized as head of the civil service (to unify what had previously been a highly departmentalized and fragmented service). The first two incumbents, respectively Sir Maurice Hankey and Sir Warren Fisher, remained in post until the late 1930s and were thus able to make their presence fully felt. Hankey established the convention whereby retiring ministers should neither retain nor remove official papers from Whitehall. They could thereafter consult them, but only in person; and they were obliged to submit to the Cabinet Secretary for approval any subsequent manuscript before publication. In other words, public records could no longer be regarded as private property, as they had been by nineteenth-century statesmen.3 Simultaneously Fisher employed his not inconsiderable energy to build up a common esprit de corps amongst senior officials, which inevitably became somewhat self-regarding and secretive. Officials (as demonstrated by the cabinet secretary’s vetting role) came to regard themselves – and were accepted – as custodians to, and arbiters of, government’s working practices. Given Britain’s unwritten constitution, these practices deliberately contained many nuanced and ill-defined assumptions.

6This legacy and, in particular, this penchant for secrecy was imbued with an underlying moral force by their successor, Sir Edward Bridges (cabinet secretary, 1938-1947, and head of the civil service, 1945-1956). As will be seen, he demonstrated remarkable boldness in the darkest days of the Second World War to commission a séries of “civil” war histories. However, subsequent problems in approving the final texts bear testimony to his determination to preserve the “mysteries” of government.4 Only normative accounts of how government should work – rather than less edifying accounts of how it actually did work – were to be permitted.

7One of the last vestiges of the formalization of central government was the appointment in 1952 of the Grigg Committee on departmental records which led to the 1958 Public Records Act (see chapterby David Leitch). This Act finally stipulated that the papers of the “private offices” of ministers and senior officials were public records. As is well recognized, many key policy decisions are taken in clubs, conclaves or corridors – with perhaps “a note for the record” appended to the relevant file providing a sanitized version of how it was reached. Private office papers provide a less sanitized account both of the deals done by ministers and the frequent political judgements made, and actions taken, by officials on their behalf. Their preservation and public accessibility are essential for anything approaching a full understanding of how particular policy decisions are reached.

8As gaps in the holdings of the National Archives (TNA) demonstrate, however, the 1958 Act in practice provided no guarantee that the files would not continue to be removed, destroyed or simply lost. For instance, official papers continue to appear in private archives, such as the major holdings of late twentieth-century political papers held at Churchill College, Cambridge. The private office papers of Mrs Thatcher at the Ministry of Education between 1970 and 1974 would appear to have been irrevocably “lost”. Bridges’ own working files were also belatedly discovered in a disused cupboard and thus came only fortuitously to be preserved in TNA (class T273). Clearly to preserve the “mysteries” of government, not to mention their own personal reputations, it was deemed to be to the mutual advantage of both ministers and officials that these papers should continue to be regarded in essence as private. Even with new and tighter Cabinet Office guidelines, the “notebooks” of private secretaries are still unlikely to survive and the permanent preservation of ministerial appointments diaries (of importance for those interested in network theory) is not guaranteed.5

9The second set of records, generated at public expense, but regarded as essentially private, are those of ministerial special advisers. Until the 1960s, ministers were advised by private offices staffed, unlike those in France, almost exclusively by young, high-flying, full-time officials. After 1964 and especially 1974, however, an increasing number of temporary advisers were appointed to provide ministers with either the political support and contacts or the expert advice which impartial and “generalist” officials were deemed incapable of providing. Successive prime ministers also appointed a No 10 Policy Unit to increase “central capability”. As is suggested by the diaries of the first two heads of the Policy Unit, and confirmed by those internal policy files which include advisers’ interventions, the impact of these advisers was considerable.6 Their own set of papers, however, did not (like them) survive the end of an administration – in the public domain at least. Until the Blair government, they were routinely shredded. This has left a major gap in the record of public policy making.

10The final set of ostensibly public records, which remain publicly inaccessible, are those of private organizations in receipt of taxpayers’ money which assist government in either policy formulation or delivery. In the political scientists’ terminology, the State has been increasingly “hollowed out”. Government (in the sense of a hierarchical command structure) has been replaced by “governance” (where power is increasingly diffuse and based on “dependency”, or the need to negotiate with other parties). The records of defence contractors have never been regarded in Britain as public. Nor have those of major industrial firms, which have received considerable subsidies to implement, for instance, regional policy. A further tranche of subsidised “private” bodies has been created with the increased privatisation, contracting ont and “executivisation” of publicly financed services. Given the market prerogative of profit maximization, what incentive is there for such bodies to preserve their own records, let alone those inherited from central government? To maintain paper in bulk is expensive. To preserve electronic information, as the public sector is only too aware, is complex. The historian of the recent past is again to face a major difficulty.

11For current historians, however, opportunities have been opened by the new FOI Act, which ended the blanket “thirty year rule” on the release of public records. Typical of such legislation world wide, this Act made certain classes of records less rather than more accessible to the public – although the presumption remains that the recently attained practice will continue of prompt release to TNA, even from the Security Service, of records over thirty years old.7 Many traditional restrictions on the release of other classes of records under thirty years also remain (as specified in Part II of the Act). They include issues of national security, commercial confidentiality and personal privacy. What is new is that all other records become available to researchers on a “prejudice” test. Would public access rather than closure better serve the public interest? All government officials, faced with a public request under FOI. are under a duty to provide helpful “advice and assistance”. Should a request be rejected, an internal appeal is possible and then a further one to an Information Commissioner, who will accordingly be able to develop case law on openness. This means, in essence, that in many areas of study historians in general will now be able to encroach upon the “secret garden” into which previously only official historians had been able to enter.

Who are the official historians?

12There are three types of historian working within government. There are: hose permanently employed within certain departments, such as the Foreign and Commonwealth Office (FCO) and the Ministry of Defence.8 There are those temporarily employed to Write departmental histories, such as Prof essor Chris Andrew who has been commissioned to Write a history of the Security Service. Finally there are the “official historians” usually seconded from universities to work to write specific official histories in a programme administered by the Cabinet Office. It is on the “civil” side of this Official History programme that the rest of this chapter largely concentrates.

13The process of selection for official historiés is relatively straightforward. An official Cabinet committee draws up a range of subjects for possible subjects. This list is then amended and ratified by a cross-party group of Privy Councillors. Finally, an individual subject is chosen and the official historian appointed by the Prime Minister; and, once the historian has been placed under contract, an announcement is made in Parliament. The “historian” need not be a professional historian. Indeed two of the current civil histories, on North Sea oil and gas and on privatisation, are being written by economists. The latest volumes to be published, the Official History of the Falklands Campaign (Routledge, London, 2 vols, 2005), was also written by Professor Sir Lawrence Freedman who might be more accurately described as a political scientist. Of the remaining three histories which have been recently commissioned (on the United Kingdom and the European Community, the Channel Tunnel and the civil service since 1968), the authors are respectively an economic, business and contemporary historian.9

What is the Official History programme?

14The Official History programme started shortly after the entente cordiale in 1908. It was exclusively military and, in the aftermath of the Boer war, was designed to improve “collective memory” about best practice in relation to both the organization and waging of war. As in most other belligerent countries the First, and then the Second, World War led to major increase in the production of such volumes.10 A “civil” programme was not commissioned until, of all times, 1941 (as part of the Official History of the Second World War); and the principle of a purely peacetime civil history was reaffirmed by Harold Wilson’s government in 1966.

15Why was such an initiative taken in 1941? Essentially it was an act of bravura. Its object, a full year before any significant victories had been achieved, was to record for the future how the war had been won on the home – as much as on the military – front. The proposed methodology represented another act of faith. To break with the conventions of historical scholarship entrenched in Oxbridge, an Australian (Keith Hancock) currently working in Britain was commissioned as the general editor.11 He wanted to Write history as it happened. Accordingly, it was to be based on full access to all documents as they were written and on regular interviews with policy makers. The results were to be presented to a popular mass audience and thereby, it was hoped, make a significant contribution not only to good government but also another ideal of the time – a core of informed and responsible citizens.

16Hancock’s ideal was hopelessly impractical. Policy-makers over the next four years had rather more pressing tasks than to be interviewed by historians. Understandably, they were reluctant to fall over themselves to divulge material in top secret memoranda. More over, although he had commissioned the programme to laud achievements on the Home Front, Edward Bridges as cabinet secretary had his own serious reservations. As already suggested, for instance, he was wholly opposed to the demystification of government and the glories of Britain’s unwritten constitution. In particular he had no intention of destroying the carefully crafted constitutional convention of ministerial responsibility, which preserved the illusion of both ministerial accountability to Parliament and the impartiality of civil servants. There was thus, he ruled, to be no exposure of the irrational and often bitterly contested way in which policy was typically made. In addition no official should be named, let alone held to account for a particular decision.12 As a resuit, the historians were severely restricted in the information they could gather. They were even more restricted in what they could publish, with final manuscripts being heavily vetted to remove ail indiscrete passages. Inevitably the published volumes were exceedingly dull and failed to find a popular audience.

17The original programme of civil histories, as adjudged by Jose Harris, nevertheless had an enduring impact – and one of which Bridges would have approved. If only in an indirect way, so she has argued, it contributed to the evolving and later predominant perception that “Britain had won the war and emancipated the people because public enterprise, rational administration and altruism had succeeded where markets and private interests [as in the interwar period] had singularly failed”.13 Intellectually, in other words, it played a part in bolstering a postwar consensus based on collective principles and the enduring perception that the War had provided a test for British institutions and for the nature of “Britishness”, which both government and society had gloriously passed. It achieved this by complementing and reinforcing the views of other professional historians, which were then transferred to the policy-making elite and – through more popular histories – to the general public. This was the long-term conséquence of permitting professional historians to Write an early “third draft” of history.14

Is Official History inevitably biased?

18There are two conventional responses to official histories as they are produced. The first is to doubt its value. It may be official, so the critics argue, but is it history? The second, rather conversely, is to fear its influence. As with the Second World War histories, the aggregate impact of the early “third” drafts are seen to create – if only subconsciously – a “contagions unanimity” which prejudices ail further drafts even by non-official historians.

19The potential for bias undoubtedly did, and continues to, exist. Formally, official historians are selected by government – albeit now under open competition. They are bound by the provisions of the Official Secrets Acts and, because they may see information which is still classified, undergo security clearance. They are also provided with a broad synopsis of the projected history and agree to abide by certain rules, mainly of a technical nature.15 Finally, their final manuscript is vetted by all interested Whitehall departments – a process which in the past has potentially conflicted with the disclaimer in each volume that “the author has been given full access to official documents. He (sic) alone is responsible for the statements made and the views expressed”.

20More serious than these formal pressures is, arguably, a danger of subconscious bias. Working in a wealth of detailed records generated by a single institution, an historian can easily be “captured” by the material and absorb the assumptions of its authors. Any effective sense of balance or perspective is thereby lost. Such a danger, however, is an occupational hazard for any historians working predominantly on the papers of one institution – be that institution government or not, and be the historian official or not. Lack of perspective is also the conventional charge of all those who maintain that no “third draft” should be written within at least fifty years of the events described.

21Real though the potential dangers of bias are, there are two powerful counterweights. The first is the existence of powerful first and second drafts of history. This means that there is, in the authors, a body of informed critics for any official history. It also means that there exists the detailed material on which others can base critical assessments. Secondly, and even more important (if somewhat less fashionable), is the principle of professional integrity. This has obviously to be exercised in the structuring and interpretation of the evidence. Alan Milward, in the preface to the first volume of his official history of the United Kingdom and the European Community, has well expressed the ideal by stating that his objective was to “eliminate simplistic condemnation and approval of what was done”.16 Rather less obviously, professional integrity has also to be exercised in the tracking down of evidence. “Unrestricted access to official material” may be promised; but the location, and even existence, of much of this material is inevitably often obscure. Effective access, in other words, depends not only on the expertise and goodwill of department record officers (whose resources are ever more stretched by the demands of FOI) but also by principled perseverance on the part of the historian. Moreover, as demonstrated earlier, there are significant gaps in the public record. Official, no less than other, historians must remain ever alert the reality that the records do not reveal everything. At times they may even have been designed to be “economical with the truth”.

22Finally, there is the issue of vetting. Given that attitudes towards the “mysteries” of government are now more relaxed than in Bridges’ day, it may be less obtrusive. It nevertheless exists and historians have to, and do, fight for what they consider right. Even if they fail, however, there is a safeguard. Lists of changes to their original manuscript are, by definition, public records. Their “unvarnished truth” is, therefore, publicly available after thirty years – or even earlier, under FOI. This is an assurance which no other commissioned historian, and especially an historian of a private institution, has.

Are Official Histories justified?

23Official histories, therefore, may reasonably be regarded not just as official but also as history. However, what – particularly given the inception of FOI – is their continuing justification? How can taxpayers be assured they are getting value for money from the – albeit relatively modest – cost of the Official History programme?17 There are three levels of justification: the public, the professional and the official.

24At the public level, Hancock’s initial ambition to write for a mass popular audience has clearly had to be abandoned. On the demand side, there are still doubts about whether such a market ever existed – the latent demand for, and ideal of, informed citizenship being, so it is now argued, another postwar delusion or myth manufactured by “consensual” historians.18 On the supply side, as FCO initiatives have shown, the internet may provide a potential medium through which to attract an audience wider than those either studying or writing history in a conventional way. However, in reality, any mass influence the official histories can achieve must continue to remain indirect – through the distillation of their arguments and evidence by more deliberately popular historians.

25At a professional level, accordingly, the first service official histories provide is as a quarry of information. This not only enables other historians to form their own independent judgements. Given that this information is gleaned from records which (despite FOI) are still typically closed to the public, it also realisés another of Hancock’s early ambitions. It “gives truth a quick start”.19 In addition, as Milward has argued, given their wide remit and sweep of evidence, official histories can provide the contextual framework not just for popular but also for more specialized studies.

26There are also two other, rather less obvious services which the Official History programme can perform at a professional level. It can both enrich and play its part in the preservation of evidence. Enrichment can occur because one of the specifie duties of official historians is to supplement archival research “by personal contact with participants [...] in order to obtain their recollections”. Given their early access to a wide range of official files, interviews can both be held while participants’ memories are still fresh and be more full. With the interviewee’s permission, the resulting transcripts can also be preserved as public records, to the benefit of all later historians.20 As for the preservation of evidence, official histories have a dual role. They can act as an informed pressure group to ensure, so far as is possible, that records generated at public expense (and especially the three groups of records earlier identified) are not treated as private. Secondly, and rather conversely, they can assist professional record officers in both departments and TNA with the formidable problem posed by the sheer bulk of public records. Only 1 % of the annual output of central government files can be permanently preserved, not least because this requires almost 3 kilometres of shelf-space.21 Under such pressure, how can selection be rational? The 1958 Public Record Act instigated a two-stage review process. Five years after the closure of a file, there was to be an initial review. The principal criterion for preservation was to be current administrative need. After twenty-five years, and in anticipation of release to TNA, there was to be a second review based on an assessment of each file’s historical value. Clearly, the Act contained a glaring anomaly: legally, the “historical value” of a file did not have to be considered at first review. As a good example of the triumph of common sense over statutory requirement, however, this anomaly appears – particularly in the recent past – to have caused little damage.

27Never the less, the actual selection of individual files for permanent preservation still poses major problems. It is here that official historians can provide some assistance. Already they are required to stamp for preservation any file from which they intend to quote. This is ostensibly to permit the vérification of any such reference although, by definition, the file should be of historical value. Having read and assessed the value of many files, or classes of files, they are also in an informed position to do more. They can offer advice on Operational Selection Policies (the broad frameworks, mutually agreed by departments and TNA, for the selection of records on particular subjects), to which any historian can contribute. They can also advise on the preservation of individual files within that framework. Their advice, admittedly, has no claim to be omniscient. They are no better qualified than many to predict long-term trends in historical research and are no more immune than any other professional historian from being blinded by their own specialism (as well as, to the frustration of record officers, from an aversion to any record being destroyed). The right to research widely, however, does incur the duty to advise wisely.

28At an official level, the role of official histories is to reinforce the “collective memory” of officials. This, as has been seen, was their purpose when they were first commissioned within the fragmented Edwardian civil service. It remained their purpose during the years of greater centralisation, when there was a clear view of how government should work and administrative officials (as much as official historians) were employed to record how it did work. Now that, with the “hollowing ont” of the State, government has again become more fragmented the need remains as great. Policy decisions, both in the formulation and delivery of policy, are typically based on precedent; and it is essential to good government that such precedents are properly remembered as well as contextualized.

29The Official History programme in the United Kingdom, particularly in its civil dimension, is almost unique.22 Other countries’ national programmes concentrate almost exclusively on military conflict. Other initiatives are driven by individual departments or research councils. Questions about its potential bias and ultimate justification are as inevitable as they are old. They are also healthy, because they provide a stimulus to ensure that a constructive relationship is maintained with the rest of the civil service (to reinforce collective memory), with record officers (to optimize the definition and selection of public records for preservation) and with other historians (in providing a quarry of enriched information). So long as subjects are chosen with imagination, the programme is administered with economy and the histories themselves are written with integrity, all such long-rehearsed doubts can therefore be dispelled. Complemented rather than challenged by FOI, the programme can continue to play a significant part in the selection of public records for the archive and in publicizing the riches that they contain.