Archives – what is our business model for the digital age?

A speech given by Director-General David Fricker at the International Council on Archives Congress in Brisbane
on 23 August 2012.

Introduction

It's a great pleasure for me to talk to you today here at the first International Council on Archives (ICA) Congress to be held in Australia and for me to be a part of this congress dealing with its themes of trust, identity and sustainability. My theme for today is sustainability – in particular, my thoughts on the Business Model of a National Archives, and what reforms we should be looking at as we operate in the digital age.

I will focus my remarks on the basic elements of our business: the records, custody, preservation, access and the legislative framework that enables all of our activities.

Our business model today

Every archive around the world has its own unique purpose or mission that defines its roles and responsibilities, within a particular scope of records, policies, regulations and of course legislation. These factors notwithstanding, we are all operating on common archival concepts.

We speak of 'records', indeed of the 'unique record' which is the 'authentic, reliable and usable' thing that can be relied upon as the 'evidence' of business activity.

The records are 'transferred' to us by agencies, donors, governments, etc. They are then in our 'custody', for our 'appraisal', 'arrangement, description', with regard to their 'provenance' and 'original order'.

We 'preserve' the records in their proper context, with all necessary arrangements to ensure they're usable by all future generations.

And of course we do all of this to ensure we can provide 'access', for without access there is no purpose to preservation.

We provide our services to researchers, genealogists, historians, government agencies and individuals, who by and large come to us for the records we hold. This could be in person or online, but by and large they are making a decision to visit the Archives to access archival records.

Our value proposition? Our differentiator? We guarantee the long-term availability of the authentic source record, the original – the reliable, irrefutable evidence.

At the National Archives of Australia we think about these basic concepts a lot – because every one of them will be turned on its head by our digital society, and this will happen well within the next decade. Which means our preparations should have already started by now!

History

First, a little history.

When the National Library Act came into force in 1961 the 'Archives Division' became an independent body, the Commonwealth Archives Office within the Department of Prime Minister. A review of the Archives in the early 1970s argued that not only should a modern archives system watch over government records at every stage of their lifecycle, but that the National Archives should also be more outward looking. It wasn't until 1983 that the organisation had enabling legislation, the Archives Act, giving the Archives the power to preserve the archival resources of the Commonwealth; providing that the public should have access to records in our custody; authorising the development of records management standards, services, products and training; and making it illegal to destroy Commonwealth records without permission from the National Archives or under the authority of some other piece of legislation.

A program of 'major change' led to the launch of the National Archives' e-permanence range of products in the year 2000. The e-permanence suite of tools and guidelines, based on the concepts and strategies recommended in AS 4390 and later ISO 15489, provided the framework for government agencies to develop systems to make and keep good records – including records 'born digital'.

From 2002 to 2006 the Australian National Audit Office released reports advising on government recordkeeping , the last of which made recommendations resulting in the 2007 launch of Check-up, and then, in 2011, Check-up 2.0 – a practical example of the role of the Archives in assisting government departments to self assess the state of their records and information management.

Also in 2011, the Australian Government established the Digital Transition Policy. For many government agencies the new policy means moving from paper-based records management to digital information and records management. To support the new government policy, the Archives developed the Digital Continuity Plan, which provides guidance to government agencies on the management of digital information for as long as it is needed.

Developments in digital archives

Of course, none of this work is done in isolation. Collaborations, both direct and indirect, are crucial to the ability of the Archives, and I would say the profession generally, to develop new products and to improve skills and abilities.

I would like to specifically mention some of our collaborations across Australian state and territory borders, across 'the ditch' to New Zealand and beyond our Pacific region that have helped position Australia and the region as standards setters. The National Archives has close links with archival and cultural organisations within Australasia, as well as with professional and industry bodies and the higher education sector.

I meet regularly with national, state and territory archivists of Australia and New Zealand through the Council of Australasian Archives and Records Authorities, or CAARA. CAARA is the peak body of government archival and records institutions in Australia and New Zealand.

One CAARA initiative is the Australasian Digital Recordkeeping Initiative, or ADRI.

ADRI was formed to articulate and promote a common approach to digital recordkeeping and to collaborate, engage and inform, and share resources in the development of practical strategies for enabling the creation and preservation of, and long-term access to, born-digital and digitised records.

The National Archives supports and liaises with a range of other groups. At the national level, we work closely with professional and industry bodies including the Australian Society of Archivists, the Records and Information Management Professionals Australasia, the Australian Institute for the Conservation of Cultural Material, and Museums Australia.

We also work with the higher education sector, providing course advice and participating in Australian Research Council collaborative research and development projects.

In more recent years, under the auspices of the ICA, the National Archives led an international project team developing globally harmonised statements of requirement for electronic records management software applications. ICA-Req – the ICA Principles and Functional Requirements for Records in Electronic Office Environments – was launched at the ICA Congress held in Kuala Lumpur in 2008. The current project ICA-Req2 is also being led by the Archives and entails the development of guidelines to put ICA-Req into practice.

In late August last year the Archives was thrilled to receive the UNESCO/Jikji Memory of the World Prize. In announcing the prize, UNESCO described the National Archives as 'a world leader in many areas, notably that of digital preservation'. The announcement also praised the National Archives for sharing with others 'the fruit of its own research and development by making available open source tools for digital preservation'.

What got us here, won't get us there

But 'what got us here, won't get us there'. There's still plenty to rethink and re-engineer if we are to take every opportunity offered by the digital age, and meet the expectations of the digital natives who are our next wave of clients.

Disruptions to the business model

So what are the megatrends that are disrupting our current business model, and how are we framing our position as Australia's National Archives?

Expectations – government and community

Let's consider the environment we are operating in: the environment our business model has to be robust enough to survive.

National culture

The Australian Government will be releasing its National Cultural Policy later this year.

One of the cultural policy's proposed goals focuses on technology, innovation and participation – using emerging technologies and rich information sources to enable more people to access and participate in the cultural development of the nation.

Crucial to this will be the ability for artists and the creative industries to locate and draw inspiration from national stories. A solid cultural identity cannot be manufactured on a selective or indeed invented record of the past – deep and lasting cultural understanding and national identity can only come from the truth, an accurate 'warts and all' account of the past. This is a clear role for Archives, and we must ensure we are among the front line of 'content providers' in the cultural life of the nation.

Open government

In 2010 the Australian Government published Ahead of the Game, a set of public service reforms that confirmed a commitment to open government, calling for openness and transparency, for example through a 'pro-disclosure' culture and greater engagement and participation of the public in agency and policy activities, particularly through online sources.

In this context we are seeing increased use of a 'co-design' approach to government policy and programs, using Web 2.0 technologies to put the citizen at the centre of planning and invite more participation in the formulation and indeed delivery of those policies and programs. Saying to the citizen that we are delivering services 'the way you want it' as opposed to the way the government departments are organised.

This move to a more inclusive, open government is matched by increasing levels of scrutiny upon the actions of organisations – particularly government organisations. People are less tolerant of poor governance, expect to find answers in government records and rightly expect to be able to trust those records, that they will be authentic and have integrity. People are not afraid to challenge, through legal frameworks, when they feel their rights have been abused or ignored. Across Australia, we are seeing more and more resources directed to responding to Freedom of Information requests, subpoenas, discovery orders and inquiries.

There is also an expectation of easy access to all information, whether official records or not, in an age where information is available 24/7 from a myriad of sources.

Government information is increasingly seen as public information, not the sole property of government but democratised and socialised through developments such as information disclosure policies, social media, Wikipedia, crowdsourcing, and blogs.

This is a globalised information marketplace, and it's not being shaped by government institutions like mine – it's being shaped by market forces.

And of course in a heated and dynamic marketplace, the right to privacy and requirements of national security must be reconciled and balanced with principles of freedom of information.

Government information management practices

The business of government is increasingly conducted in cyberspace, with a consequential fragmentation of information holdings across many digital platforms including email servers, social media accounts, electronic business systems, shared drives, mobile devices and Electronic Document and Records Management Systems (EDRMS) – and the list is growing. For more than a decade, government policy has preferred that services and information be provided to citizens by digital means. Recent policy initiatives such as the Declaration of Open Government and legislative changes are accelerating the requirement.

Government policy and Archives' strategies are facing the reality of the tide of digital information washing over us all. A survey conducted by the Archives in 2010 showed that by 2014 the total volume of electronic records which agencies expect to create will be more than 10.7 million gigabytes – and that's just the new records, not the overall amount of information that needs to be managed.

These are exciting times and there is no doubt that huge leaps are being made in day-to-day efficiency and utility of information, but we are at grave risk of losing any substantive archival memory of our time, this fragmented information could evaporate as easily as it was created.

Here in Australia, the Australian National Audit Office has just released a report into recordkeeping practices of government departments.

It found that Australian Government agencies create a substantial amount of electronic information and records as part of their normal operations. However, in 2009 less than 30 per cent of these agencies and bodies managed the majority of their records digitally, even though more than half reported having an EDRMS and using other electronic business systems to manage records. Many other electronic business systems that were not identified and functioning as 'records management systems' were also used by the agencies to create, capture and manage records. These systems did not generally meet legal requirements relating to the management, and destruction or transfer, of records.

Establishing effective records management, particularly digital records management, represents a significant business issue for many agencies.

Archival concepts for a digital age

What is a record?

At the heart of everything we do is this concept of 'the record'. Since taking up my position with the Archives, I have been most surprised by how mercurial this concept is, and how subtle differences in our definitions can lead to major policy dilemmas. So what is a 'record' in our business model for the digital age?

My view is that the record is the 'thing' that provides evidence of business activity.

It exists in a virtual space.  It is the information, the knowledge, or the experience – but it is not the medium that carries it.

In the digital environment a record is 'made' by a business process. As far as the record's creator is concerned, it might be produced deliberately, incidentally, accidentally, or even invisibly. But my view is that it doesn't wait to ‘be made' by a separate act. It exists and has meaning well before anyone has separately 'put it to file'.

It has a digital context that is global, interconnected and commercialised – comprising collections which have no meaningful physical presence. A business transaction can traverse an email, a tweet, a business system, a voicemail, a website. The provenance of the records is clear; however, the ownership is not, and may present new challenges for 21st-century archivists.

A record typically comes loaded with metadata offering many new opportunities for more comprehensive and accurate appraisal, description, arrangement, discovery and retrieval.

A digital record will move from one medium to another, as it cycles through ICT maintenance and data migration projects, from disk to tape and back again. In order to use the record, it will also be rendered many times over: sometimes on a computer screen, projected on a wall, or printed on a sheet of paper. This movement or viewing of the record does not alter it, nor is it 'reborn' – the magnetic tape, computer screen, wall or paper does not become part of the record.

Our business model must recognise this, both in in practice and in law. It is essential that the record maintains its unique identity as it moves from one medium to the next – provided that:

  • It is not altered
  • All of it is preserved
  • It is authentic
  • It reliably provides evidence of business activity

What are we preserving?

If you accept this definition of a record, a somewhat mercurial thing that maintains its identity but changes its form, then you must be clear about what it is we are preserving.

We are framing our thinking around the concept of the 'essential performance'. This is a concept that accepts that the digital information must endure well beyond the utility of the medium that carries the information.

In a way we're trying to preserve the 'song' long after the 'singer' has departed. But can you separate the song from the singer? And is it the same song? Is it enough to preserve the music sheet, or is it the performance that matters? Will future generations have a different view to ours? In short – what are we losing even when we think we've preserved everything?

Our response to this is the concept of 'essential performance', a formal mechanism for determining the characteristics that must be preserved for the record to retain its meaning over time.

For example, the essential characteristics of a word processing document may include textual content; formatting such as bolded text, font type and size; layout; bulleting; colour; embedded graphics and of course elements of the metadata such as author and creation date/time.

Non-essential elements may include the characteristics of the software that created the document, ordering of bytes in the data file or the specific format of the electronic document, since these are not essential to the 'performance' of the record.

As you would have heard from Michael Carden earlier in the program, at the National Archives of Australia we make decisions to preserve the essential performance, but we also keep a complete copy of the original form, 'just in case'.

Of course, there are some physical documents and objects that must be preserved. There are some exceptional historical records that connect us with the past, both by virtue of the words on the page but also through a more emotional experience of seeing, perhaps holding, the very paper that the original record is on. So I stress that we'll still need to keep some physical objects to recreate the 'essential performance' – but I'm equally certain that as time moves on these will be very few by comparison.

In our Digital Business Model, preservation must start well before the originator thinks about transferring it to the Archives. It must start the moment the record is created.

Increasingly the issue is that records are initially created in digital form but are printed to paper for records management and preservation purposes. This introduces many inefficiencies and costs into the agency, but also loses much of the important metadata originally held by the digital record. To address this, in 2011 the Australian Government established the Digital Transition Policy, which will see all Commonwealth Government agencies moving from paper-based records management to digital information and records management. To support the new government policy, the Archives developed the Digital Continuity Plan, which provides guidance to government agencies on the management of digital information for as long as it is required. Importantly it also ensures that it is not kept for longer than it is needed.

The Digital Continuity Plan (which is published on our website) has at its core six principles:

  1. The value of digital information as a business, evidentiary and community resource is understood and the information is managed accordingly.
  2. The governance of digital information is integrated with agency governance, with roles and responsibilities clearly defined and allocated.
  3. Digital information is authentic and reliable.
  4. Digital information is discoverable, accessible and usable.
  5. Digital information is managed digitally.
  6. Digital information is managed, protected and preserved for as long as required and then disposed of appropriately.

Getting digital preservation right is an enormous challenge; obsolete technology can render digitised documents inaccessible for future generations. The Archives has developed a suite of software to deal with this challenge. This suite of products is referred to as the Digital Preservation Software Platform (DPSP), made up of four applications: Xena, Manifest Maker, Digital Preservation Recorder, and Checksum Checker. These tools are on the Archives website and are available for download and use. This open source development allows the Archives to share results with other users, so we can achieve our goals more quickly with fewer resources.

What is custody?

Another aspect of the digital business model is the 'transfer' or 'accessioning' of records into our custody.

Custody in the virtual world is more about 'control', less reliant on physical location or proximity. Data belongs to an enterprise but is usually housed in a remote data centre, a dark room many kilometres away. Or, indeed, in a cloud, which may itself draw upon a geographically diverse array of servers and data stores. The point here is that 'custody' means protecting, monitoring and controlling records – not building a brick wall around them.

So how will this shape our business model?

For one thing, we won't have to wait for an agency to transfer records to the Archives. Based on the metadata and the business rules that we embed within the agency's systems, the record could 'transfer itself'.

More and more, an organisation's information management is conducted in the digital environment – whether it be the specific recordkeeping system or any of the electronic business systems running within the enterprise.

Our future in this environment, in particular to guarantee preservation of the record and proper and timely custody of the record by the Archives, is to embed business rules within those systems; in effect, to automate as much as possible such that the sound practices and proper information governance are a natural by-product of the business process, not a separate activity occurring several years after the event.

In this model, the Archives could have the record as soon as it's created. Why wait until it's too late?

Taking the digital record means that we're not preventing the ongoing use of the record by the originating agency. No constant requests to lend or transfer the records between physical locations.

The record may still be in administrative use, but its preservation is assured.

The embedding of business rules in the source systems will also ensure collection of the optimum level of metadata. This will provide enormous benefits as we are able to automate many aspects of accessioning, arrangement and description of our archival resources.

Access examination, to properly treat privacy, security and public interest issues, will be far more efficient and effective, supported by advanced data analysis and retrieval tools – taxonomies and ontologies that provide for concept-based searching to fully appreciate the significance and interrelationships of the information in our care.

And of course the originating agency will have a more complete and accurate repository available whenever it requires access to its preserved records. Current lending and transfer processes will be replaced with continuous online access with no risk to archival material.

What is access?

Make accessible means render the essential performance.
The expectation is that this will not require a trip to a reading room or a wait until an exhibition is mounted.

From our citizens' point of view it must be 'when I want it', and 'where I am, right now' – be it at home, in the office or riding on the train.

In this environment, our records will be intermingled with all the other information out there, and the consumer will take the easiest, most convenient path to satisfy his or her query or research requirement.

Queries won't always start with a virtual visit to that Archives website, but instead will be serviced by third party federated searches and concept-based algorithms for discovery and assembly of information into highly customised factsheets or result sets, by commercial portal services influenced by advertising revenue or a paywall.

This is, as I said earlier, an information marketplace. A crowded marketplace at that, with more and more choice and power to the consumer.

I think the majority of people in this marketplace won't be 'serious' researchers – they won't want to retrieve a series, or perhaps even an item or a record. They will want a 'factoid', a piece of information that fills a specific gap in their knowledge or answers a narrow question – a date, a name, a place or a quote. They might stop looking as soon as they believe they have what they want.

A really interesting development is augmented reality. Anyone with a smart device can simply point at a document, a building, a monument or a landscape and enjoy a rich array of audiovisual information that brings the object to life.

This means that people will not even have to type a query, or think of a question to ask. They won't have to know that the Archives has relevant and interesting information. It will be possible for us to augment the object itself with our archival resources, presenting it in an interesting and engaging way – perhaps inviting the person in to explore our collection even further.

Another key development in expanding access is crowdsourcing. As many of us are now seeing, by allowing the public to contribute to the description of archival resources we are enhancing the ability of future generations to discover and learn from our archives. I also think it is a wonderful opportunity for the public to be more engaged with us as archives and to share in the work we do – preserving the memories of our nations. There is still some work to do here, in order to maximise the value of contributions and to maintain the integrity of our archives as authentic and accurate. However, I do not believe these problems are insurmountable, and indeed I believe these systems can to some extent be self-correcting.

This is a type of the co-design, citizen first activity I mentioned earlier: drawing on the interest and enthusiasm of the community to bring more of our archives into view – discoverable and retrievable.

The slide I have here shows the National Archives of Australia's most recent adventure into this area, Destination: Australia.

It was unofficially implemented in June this year, and is now building a user community through word of mouth and some mainstream media attention. We have around 50 people adding tags, with around 150–200 visits per day.

The featured photographs come from a promotional series of photographs taken by the Department of Immigration which is now stored as the Immigration Photographic Archive collection.

The series contains more than 25,000 photographs and over 21,000 of those are featured on this site. With nearly six million migrants to Australia since 1945 it would be challenging to try to identify everyone who might appear in the photographs, and to collect and share their stories. But with the help of all of Australia, we are going to try.

Increasingly our information and our services are contestable – that is, the records we hold will be held in any number of other places, and will be discoverable and retrievable by any number of other content providers. Because of contemporary legislative developments in freedom of information and open government, not to mention greater scrutiny of government institutions by investigative journalism, much more of the information we hold will be publicly available well before the statutory periods that control the release of archival records.

In this information marketplace, our brand, our differentiator, is everything – we may not be the first port of call, but we will be the last port of call when the authenticity of the fact is more important than the convenience of the search, and when the evidence of an act or decision is more important than simply the understanding or knowledge of it.

We also need to consider the most basic principles of provenance and original order. It's a world of contestable information and competitive information services, and this world is inhabited by people who consider themselves to be 'time poor', lacking perhaps the motivation to go that extra step to corroborate their findings or understand the historical or business context of events.

It will be important, therefore, to make our archival resources discoverable in as many portals and search results as possible, but the next step of opening the details should, in my view, bring them in to an online archive, using rich data-visualisation techniques to encourage a broader view of the information and the context in which it was created.

Productive partnerships with like-minded organisations from the public and private sector will be a key to our success.

In summary: the new features of our business model

We will be embedded in business systems to ensure complete, authentic records are properly preserved – at the time they are made.

We will have legislative and regulatory frameworks that differentiate between a record's 'essential performance' and the medium upon which it is held. Those frameworks will also recognise that an organisation's records exist across many data storage platforms, potentially across many legal jurisdictions.

Our relationship with originating agencies will be changed – business rules embedded in an agency's systems will allow the preservation of records at the time they're created, without constraining the ongoing administrative use by the owning agency.

We will exist alongside, and intermingled with, many other information portals, offering much of the same information that we hold, sometimes in formal partnerships. Uniquely, however, we will be a rich content provider in a global information marketplace, ensuring that the 'whole story' is remembered and can be retold.

And, of course, access will be online and everywhere, improved by rich new data visualisation techniques and expanded descriptive contributions from an engaged citizenry.

But I am equally confident that some things will not change: in a world full of information sources, we will be consulted for the authentic, complete record – that can truly be regarded and relied upon as evidence.

And the 'true record' will preserve the cultural heritage of the nation, support the rights and entitlements of our citizens and provide accountability and transparency of government.

Thank you.

Copyright National Archives of Australia 2017