The National Archives of Australia Digital Business Roadmap
Presented to the International Council on Archives 2013 Conference held in Brussels, Belgium
23 November 2013
The Archives and the Digital World
The National Archives of Australia is tasked under the Archives Act 1983 to, among other things, ensure the conservation and preservation of the existing and future archival resources of the Commonwealth of Australia, to provide advice and assistance to Commonwealth institutions on the creation, keeping and management of records, and to promote the value and use of the archival resources of the Commonwealth. Traditionally these functions translated to the physical management of analogue, predominantly paper, materials.
The changing definition of ‘record’ and ‘information’ in the digital world presents challenges for the Archives in its ability to manage an increasingly broad and complex range of information types, formats and systems. With greater diversity and complexity comes greater cost in receiving, preserving, managing and providing access.
The Archives management of analogue records has traditionally been based around defined points of intervention in the life of the record. In many cases this intervention commenced at the time an agency wished to transfer records to the Archives. Points of intervention included descriptive work, repackaging and conservation, access examination and more recently, digitisation. As long as the Archives had custody of the record, the intervention could happen at any time. This intervention was generally driven by preservation needs and public use priorities; in many cases by a public request for access, necessitating a reactive business process to satisfy that request. The ultimate result of numerous interventions or actions by Archives is a record publicly discoverable and deliverable online.
This luxury of time is not, however, a characteristic of the digital archive. Put simply, paper is patient but digital won’t wait!
Information created and managed digitally will not remain intact waiting for Archives’ intervention. Digital government information must be prepared for preservation, discovery and delivery as soon as practicable. Postponing the necessary steps to make information available to the public until such time as that information is requested exposes the Archives to an unpredictable demand on its resources and places that information at significant risk of data loss through obsolescence of format and corruption. Increasing user expectations of fast online access to government information also makes this approach unsustainable.
In the digital records environment our focus to date has been on the preservation of digital information for future use. This is a reflection of the traditional custodian role of an archive. However this approach postpones the work required for the preparation of digital records for discovery, delivery and use, and it misses some of the extraordinary advantages digital information has over analogue records by nature of inherent qualities of transportability, discoverability and usability.
Digital Transition Policy
A major turning point of the Archives strategic direction in managing digital information was driven by the Australian Government's Digital Transition Policy. The Policy aims to move Australian Government agencies to digital recordkeeping for efficiency purposes.
Digital information management means that the majority of an agency's records will be created, stored and managed digitally and, where possible, incoming paper records will be scanned so that new paper files are no longer created. For many agencies this policy means a ‘digital transition’ from paper-based records management to digital information and records management.
In accordance with the Digital Transition Policy, records that are created digitally after 2015 will be accepted for transfer to the National Archives in digital formats only.
This is one of the business drivers that caused the Archives to re-assess its directions and work to provide an end-to-end business solution for the management of government digital information from its creation in an agency to its eventual use and re-use by public, government and commercial users.
While the Digital Transition Policy was a turning point, other whole of government and societal changes demand an archives response. Other drivers include:
- Open Government – The Australian Government’s open government principles , published by the Office of the Australian Information Commissioner, drive, as well as recognise, an overall push to make government more accountable through a more open approach to government information. The Archives must provide clear advice to agencies on recordkeeping and information management, as well as change the management of our own business information.
- The Australian Government Information Management Office (AGIMO) aims to improve the efficiency and effectiveness of Australian Government Information and Communications Technology (ICT) activities through whole of government solutions, standards and guidelines. It guides the government’s response to and adaption of emerging technology. The Archives must provide information management solutions in response to government use of new technology. Archives can leverage the reach and power of AGIMO to embed recordkeeping and information solutions across Government.
- Australian Public Service Information and Communications Technology Strategy 2012-2015 – The purpose of this strategy is to simplify and join together services that government provides; use ICT innovatively to improve the efficiency and effectiveness of service delivery; and to engage more openly with citizens. This is a convergence of services across government and a standardisation of public interaction.
- National Cultural Policy – This policy outlines the government’s requirement to expand our outreach and engagement, and to improve access to our collections and programs. Digital delivery is an efficient and creative approach for the Archives to respond to this requirement using innovative products and delivery methods, including mobile device delivery.
- Australian Public Service Mobile Roadmap  – The recent APS Mobile Roadmap drives the Archives to deliver services and products to users via mobile devices. It recognises the increasing use of mobile technology – over 50% of adult Australians use smart phones to access the internet ; the expectation of users for government to deliver services on mobile devices; and the potential for broader engagement, greater access and operational efficiency. This drives the Archives to review the delivery methods of its new and existing products and services, and to provide information management advice about the use of mobile platforms in government.
- e-Government – More than 80% of Australians expect the Government to deliver services online . This strategy recognises and responds to the wholesale move toward digital life – covering personal, business, health, education, and government. It recognises that effective participation in the digital economy by government can reduce costs, broaden the delivery base to reach a greater proportion of the community, increase customer satisfaction and engagement, and promote innovation. Central to the Strategy is the commitment of Australian Government agencies to a policy of Digital First ‑ using digital channels as the primary or preferred means of delivery for government services. This changes the way society interacts with government, and the records of those interactions, and it changes the way the Archives provides its services.
- Archives Act amendments – The reduction in the open access period from 30 to 20 years is driving a faster resolution of the difficulties in digitally redacting sensitive information from the public record, and brings forward our need to deliver high volume access to born-digital records.
- Big Data – The rapid growth in the volume of digital information, in particular information in agencies, is driving the Archives to deliver whole-scale approaches to what were traditionally case-by-case issues. This presents particular challenges in relation to appraisal, release for public access, discoverability, and delivery.
- National Broadband Network – The expectation is that the Australia-wide rollout of high speed broadband will enable government agencies to broaden the extent of their online products by having access to greater delivery opportunities. This is coupled with the user expectations of a broader and more complex range of information being readily available.
Developing the Archives Digital Business Roadmap
Providing leadership and support for the transition to digital information and records management across the Australian Government is a strategic priority for the Archives. This includes leading implementation of the Government’s Digital Transition Policy and developing the ability to ingest, manage and deliver the digital records and archival resources of the Commonwealth.
In 2012 the Archives engaged external consultants to assess our digital capacity and readiness for the digital transition. The study’s aim was to identify opportunities where access to the digital records in the Archives collection could be improved. It recognised that access issues need to be tackled at creation, and embedded in the business systems that create and manage information. The highest priority recommendation from the consultant’s report was of the Archives to develop a Digital Business Roadmap.
Establishing the Vision – the Digital Destination
The first step in establishing the Archives Digital Business Roadmap was determining the destination. This is how the Archives envision digital government information will be best managed in the medium term future. This ‘Digital Destination’ was centred on the following five key activities.
Precreation of Records
The Archives is actively involved in the development and procurement of hardware, software, business systems and standards that influence the existence and creation of information used for Australian government business. This involvement prior to record creation gives government information inherent qualities and characteristics that better enable it to be managed for its intended business use and beyond, including its archival preservation, public discoverability, access, and delivery. It includes using our knowledge and influence to embed information management principles, practices and functionality into whole of government digital economy initiatives such as whole of government ICT and device procurement policies, use of the cloud by government, and data centres.
Australian Government agencies are supported to create and manage information that enables them to better deliver their business with accountability and integrity, in common/core systems, and applying metadata that facilitates integration into the Archives business and systems.
Ingest and Custody
The Archives has the capacity to receive digital information at the rate agencies wish to transfer, to process digital transfers and to store them securely. It has a robust policy on the custody of the diverse range of digital government information which may include management of a distributed collection. Australian Government agencies are satisfied with the transfer, preservation and access processes and are ready, willing and able to transfer digital information. Automatic ingest of digital information is facilitated through business and recordkeeping systems that engage with Archives data harvesting. The Australian people have access to digital archival information held across the government in a fast, equitable and open online environment.
The Archives digital information management is recognised by the government and the public to be secure, to have integrity and authenticity, to be adequate for Australian Government agency needs, and is being continually developed to cater for future needs. Preservation and normalisation of digital information is achieved in the most cost effective manner at a time that provides the greatest efficiency. Digital information held outside the Archives satisfies the integrity, authenticity, accessibility and delivery needs of the Australian Government and the Australian people.
The Archives is able to provide meaningful access for the public to born-digital records. The level of description of analogue records is continually improving towards the descriptive standards applicable to newly created information. The Archives has the systems, networks and processes to securely examine and release information to the public in a format that satisfies their information needs and with information that is accepted as authentic. The Archives provides information to users that meets their needs in relation to: format and content; is recognised by them as authentic; is usable and re-usable; is linked to their digital identity for continuity of use; is searchable by content; is accessible using the interface of their choice; and their interaction in locating and using information provides them with the tools and functions to assist their use and meet their social sharing requirements.
Transitioning towards these goals will require the Archives to focus on three key areas:
Defining the Archival Resources of the Australian Government
The Archives will develop a clear and evolving policy supported by legislation that sets out the limitations of the types of information that can be examined, stored and delivered, including:
- The principle that digital transfer to Archives of RNA material is the preferred action, but certain exceptions mean that some material will be best managed within the creating agency for a period of time;
- Determining criteria to assess material that is best managed within the creating agency. This includes information requiring specialist management or preservation; the interpretation by agency experts in order to deliver meaningful and useable information; and outlining the avenues for preservation, public discovery, security and delivery of that information;
- Preparing guidance on cases where data extraction or generation from proprietary formats or systems may be preferable for access, usability or preservation requirements, rather than preserving the business system or database as a whole;
- Developing criteria outlining restrictions on the types of data unsuitable for general public use due to the specialised form or nature of the information, its reliance on tools or systems to extract meaningful or useful information packages or the need for specialist interpretation of the information;
- Determining if and when embedded information such as previous versions, tracked changes, management history including when viewed or changed, linked information and underlying formulas must be included in the digital archival record and the extent to which earlier versions, drafts and working papers need be retained as part of the national archive;
- Determining criteria to set the limitations on investment in attempting to discover information from obsolete formats; and
- Ensuring the right of public access to information held outside the Archives, its preservation, integrity, discoverability and delivery.
Using metadata standards to advance archives business requirements
The Archives will develop mandated minimum metadata component sets for transfer for common record types, databases, and websites. It will publish optimal metadata sets that indicate future mandated components, for both born-digital information and digitised analogue records. The sets will be developed over time to support automated sentencing, transfer processes, preservation, and management within the agency and in the Archives, and allow for greatest possible discoverability such as machine readability and content searching. Archives will work with agencies and the providers of business systems to incorporate the required metadata management functionality into information management systems. The metadata sets will be regularly reviewed to capture changes in the way Government content is created, organised and distributed. The regular review will seek a continual improvement in the transfer set to better facilitate archives business requirements, and the seamless flow of information from agencies to the public.
Investing in technological solutions
To move forward with delivering and end-to-end business solution, the Archives will:
- Invest in the digital storage capacity to cater for the demands of increasing digital information being created and managed by the government;
- Partner with agencies and systems developers to provide agencies with the systems to best manage their information;
- Design the systems architecture that will meet the needs of agencies, users and the Archives, covering the functionality to ingest automatically, securely store to the required level, prevent loss and degradation, and provide tools for online transfer, discovery and delivery;
- Integrate into the system preservation systems and processes;
- Accredit the security of system components to allow for the management of material of National Security classification, and its de-classification for full or partial public release;
- Expand the ability to search analogue records through the application of technological solutions to traditional analogue activities, including digitisation to a standard that enables content searching, and the use of crowd-sourced input; and
- Integrate user contact into a single interface allowing for linked searching, access application, purchasing, user input, sharing, tagging, transcription, publicity, and events notification.
Critical Business Decisions
Integral to developing an end-to-end business model for the management of digital government information are several critical business decisions. These decisions will determine the directions and strategies we employ, and will drive our policy and development of business solutions.
- Digital appraisal – In regard to digital information, is the value in the information (content), or the visual depiction of that information in the manner it was created (the digital object)? Is the answer the same for all digital information? Should appraisal decisions take into account the cost of normalising or managing complex or specialised digital information? If not, there is a risk that it will be beyond our resources to interpret, extract or deliver information from that data in any useful form. If we do reject data solely on these grounds, we must determine custody, security, preservation, and public discovery, access and delivery arrangements for this information. If appraisal decisions determine that some information is best held by the creating agency, a distributed collection model will require agencies to take on the responsibility of maintaining, preserving, securing, providing discovery metadata, and public access to their information. Will they be willing to do so? Can and should we sub-contract our legislated responsibilities?
- Digital transfer – Is an agency-accessible online transfer repository a sustainable model to serve agency transfer needs and to allow agencies to access to their information online? If agencies transfer ‘open’ information, should the Archives take on the responsibility to provide public access to this current or near current information? Will more recent digital data add complexity to our discovery and delivery pathways?
- Transfer formats – Will the Archives accept every digital information format for transfer? For obscure formats, are the resources required to preserve and provide access sustainable?
- Normalisation – Do all records need to be normalised to ensure preservation? Are the benefits from normalisation greater than the costs, especially for rare or bespoke formats?
- Digital delivery – What public delivery format of digital information should the Archives provide – as close to the original as possible, or in standard or normalised formats?
- Digital redaction – Will the Archives partially release digital information by redacting sensitive parts? What is the risk of forensic discovery of the redacted part?
- Resourcing – Given increasing resource constraints, what current functions or activities can be suspended, ceased, or reduced to release resources to advance digital goals?
To move forward on delivering the digital solution, the Archives has initiated seven strategic projects:
- Agency Forum – The aim of the Agency Forum is to create a group of willing agencies who will initially provide functionality input into our systems design, and then provide a safe real-world environment to test experimental and emerging digital capabilities such as online transfer and access, metadata management, automated sentencing and data harvesting.
- Agency Audit – An audit of agency information will provide the Archives with data on:
- volumes in terms of both digital storage for capacity planning and digital objects, for discovery and delivery strategies;
- volumes and levels of classified digital information;
- the range of metadata used by agencies and how it is managed;
- business and information systems used by agencies and their functionality, especially in terms of sentencing and transfer; and
- common document, database and web formats.
- Internal Digital Transition – This project responds to the Government’s digital Economy and is designed to show the Archives as an exemplar agency for digital business. It involves converting as far as practical the Archives offline and analogue business transaction – both internal and external – to digital and online. This project will cover our dealings with staff, contractors, service providers, government departments, agency clients and the public.
- Online Access – We are working to provide public users with better online access for digitised analogue material, and progressing towards functionality that improves record description by technology such as optical character recognition, user tagging and transcription and voice recognition. This will guide our provision of access to born digital records.
- Strategic Partnerships – We are working to build an overall strategy to engage professional, government, educational, commercial and community organisation to achieve common goals. The strategy will outline how we assess the risks, costs and benefits of potential partnerships, and allow us to respond faster to opportunities. It recognises the benefits of shared costs, capabilities and knowledge.
- Mandated Metadata Sets – Metadata standards have no value if they are not used for advancing Archives and agencies business needs. Mandating a standard minimum set of metadata elements for acceptance of agency record transfers will enable the Archives to have custody of material that is better prepared for long term preservation, discovery and access. Adapting, expanding and improving the metadata sets over time will lead to a continual improvement in agency record quality. Partnering with industry to develop the systems that support the standard sets will enable agencies to meet our transfer needs and to improve their information quality. This project has its foundations in three functional areas:
- Metadata standards, their use and application within agencies;
- Specialised metadata used for Audio-visual management and preservation; and
- Systems capabilities and designing the systems architecture to manage metadata for Archives business.
- Systems Architecture Design
This project will have two components:
- Systems design, testing, and proof of concept will be a combined effort from our internal ICT experts and contracted solutions architects. The first stage of the project will be to procure or design, and then test systems components and their integration, and recommend an achievable solution. The design will be informed by each of the preceding projects, and our experience with the existing digital archive and its current functionality.
- System architecture implementation and investment is a project requiring funding support to deliver the recommended system requirements in terms of hardware and software, infrastructure, support and maintenance. This stage of the project will seek to fund the systems solution proposed in the Systems Design, Testing, and Proof of Concept stage, and to build, test and implement the solution.
Responding to the digital challenge is a whole of Archives responsibility, and will require a cultural shift to view ourselves as managing digital government information rather than just a legacy collection. It relies on the Archives embracing technological solutions to new challenges, and on using technology to improve our existing service and product management and delivery.
The Archives Digital Business Roadmap recognises the need to push for reform in the culture of the Archives. It recognises the changes required, in recordkeeping and business systems used across government; the whole of government and societal changes that are driving these reforms; and the existing powers, technology and knowledge that will enable the Archives to delivery an end-to-end digital business solution.
- Office of the Australian Information Commissioner, Principles on open public sector information: Report on review and development of principles (2011).
- Australian Public Service Information and Communications Technology Strategy 2012-2015, Copyright Australian Government 2012
- Australian Public Service Mobile Roadmap, © Commonwealth of Australia 2013
- Australian Communications Media Authority Report 3—Smartphones and tablets - Take-up and use in Australia January 2013
- Australian Communications Media Authority Report 3—Smartphones and tablets - Take-up and use in Australia January 2013