Archives End-to-End Digital Business Solution Design

Organisational context

The National Archives of Australia’s role is to promote the creation, management and preservation of authentic, reliable and usable Commonwealth records and enable ongoing access to the archival resources of the Commonwealth.

The business problem

Records currently being created by Commonwealth agencies are predominantly digital. Under the Digital Transition Policy, any records that are created digitally after 2015 must be kept in a digital format, and any that are identified as RNA (Retain as National Archives) must be transferred to the National Archives in digital format.

Managing digital records brings a range of challenges, and the National Archives must further develop its business capabilities in order to sustainably harvest, preserve and provide access to its born-digital collection, as well as increase its capacity to provide online access to its analogue collection.

The primary challenges of digital records for the National Archives in implementing sustainable business processes for managing the lifecycle of digital records from their creation to providing access are outlined in the table below:

Prepare

Multiple Formats

Structured and Unstructured

Static and Dynamic

For humans and for computers

Size of each item and volume

Ingest

Proprietary Formats

Transfer

Reliability of Storage

Metadata Extraction and Generation

Security Threats (Viruses)

Authentication and Access Control

Preserve

Normalising and Standardising Formats

Licensing for Commercial Formats

Storage of Structure Data

Items that link to external services

Capturing behaviour as well as structure

Manage

Storage of Resources that can be arbitrarily copied and moved electronically

Dynamic content could 'change itself'

Dependency management

Copyright and Intellectual Property

Access

Classification

Copying

Distribution

Size

Decisions around access

Authentication and Access Control

Validating Generated Data and Metadata

 

Project objectives

The Archives End-to-End Digital Business Solution Design has been created to provide an end-to-end system for digital records that will enable the National Archives to effectively meet the needs of Commonwealth agencies and public clients. Based on the Open Archival Information System (OAIS) framework, it will:

  • Provide appropriate online access to all records as soon as possible, to clients who are anywhere, at the time the records are required, and in a format and on platforms that meet their requirements.
  • Preserve and manage the Commonwealth’s digital records ensuring their long term integrity and authenticity.
  • Provide cost effect and efficient sentencing of Commonwealth agency digital records and their transfer to the Archives.

The target state architecture envisaged in the Archives Digital Business System will meet the challenges within the archival lifecycle for digital records in a scalable, flexible and sustainable way – with as much automation of business processes as possible.

Desired outcomes

From Agency to Ingest

Commonwealth Agencies will have their record producing and management systems integrated into the Archives Distributed Digital Record System (ADDRS). Records will be exported and/or harvested, batched and transferred to the Archives via an automated process over the most appropriate channel.

The records will then be quality assured and stored within the Store Cloud for further processing.

Figure 1: Digital Records – From Agency to ingest within the Archives

Digital Records – From Agency to ingest within the Archives

Archives Distributed Digital Record System

Agency

Manage Agency Record Authority & Exception Handling

Archives

Monitor Agency Record Keeping Notify of Disposal Freeze Government wide Rule Implementation

Agency Record Producing and Management Systems

Agency Business System, Agency EDRMS, ERP and Agency Websites all feed into ADDRS Pop which consists of:

  • Business Functions
  • Business Processes
  • Dictionary
  • Series
  • Document Categories
  • Categorisation Rules
  • Analytics Document Classification
  • Transfer Rules
  • Disposal Freezes
  • Statistics on Agency Holdings
  • Forward Plan for Transfers
  • Web Archiving
  • Format Normalisation
  • File Transfer Cache
  • Checksums

This feeds between the Internal DMZ, NAA Internal DMZ, ADDRS Net and a Submit Information Package to the ADDRS Store.

Ingest Process

Scheduled Scanning of EDRMS & Agency Websites

Change Detection

Item Categorisation

Item Classification

Item Preservation Format

Exception Handling

Transfer Manifest & Schedule

Transfer – Cache and Send - Channel Supports network or physical

Reports flows forward

Transfer Requests flows forward

Metadata flows forward

Items flows forward

Transfer Report flows backward

Governance flows backward

Archive Notifications flows backward

Transfer - Receive & Acknowledge

Scan

Permanent Store

Notify Receipt

From Archives to external clients

Clients will search for the records via a find and display function. The records will be retrieved by a workflow running within the workflow controller. The workflow will retrieve the records, converting them into an appropriate access format.

The retrieved records will be published to a delivery platform most appropriate for the client. For example, a small digital file for consumption over the Internet, would be published to the Local Content Distribution Server and be accessible via a web download; whereas a large video may be pushed to the Content Distribution Network for retrieval by the client using a streaming video player.

Figure 2: Digital Records – From Archives to external clients

Digital Records – From Archives to external clients

Consumer

Retrieve Information Package

Digital Find & Display

Retrieval Request

Digital Workflow Controller

Publish Information Package Request

ADDRS Store Cloud

Ingest

Information Package

Digital Execution Cloud

Publish to Offline Copy Creation

Publish to Local Content Distribution Server

Publish to Content Distribution Network (Public)

The Archives environment

The Archives has been delivering its services online for many years. Currently 99% of collection access occurs online, in digital format. A key priority for the Archives is to build business effectiveness to deliver innovative, responsive and appropriate services that meet the needs of government, the Australian people and other key stakeholders. These online services will continue to be expanded through:

  • Provision of information policy, advice and training to government agencies, to ensure that digital records are created and managed appropriately.
  • Transfer of digital records of permanent value (RNA) to the National Archives in digital format.
  • Digitisation of analogue records already held by the Archives – this includes paper, microform, three dimensional and audiovisual collection.
  • The preservation and management of records in digital format, with a rich metadata structure that allows for enhanced search and discovery.

Scope

The scope of the Archives Digital Business System will support the Archives' business needs by:

  • Providing a scalable, extensible platform for digital business.
  • Providing support for access from multiple channels, including mobile.
  • Enabling the user to search across multiple internal Archives and potentially external agency data sources and secure Cloud environments.
  • Delivering reliable, robust, flexible and sustainable business processes, clearly defining how the business operates, who owns the processes and how this links into the overall operation of the Archives’ business.
  • Having an integrated IT application platform that supports the business processes that are flexible enough to implement rapid change and speed of delivery to meet market and stakeholder demands whilst maintaining data integrity.
  • Providing productivity improvements in processes and systems, including the implementation of proper process and change governance.
  • Increasing the Archives’ ability to manage the business performance end-to-end, utilising collation/collaboration of data from multiple sources to provide meaningful content and related information.

Benefits

Benefits delivered by the Archives Digital Business System over time include:

  • Standardised and automated business processes across the Digital Business value chain.
  • Support for consistent digital access to the Archives' collection via multiple channels (including mobile access); and potentially value-added partners.
  • Common metadata standards that can be built upon and utilised to drive and enhance workflow automation and search.
  • Cost savings via:
    • Increased efficiencies and higher productivity (e.g. more automated processes);
    • Operation on a low cost / high availability platform
    • Better/more reliable management reporting;
    • Lower cost of system ownership (e.g. reduced maintenance/support effort, lower support costs); and
    • Flexibility & adaptability to handle changing business requirements without necessitating development and support of new systems.
  • Provide online access to the records as soon as possible, to clients who are anywhere, at the time the records are required, and in a format and on platforms that meets their requirements.
  • Digital transfer, ingest and preservation continue to meet integrity and authenticity requirements.

High Level Schedule

The project is to be delivered in four iterations. Each of those iterations will deliver increased capability over time, before the entire end-to-end system becomes entirely business as usual.

Iteration 1: Proof of Concept

Design and trial using Archives data from its internal Electronic document and record management system (EDRMS)

Iteration 2: Transfer / Ingest Scalability

Extend the trial to transferring data from selected partner agencies to test scalability & automation of transfer & storage

Iteration 3: Access Scalability

Further extend the trial to ensure all Archives' clients can access digital records in an appropriately controlled environment

Iteration 4: Moving to Full Production

Moving into full production, trialling extraction of agency data directly from the agency and/or gateways

Business as Usual

System in place as business as usual for transferring, preserving, managing and providing access to digital records

Current Progress

June 2014 – July 2015 was dedicated to undertaking a Archives Digital Business System proof of concept stage in order to identify and mitigate the highest levels of risks that were identified in the Architectural Blueprint. This was broken into five projects, each with its own purpose and scope. Each separate project was scoped. The projects were:

  • Project 1: Records Extraction
  • Project 2: Transfer, Ingest and Store Records
  • Project 3: Search and Discovery
  • Project 4: Development and Test Environment
  • Project 5: Metadata Model and Mapping

The proof of concept has used the Archives’ data from its internal TRIM EDRMS as the basis for this work, with the addition of other collection and external data sets to test the Access Examination process as well as Search and Discovery.

July 2015 marked a successful end to the proof of concept stage of the project with all major objectives and milestones being met. In précis, the project team and associated partners within the Archive have demonstrated the viability of Chrysalis by proving it is possible to develop a suite of software to assist record keepers in the digital age by:

  • At the Client Agency: digitally selecting and "packing" RNA records stored in a client agency’s EDRMS and transmitting that package to the National Archives of Australia for management.
  • At the Archives: receiving, "unpacking", storing, preserving and digital records received from client agencies in a secure environment.
  • In the world, online: providing Agencies with secure private access to their transferred records, and providing the Australian public with greatly enhanced discovery and interaction opportunities through a federated and faceted discovery experience.

Lessons learnt from the build of the proof of concept will inform decisions, processes and planning required for the development of the prototype. The end-to-end prototype is scheduled to be completed by June 2016.

Copyright National Archives of Australia 2016