Child pages
  • Preservation Policies

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Next steps for moving towards preservation

  1. Start documenting policy decisions. This page, its siblings, and its children are good starting points.
    1. As we make decisions, determine what documents we should sort the decisions into, and what person/group should manage each document.
  2. Determine who should be invited to join a "repository policy group". This group should meet regularly to ensure that policies meet the needs of the repository's users, and ensure that policies are being followed.

RLG "Trusted Digital Repository" checklist

The most important document related to preservation is RLG's Audit Checklist for the Certification of Trusted Digital Repositories. While this checklist is a good place to start working on our preservation system, we won't treat it as a mandate.

...

  • 3.6 Repository commits to define, collect, track, and provide, on demand, its information
    integrity measurements.
  • B1.1 Repository identifies properties it will preserve for each class of digital object.
  • B1.5 Repository obtains sufficient physical control over the digital objects to preserve
    them. (The description of this is confusing, encompassing many different aspects of repository functionality.)
  • B2.1 Repository has an identifiable, written definition for each AIP or class of information
    preserved by the repository.
  • B2.4. Repository has and uses a naming convention that can be shown to generate visible,
    unique identifiers for all AIPs.
  • B3.7 Repository actively monitors AIP integrity. (This includes maintaining checksums in a location separate from media files, and maintaining logs of the integrity checks.)
  • B4.2 Repository can demonstrate that referential integrity is created between all AIPs and
    associated descriptive information.
  • B5.2 Repository logs all access management failures, and staff review inappropriate
    "access denial" incidents. (This should probably apply both to Fedora and to HPSS.)
  • D1.1 Repository functions on well-supported operating systems and other core
    infrastructural software. (Not sure if HPSS will ever meet this...)
  • D1.2 Repository ensures that all platforms have a backup function sufficient for the
    repository's services and for the data held, e.g., metadata associated with access controls,
    repository main content, etc.
  • D1.4 Repository has mechanisms in place to insure any/multiple copies of digital objects
    are synchronized.
  • D1.5 Repository has effective mechanisms to detect data corruption or loss. For example, if the policy were the repository could not lose more than 0.001% of the collection per year, then the repository would need to confirm media integrity at least every 72 days to achieve an average time-to-recover of 36 days or about one tenth of a year.
  • D1.6 Repository reports to its administration all incidents of data corruption or loss, and
    steps taken to repair/replace corrupt or lost data.
  • D1.7 Repository has defined processes for storage media migration.
  • D1.8 Repository has a documented change management process that identifies changes to
    critical processes.
  • D1.9 Repository has a process for testing the effect of critical changes to the system.
  • D1.10 Repository has a process to stay current with the latest operating system security
    fixes
  • D3.4 Repository has written disaster preparedness and recovery plan(s), including at least
    one off-site copy of all deposited data. Multiple off-sites copies are expected of most repositories, but others may be able to justify not providing these.
  • D3.5 Repository tests disaster plans regularly.
  • D3.6 Repository has defined processes for service continuity and disaster recovery.

Possible threats to the preservation repository

  • Failure to access HPSS
  • Failure of HPSS tapes
    • Simultaneous failure in both locations is possible (nuclear war)
  • Fire, tornado, or other catastrophic failure in WCC machine room
  • Disgruntled employee attempts to delete everything from HPSS
  • Disgruntled employee attempts to delelte random sets of items from HPSS

Other preservation-related items

A paper on using Fedora as a preservation system

...

Another paper on preservation issues.

Building an integrity checker

If all goes well, an integrity checker will never report a problem. But a non-existent or non-running integrity checker would exhibit the same behavior. So we must set up tests that ensure the integrity checker is regularly exercised. One simple way would be to put an object in the repository and purposefully corrupt it, then measure how long it takes for the integrity checker to notice.

...

  • a basic ping service that checks whether a file appears to be in place (e.g., HPSS reports that the file is there and is the correct size, but we don't take the time to download it)
  • a full checksum service that downloads a copy of the file and verifies its checksum

Other thoughts

  • We need a system to provide periodic reports on the existence of orphaned and incomplete objects, so we can keep the contents of the repository "clean".
  • Eventually, we would like to set up a "digital preservation advisory board" to proof our decisions. This page and its children will eventually be maintained by the board.

Open questions

  1. Can we eventually store lots of small files (page images) directly in HPSS via the filesystem interface without aggregation, or will it simply take too long to retrieve these files?
  2. Should we store copies of our metadata in HPSS, or are the regular server backups good enough for this?
  3. How do we manage preservation packages for materials that will be accessed through Variations? We don't want to store duplicate copies of the derivative files, because they are quite large. Perhaps Fedora can store these as Redirect datastreams, and just keep the appropriate metadata in the actual repository directories.