Child pages
  • 2015-10-23 HydraDAM2 at IU Meeting notes
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »




  • Review previous meeting notes
    • corrections and links added - thanks!
  • HPSS meeting (with Kristy Kallback-Rose and Danko Antolovic) - Heidi and Brian
    • next version will solve some of our issues but no date for migration to next version
    • next version has checksums going through entire workflow vs now (no checksum between things going from disk cache to tape)
    • drives HPSS purchased do check as things are being written to tape, but errors aren’t passed up the stack to know it was part of a transfer
    • 3rd party products can do this work but have never been involved in IU's HPSS environment, other places use them
      • listens for SCSI commands going by
    • reader and writer will both be checked in new workflow (both sides of where HPSS gets a file and where it puts that file)
    • regularly scheduled fixity validation (suggested every 2-5 years if it can't be yearly)
    • derivatives - check fixity on these? could bargain with these to have all masters checked if derivatives are not
      • Brian running tests on MDPI data to do fixity checking and see how much bandwidth is required (estimate)
    • HPSS folks will work with us whatever we need
    • what do you do when you find an error? go to original source to get another copy
      • if there are 2 copies in HPSS (IUB and IUPUI), go to other copy, but requires manual intervention so file on IUB tape is marked unavailable
    • also assuming another copy is stored in non-HPSS system (DPN) - need for fixity checking goes down at that point, assuming fixity checking occurs regularly in non-HPSS system
    • migrating into new formats “relatively often"
    • network is real problem - storage silos everywhere but getting it from one place to another is a problem
  • Additional discussion
    • HydraConnect meeting on digital preservation - Jon brought up idea of using Fedora/Hydra functions to manage checksum fixity checking; folks said that wasn’t doing preservation because that is trusting a different system than where preserved items are located (same can be said for HPSS environment to certain degree)
      • can’t do all fixity checking in place - takes too much bandwidth, have to farm that out
    • Survey of fixity checking practices - NDSA storage survey 2012 
    • Fixity checking on regular schedule to start but hopefully don’t need as much regular checking and can do random sampling or as files are pulled for use in future
    • At this point, have to be good HPSS citizens and not lock everyone out; also need to make sure SDA meets our needs and advocate for more funding if necessary
  • Next steps / plan moving forward
    • January - Dark Avalon can take in HPSS data; will need a download master feature/tool of some kind
    • long term - HydraDAM2 comes in between HPSS and Dark Avalon and will also have that master download feature/tool
    • HydraDAM2 - manual fixity check calls, might need to be requests with reports you can get at later time
      • ingest to HydraDAM2 probably needs to take static XML and convert to RDF properties - need data model
    • Brian can get bag examples to Julie for documenting bag examples on wiki
    • what metadata are we mapping into RDF for HydraDAM2?
    • for items with IUCAT records - use catalog key to put into Avalon; otherwise POD info in MODS in bag has to be called up
    • USE CASE [not in Func Req - HDM-234]: HydraDAM2 - might need descriptive metadata versioning, ability to compare current IUCAT record with preserved MODS, but low priority
    • on-demand fixity checking based on events - are there standards for that? batch our fixity checking requests (looking on a single tape for certain criteria - have to do things somehow based on physical underlying system); need to gather up requests as they occur; if there is checksum question on master file, it needs to be taken out of circulation
      • AHEYM 5th video file example - truncated on upload to HPSS, that changed the file and checksum didn’t match in IUB; no validation after upload; different than process we do now (earlier use of HPSS for Sound Directions)
    • when user goes through HydraDAM2 to retrieve master file, there should be fixity checking happening at that point
      • would need fixity check message stored somewhere
      • call to download master file will grab from HPSS and place in dropbox, fixity check happens then automatically through HPSS and transfer fails if fixity fails
      • would be up to the end user when downloading from dropbox to check the checksum at that point
    • see if there are additional use cases for HydraDAM2; create stories for them in backlog
    • see if any of those stories need to be reflected in Dark Avalon (separate group working on Dark Avalon now)
    • review HydraDAM2 functional requirements page and see if these use cases are included there already
    • might need meeting again this year depending on how HydraDAM2 work and Dark Avalon work goes
    • Dark Avalon work being worked into current Avalon development in terms of more automated deposit into Avalon; will need a group for actual process of getting MDPI items into Dark Avalon

Action items

  • MDPI bag examples for audio formats to create wiki documentation - Brian and Julie
  • Consider metadata to map into HydraDAM2 RDF from example MDPI bags - Julie
  • Review HydraDAM2 functional requirements to see if use cases brought up in these meetings already included there - Julie [DONE]
  • Add use cases not already covered by functional requirements to HydraDAM2 backlog - Julie
  • No labels