Skip to end of metadata
Go to start of metadata

For the OR 2016 Demo, the development team was able to demonstrate the proof of concept of an asynchronous storage proxy and storage proxy client. This was done by building the storage proxy functionality in a simple Rails application, and then building a thin client library within the HydraDAM application to interact with it.

From the client's point of view, the process is simple:

  1. Use the API to get status of a file in the cache
  2. Use the API to create a job to stage a file or run a fixity check on a file
  3. Keep using the API to get the status of the file
  4. Use the API to get the desired details (i.e download url or fixity info)
  5. Optionally, use the API to "unstage" and reset a file's status

This "mock" storage proxy is still deployed on the HydraDAM shared server at: http://helium.dlib.indiana.edu:3001/storage_api . The available interactions and their responses are documented below.

Get status and detail for a named file in the cache 

$ curl http://helium.dlib.indiana.edu:3001/storage_api/caches/SDADisk/cache_files/MDPI_40000002054841_01_access.mp4
{"id":47,"name":"MDPI_40000002054841_01_access.mp4","status":null,"url":null,"cache_id":1,"created_at":"2016-06-08T14:51:43.305Z",
  "updated_at":"2016-06-08T14:51:43.305Z","checksum":"","fixity_available":null,
  "fixity_date":null,"fixity_type":null}

 

 List all files and their status in the cache called 'SDADisk'

$ curl http://helium.dlib.indiana.edu:3001/storage_api/caches/SDADisk/cache_files
{"id":47,"name":"MDPI_40000002054841_01_access.mp4","status":null,"url":null,"cache_id":1,"created_at":"2016-06-08T14:51:43.305Z",
  "updated_at":"2016-06-08T14:51:43.305Z","checksum":"","fixity_available":null,
  "fixity_date":null,"fixity_type":null}
(list of all available files...)
{"id":60,"name":"MDPI_40000002269555_01_access.mp4","status":null,"url":null,"cache_id":1,"created_at":"2016-06-08T14:51:43.443Z",
  "updated_at":"2016-06-08T14:51:43.443Z","checksum":"","fixity_available":null,
  "fixity_date":null,"fixity_type":null} 

 

Post a job to mimic staging of file in the store to the cache, then view changed status

$ curl -X POST http://helium.dlib.indiana.edu:3001/storage_api/jobs/SDADisk/MDPI_40000002054841_01_access.mp4 -d "type=stage"
{"cache_name":"SDADisk","cache_file_name":"MDPI_40000002054841_01_access.mp4","type":"stage"}
 
# Notice 'status' field changes to "staging"
$ curl http://helium.dlib.indiana.edu:3001/storage_api/caches/SDADisk/cache_files/MDPI_40000002054841_01_access.mp4
{"id":47,"name":"MDPI_40000002054841_01_access.mp4","status":"staging","url":null,"cache_id":1,"created_at":"2016-06-08T14:51:43.305Z",
  "updated_at":"2016-06-08T14:51:43.305Z","checksum":"","fixity_available":null,
  "fixity_date":null,"fixity_type":null}

# Notice 'status' field changed to "staged and 'url' contains a location to a file"
$ curl http://helium.dlib.indiana.edu:3001/storage_api/caches/SDADisk/cache_files/MDPI_40000002054841_01_access.mp4
{"id":47,"name":"MDPI_40000002054841_01_access.mp4","status":"staged",
  "url":"http://hydradam.dlib.indiana.edu/cache/SDADisk/MDPI_40000002054841_01_access.mp4","cache_id":1,
  "created_at":"2016-06-08T14:51:43.305Z","updated_at":"2016-06-08T14:51:43.305Z",
  "checksum":"","fixity_available":null,"fixity_date":null,"fixity_type":null}


Post a job to mimic unstaging of file from the cache, then view changed status

$ curl -X POST http://helium.dlib.indiana.edu:3001/storage_api/jobs/SDADisk/MDPI_40000002054841_01_access.mp4 -d "type=unstage"
{"cache_name":"SDADisk","cache_file_name":"MDPI_40000002054841_01_access.mp4","type":"unstage"}

# Notice 'status' field changes back to "null"
$ curl http://helium.dlib.indiana.edu:3001/storage_api/caches/SDADisk/cache_files/MDPI_40000002054841_01_access.mp4
{"id":47,"name":"MDPI_40000002054841_01_access.mp4","status":null,"url":null,"cache_id":1,"created_at":"2016-06-08T14:51:43.305Z",
  "updated_at":"2016-06-08T14:51:43.305Z","checksum":"","fixity_available":null,
  "fixity_date":null,"fixity_type":null}

 

Post a job to mimic fixity check of file from the cache, then view changed status

$ curl -X POST http://helium.dlib.indiana.edu:3001/storage_api/jobs/SDADisk/MDPI_40000002054841_01_access.mp4 -d "type=fixity"
{"cache_name":"SDADisk","cache_file_name":"MDPI_40000002054841_01_access.mp4","type":"fixity"}

# Notice 'status' field changes to "staging" because it has to stage in order to run a fixity check
$ curl http://helium.dlib.indiana.edu:3001/storage_api/caches/SDADisk/cache_files/MDPI_40000002054841_01_access.mp4
{"id":47,"name":"MDPI_40000002054841_01_access.mp4","status":"staging","url":null,"cache_id":1,"created_at":"2016-06-08T14:51:43.305Z",
  "updated_at":"2016-06-08T14:51:43.305Z","checksum":"","fixity_available":false,
  "fixity_date":null,"fixity_type":null}

# Notice 'status' field changes to "calculating...", but it also contains a value in 'url' since it was staged first
$ curl http://helium.dlib.indiana.edu:3001/storage_api/caches/SDADisk/cache_files/MDPI_40000002054841_01_access.mp4
{"id":47,"name":"MDPI_40000002054841_01_access.mp4","status":"calculating checksum",
  "url":"http://hydradam.dlib.indiana.edu/cache/SDADisk/MDPI_40000002054841_01_access.mp4","cache_id":1,
  "created_at":"2016-06-08T14:51:43.305Z","updated_at":"2016-06-08T14:51:43.305Z","checksum":"",
  "fixity_available":false,"fixity_date":null,"fixity_type":null}

# Notice 'status' is now "staged", and 'fixity_available' is true, along with a value in other fixity fields
$ curl http://helium.dlib.indiana.edu:3001/storage_api/caches/SDADisk/cache_files/MDPI_40000002054841_01_access.mp4
{"id":47,"name":"MDPI_40000002054841_01_access.mp4","status":"staged",
  "url":"http://hydradam.dlib.indiana.edu/cache/SDADisk/MDPI_40000002054841_01_access.mp4","cache_id":1,
  "created_at":"2016-06-08T14:51:43.305Z","updated_at":"2016-06-08T14:51:43.305Z",
  "checksum":"44e5349c65f6df1fd0f87d69e8e2b9dd","fixity_available":true,"fixity_date":"2016-08-19T20:21:17.129Z","fixity_type":"md5"}

# Optionally, an additional parameter can be passed to specify a fixity type to be calculated (MD5 is default if none given)
$ curl -X POST http://localhost:3001/storage_api/jobs/SDADisk/MDPI_40000002054841_01_access.mp4 -d "type=fixity" -d "fixity_type=sha1" 



  • No labels