This documentation is for Release 3.x. For the Release 1 version of this page, see v.43. For the Release 2 version of this page, see v.71.

Introduction

Avalon's batch ingest feature provides a method of building one or more media items at a time from uploaded content and metadata outside the user interface. A batch ingest is started by uploading an ingest package consisting of one manifest file and zero or more content files to the Avalon dropbox. For your convenience there is a demo ingest package available to download and import into test systems. Follow the instructions below to ensure a successful batch upload.

Ingest Packages

An ingest package is the combination of content and metadata that make up a single batch.

Package Layout

When a new collection is created, Avalon creates a subdirectory with the name of that collection (substituting underscores for any blanks), beneath the Avalon dropbox directory. The package (manifest file and associated content files) must be uploaded to that collection-named subdirectory or in a subdirectory beneath it. All items included in a single ingest package will be uploaded to the same collection. The following is a very simple package that has been uploaded:

Manifest File Format

The manifest file is a spreadsheet (xls, xlsx, csv, or ods) containing the metadata for the items to be created, as well as the names of the content files that make up each item. In this case, the manifest file is named batch_manifest.xlsx. See batch_manifest_template_R3.xlsx for an Excel example file. Required fields are in bold. Note: Neither the spreadsheet filename nor any folder/directory names above it can have blanks in them–substitute underscores.

 ABCDEFG
1Michael's First Test Batchmichael.klein@northwestern.edu     
2Main TitleCreatorDate IssuedFileLabelFileLabel
3Test item 1Klein, Michael B.2012content/file_1.mp3Part 1content/file_2.mp4Part 2
4Test item 2Northwestern1951content/file_3.mp4   

Row 1, Column A contains a reference name for the batch. This is mostly for your reference so we recommend naming the batch file according to what will help you remember the contents.

Row 1, Column B contains the submitter's email address (or username, depending on how your system is set up) to be used for notifications and exceptions. The submitter's email or user name must be listed as a manager, editor, or depositor for the collection in which this batch is deposited in the Avalon dropbox.

Row 2 specifies the names of the metadata fields supplied in the following rows. Main TitleCreator, Date Issued, and File are required. These fields are shown in bold in the Excel example file. Each subsequent row represents a single media item to be created. Metadata values are specified first, followed by a list of content files to be attached to each item. Note: Make sure none of the field names in row 2 have leading or trailing blanks, or the field names will not be recognized by Avalon and will report an error.

Content files listed in the manifest file must have the correct path noted for where those files are located in the Avalon dropbox, relative to the manifest file. Additionally, all content files must include a file extension. If necessary, include any directories or subdirectories (note the paths listed in columns D and F in the above example).

Multivalued fields are specified by multiple columns with the same header, e.g. Topical Subject in the following example:

 ABCDEF
1Michael's Second Test Batchmichael.klein@northwestern.edu    
2Main TitleCreatorDate IssuedTopical SubjectTopical SubjectFile
3Nachos: A MemoirKlein, Michael B.2012-12-22MeatCheesecontent/tasty_tasty_nachos.mp4

Supported Field Names (required fields in bold)

In addition to the descriptive fields, there are five supported operational fields:

Batch Processing Notes

Each batch will generate 2 emails to the user listed at the top of the manifest.

Once Avalon detects the presence of an unprocessed manifest file, it will first verify that the metadata columns are recognizable, that the required columns are present and have values in them, and that the package is complete (i.e., all content files specified in the manifest are present and not open by any other processes) before attempting to ingest.

If the package is incomplete or in error, it will not be processed and an error file will be generated in the same directory as the manifest file (e.g., batch_manifest.xlsx.error). The error file will contain details of what was missing, and will email the same information to the user specified in the manifest.

To re-run a successfully completed batch, remove the *.processed file from the batch directory (e.g., batch_manifest.xlsx.processed).