For VOV-902, I used ffmpeg to produce ten 20GB chunks from the Gandhi movie using the same codec and ingested them all at the same time. The system queued them up quickly with no apparent slowness. I could also start ingesting additional files with no problems. However, Matterhorn only processes 4 files at a time by default, which means subsequent files, no matter how small, will have to wait. We can:
- Increase the number of max concurrent jobs in Matterhorn (currently 4)
- For batch items, ingest a number of files at a time, leaving room for web items (I assume users expect batch to be time-consuming and web upload to be immediate)
For now the number of Matterhorn workers on Mallorn has been increased to 10, which does slow the server down quite a bit if we were ingesting 10 large files at once.