How to Upload Large Media Files Via the Digimarc Barcode Services API
09 April 2021
Using the Web API to upload >5M files
This article describes how to upload a large media file (> 5MB) via the Digimarc Barcode Services API. The process is generally the same for audio and image files, differing only in some details. The steps below illustrate the process.
NOTE: The HTTP headers for all the methods discussed here must include the Authorization header:
Authorization: <encodedCredentials> |
where <encodedCredentials> is your encoded Application Name and API Key. For more details on authorization, see the Overview of the Digimarc Barcode Services API [pdf].
Also, since a single user may have access to several accounts, all methods must include the query parameter account=<account> to identify the account of interest.
The following steps demonstrate uploading a large file via the Web API:
- Using the tool of your choice, split your file into chunks smaller than 5MB.For tracking individual chunks, the API numbers them starting at 0. So if a file is split into N chunks, the chunks are numbered 0 through N-1. (See the bottom of this page for a brief discussion of tools for splitting large files.)
- Upload the first (0th) chunk using the file
upload method.
- Headers (plus
Authorization):
Content-Type: <mediaType> Content-Length: <lengthOfChunk(0)> |
where:
<mediaType> indicates the type of media.<lengthOfChunk(0)> is the length in bytes of the first (0th) chunk of the file.
Supported media types include image/tiff, image/jpeg, audio/wav, audio/mp3, audio/mpeg, audio/m4a, and audio/mp4. - URL:
POST /v2/file/upload?type=chunk&name=<filename>&account=<account> |
where:
<filename> is the name of the file being created (not the file name of the chunk).<account> identifies the account.
- Status code on success:
- Response body on success:
where:
<fileId> identifies the file being created from the chunks.
The <fileId> should be saved for use in the following methods.
- Use the file
chunk method to upload each remaining chunk K, where K ranges from 1 through N-1 (N = total number of chunks).
- Headers (plus
Authorization):
Content-Type: application/octet-stream Content-Length: <lengthOfChunk(K)> |
where:
<lengthOfChunk(K)> is the length in bytes of chunk number K.
- URL:
POST /v2/file/<fileId>/chunk/<K>?account=<account> |
where:
<fileId> is the file identifier returned from the upload method.<K> is 1 through N-1.<account> identifies the account.
- Status code on success:
- It is optional but recommended that you enumerate the uploaded chunks using the file
chunks method. Enumerating the chunks enables you to double-check that all chunks are successfully uploaded.
- Headers (plus
Authorization): none - URL:
GET /v2/file/<fileId>/chunks?account=<account> |
where:
<fileId> is the file identifier used in the previous steps.<account> identifies the account.
- Status code on success:
- Response body on success:
{ NumChunks: <N> Chunks: [ { Id: 0 ContentLength: <lengthOfChunk(0)> }, { Id: 1 ContentLength: <lengthOfChunk(1)> }, ... { Id: <N-1> ContentLength: <lengthOfChunk(N-1)> } ] } |
where:
<N> is the number of chunks.<N-1> is the enumerator of the last chunk.<lengthOfChunk(0)> is the length in bytes of chunk 0.<lengthOfChunk(1)> is the length in bytes of chunk 1.- ...
<lengthOfChunk(N-1)> is the length in bytes of chunk N-1.
NOTE: It is possible that the sum of the lengths of the enumerated chunks is different from the length of the original file on your system. You should use the sum of the enumerated chunks as the total file length in the next step. - Once all chunks have been successfully uploaded, use the
commit method to combine the chunks and create the file.
- Headers (plus
Authorization):
Content-Type: application/json |
- Request body:
{ "Name": "<filename>", "ContentType": "<mediaType>", "ContentLength": <totalLength>, "Chunks": [0,1, ... <N-1>] } |
where:
<totalLength> is the sum of the lengths of the enumerated chunks in the previous step.<mediaType> indicates the type of media.<filename> is the name of the file being created.<N-1> is the enumerator of the last chunk.
- URL:
PUT /v2/file/<fileId>/chunks/commit?account=<account> |
where:
<fileId> is the file identifier used in the previous steps.<account> identifies the account.
- Status code on success:
At this point, the separate chunks have been recombined into a complete file and you can proceed with further operations on the file, such as embedding.
Tools for splitting files
We have found the following tools to work effectively in splitting large media files into smaller chunks for uploading. NOTE: Digimarc does not endorse any specific tool for splitting files. These tools are mentioned here solely for your convenience.
- On the Mac, you can use the
split command in a Terminal window. For example, assuming your working directory contains an 11MB file named imageFile.tif, the following command splits imageFile.tif into 4MB chunks:
split -b 4m imageFile.tif imageFile.tif_ |
The -b option specifies the size of the chunks to be created. The second argument, imageFile.tif_, is the leading portion of the names of the generated chunks, where each chunk is distinguished by a unique appended string. See the man page for the split command for more details on its use. The result in this case is three files:
imageFile.tif_aa imageFile.tif_ab imageFile.tif_ac |
Each of the first two files is 4MB (4,194,304 bytes) in length; the third is ~3MB. You must determine the exact length in bytes of each chunk before uploading. - For Windows, there are several free file-splitting apps available online. We have tested HJSplit and verified that it works on both TIFF and WAV files. Given an 11MB file named
imageFile.tif, HJSplit produces three files as follows:
imageFile.tif.001 imageFile.tif.002 imageFile.tif.003 |
Each of the first two files is 4MB (4,194,304 bytes) in length; the third file is ~3MB. You must determine the exact length in bytes of each chunk before uploading.