Tooltool

In most cases, you will use tooltool with the [tooltool client](https://github.com/mozilla/build-tooltool/blob/master/tooltool.py). Just give it the right base URL and manifest and you should be on your way.

If you need more control, the API details are below.

Periodic Tasks

When a user uploads a file to tooltool, the file is transferred directly to Amazon S3. This occurs via a "signed URL" which temporarily grants the user write access to the file.

There is a periodic task named relengapi.blueprints.tooltool.grooming.check_pending_uploads which runs every 10 minutes. It verifies any newly uploaded files and records their presence for subsequent download. Uploads can only be verified after the signed URL has expired -- otherwise they could be changed after the fact!

Separately from verifying uploads, a task named relengapi.blueprints.tooltool.grooming.replicate runs every hour to replicate content between AWS regions. Any files which are not in at least one, but not all configured AWS regions are copied to the remaining regions. Once the copies are complete, they become available to clients for download.

Thus there is a short period after a file is uploaded where it is available in zero, and then only one, region.

Types

REST type File

A complex type that represents a file.

In the particular case of protocol accepting form encoded data as input, File can be loaded from a form file field.

Keys:
  • filename (unicode) -- The file name

  • contenttype (unicode) -- Mime type of the content

  • content (binary) -- File content

REST type UploadBatch

An upload batch describes a collection of related files that are uploaded together -- similar to a version-control commit. The message and files list must be non-empty.

Keys:
  • id (int) -- Identifier for this batch

  • uploaded (datetime) -- The date and time when this upload occurred. This will be added by the server and need not be specified when making a new upload.

  • author (unicode) -- The author (uploader) of the batch. Do not include this when submitting a batch for upload; it will be filled in based on the request authentication.

  • message (unicode) -- The message for the batch. Format this like a version-control message.

  • files ({"...": File}) -- The collection of files in this batch, keyed by filename. Note that filenames containing path separators (\ and /) will be rejected the tooltool client.

Endpoints

endpoint GET /tooltool/file
Parameters:
  • q -- unicode

Response Body:

[File]

Search for files matching the query q. The query matches against prefixes of hashes (at least 8 characters) or against filenames.

endpoint PATCH /tooltool/file/sha512/<digest>
Parameters:
  • digest -- unicode

Request Body:

[{"...": unicode}]

Response Body:

File

Make administrative changes to an existing file. The body is a list of changes to apply, each represented by a JSON object.

The object {"op": "delete_instances"} will cause all instances of the file to be deleted. The file record itself will not be deleted, as it is still a part of one or more upload batches, but until and unless someone uploads a new copy, the content will not be available for download.

If the change has op "set_visibility", then the file's visibility will be set to the value given by the change's visibility attribute. For example, {"op": "set_visibility", "visibility": "internal"} will mark a file as "internal" after someone has accidentally uploaded it with public visibility.

The returned File instance contains an instances attribute showing any changes.

endpoint GET /tooltool/file/sha512/<digest>
Parameters:
  • digest -- unicode

Response Body:

File

Get a single file, by its digest. Filenames are associated with upload batches, not directly with files, so use GET /uploads to find files by filename.

The returned File instance contains an instances attribute showing the regions in which the file exists.

endpoint GET /tooltool/sha512/<digest>
Parameters:
  • digest -- unicode

  • region -- unicode - optional

Fetch a link to the file with the given sha512 digest. The response is a 302 redirect to a signed download URL.

The query argument region=us-west-1 indicates a preference for a URL in that region, although if the file is not available in tht region then a URL from another region may be returned.

endpoint GET /tooltool/upload
Parameters:
  • q -- unicode

Response Body:

[UploadBatch]

Search upload batches. The required query parameter q can match a substring of an author's email or a batch message.

endpoint POST /tooltool/upload
Parameters:
  • region -- unicode - optional

Request Body:

UploadBatch - optional

Response Body:

UploadBatch

Create a new upload batch. The response object will contain a put_url for each file which needs to be uploaded -- which may not be all! The caller is then responsible for uploading to those URLs. The resulting signed URLs are valid for one hour, so uploads should begin within that timeframe. Consider using Amazon's MD5-verification capabilities to ensure that the uploaded files are transferred correctly, although the tooltool server will verify the integrity anyway. The upload must have the header Content-Type: application/octet-stream`.

The query argument region=us-west-1 indicates a preference for URLs in that region, although if the region is not available then URLs in other regions may be returned.

The returned URLs are only valid for 60 seconds, so all upload requests must begin within that timeframe. Clients should therefore perform all uploads in parallel, rather than sequentially. This limitation is in place to prevent malicious modification of files after they have been verified.

endpoint GET /tooltool/upload/<int:id>
Parameters:
  • id -- int

Response Body:

UploadBatch

Get a specific upload batch by id.

endpoint GET /tooltool/upload/complete/sha512/<digest>
Parameters:
  • digest -- unicode

Response Body:

unicode

Signal that a file has been uploaded and the server should begin validating it. This is merely an optimization: the server also polls occasionally for uploads and validates them when they appear.

Uploads cannot be safely validated until the upload URL has expired, which occurs a short time after the URL is generated (currently 60 seconds but subject to change).

If the upload URL has expired, then the response is an HTTP 202 indicating that the signal has been accepted. If the URL has not expired, then the response is an HTTP 409, and the X-Retry-After header gives a time, in seconds, that the client should wait before trying again.