Release 2.3.2 This dot release includes a fix for a change that Docker implemented to their registry RESTful API which broke compatibility with Singularity (among several other low minor fixes).

What happened?

Where does Docker image metadata come from? The Docker Registry serves metadata about images via manifests, where each image has a manifest that tells Singularity important information like environment, labels, and entrypoints and running command. Importantly, the image manifest serves the actual layers that should be obtained to create the image. The manifests come in different flavors, or schema versions. Version 1 serves the primary load of metadata (labels, environment) while the first release of version 2 served layers, and had the addition of size. This addition made it possible to pull an image with Singularity and calculate the size on the fly for images generated with support for this manifest.

Singularity had an old default to retrieve the version 2 (done by way of a header asking for it), and ask it for layers. If the remote registry could only offer version 1, we were still able to obtain a list of layers (under a different key, fsLayers instead of layers) and metadata via the older (schema 1) manifest. However, with the update, the API version 2 schema can now return a list of manifests. This meant that when we checked the response for the keys layers or fslayers, they wouldn’t be found, because we needed to look under manifests, and then do a second call to retrieve the manifest of interest based on a system architecture and OS. This of course broke most import, shell, pull, exec, because all of these commands require retrieving the layers.

The Patch

A super quick fix would have been to fall back to the version 1 manifest, always, but then we would lose the automatic calculation of size, which is important to many users. The “better” fix is obvious - the code needs to:

  • check for a manifests key
  • if found, select a default manifest to use
  • retrieve it, and continue!
  • if not found, fall back to version 1

This means that we’ve added environment variables SINGULARITY_DOCKER_OS and SINGULARITY_DOCKER_ARCHITECTURE to specify this choice, with defaults linux and amd64. This is a pretty exciting change, because it means down the line you (as a user!) can specify the specifics of the layers you want returned, given an image that has this metadata available. We can see the fix running as follows:

DEBUG 
*** STARTING DOCKER IMPORT PYTHON  ****

This is the initialization of the Docker client, it’s parsing the image name provided

DEBUG Starting Docker IMPORT, includes env, runscript, and metadata.
VERBOSE Docker image: ubuntu:14.04
VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
DEBUG Headers found: Content-Type,Accept
VERBOSE Registry: index.docker.io
VERBOSE Namespace: library
VERBOSE Repo Name: ubuntu
VERBOSE Repo Tag: 14.04
VERBOSE Version: None

The initial poke to Docker hub asks for tags, to determine if we need some kind of token

VERBOSE Obtaining tags: https://index.docker.io/v2/library/ubuntu/tags/list
DEBUG GET https://index.docker.io/v2/library/ubuntu/tags/list

401 means that we do, and with the response the API sends back the scope and other details we need to make to request it

DEBUG Http Error with code 401

Here we are requesting it

DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/ubuntu:pull
DEBUG Headers found: Content-Type,Authorization,Accept

And here is the new bit. Instead of blindly doing one call, we have a function to update two versions of the manifest - the older (with metadata) and some version of the newer (with layers and size) with a fallback to use the version 1

DEBUG Updating manifests.

# Here is version 2+
DEBUG MANIFEST (Primary): not found, making initial call.
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest
DEBUG GET https://index.docker.io/v2/library/busybox/manifests/latest

# Here is version 1 (Metadata)
DEBUG MANIFEST (Metadata): not found, making initial call.
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest
DEBUG GET https://index.docker.io/v2/library/busybox/manifests/latest

Notice that the two calls are the same - that’s because the headers are actually different, to request different ones.

And here is the (new) indication that we found a list

DEBUG Image manifest version 2.2 list found.

Here is the default architecture / os

DEBUG Obtaining architecture: amd64, OS: linux

And the specific call to get it

VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/sha256:030fcb92e1487b18c974784dcc110a93147c9fc402188370fbfd17efabffc6af
DEBUG GET https://index.docker.io/v2/library/busybox/manifests/sha256:030fcb92e1487b18c974784dcc110a93147c9fc402188370fbfd17efabffc6af

With this fix - we can again use pull/import, etc, and also have a working singularity pull docker://busybox that estimates the size automatically.

Please report any additional bugs to our issues board.