Wrangling the Different Docker APIs
Join the DZone community and get the full member experience.
Join For Free[This article was written by Alex Harford.]
Docker APIs are a convenient way for your systems to talk to Docker infrastructure. But sometimes there are challenges associated with them. I've outlined in this blog the steps you need to take and the items you need to look out for when working with Docker APIs.
Initial Docker Setup
Ensure you have the latest Docker client installed. It should be v1.6 or newer.
[alexh:~/work] docker pull ubuntu
latest: Pulling from ubuntu
428b411c28f0: Pull complete
435050075b3f: Pull complete
9fd3c8c9af32: Pull complete
6d4946999d4f: Already exists
ubuntu:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:45e42b43f2ff4850dcf52960ee89c21cda79ec657302d36faaaa07d880215dd9
Status: Downloaded newer image for ubuntu:latest
[alexh:~/work] docker run -ti ubuntu /bin/bash
root@1092e8ca2ead:/# ps
PID TTY TIME CMD
1 ? 00:00:00 bash
14 ? 00:00:00 ps
root@1092e8ca2ead:/# exit
exitDaemons, Registries, Hubs
The Docker registry is used to host docker images for download. In the most simple case, it can be a process serving static images. This would be a read-only registry supporting GET operations only. If you need something more complex, you need to use a Docker registry web service. You can [a target="_blank" href="http://www.activestate.com/blog/2014/01/deploying-your-own-private-docker-registry"]run your own private Docker registry or use the public official Docker Hub. The Docker Hub contains a Docker registry, but also includes other features, like user authentication. In our examples, we will run an unauthenticated Docker registry.
Setup
If you are using standard Docker images, most people will pull from the Docker Hub, which is a publically accessible Docker registry. However, a more complicated service may be talking to private Docker registries running different versions of the API. Let’s assemble a test environment with both versions of the docker registry API so we can see the different ways you can access it.
First, pull down two versions of the docker registry from the Docker Hub:
docker pull registry:0.9.1 0.9.1: Pulling from registry e9e06b06e14c: Pull complete a82efea989f9: Pull complete 37bea4ee0c81: Pull complete 07f8e8c5e660: Pull complete 1f4ab7282e19: Pull complete 3c27027cdae8: Pull complete 7e0e5314436e: Pull complete 2696504d3685: Pull complete 012772dbb1c6: Pull complete e24d9fce1d00: Pull complete fd2726a79da8: Pull complete bffc32d7113a: Pull complete 0cd49aa0e23c: Pull complete 4e698fa80441: Already exists registry:0.9.1: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Digest: sha256:98937757728eecbd72c9276bf711260aa29896f15217ce05be0562287e73232d Status: Downloaded newer image for registry:0.9.1 [alexh:~/work] docker pull registry:2.0.1 2.0.1: Pulling from registry 39bb80489af7: Pull complete df2a0347c9d0: Pull complete 7a3871ba15f8: Pull complete a2703ed272d7: Pull complete 68769176e114: Pull complete ab2ab59d7d1b: Pull complete 882ecee9f360: Pull complete 40de65f8e79f: Pull complete 0c4f9c7d798f: Pull complete ca29675fe853: Pull complete 89d10e9463e5: Pull complete 1a5aa415e484: Pull complete 3ea7a9e93b04: Pull complete 769d811a57fd: Pull complete ae8a4a3af1aa: Pull complete 85cc9a791bb5: Pull complete 9cd2c8646022: Pull complete 048c32c549b9: Pull complete cbbbda28c189: Pull complete 2602c005e534: Pull complete 136beb445cfa: Pull complete 0c5e5ef1d7da: Already exists registry:2.0.1: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Digest: sha256:0cd177d687589aff586aa2c66c64d1c25657b8d09cff9e1492192f496e7786c3 Status: Downloaded newer image for registry:2.0.1
The next step is to start them. We will start the v1 registry on port 5000, and the v2 registry on port 6000. The v1 registry occasionally fails when starting due to a lock file race condition, so tell it to restart if necessary.
[alexh:~/work] docker run -p 5000:5000 -d --restart=on-failure:3 registry:0.9.1 896c651b9bfa9780b14e3710d20428baab8497c30b9bc89946b192e1d1c145aa [alexh:~/work] docker run -p 6000:5000 -d registry:2.0.1 e09d4204921c732879ee9b7544cd40a25275e0d1f1702cacd954412cfd586ffb [alexh:~/work] docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e09d4204921c registry:2.0.1 "registry cmd/regist 4 seconds ago Up 3 seconds 0.0.0.0:6000->5000/tcp silly_albattani 896c651b9bfa registry:0.9.1 "docker-registry" 35 seconds ago Up 34 seconds 0.0.0.0:5000->5000/tcp jovial_leakey
Understanding Docker Namespaces
Docker has a concept of namespaces for its repositories which can be confusing.
[a target="_blank" href="https://docs.docker.com/docker-hub/official_repos/"]Official Repositories can be referred to without a username prefix:
Internally these are prefixed by library/. This means that command like docker pull ubuntu:15.10 and docker pull library/ubuntu:15.10 are equivalent.
If the name includes a '/' character (samalba/docker-registry), the left side refers to the username, and the right side refers to the image name in their public repository. It gets more complex when accessing private registries. The format becomes HOST:PORT/[USERNAME/]IMAGE. However, you should note that there is no authentication performed at this layer of our docker registry environment: anyone can push, pull, or delete from any 'user'. If the USERNAME is omitted, it is internally treated as being an 'official' image, and prefixed with library/.
docker pull 127.0.0.1:5000/library/test-ubuntu
Pulling repository 127.0.0.1:5000/library/test-ubuntu
FATA[0004] Error: image library/test-ubuntu:latest not found
[alexh:~/work] docker tag 0fe5a10d2cf8 127.0.0.1:5000/test-ubuntu
[alexh:~/work] docker push 127.0.0.1:5000/test-ubuntu
The push refers to a repository [127.0.0.1:5000/test-ubuntu] (len: 1)
Sending image list
Pushing repository 127.0.0.1:5000/test-ubuntu (1 tags)
Image 5c1d0c04c3b8 already pushed, skipping
Image 8c63e4ac9a5f already pushed, skipping
Image 5fc05c0feaea already pushed, skipping
Image 0fe5a10d2cf8 already pushed, skipping
Pushing tag for rev [0fe5a10d2cf8] on {http://127.0.0.1:5000/v1/repositories/test-ubuntu/tags/latest}
[alexh:~/work] docker pull 127.0.0.1:5000/library/test-ubuntu
Pulling repository 127.0.0.1:5000/library/test-ubuntu
0fe5a10d2cf8: Download complete
5c1d0c04c3b8: Download complete
8c63e4ac9a5f: Download complete
5fc05c0feaea: Download complete
Status: Image is up to date for 127.0.0.1:5000/library/test-ubuntu:latestIn the v2 Docker registry, the [a target="_blank" href="https://docs.docker.com/registry/spec/api/#overview"]URI scheme has changed to allow the repository name to be broken up into multiple components. However, the Docker client does not yet support this flexibility. In the future, you should be able to extend the namespace of your registries, ie `redhat/centos/beta or redhat/fedora/stable.
Populating the Registries
We'll use Ubuntu 15.10 as our example image:
docker pull ubuntu:15.10 15.10: Pulling from ubuntu 5c1d0c04c3b8: Pull complete 8c63e4ac9a5f: Pull complete 5fc05c0feaea: Pull complete 0fe5a10d2cf8: Already exists ubuntu:15.10: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security. Digest: sha256:d569b6ebfc62f35f9792392724bd4a74a4f5f5af10ccbc1880974ae2f0660898 Status: Downloaded newer image for ubuntu:15.10
It needs to be tagged with the new URL in order to push it to the private registries:
[alexh:~/work] docker tag ubuntu:15.10 127.0.0.1:5000/ubuntu:15.10
[alexh:~/work] docker tag ubuntu:15.10 127.0.0.1:6000/ubuntu:15.10
[alexh:~/work] docker push 127.0.0.1:5000/ubuntu:15.10
The push refers to a repository [127.0.0.1:5000/ubuntu] (len: 1)
Sending image list
Pushing repository 127.0.0.1:5000/ubuntu (1 tags)
5c1d0c04c3b8: Image successfully pushed
8c63e4ac9a5f: Image successfully pushed
5fc05c0feaea: Image successfully pushed
0fe5a10d2cf8: Image successfully pushed
Pushing tag for rev [0fe5a10d2cf8] on {http://127.0.0.1:5000/v1/repositories/ubuntu/tags/15.10}
[alexh:~/work] docker push 127.0.0.1:6000/ubuntu:15.10
The push refers to a repository [127.0.0.1:6000/ubuntu] (len: 1)
0fe5a10d2cf8: Image already exists
5fc05c0feaea: Image successfully pushed
8c63e4ac9a5f: Image successfully pushed
5c1d0c04c3b8: Image successfully pushed
Digest: sha256:1f93077ce8f2fa1da8aae87735f395eae93a1c21928d3e2d130717c9aeff177dNote that the output between the v1 registry (on port 5000) and v2 (port 6000) are slightly different, but the result is the same: the Ubuntu image is now available on each registry.
Docker Registry APIs
At this point, we're able to compare the different APIs. In April 2015, Docker [a target="_blank" href="http://docs.docker.com/v1.6/release-notes/"]released version 1.6 and this included v2 of the Registry. Your software should be aware of the different versions of the Docker Registry API to handle these differences. Let's look at what it takes to download the image layers through the various APIs in order to make an offline cache.
First, we'll prepare our environment:
[alexh:~/work] export image=ubuntu [alexh:~/work] export tag=15.10
v1
The v1 private registry can be examined at this point:
[alexh:~/work] curl -s http://127.0.0.1:5000/v1/repositories/library/$image/tags/$tag | python -m json.tool "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547" export v1_image_id=`curl -s http://127.0.0.1:5000/v1/repositories/library/$image/tags/$tag | sed 's/"//g'` [alexh:~/work] curl -s http://127.0.0.1:5000/v1/images/$v1_image_id/ancestry | python -m json.tool [ "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547", "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1", "8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824", "5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73" ] [alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547/layer > 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1/layer > 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824/layer > 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:5000/v1/images/5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73/layer > 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73.tar.gz
v1 on Docker Hub
The Docker Hub currently implements the v1 API, but requires an authentication token for certain operations. It also allows multiple endpoints to be returned by the server. We'll take the simple approach of always using the first endpoint:
[alexh:~/work] export endpoint=`curl -sSL -o /dev/null -D- "https://index.docker.io/v1/repositories/$image/images" | awk '/X-Docker-Endpoints/{print $2}' | tr -d '\r' | sed 's/,//'`
[alexh:~/work] echo $endpoint
registry-1.docker.io
[alexh:~/work] export token=`curl -sSL -o /dev/null -D- -H 'X-Docker-Token: true' "https://index.docker.io/v1/repositories/$image/images" | tr -d '\r' | awk '/X-Docker-Token/{print $2}'`The token needs to be used for authentication for the rest of the commands, but otherwise they are the same as the v1 private registry:
[alexh:~/work] export v1_image_id=`curl -s -H "Authorization: Token $token" https://$endpoint/v1/repositories/library/$image/tags/$tag | sed 's/"//g'` [alexh:~/work] curl -sSL -H "Authorization: Token $token" "https://registry-1.docker.io/v1/images/$v1_image_id/ancestry" | python -m json.tool [ "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547", "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1", "8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824", "5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73" ] [alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547/layer > 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547.tar.gz [alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1/layer > 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1.tar.gz [alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824/layer > 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824.tar.gz [alexh:~/work] curl -sSL -H "Authorization: Token $token" https://$endpoint/v1/images/5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73/layer > 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73.tar.gz
v2 API
The v2 API works with manifest files that include checksums. It's also slightly simpler. A manifest file for a tag contains all of the layer information, rather than requiring an image ID to be looked up for a tag, and then the ancestry for that image to be looked up.
[alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/manifests/$tag | python -c 'import sys, json, pprint; pprint.pprint(json.load(sys.stdin)["fsLayers"])' [{u'blobSum': u'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'}, {u'blobSum': u'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'}, {u'blobSum': u'sha256:d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d'}, {u'blobSum': u'sha256:23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca'}, {u'blobSum': u'sha256:7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107'}] [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 > a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d > d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca > 23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca.tar.gz [alexh:~/work] curl -sSL http://127.0.0.1:6000/v2/$image/blobs/sha256:7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107 > 7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107.tar.gz
We can get the checksum for these files to verify that they are what is described in the manifest file:
[alexh:~/work] sha256sum *.tar.gz a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d d4d342aa9da086ca4b7f7273858072e81021f4379c486223bc4708df6862b55d.tar.gz 23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca 23dc26e1038ae691b1a7e8e0152f974a358c42c929104c18c8e20b6d363c41ca.tar.gz 7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107 7772c716a45a828e124d20bc67199e77f2e63fb62589d0046f974f99b406e107.tar.gz
The Remote (daemon) API
Another API that is available is the Docker daemon running locally. It can be accessed over a Unix socket, or over TCP if the daemon is configured to allow it.
[alexh:~/work] echo -e "GET /images/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock | tail -n +6 | python -m json.tool
[
{
"Created": 1433116930,
"Id": "0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547",
"Labels": {},
"ParentId": "5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1",
"RepoDigests": [],
"RepoTags": [
"127.0.0.1:6000/ubuntu:15.10",
"ubuntu:15.10",
"127.0.0.1:5000/ubuntu:15.10"
],
"Size": 0,
"VirtualSize": 132392276
},
{
"Created": 1432704049,
"Id": "0c5e5ef1d7dac23c7164ea48faafc79f0c921f6cf87d2d8ea7469832ea31e4ca",
"Labels": {},
"ParentId": "136beb445cfa7f48dbe4e36a80a83d4b7945682827fd8bfb1510ac17b6a200c0",
"RepoDigests": [],
"RepoTags": [
"registry:2.0.1"
],
"Size": 0,
"VirtualSize": 548626543
},
{
"Created": 1432703977,
"Id": "4e698fa804417b34b334793bab8a143403be9384e0651067b0c3933fe8d90eb2",
"Labels": {},
"ParentId": "0cd49aa0e23cfe176cbea4bf622d552a6f16b21965cf52d633f8c9e27438f52c",
"RepoDigests": [],
"RepoTags": [
"registry:0.9.1"
],
"Size": 0,
"VirtualSize": 413940033
}
]A tarball containing all of the layers for a tag can be generated:
[alexh:~/work] echo -e "GET /images/get?names=$image:$tag HTTP/1.0\r\n" | nc -U /var/run/docker.sock | tail -n +5 > $image-$tag.tar [alexh:~/work] mkdir tmp [alexh:~/work] tar -C tmp -xf ubuntu-15.10.tar [alexh:~/work] ls -l tmp total 20 drwxr-xr-x 2 alexh alexh 4096 Jun 2 15:33 0fe5a10d2cf8cdb378a39a81d87b0c8fcfa8fcaaf11bba895a1b6f72baf9a547 drwxr-xr-x 2 alexh alexh 4096 Jun 2 15:33 5c1d0c04c3b846fffd1d70886c956927a5c5f6a1c96f5e9f61c02f2ec1a45a73 drwxr-xr-x 2 alexh alexh 4096 Jun 2 15:33 5fc05c0feaeab977e52b7c2490bffacaba0e3d58e7955b683f271041d3558ad1 drwxr-xr-x 2 alexh alexh 4096 Jun 2 15:33 8c63e4ac9a5f31e482d25a149b022209653b5948cb4f045c2ede9331a18e5824 -rw-r--r-- 1 alexh alexh 87 Jun 2 15:33 repositories
Conclusions
Docker is a great technology and there are a lot of improvements and new features coming out at a rapid pace. Fortunately it's well documented and discussions about bugs are in the open on GitHub. However, there are still some edge cases to be aware of when talking to the Docker APIs. With some good design choices, your applications can be made backwards and forwards compatible, and will be able to use a wide range of Docker client versions and remote APIs.
Opinions expressed by DZone contributors are their own.
Comments