OCI Registry Mirroring With zot¶

A zot registry can mirror one or more upstream OCI registries, including popular cloud registries such as Docker Hub and Google Container Registry (gcr.io).

A key use case for zot is to act as a mirror for upstream registries. If an upstream registry is OCI distribution-spec conformant for pulling images, you can use zot's sync feature to implement a downstream mirror, synchronizing OCI images and corresponding artifacts.

As with git, wherein every clone is a full repository, you can configure a local zot instance to be a full OCI mirror registry. This allows for a fully distributed disconnected container image build pipeline.

Synchronization between upstream and downstream registries can be implemented by periodic polling of the upstream registry or synchronization can occur on demand, when a user pulls an image from the downstream registry.

Because Docker Hub rate-limits pulls and does not support catalog listing, do not use polled mirroring with Docker Hub. Use only on-demand mirroring with Docker Hub.

Basic configuration for mirroring with sync¶

The sync feature of zot is an extension of the OCI-compliant registry implementation. You can configure the sync feature under the extensions section of the zot configuration file, as shown in this example:

  "extensions": {
    "sync": {
      "credentialsFile": "./examples/sync-auth-filepath.json",
      "registries": [
        {
          "urls": [
            "https://registry1:5000"
          ],
          "onDemand": false,
          "pollInterval": "6h",
          "tlsVerify": true,
          "certDir": "/home/user/certs",
          "maxRetries": 3,
          "retryDelay": "5m", 
          "onlySigned": true,
          "content": [
            {
              "prefix": "/repo2/repo",
              "tags": {
                "regex": "4.*",
                "semver": true
              }
              "destination": "/repo2",
              "stripPrefix": true
            }
          ]
        }
      ]
    }
  }

The following table lists the configurable attributes for the sync feature:

Attribute	Description
credentialsFile	The location of a local file containing credentials for other registries, as in the following example: { "127.0.0.1:8080": { "username": "user", "password": "pass" }, "registry2:5000": { "username": "user2", "password": "pass2" } }
urls	A list of one or more URLs to an upstream image registry. If the main URL fails, the sync process will try the next URLs in the listed order.
onDemand	`false`: Pull all images not found in the local registry. `true`: Pull any image not found in the local registry only when needed.
pollInterval	The period in seconds between polling of remote registries. If no value is specified, no periodic polling will occur. If a value is set and the content attributes are configured, periodic synchronization is enabled and will run at the specified value. Note: Because Docker Hub rate-limits pulls and does not support catalog listing, do not use polled mirroring with Docker Hub. Use only onDemand mirroring with Docker Hub.
tlsVerify	`false`: TLS will not be verified. `true`: (Default) The TLS connection to the destination registry will be verified.
certDir	If a path is specified, use certificates (.crt, .cert, *.key files) at this path when connecting to the destination registry or daemon. If no path is specified, use the default certificates directory.
maxRetries	The maximum number of retries if an error occurs during either an on-demand or periodic synchronization. If no value is specified, no retries will occur.
retryDelay	The interval in seconds between retries. This attribute is mandatory when maxRetries is configured.
onlySigned	`false`: Synchronize signed or unsigned images. `true`: Synchronize only signed images (either notary or cosign).
content	The included attributes in this section specify which content will be pulled. If this section is not populated, periodic polling will not occur. The included attributes can also filter which on-demand images are pulled.
prefix	On the remote registry, the path from which images will be pulled. This path can be a string that exactly matches the remote path, or it can be a glob pattern. For example, the path can include a wildcard () or a recursive wildcard (*).
tags	The included attributes in this optional section specify how remote images will be selected for synchronization based on image tags.
tags.regex	Specifies a regular expression for matching image tags. Images whose tags do not match the expression are not pulled.
tags.semver	Specifies whether image tags are to be filtered by semantic versioning (semver) compliance. `false`: Do not filter by semantic versioning. `true`: Filter by semantic versioning.
destination	Specifies the local path in which pulled images are to be stored.
stripPrefix	Specifies whether the prefix path from the remote registry will be retained or replaced when the image is stored in the zot registry. `false`: Retain the source prefix, append it to the destination path. `true`: Remove the source prefix. Note: If the source prefix was specified with meta-characters (such as **), only the prefix segments that precede the meta-characters are removed. Any remaining path segments are appended to the destination path.

Example: Multiple repositories with polled mirroring¶

The following is an example of sync configuration for mirroring multiple repositories with polled mirroring.

"sync": {
  "enable": true,
  "credentialsFile": "./examples/sync-auth-filepath.json",
  "registries": [
    {
      "urls": ["https://registry1:5000"],
      "onDemand": false,
      "pollInterval": "6h",
      "tlsVerify": true,
      "certDir": "/home/user/certs",
      "maxRetries": 3,
      "retryDelay": "5m",
      "onlySigned": true,
      "content": [
        {
          "prefix": "/repo1/repo",
          "tags": {
            "regex": "4.*",
            "semver": true
          }
        },
        {
          "prefix": "/repo2/repo",
          "destination": "/repo2",
          "stripPrefix": true
        },
        {
          "prefix": "/repo3/repo"
        }
      ]
    }
  }

The configuration in this example will result in the following behavior:

Only signed images (notation and cosign) are synchronized.
The sync communication is secured using certificates in certDir.
This registry synchronizes with upstream registry every 6 hours.
On-demand mirroring is disabled.
Based on the content filtering options, this registry synchronizes these images:
- From /repo1/repo, images with tags that begin with "4." and are semver compliant.
  Files are stored locally in /repo1/repo on localhost.
- From /repo2/repo, images with all tags.
  Because stripPrefix is enabled, files are stored locally in /repo2. For example, docker://upstream/repo2/repo:v1 is stored as docker://local/repo2:v1.
- From /repo3/repo, images with all tags.
  Files are stored locally in /repo3/repo.

Example: Multiple registries with on-demand mirroring¶

The following is an example of sync configuration for mirroring multiple registries with on-demand mirroring.

{
  "distSpecVersion": "1.0.1",
  "storage": {
    "rootDirectory": "/tmp/zot",
    "gc": true
  },
  "http": {
    "address": "0.0.0.0",
    "port": "8080"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "sync": {
      "enable": true,
      "registries": [
        {
          "urls": ["https://k8s.gcr.io"],
          "content": [
            {
              "prefix": "**", 
              "destination": "/k8s-images"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        },
        {
          "urls": ["https://docker.io/library"],
          "content": [
            {
              "prefix": "**", 
              "destination": "/docker-images"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        }
      ]
    }
  }
}

With this zot configuration, the sync behavior is as follows:

This user request for content from the zot registry:
skopeo copy --src-tls-verify=false docker://localhost:8080/docker-images/alpine <dest>
causes zot to synchronize the content with the docker.io registry:
docker.io/library/alpine:latest
to the zot registry:
localhost:8080/docker-images/alpine:latest
before delivering the content to the requestor at <dest>.
This user request for content from the zot registry:
skopeo copy --src-tls-verify=false docker://localhost:8080/k8s-images/kube-proxy:v1.19.2 <dest>
causes zot to synchronize the content with the gcr.io registry:
k8s.gcr.io/kube-proxy:v1.19.2
to the zot registry:
localhost:8080/k8s-images/kube-proxy:v1.19.2
before delivering the content to the requestor at <dest>.

You can use this command:
curl http://localhost:8080/v2/_catalog
to display the local repositories:

  {
    "repositories":[
      "docker-images/alpine",
      "k8s-images/kube-proxy"
    ]
  }

Example: Multiple registries with mixed mirroring modes¶

The following is an example of a zot configuration file for mirroring multiple upstream registries.

{
  "distSpecVersion": "1.1.0-dev",
  "storage": {
    "rootDirectory": "/tmp/zot"
  },
  "http": {
    "address": "127.0.0.1",
    "port": "8080"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "sync": {
      "enable": true,
      "credentialsFile": "./examples/sync-auth-filepath.json",
      "registries": [
        {
          "urls": [
            "https://registry1:5000"
          ],
          "onDemand": false,
          "pollInterval": "6h",
          "tlsVerify": true,
          "certDir": "/home/user/certs",
          "maxRetries": 3,
          "retryDelay": "5m",
          "onlySigned": true,
          "content": [
            {
              "prefix": "/repo1/repo",
              "tags": {
                "regex": "4.*",
                "semver": true
              }
            },
            {
              "prefix": "/repo1/repo",
              "destination": "/repo",
              "stripPrefix": true
            },
            {
              "prefix": "/repo2/repo"
            }
          ]
        },
        {
          "urls": [
            "https://registry2:5000",
            "https://registry3:5000"
          ],
          "pollInterval": "12h",
          "tlsVerify": false,
          "onDemand": false,
          "content": [
            {
              "prefix": "/repo2",
              "tags": {
                "semver": true
              }
            }
          ]
        },
        {
          "urls": [
            "https://docker.io/library"
          ],
          "onDemand": true,
          "tlsVerify": true,
          "maxRetries": 6,
          "retryDelay": "5m"
        }
      ]
    }
  }
}

Example: Support for subpaths in local storage¶

With this zot configuration, the sync behavior name=__codelineno-5-1 href=#__codelineno-5-1>{ "distSpecVersion": "1.0.1", "storage": { "subPaths":{ "/kube-proxy":{ "rootDirectory": "/tmp/kube-proxy", "dedupe": true, "gc": true } }, "rootDirectory": "/tmp/zot", "gc": true }, "http": { "address": "0.0.0.0", "port": "8080" }, "log": { "level": "debug" }, "extensions": { "sync": { "enable": true, "registries": [ { "urls": ["https://k8s.gcr.io"], "content": [ { "destination": "/kube-proxy", "prefix": "**" } ], "onDemand": true, "tlsVerify": true, "maxRetries": 2, "retryDelay": "5m" } ] } } } is as follows:

This user request for content from the zot registry:
skopeo copy --src-tls-verify=false docker://localhost:8080/kube-proxy/kube-proxy:v1.19.2 <dest>
causes zot to synchronize the content with this remote registry:
k8s.gcr.io/kube-proxy:v1.19.2
to the zot registry:
localhost:8080/kube-proxy/kube-proxy:v1.19.2
before delivering the content to the requestor at <dest>.

You can use this command:
curl http://localhost:8080/v2/_catalog
to display the local repositories:

  {
    "repositories":[
      "docker-images/alpine",
      "k8s-images/kube-proxy",
      "kube-proxy/kube-proxy"
    ]
  }

In zot storage, the requested content is located here:
/tmp/zot/kube-proxy/kube-proxy/kube-proxy/
This subpath is created from the following path components:

/tmp/zot is the rootDirectory of the zot registry
kube-proxy is the rootDirectory of the storage subpath
kube-proxy is the sync destination parameter
kube-proxy is the repository name

Last update: March 6, 2023