无法将 docker 镜像推送到 Nexus

问题描述

我正在运行 Nexus OSS 版本 3.29.2-02,我遇到了一些奇怪的行为。我正在 CI 级别 (GitLab) 构建各种图像,并将它们推送到自定义存储库。

在大多数情况下,一切正常,我在标记和推送我制作的图像时没有问题。但是最近,其中一个生成 Docker 镜像的项目无法推送,并出现以下错误

$ TAGGED=${NEXUS_DOCKER_URL}/${BASE_IMAGE_NAME}:snapshot-MR${CI_MERGE_REQUEST_IID}
$ docker tag ${BASE_IMAGE_NAME}:latest ${TAGGED}
$ docker push ${TAGGED}
The push refers to repository [<custom-repository-url>/<image-name:tag>]
cade37b0f9c9: Preparing
578ec024f17c: Preparing
fe0b994190e8: Preparing
b24d08ca4359: Preparing
9a14db3b513b: Preparing
777b2c648970: Preparing
777b2c648970: Waiting
b24d08ca4359: Layer already exists
9a14db3b513b: Layer already exists
777b2c648970: Layer already exists
fe0b994190e8: Pushed
cade37b0f9c9: Pushed
578ec024f17c: Pushed
[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the <custom-repository-url> registry Now to avoid future disruption.
errors:
blob unkNown: blob unkNown to registry
blob unkNown: blob unkNown to registry
ERROR: Job Failed: exit code 1

我已经尝试调试此行为并在线搜索解决方案,但我还没有找到任何东西。似乎由于某种原因,这个特定的 Docker 镜像无法上传。我已经在本地机器和无状态 CI 构建器中尝试了相同的过程,并且行为是一致的,即我只能推送一次,然后该过程一直失败。

作为参考,我的 Dockerfile 如下:

FROM <custom-repository-url>/adoptopenjdk/openjdk11:jre-11.0.10_9-alpine
workdir /home/app
copY build/libs/email-service.jar application.jar

# Set the appropriate timezone
RUN apk add --no-cache tzdata && \
    cp /usr/share/zoneinfo/America/New_York /etc/localtime && \
    echo "America/New_York" > /etc/timezone

EXPOSE 8080
CMD java -jar ${OPTS} application.jar

这很简单,没有隐藏任何复杂的东西。我最初认为这个问题可能是由于使用了基于代理的图像(即 FROM),但其他几个项目都这样做了,没有任何问题。

我也尝试过检查 Nexus 的日志,但我看到的唯一内容如下:

2021-02-05 17:12:27,441+0000 ERROR [qtp1025847496-15765]  ci-deploy org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:66db482b5034f8eda0b18533d4eddb0012f4940bf3d348b08ac3bac8486bb2ee for: fts/marketing/email-service/snapshot-MR40 in repository RepositoryImpl$$EnhancerByGuice$$4d5af99c{type=hosted,format=docker,name='docker-hosted-s3'}
2021-02-05 17:12:27,443+0000 ERROR [qtp1025847496-15765]  ci-deploy org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl - Manifest refers to missing layer: sha256:2ec25ba939258edb2e85293896c5126478d79fe416d3b60feb20426755bcea5a for: fts/marketing/email-service/snapshot-MR40 in repository RepositoryImpl$$EnhancerByGuice$$4d5af99c{type=hosted,445+0000 WARN  [qtp1025847496-15765]  ci-deploy org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/fts/marketing/email-service/manifests/snapshot-MR40: 400 - org.sonatype.nexus.repository.docker.internal.V2Exception: Invalid Manifest

所以我的问题是:

  1. 这个错误究竟意味着什么?我觉得它不是很有用:
errors:
blob unkNown: blob unkNown to registry
blob unkNown: blob unkNown to registry
  1. 究竟是什么导致了这种行为,我该如何解决这个问题?

注意(不是它应该有任何区别),该图像是一个 dockerized Micronaut 应用程序,使用最新版本的框架。

作为参考,该图像的 docker inspect 输出如下:

[{
    "Id": "sha256:fec226a68e3b744fc792e47d3235e67f06b17883e60df52c8ae82c5a7ba9750f","RepoTags": [
        "<custom-repository-url>/fts/marketing/email-service:mes-33-3","test-mes-33:latest"
    ],"RepoDigests": [],"Parent": "sha256:ddd8e2235b60d7636283097fc61e5971c32b3006ee52105e2a77e7d4ee7e709e","Comment": "","Created": "2021-02-06T21:06:59.987108458Z","Container": "8ab70692b75aac21d0866816aa52af5febf620744282d71a39dce55f81fe3e44","ContainerConfig": {
        "Hostname": "8ab70692b75a","Domainname": "","User": "","AttachStdin": false,"AttachStdout": false,"AttachStderr": false,"ExposedPorts": {
            "8080/tcp": {}
        },"Tty": false,"OpenStdin": false,"StdinOnce": false,"Env": [
            "PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","LANG=en_US.UTF-8","LANGUAGE=en_US:en","LC_ALL=en_US.UTF-8","JAVA_VERSION=jdk-11.0.10+9","JAVA_HOME=/opt/java/openjdk"
        ],"Cmd": [
            "/bin/sh","-c","#(nop) ","CMD [\"/bin/sh\" \"-c\" \"java -jar ${OPTS} application.jar\"]"
        ],"Image": "sha256:ddd8e2235b60d7636283097fc61e5971c32b3006ee52105e2a77e7d4ee7e709e","Volumes": null,"WorkingDir": "/home/app","Entrypoint": null,"OnBuild": null,"Labels": {}
    },"DockerVersion": "19.03.13","Author": "","Config": {
        "Hostname": "","java -jar ${OPTS} application.jar"
        ],"Labels": null
    },"Architecture": "amd64","Os": "linux","Size": 220998577,"VirtualSize": 220998577,"GraphDriver": {
        "Data": {
            "LowerDir": "/var/lib/docker/overlay2/78561c2e477b099a547bead4ea17b677bb01376fc1ed1ce1cd942157d35c0329/diff:/var/lib/docker/overlay2/af8ac4feace0cbecd616e2a02850ec366715aaa5ac8ad143cb633f52b0f6fbe2/diff:/var/lib/docker/overlay2/211a8e68c833f664de5d304838b8cd98b8e5e790f79da8b8839a4d52d02a8d66/diff:/var/lib/docker/overlay2/cbc98e7274ff8266425aed31989066ff7c5f7a46d9334b84110fc57d8b1d942c/diff:/var/lib/docker/overlay2/c773dedbc53b81c2e68ad61811445c0377271db3af526dbf5a6aa6671d0b2b71/diff","MergedDir": "/var/lib/docker/overlay2/04240d9f745382480e52e04d8088de6f65a9ece0cd6e091953087f3d06fcc93c/merged","UpperDir": "/var/lib/docker/overlay2/04240d9f745382480e52e04d8088de6f65a9ece0cd6e091953087f3d06fcc93c/diff","workdir": "/var/lib/docker/overlay2/04240d9f745382480e52e04d8088de6f65a9ece0cd6e091953087f3d06fcc93c/work"
        },"Name": "overlay2"
    },"RootFS": {
        "Type": "layers","Layers": [
            "sha256:777b2c648970480f50f5b4d0af8f9a8ea798eea43dbcf40ce4a8c7118736bdcf","sha256:9a14db3b513b928759670c6a9b15fd89a8ad9bf222c75e0998c21bcb04e25e48","sha256:b24d08ca43598c9ea44f73c3f5dfca2b4897c475b2cc480bac98cccc42dce10f","sha256:11d1fa1ad1ef523c60369c11b1096baf89c8d43afa53813e84c73d0926848598","sha256:30001f69fd3b3b08fdbf6d843e38d0a16d0e46e84923f92480ac88603c0eb680","sha256:b2d3c5f57d1d626a7501b8871f548fd7e1f7625fe05c1885c27ec415b14e9915"
        ]
    },"Metadata": {
        "LastTagTime": "2021-02-06T23:08:30.440032169+02:00"
    }
}]

解决方法

每当遇到图像中缺少/无效的图层时,docker 注册表(在您的情况下为 Nexus)都会抛出该错误。

Nexus 曾经在处理外部层时遇到困难,但这应该不是问题,因为您运行的是相当新的版本。

我认为您只需要在 Nexus 中启用“外部层缓存”即可使其正常工作。

包含 docker manifest inspect <custom-repository-url>/adoptopenjdk/openjdk11:jre-11.0.10_9-alpinedocker manifest inspect ${TAGGED}

的输出会很有帮助

Docker registry API spec