Thank you Ina! I manually applied this patch and looks like it actually works :)
ср, 25 сент. 2019 г. в 19:24, Ina Panova <ipan...@redhat.com>: > Hi Konstantin, > > surprisingly today i was able to reproduce the issue and was able to find > the root cause of the problem. > Here's the PR that should fix the upload issue > https://github.com/pulp/pulp_docker/pull/418 > > > -------- > Regards, > > Ina Panova > Senior Software Engineer| Pulp| Red Hat Inc. > > "Do not go where the path may lead, > go instead where there is no path and leave a trail." > > > On Tue, Sep 17, 2019 at 11:58 AM Konstantin M. Khankin < > khankin.konstan...@gmail.com> wrote: > >> Sorry for necroposting. I'm not sure if I still have the same issue, but >> I have the same symptoms. Could you please have a look and advice? Thanks! >> >> Whenever I try to create and upload another version of a container to the >> same repo, I get an error message like this: >> >> $ /usr/bin/pulp-admin docker repo uploads upload --repo-id gsk_postfix -f >> /tmp/postfix.tar >> ... >> Uploading: postfix.tar >> [==================================================] 100% >> 283176960/283176960 bytes >> ... completed >> >> Importing into the repository... >> This command may be exited via ctrl+c without affecting the request. >> >> >> [-] >> Running... >> >> Task Failed >> >> The importer docker_importer indicated a failed response when uploading >> docker_manifest unit to repository gsk_postfix. Summary: 'Cannot resolve >> field >> "digest"'. Details: '{}' >> >> I run pulp 2.20.1 and use buildah 1.9.0 to build container images. First >> image in the repository uploads fine, but the second always fails. >> >> Current repo status: >> Id: gsk_postfix >> Display Name: None >> Description: None >> Content Unit Counts: >> Docker Blob: 4 >> Docker Manifest: 1 >> Docker Tag: 1 >> >> # pulp-admin docker repo search image --repo-id gsk_postfix >> <returns nothing> >> >> # pulp-admin docker repo search manifest --repo-id gsk_postfix >> Created: 2019-08-20T22:58:50Z >> Metadata: >> Config Layer: >> sha256:d2b497c83fe099624567046432468d84a5fdb49139b5192c9b332f0b10784695 >> Digest: >> sha256:d17067c71cb229f703b0f43ecd8a85c55995f3007b99570f580f9f6ebc5dbccc >> Downloaded: True >> Fs Layers: >> Blob Sum: >> sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95 >> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip >> Size: 210204160 >> Blob Sum: >> sha256:204cafbfdb1d31cb427478102377a99ceccfd7cc7be19cc191568fb9008fd4af >> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip >> Size: 12288 >> Blob Sum: >> sha256:a85e1551e989545ca081c9ed0584e2db535a1484b91060eed88a17991e4b25cf >> Layer Type: application/vnd.docker.image.rootfs.diff.tar.gzip >> Size: 72946688 >> Pulp User Metadata: >> Schema Version: 2 >> Repo Id: gsk_postfix >> Unit Id: 000c0b4a-fb98-4ca3-a3ec-3b2a42521949 >> Unit Type Id: docker_manifest >> Updated: 2019-08-20T22:58:50Z >> >> # pulp-admin docker repo search tag --repo-id gsk_postfix >> Created: 2019-08-20T22:58:55Z >> Metadata: >> Manifest Digest: >> sha256:d17067c71cb229f703b0f43ecd8a85c55995f3007b99570f580f9f6ebc5dbccc >> Manifest Type: image >> Name: latest >> Pulp User Metadata: >> Repo Id: gsk_postfix >> Schema Version: 2 >> Repo Id: gsk_postfix >> Unit Id: 26c40bb1-4f3b-438f-8f50-cc5afe6baeb6 >> Unit Type Id: docker_tag >> Updated: 2019-08-20T22:58:55Z >> >> To build and upload containers, I wrote a simple script. Here is the >> command sequence: >> ### STEP 1: Build an image from Dockerfile. >> $ /usr/bin/buildah bud -t gsk_postfix . >> ... >> Writing manifest to image destination >> Storing signatures >> db008cf80da01afe8ca11898e566279a7765e682bf56070857d40ffd469e2486 >> ### STEP 2: Get the image digest. >> $ /usr/bin/buildah images --quiet --digests --no-trunc >> localhost/gsk_postfix:latest >> sha256:db008cf80da01afe8ca11898e566279a7765e682bf56070857d40ffd469e2486 >> ### STEP 3: Pull blobs from the storage. >> $ /usr/bin/skopeo copy --format v2s2 >> containers-storage:localhost/gsk_postfix:latest dir:/tmp/tmp.NYhxiiOsOe >> Getting image source signatures >> Copying blob b05580fca2f9 done >> Copying blob 8d136bb340b2 done >> Copying blob 7a406e7184ab done >> Copying config db008cf80d done >> Writing manifest to image destination >> Storing signatures >> ### STEP 4: Tar the archive. >> $ tar cf /tmp/postfix.tar * >> ### STEP 5: Upload the archive to the repo. >> $ /usr/bin/pulp-admin docker repo uploads upload --repo-id gsk_postfix -f >> /tmp/postfix.tar >> <failure here> >> >> This happens no matter if I use v2s1 or v2s2. Resulting manifest.json >> contains digests for all blobs (different image but built at the same time >> with the same script and failed to get uploaded with the same error): >> {'config': {'digest': >> 'sha256:fef5b5f9bb6fca5f1909303822ae34ae9d3ac1d88da84144fa1a7b83229652f3', >> 'mediaType': 'application/vnd.docker.container.image.v1+json', >> 'size': 2339}, >> 'layers': [{'digest': >> 'sha256:b05580fca2f9aabb2d8fa975b29146c9147c8418e559f197c54a4fac04babb95', >> 'mediaType': >> 'application/vnd.docker.image.rootfs.diff.tar.gzip', >> 'size': 210204160}, >> {'digest': >> 'sha256:bbd20206b01e4bf5d521890bf5ffe2d5f565bbf70eda987b059b267c80f4f9a1', >> 'mediaType': >> 'application/vnd.docker.image.rootfs.diff.tar.gzip', >> 'size': 39936}], >> 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', >> 'schemaVersion': 2} >> >> ср, 24 окт. 2018 г. в 16:05, Konstantin M. Khankin < >> khankin.konstan...@gmail.com>: >> >>> Thank you, https://pulp.plan.io/issues/4095 filed (why didn't I wait a >>> bit to get 4096...) >>> >>> ср, 24 окт. 2018 г. в 15:35, Austin Macdonald <amacd...@redhat.com>: >>> >>>> The other projects are sometimes hidden. >>>> https://pulp.plan.io/projects/pulp_docker >>>> >>>> On Wed, Oct 24, 2018 at 2:55 AM Konstantin M. Khankin < >>>> khankin.konstan...@gmail.com> wrote: >>>> >>>>> Dennis, >>>>> >>>>> Which project I should use? I don't see a separate project for >>>>> pulp_docker. >>>>> >>>>> Thanks! >>>>> >>>>> вт, 23 окт. 2018 г. в 23:58, Dennis Kliban <dkli...@redhat.com>: >>>>> >>>>>> Konstantin, >>>>>> >>>>>> Could you please file an issue in pulp.plan.io? >>>>>> >>>>>> Thanks, >>>>>> Dennis >>>>>> >>>>>> On Sat, Oct 20, 2018 at 5:40 AM Konstantin M. Khankin < >>>>>> khankin.konstan...@gmail.com> wrote: >>>>>> >>>>>>> *Forgot to mention versions:* >>>>>>> >>>>>>> pulp-admin-client-2.17.1-1.el7.noarch >>>>>>> pulp-agent-2.17.1-1.el7.noarch >>>>>>> pulp-consumer-client-2.17.1-1.el7.noarch >>>>>>> pulp-deb-admin-extensions-1.8.0-1.el7.noarch >>>>>>> pulp-deb-plugins-1.8.0-1.el7.noarch >>>>>>> pulp-docker-admin-extensions-3.2.1-1.el7.noarch >>>>>>> pulp-docker-plugins-3.2.1-1.el7.noarch >>>>>>> pulp-puppet-consumer-extensions-2.17.1-1.el7.noarch >>>>>>> pulp-puppet-handlers-2.17.1-1.el7.noarch >>>>>>> pulp-python-admin-extensions-2.0.3-1.el7.noarch >>>>>>> pulp-python-plugins-2.0.3-1.el7.noarch >>>>>>> pulp-rpm-admin-extensions-2.17.1-1.el7.noarch >>>>>>> pulp-rpm-consumer-extensions-2.17.1-1.el7.noarch >>>>>>> pulp-rpm-handlers-2.17.1-1.el7.noarch >>>>>>> pulp-rpm-plugins-2.17.1-1.el7.noarch >>>>>>> pulp-rpm-yumplugins-2.17.1-1.el7.noarch >>>>>>> pulp-selinux-2.17.1-1.el7.noarch >>>>>>> pulp-server-2.17.1-1.el7.noarch >>>>>>> python-pulp-agent-lib-2.17.1-1.el7.noarch >>>>>>> python-pulp-bindings-2.17.1-1.el7.noarch >>>>>>> python-pulp-client-lib-2.17.1-1.el7.noarch >>>>>>> python-pulp-common-2.17.1-1.el7.noarch >>>>>>> python-pulp-deb-common-1.8.0-1.el7.noarch >>>>>>> python-pulp-docker-common-3.2.1-1.el7.noarch >>>>>>> python-pulp-oid_validation-2.17.1-1.el7.noarch >>>>>>> python-pulp-puppet-common-2.17.1-1.el7.noarch >>>>>>> python-pulp-python-common-2.0.3-1.el7.noarch >>>>>>> python-pulp-repoauth-2.17.1-1.el7.noarch >>>>>>> python-pulp-rpm-common-2.17.1-1.el7.noarch >>>>>>> >>>>>>> buildah-1.2-2.gitbe87762.el7.x86_64 >>>>>>> >>>>>>> *Manifest v2s1 from one of the images created by buildah:* >>>>>>> {'architecture': 'amd64', >>>>>>> 'fsLayers': [{'blobSum': >>>>>>> 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8'}, >>>>>>> {'blobSum': >>>>>>> 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'}, >>>>>>> {'blobSum': >>>>>>> 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'}, >>>>>>> {'blobSum': >>>>>>> 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f'}], >>>>>>> 'history': [{'v1Compatibility': >>>>>>> '{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Entrypoint":["/bin/bash"],"WorkingDir":"/","Labels":{"org.label-schema.build-date":"20181006","org.label-schema.license":"GPLv2"," >>>>>>> org.label-schema.name":"CentOS Base >>>>>>> Image","org.label-schema.schema-version":"1.0","org.label-schema.vendor":"CentOS"}},"created":"2018-10-20T07:49:56.084883454Z","id":"4591c91f4c223fa61c800a84bcf711647786f3fe71e589606f17b4d11b9c5807","os":"linux","parent":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904"}'}, >>>>>>> {'v1Compatibility': >>>>>>> '{"id":"f871460fb057cfa08e7268df9b884eecedeed72bc6dbc3262deb19a25e66a904","parent":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","created":"2018-10-09T18:19:48.447478476Z","container_config":{"Cmd":["/bin/sh >>>>>>> -c #(nop) CMD [\\"/bin/bash\\"]"]},"throwaway":true}'}, >>>>>>> {'v1Compatibility': >>>>>>> '{"id":"8a4f409d848e6f693d5a4e6c260fbd8ec6a4dc99fc9915e3210bd5ed9c22e700","parent":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:48.258784779Z","container_config":{"Cmd":["/bin/sh >>>>>>> -c #(nop) LABEL org.label-schema.schema-version=1.0 >>>>>>> org.label-schema.name=CentOS Base Image >>>>>>> org.label-schema.vendor=CentOS org.label-schema.license=GPLv2 >>>>>>> org.label-schema.build-date=20181006"]},"throwaway":true}'}, >>>>>>> {'v1Compatibility': >>>>>>> '{"id":"961164371d3af72a151063f656820bc835c695fa44af66c2845d0850f01ce0d9","created":"2018-10-09T18:19:47.884079751Z","container_config":{"Cmd":["/bin/sh >>>>>>> -c #(nop) ADD >>>>>>> file:fbe9badfd2790f0747a25fbe5c94a6daa78969511ca08c8d4ac654f3442570de >>>>>>> in / >>>>>>> "]}}'}], >>>>>>> 'name': '', >>>>>>> 'schemaVersion': 1, >>>>>>> 'signatures': [{'header': {'alg': 'ES256', >>>>>>> 'jwk': {'crv': 'P-256', >>>>>>> 'kid': '...', >>>>>>> 'kty': 'EC', >>>>>>> 'x': '...', >>>>>>> 'y': '...'}}, >>>>>>> 'protected': '...', >>>>>>> 'signature': '...'}], >>>>>>> 'tag': ''} >>>>>>> >>>>>>> *Manifest v2s2 for the same image:* >>>>>>> {'config': {'digest': >>>>>>> 'sha256:c17d63d23edcbe46b14fbe531e2e9b4411483947669087e90bdba3372eb2684d', >>>>>>> 'mediaType': >>>>>>> 'application/vnd.docker.container.image.v1+json', >>>>>>> 'size': 1217}, >>>>>>> 'layers': [{'digest': >>>>>>> 'sha256:f972d139738dfcd1519fd2461815651336ee25a8b54c358834c50af094bb262f', >>>>>>> 'mediaType': >>>>>>> 'application/vnd.docker.image.rootfs.diff.tar.gzip', >>>>>>> 'size': 208811520}, >>>>>>> {'digest': >>>>>>> 'sha256:4d85b6fa2344380743db8ccf4f0404603f56c8289db0824c13bfc309b2009be8', >>>>>>> 'mediaType': >>>>>>> 'application/vnd.docker.image.rootfs.diff.tar.gzip', >>>>>>> 'size': 589285888}], >>>>>>> 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', >>>>>>> 'schemaVersion': 2} >>>>>>> >>>>>>> сб, 20 окт. 2018 г. в 11:56, Konstantin M. Khankin < >>>>>>> khankin.konstan...@gmail.com>: >>>>>>> >>>>>>>> Hi! >>>>>>>> >>>>>>>> I tried the tool Buildah ( >>>>>>>> https://www.projectatomic.io/blog/2017/06/introducing-buildah/) to >>>>>>>> build containers without having docker binary. Containers are built >>>>>>>> (not >>>>>>>> sure if they work), but I can't upload them to the pulp repo: >>>>>>>> >>>>>>>> 1) I copy container from "container-storage:" to "dir:", then >>>>>>>> create tar archive per >>>>>>>> https://docs.pulpproject.org/plugins/pulp_docker/user-guide/recipes.html#upload-v2-schema-2-and-schema-1-images-to-pulp >>>>>>>> 2) pulp-admin -vvv docker repo uploads upload --repo-id >>>>>>>> docker_centos7 -f centos7.tar >>>>>>>> >>>>>>>> [2] results in: >>>>>>>> "traceback": "Traceback (most recent call last):\n File >>>>>>>> \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in >>>>>>>> trace_task\n R = retval = fun(*args, **kwargs)\n File >>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line >>>>>>>> 529, >>>>>>>> in __call__\n return super(Task, self).__call__(*args, **kwargs)\n >>>>>>>> File >>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line >>>>>>>> 107, >>>>>>>> in __call__\n return super(PulpTask, self).__call__(*args, >>>>>>>> **kwargs)\n >>>>>>>> File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line >>>>>>>> 622, in >>>>>>>> __protected_call__\n return self.run(*args, **kwargs)\n File >>>>>>>> \"/usr/lib/python2.7/site-packages/pulp/server/managers/content/upload.py\", >>>>>>>> line 223, in import_uploaded_unit\n unit_type=unit_type_id, >>>>>>>> summary=result['summary'], >>>>>>>> details=result['details']\nPulpCodedException: >>>>>>>> The importer docker_importer indicated a failed response when uploading >>>>>>>> docker_manifest unit to repository docker_centos7.\n", >>>>>>>> >>>>>>>> "error": { >>>>>>>> "code": "PLP0047", >>>>>>>> "data": { >>>>>>>> "unit_type": "docker_manifest", >>>>>>>> "importer_id": "docker_importer", >>>>>>>> "repo_id": "docker_centos7", >>>>>>>> "details": {}, >>>>>>>> "summary": [ >>>>>>>> "Cannot resolve field \"digest\"" >>>>>>>> ] >>>>>>>> }, >>>>>>>> "description": "The importer docker_importer indicated a failed >>>>>>>> response when uploading docker_manifest unit to repository >>>>>>>> docker_centos7.", >>>>>>>> >>>>>>>> I tried both v2s2 and v2s1, both don't work. I tried 2 different >>>>>>>> Dockerfiles and both produce the same error. >>>>>>>> >>>>>>>> Is it an issue in Buildah or in Pulp? Or am I doing something wrong? >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> -- >>>>>>>> Konstantin Khankin >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Ханкин Константин >>>>>>> _______________________________________________ >>>>>>> Pulp-list mailing list >>>>>>> Pulp-list@redhat.com >>>>>>> https://www.redhat.com/mailman/listinfo/pulp-list >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Konstantin Khankin >>>>> _______________________________________________ >>>>> Pulp-list mailing list >>>>> Pulp-list@redhat.com >>>>> https://www.redhat.com/mailman/listinfo/pulp-list >>>> >>>> >>> >>> -- >>> Konstantin Khankin >>> >> >> >> -- >> Ханкин Константин >> _______________________________________________ >> Pulp-list mailing list >> Pulp-list@redhat.com >> https://www.redhat.com/mailman/listinfo/pulp-list > > -- Ханкин Константин
_______________________________________________ Pulp-list mailing list Pulp-list@redhat.com https://www.redhat.com/mailman/listinfo/pulp-list