Re: prune images error - 503
On Wed, Oct 26, 2016 at 12:09 AM, Lionel Orellana wrote: > It might be trying to go through the proxy. > If you have one configured it might, can you try without it? The logs from the docker registry does not look like are connected with pruning. Usually you'll see a direct messages on each side of the call, that's not happening in your case. > > From the master this works. > > curl -v --noproxy 172.19.38.253 http://172.19.38.253:5000/healthz > > On 26 October 2016 at 09:04, Lionel Orellana wrote: > >> Hi Maciej >> >> Here's what I got from the prune logs. >> >> I1026 08:54:32.200913 20095 prune.go:794] Using registry: >> 172.19.38.253:5000 >> I1026 08:54:32.200938 20095 prune.go:181] Trying https for >> 172.19.38.253:5000 >> I1026 08:54:32.200995 20095 round_trippers.go:296] GET >> https://172.19.38.253:5000/healthz >> I1026 08:54:32.201010 20095 round_trippers.go:303] Request Headers: >> I1026 08:54:32.201021 20095 round_trippers.go:306] Authorization: >> Basic dW51c2VkOk53cmwtZ1FkM0x3Vm9CQU9hai1VdnZkTkljWG1NQ2QwRUN3UFVI >> N1c1LU0= >> I1026 08:55:32.483709 20095 round_trippers.go:321] Response Status: in >> 60282 milliseconds >> I1026 08:55:32.483838 20095 prune.go:186] Error with https for >> 172.19.38.253:5000: Get https://172.19.38.253:5000/healthz: Service >> Unavailable >> I1026 08:55:32.483878 20095 prune.go:181] Trying http for >> 172.19.38.253:5000 >> I1026 08:55:32.483939 20095 round_trippers.go:296] GET >> http://172.19.38.253:5000/healthz >> I1026 08:55:32.483961 20095 round_trippers.go:303] Request Headers: >> I1026 08:55:32.483974 20095 round_trippers.go:306] Authorization: >> Basic dW51c2VkOk53cmwtZ1FkM0x3Vm9CQU9hai1VdnZkTkljWG1NQ2QwRUN3UFVI >> N1c1LU0= >> I1026 08:56:35.078825 20095 round_trippers.go:321] Response Status: 503 >> Service Unavailable in 62594 milliseconds >> I1026 08:56:35.078925 20095 prune.go:186] Error with http for >> 172.19.38.253:5000: unexpected status code 503 >> F1026 08:56:35.079066 20095 helpers.go:110] error: error communicating >> with registry: unexpected status code 503 >> -bash-4.2$ >> >> >> The address of the registry is fine. The registry is insecure. >> >> On the registry logs the following came up. Not sure if it's related to >> the prune but it happened at the very same time. >> >> time="2016-10-25T21:56:37.829909567Z" level=error msg="error authorizing >> context: authorization header required" go.version=go1.6.3 >> http.request.host="172.19.38.253:5000" >> http.request.id=7df9debf-45eb-48fa-ae07-57a09de9b3b4 >> http.request.method=GET http.request.remoteaddr="172.19.39.129:56350" >> http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.6.2 >> git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 >> os/linux arch/amd64" instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c >> 172.19.39.129 - - [25/Oct/2016:21:56:37 +] "GET >> /openshift/token?account=serviceaccount&scope=repository% >> 3Atransaction-authority-lite%2Ftransaction-auth-lite%3Apull HTTP/1.1" >> 200 1823 "" "docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported >> kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64" >> >> time="2016-10-25T21:56:37.841655976Z" level=info msg="response >> completed" go.version=go1.6.3 http.request.host="172.19.38.253:5000" >> http.request.id=213f9613-c430-4386-bef2-cb67a7069f52 >> http.request.method=GET http.request.remoteaddr="172.19.39.129:56352" >> http.request.uri="/openshift/token?account=serviceaccount&sc >> ope=repository%3Atransaction-authority-lite%2Ftransaction-auth-lite%3Apull" >> http.request.useragent="docker/1.10.3 go/go1.6.2 >> git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 >> os/linux arch/amd64" http.response.contenttype="application/json" >> http.response.duration=6.642711ms http.response.status=200 >> http.response.written=1823 instance.id=9e99450d-8197-4cce >> -ab17-1cf256d32a7c >> >> time="2016-10-25T21:56:37.844393921Z" level=debug msg="authorizing >> request" go.version=go1.6.3 http.request.host="172.19.38.253:5000" >> http.request.id=fdcf1a81-7b93-4981-a957-7417386cf7de >> http.request.method=GET http.request.remoteaddr="172.19.39.129:56354" >> http.request.uri="/v2/transaction-authority-lite/transaction >> -auth-lite/manifests/sha256:591f586760735ddf513b8bad2f98aaa7 >> 7d079050d6f0bd392eb4b93d6aa3e21b" http.request.useragent="docker/1.10.3 >> go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 >> os/linux arch/amd64" instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c >> vars.name="transaction-authority-lite/transaction-auth-lite" >> vars.reference="sha256:591f586760735ddf513b8bad2f98aaa77d079 >> 050d6f0bd392eb4b93d6aa3e21b" >> >> time="2016-10-25T21:56:37.844554174Z" level=debug msg="Origin auth: >> checking for access to repository:transaction-authori >> ty-lite/transaction-auth-lite:pull" go.version=go1.6.3 >> http.request.host="172.19.38.253:5000" >> http.request.id=fdcf1a81-7b93-4981-a957-7417386cf7de >> http.reques
Re: prune images error - 503
It might be trying to go through the proxy. >From the master this works. curl -v --noproxy 172.19.38.253 http://172.19.38.253:5000/healthz On 26 October 2016 at 09:04, Lionel Orellana wrote: > Hi Maciej > > Here's what I got from the prune logs. > > I1026 08:54:32.200913 20095 prune.go:794] Using registry: > 172.19.38.253:5000 > I1026 08:54:32.200938 20095 prune.go:181] Trying https for > 172.19.38.253:5000 > I1026 08:54:32.200995 20095 round_trippers.go:296] GET > https://172.19.38.253:5000/healthz > I1026 08:54:32.201010 20095 round_trippers.go:303] Request Headers: > I1026 08:54:32.201021 20095 round_trippers.go:306] Authorization: > Basic dW51c2VkOk53cmwtZ1FkM0x3Vm9CQU9hai1VdnZkTkljWG1NQ2QwRUN3UFVIN1c1LU0= > I1026 08:55:32.483709 20095 round_trippers.go:321] Response Status: in > 60282 milliseconds > I1026 08:55:32.483838 20095 prune.go:186] Error with https for > 172.19.38.253:5000: Get https://172.19.38.253:5000/healthz: Service > Unavailable > I1026 08:55:32.483878 20095 prune.go:181] Trying http for > 172.19.38.253:5000 > I1026 08:55:32.483939 20095 round_trippers.go:296] GET > http://172.19.38.253:5000/healthz > I1026 08:55:32.483961 20095 round_trippers.go:303] Request Headers: > I1026 08:55:32.483974 20095 round_trippers.go:306] Authorization: > Basic dW51c2VkOk53cmwtZ1FkM0x3Vm9CQU9hai1VdnZkTkljWG1NQ2QwRUN3UFVIN1c1LU0= > I1026 08:56:35.078825 20095 round_trippers.go:321] Response Status: 503 > Service Unavailable in 62594 milliseconds > I1026 08:56:35.078925 20095 prune.go:186] Error with http for > 172.19.38.253:5000: unexpected status code 503 > F1026 08:56:35.079066 20095 helpers.go:110] error: error communicating > with registry: unexpected status code 503 > -bash-4.2$ > > > The address of the registry is fine. The registry is insecure. > > On the registry logs the following came up. Not sure if it's related to > the prune but it happened at the very same time. > > time="2016-10-25T21:56:37.829909567Z" level=error msg="error authorizing > context: authorization header required" go.version=go1.6.3 > http.request.host="172.19.38.253:5000" > http.request.id=7df9debf-45eb-48fa-ae07-57a09de9b3b4 > http.request.method=GET http.request.remoteaddr="172.19.39.129:56350" > http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.6.2 > git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 os/linux > arch/amd64" instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c > 172.19.39.129 - - [25/Oct/2016:21:56:37 +] "GET > /openshift/token?account=serviceaccount&scope=repository%3Atransaction- > authority-lite%2Ftransaction-auth-lite%3Apull HTTP/1.1" 200 1823 "" > "docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported > kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64" > > time="2016-10-25T21:56:37.841655976Z" level=info msg="response completed" > go.version=go1.6.3 http.request.host="172.19.38.253:5000" http.request.id > =213f9613-c430-4386-bef2-cb67a7069f52 http.request.method=GET > http.request.remoteaddr="172.19.39.129:56352" > http.request.uri="/openshift/token?account=serviceaccount& > scope=repository%3Atransaction-authority-lite%2Ftransaction-auth-lite%3Apull" > http.request.useragent="docker/1.10.3 go/go1.6.2 > git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 os/linux > arch/amd64" http.response.contenttype="application/json" > http.response.duration=6.642711ms http.response.status=200 > http.response.written=1823 instance.id=9e99450d-8197- > 4cce-ab17-1cf256d32a7c > > time="2016-10-25T21:56:37.844393921Z" level=debug msg="authorizing > request" go.version=go1.6.3 http.request.host="172.19.38.253:5000" > http.request.id=fdcf1a81-7b93-4981-a957-7417386cf7de > http.request.method=GET http.request.remoteaddr="172.19.39.129:56354" > http.request.uri="/v2/transaction-authority-lite/transaction-auth-lite/ > manifests/sha256:591f586760735ddf513b8bad2f98aa > a77d079050d6f0bd392eb4b93d6aa3e21b" http.request.useragent="docker/1.10.3 > go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 > os/linux arch/amd64" instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c > vars.name="transaction-authority-lite/transaction-auth-lite" > vars.reference="sha256:591f586760735ddf513b8bad2f98aa > a77d079050d6f0bd392eb4b93d6aa3e21b" > > time="2016-10-25T21:56:37.844554174Z" level=debug msg="Origin auth: > checking for access to > repository:transaction-authority-lite/transaction-auth-lite:pull" > go.version=go1.6.3 http.request.host="172.19.38.253:5000" http.request.id > =fdcf1a81-7b93-4981-a957-7417386cf7de http.request.method=GET > http.request.remoteaddr="172.19.39.129:56354" http.request.uri="/v2/ > transaction-authority-lite/transaction-auth-lite/manifests/sha256: > 591f586760735ddf513b8bad2f98aaa77d079050d6f0bd392eb4b93d6aa3e21b" > http.request.useragent="docker/1.10.3 go/go1.6.2 > git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 os/linux > arch/amd64" instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c
Re: prune images error - 503
Hi Maciej Here's what I got from the prune logs. I1026 08:54:32.200913 20095 prune.go:794] Using registry: 172.19.38.253:5000 I1026 08:54:32.200938 20095 prune.go:181] Trying https for 172.19.38.253:5000 I1026 08:54:32.200995 20095 round_trippers.go:296] GET https://172.19.38.253:5000/healthz I1026 08:54:32.201010 20095 round_trippers.go:303] Request Headers: I1026 08:54:32.201021 20095 round_trippers.go:306] Authorization: Basic dW51c2VkOk53cmwtZ1FkM0x3Vm9CQU9hai1VdnZkTkljWG1NQ2QwRUN3UFVIN1c1LU0= I1026 08:55:32.483709 20095 round_trippers.go:321] Response Status: in 60282 milliseconds I1026 08:55:32.483838 20095 prune.go:186] Error with https for 172.19.38.253:5000: Get https://172.19.38.253:5000/healthz: Service Unavailable I1026 08:55:32.483878 20095 prune.go:181] Trying http for 172.19.38.253:5000 I1026 08:55:32.483939 20095 round_trippers.go:296] GET http://172.19.38.253:5000/healthz I1026 08:55:32.483961 20095 round_trippers.go:303] Request Headers: I1026 08:55:32.483974 20095 round_trippers.go:306] Authorization: Basic dW51c2VkOk53cmwtZ1FkM0x3Vm9CQU9hai1VdnZkTkljWG1NQ2QwRUN3UFVIN1c1LU0= I1026 08:56:35.078825 20095 round_trippers.go:321] Response Status: 503 Service Unavailable in 62594 milliseconds I1026 08:56:35.078925 20095 prune.go:186] Error with http for 172.19.38.253:5000: unexpected status code 503 F1026 08:56:35.079066 20095 helpers.go:110] error: error communicating with registry: unexpected status code 503 -bash-4.2$ The address of the registry is fine. The registry is insecure. On the registry logs the following came up. Not sure if it's related to the prune but it happened at the very same time. time="2016-10-25T21:56:37.829909567Z" level=error msg="error authorizing context: authorization header required" go.version=go1.6.3 http.request.host="172.19.38.253:5000" http.request.id=7df9debf-45eb-48fa-ae07-57a09de9b3b4 http.request.method=GET http.request.remoteaddr="172.19.39.129:56350" http.request.uri="/v2/" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64" instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c 172.19.39.129 - - [25/Oct/2016:21:56:37 +] "GET /openshift/token?account=serviceaccount&scope=repository%3Atransaction-authority-lite%2Ftransaction-auth-lite%3Apull HTTP/1.1" 200 1823 "" "docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64" time="2016-10-25T21:56:37.841655976Z" level=info msg="response completed" go.version=go1.6.3 http.request.host="172.19.38.253:5000" http.request.id=213f9613-c430-4386-bef2-cb67a7069f52 http.request.method=GET http.request.remoteaddr="172.19.39.129:56352" http.request.uri="/openshift/token?account=serviceaccount&scope=repository%3Atransaction-authority-lite%2Ftransaction-auth-lite%3Apull" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64" http.response.contenttype="application/json" http.response.duration=6.642711ms http.response.status=200 http.response.written=1823 instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c time="2016-10-25T21:56:37.844393921Z" level=debug msg="authorizing request" go.version=go1.6.3 http.request.host="172.19.38.253:5000" http.request.id=fdcf1a81-7b93-4981-a957-7417386cf7de http.request.method=GET http.request.remoteaddr="172.19.39.129:56354" http.request.uri="/v2/transaction-authority-lite/transaction-auth-lite/manifests/sha256:591f586760735ddf513b8bad2f98aaa77d079050d6f0bd392eb4b93d6aa3e21b" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64" instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c vars.name="transaction-authority-lite/transaction-auth-lite" vars.reference="sha256:591f586760735ddf513b8bad2f98aaa77d079050d6f0bd392eb4b93d6aa3e21b" time="2016-10-25T21:56:37.844554174Z" level=debug msg="Origin auth: checking for access to repository:transaction-authority-lite/transaction-auth-lite:pull" go.version=go1.6.3 http.request.host="172.19.38.253:5000" http.request.id=fdcf1a81-7b93-4981-a957-7417386cf7de http.request.method=GET http.request.remoteaddr="172.19.39.129:56354" http.request.uri="/v2/transaction-authority-lite/transaction-auth-lite/manifests/sha256:591f586760735ddf513b8bad2f98aaa77d079050d6f0bd392eb4b93d6aa3e21b" http.request.useragent="docker/1.10.3 go/go1.6.2 git-commit/2a93377-unsupported kernel/3.10.0-327.28.3.el7.x86_64 os/linux arch/amd64" instance.id=9e99450d-8197-4cce-ab17-1cf256d32a7c vars.name="transaction-authority-lite/transaction-auth-lite" vars.reference="sha256:591f586760735ddf513b8bad2f98aaa77d079050d6f0bd392eb4b93d6aa3e21b" time="2016-10-25T21:56:37.852266154Z" level=debug msg=GetImageManifest go.version=go1.6.3 http.request.host="172.19.38.253:5000" http.request.id=fdcf1a81-7b93-4981-a957-7417386cf7de http.request.method=GET http.request.remoteadd
Re: prune images error - 503
Upon pruning images this command reaches out to registry to actually remove them from there. Apparently, it has has problems reaching out to your registry. There are two flags you might want to check out --certificate-authority - which points to a CA when self-signed cert is used --registry-url - to specify which registry to talk to. Additionally you can increase the verbosity (--loglevel=7) and see the communication between oadm prune images and the server, including the registry. If still experiencing problems, please lemme know with that logs attached, so I could help more. Maciej On Tue, Oct 25, 2016 at 9:29 AM, Lionel Orellana wrote: > Hi. > > I'm trying to run > > oadm prune images --keep-tag-revisions=3 --keep-younger-than=24h --confirm > > but after a while I get > > error: error communicating with registry: unexpected status code 503 > > I am able to prune builds and deployments. prune images without --confirm > also returns Ok. > > I've been building a ~6GB image repeatedly in the last couple of days. > Seems like it's struggling to delete some of those? > > Thanks > > Lionel > > ___ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
prune images error - 503
Hi. I'm trying to run oadm prune images --keep-tag-revisions=3 --keep-younger-than=24h --confirm but after a while I get error: error communicating with registry: unexpected status code 503 I am able to prune builds and deployments. prune images without --confirm also returns Ok. I've been building a ~6GB image repeatedly in the last couple of days. Seems like it's struggling to delete some of those? Thanks Lionel ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
