Hi, This seems to be a general problem with the missing image. There is an open issue on github.
https://github.com/openshift/origin/issues/20676 Sounds really strange, especially due to the fact that this issue was opened around 27 days ago... I‘m going to try to replicate the issues as soon as I got some spare time on my hands. Is there anyone who experienced similar issues with the 3.10 install? On Wed 12. Sep 2018 at 21:37, Peter Heitman <pe...@heitman.us> wrote: > Thanks to Alexander, I found out that a major part of my problem is that > my nodes have a poor internet connection and pulling images from docker.io > is either slow or docker.io reports that the manifest is not found. > Pulling the images locally, pushing them to a local registry and changing > system_images_registry to my local registry helped a lot. > > However, it seems to consistently fail the first time I run > deploy_cluster.yml (the control plane pods do not come up completely - they > come up, become ready and then are deleted and started over again in a > cycle every 5 seconds or so). If I run deploy_cluster.yml again (without > changing anything) the deploy seems to go better the second time. > > I am unable to enable metrics. First, the ansible installer seems to want > to get the metrics images with the tag v3.10.0 which doesn't exist. I tried > pulling them down, labeling latest as v3.10.0 and pushing them to my local > registry, but the image for openshift/origin-metrics-schema-installer > doesn't seem to exist with any label. > > Anyway, thanks again Alexander - this is significant progress even though > I'm definitely not ready to move off of 3.9.0 yet > > On Tue, Sep 11, 2018 at 1:42 PM Peter Heitman <pe...@heitman.us> wrote: > >> Thanks for the reply. I was pinning the release only because I was >> updating a working inventory from 3.9 and forgot that I had pinned that >> release to avoid upgrading to 3.10. I've updated the inventory to set >> openshift_release="3.10" and commented out openshift_image_tag and >> openshift_pkg_version so that the ansible scripts will derive the correct >> values. I have re-run the installer using a fresh version of the master and >> minion VMs (CentOS 7.5 with docker installed). I get the same error. The >> output of systemctl status origin-node on the master is: >> >> ● origin-node.service - OpenShift Node >> Loaded: loaded (/etc/systemd/system/origin-node.service; enabled; >> vendor preset: disabled) >> Active: active (running) since Tue 2018-09-11 10:31:51 PDT; 3min 29s >> ago >> Docs: https://github.com/openshift/origin >> Main PID: 21183 (hyperkube) >> CGroup: /system.slice/origin-node.service >> └─21183 /usr/bin/hyperkube kubelet --v=2 --address=0.0.0.0 >> --allow-privileged=true --anonymous-auth=true >> --authentication-token-webhook=true >> --authentication-token-webhook-cache-ttl=5m --authorization-mode=Webhook >> --authorization-webhook-cache-authorized-ttl=5m >> --authorization-webhook-cache-unauthorized-ttl=5m >> --bootstrap-kubeconfig=/etc/origin/node/bootstrap.kubeconfig >> --cadvisor-port=0 --cert-dir=/etc/origin/node/certificates >> --cgroup-driver=systemd --client-ca-file=/etc/origin/node/client-ca.crt >> --cluster-dns=10.93.233.126 --cluster-domain=cluster.local >> --container-runtime-endpoint=/var/run/dockershim.sock --containerized=false >> --enable-controller-attach-detach=true >> --experimental-dockershim-root-directory=/var/lib/dockershim >> --fail-swap-on=false >> --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true >> --file-check-frequency=0s --healthz-bind-address= --healthz-port=0 >> --host-ipc-sources=api --host-ipc-sources=file --host-network-sources=api >> --host-network-sources=file --host-pid-sources=api --host-pid-sources=file >> --hostname-override= --http-check-frequency=0s >> --image-service-endpoint=/var/run/dockershim.sock >> --iptables-masquerade-bit=0 --kubeconfig=/etc/origin/node/node.kubeconfig >> --max-pods=250 --network-plugin=cni --node-ip= --pod-infra-container-image= >> docker.io/openshift/origin-pod:v3.10.0 >> --pod-manifest-path=/etc/origin/node/pods --port=10250 --read-only-port=0 >> --register-node=true --root-dir=/var/lib/origin/openshift.local.volumes >> --rotate-certificates=true --tls-cert-file= >> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 >> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 >> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 >> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 >> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 >> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 >> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 >> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 >> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA >> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA >> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA >> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA >> --tls-cipher-suites=TLS_RSA_WITH_AES_128_GCM_SHA256 >> --tls-cipher-suites=TLS_RSA_WITH_AES_256_GCM_SHA384 >> --tls-cipher-suites=TLS_RSA_WITH_AES_128_CBC_SHA >> --tls-cipher-suites=TLS_RSA_WITH_AES_256_CBC_SHA >> --tls-min-version=VersionTLS12 --tls-private-key-file= >> >> Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:17.667696 21183 reflector.go:205] >> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: >> Failed to list *v1.Pod: Get >> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >> dial tcp 10.93.233.126:8443: getsockopt: connection refused >> Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:17.668264 21183 reflector.go:205] >> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: >> Failed to list *v1.Node: Get >> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >> dial tcp 10.93.233.126:8443: getsockopt: connection refused >> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: W0911 >> 10:35:18.518516 21183 cni.go:171] Unable to update cni config: No >> networks found in /etc/cni/net.d >> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:18.518716 21183 kubelet.go:2143] Container runtime network not >> ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: >> network plugin is not ready: cni config uninitialized >> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:18.667894 21183 reflector.go:205] >> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: >> Failed to list *v1.Service: Get >> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&resourceVersion=0: >> dial tcp 10.93.233.126:8443: getsockopt: connection refused >> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:18.669169 21183 reflector.go:205] >> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: >> Failed to list *v1.Pod: Get >> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >> dial tcp 10.93.233.126:8443: getsockopt: connection refused >> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:18.670127 21183 reflector.go:205] >> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: >> Failed to list *v1.Node: Get >> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >> dial tcp 10.93.233.126:8443: getsockopt: connection refused >> Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:19.669734 21183 reflector.go:205] >> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: >> Failed to list *v1.Service: Get >> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&resourceVersion=0: >> dial tcp 10.93.233.126:8443: getsockopt: connection refused >> Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:19.670769 21183 reflector.go:205] >> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: >> Failed to list *v1.Pod: Get >> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >> dial tcp 10.93.233.126:8443: getsockopt: connection refused >> Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >> 10:35:19.671644 21183 reflector.go:205] >> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: >> Failed to list *v1.Node: Get >> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >> dial tcp 10.93.233.126:8443: getsockopt: connection refused >> >> >> On Tue, Sep 11, 2018 at 10:41 AM Alexander Bartilla < >> alexander.barti...@cloudwerkstatt.com> wrote: >> >>> Hi Peter, >>> >>> Is there a reason behind pinning the release, image_tag and pkg_version >>> variables to this release version? I would recommend you use just 3.10, >>> this will ensure that you get the latest version of Openshift installed >>> >>> Futhermore I found several bugreports with this issue: >>> >>> https://github.com/openshift/openshift-ansible/issues/7967 >>> https://bugzilla.redhat.com/show_bug.cgi?id=1568583 >>> https://bugzilla.redhat.com/show_bug.cgi?id=1568450#c7 >>> >>> Some more logs from the node would help to troubleshoot the problem. >>> >>> Best regards, >>> Alexander >>> >>> On Tue, Sep 11, 2018 at 3:50 PM, Peter Heitman <pe...@heitman.us> wrote: >>> >>>> I am attempting to use the openshift-ansible installer for 3.10 to >>>> deploy openshift on 1 master and 3 minions. I am using the same inventory I >>>> have been using for 3.9 with the changes shown below. I'm consistently >>>> hitting a problem with the control plane pods not appearing. Looking in to >>>> it, it seems that the cni plugin is not being configured properly. From >>>> systemctl status origin-node, I see the following: >>>> >>>> E0911 06:19:25.821170 18922 kubelet.go:2143] Container runtime >>>> network not ready: NetworkReady=false reason:NetworkPluginNotReady >>>> message:docker: network plugin is not ready: cni config uninitialized >>>> >>>> Is there something I need to add to my 3.10 inventory to address this? >>>> Are there other workarounds? >>>> >>>> - openshift_release=v3.9.0 >>>> + openshift_release=v3.10.0 >>>> >>>> - openshift_image_tag=v3.9.0 >>>> - openshift_image_tag=v3.10.0 >>>> + openshift_pkg_version=-3.10.0 >>>> + openshift_pkg_version=-3.9.0 >>>> >>>> - openshift_metrics_image_version=v3.9 >>>> + openshift_metrics_image_version=v3.10 >>>> >>>> - [masters] >>>> - <master fqdn> openshift_node_labels="{'node-role.kubernetes.io/master': >>>> 'true', 'node-role.kubernetes.io/infra': 'true'}" >>>> openshift_schedulable=true >>>> >>>> + [masters] >>>> + <master fqdn> >>>> >>>> + [masters:vars] >>>> + #openshift_node_group_name="node-config-master" >>>> + openshift_node_group_name="node-config-master-infra" >>>> + openshift_schedulable=true >>>> >>>> - [compute-nodes] >>>> - <minion1 fqdn> openshift_node_labels="{' >>>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>>> - <minion2 fqdn> openshift_node_labels="{' >>>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>>> - <minion3 fqdn> openshift_node_labels="{' >>>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>>> >>>> + [compute-nodes] >>>> + <minion1 fqdn> >>>> + <minion2 fqdn> >>>> + <minion3 fqdn> >>>> >>>> + [compute-nodes:vars] >>>> + openshift_node_group_name="node-config-compute" >>>> + openshift_schedulable=true >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> users@lists.openshift.redhat.com >>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>>> >>>> >>> >>> >>> -- >>> >>> Alexander Bartilla >>> >>> IT-Consultant >>> >>> Cloudwerkstatt GmbH - Lassallestraße 7b >>> <https://maps.google.com/?q=Lassallestra%C3%9Fe+7b&entry=gmail&source=g> >>> – A-1020 Wien >>> >>> +43-660-8989058 <+43%20660%208989058> >>> >>> alexander.barti...@cloudwerkstatt.com >>> >>> >>> *[image: id:image001.png@01D24B57.D1D08F70]* >>> >>> ------------------------------ >>> >>> Cloudwerkstatt GmbH - Lassallestraße 7b >>> <https://maps.google.com/?q=Lassallestra%C3%9Fe+7b&entry=gmail&source=g> >>> - A-1020 Wien - ATU68384759 - FN408516i - Handelsgericht Wien >>> >>> >>> -- Alexander Bartilla IT-Consultant Cloudwerkstatt GmbH - Lassallestraße 7b – A-1020 Wien +43-660-8989058 alexander.barti...@cloudwerkstatt.com *[image: id:image001.png@01D24B57.D1D08F70]* -- Cloudwerkstatt GmbH - Lassallestraße 7b - A-1020 Wien - ATU68384759 - FN408516i - Handelsgericht Wien
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
