Regarding how I've made progress, I forgot to mention that I also had to set
oreg_url=<local registry url>/openshift/origin-${component}:${version} otherwise the ansible script still tried to pull openshift/origin-node:v3.10 from docker.io instead of from my local registry On Wed, Sep 12, 2018 at 3:47 PM Alexander Bartilla < alexander.barti...@cloudwerkstatt.com> wrote: > Hi, > > This seems to be a general problem with the missing image. There is an > open issue on github. > > https://github.com/openshift/origin/issues/20676 > > Sounds really strange, especially due to the fact that this issue was > opened around 27 days ago... > > I‘m going to try to replicate the issues as soon as I got some spare time > on my hands. > > Is there anyone who experienced similar issues with the 3.10 install? > > > On Wed 12. Sep 2018 at 21:37, Peter Heitman <pe...@heitman.us> wrote: > >> Thanks to Alexander, I found out that a major part of my problem is that >> my nodes have a poor internet connection and pulling images from >> docker.io is either slow or docker.io reports that the manifest is not >> found. Pulling the images locally, pushing them to a local registry and >> changing system_images_registry to my local registry helped a lot. >> >> However, it seems to consistently fail the first time I run >> deploy_cluster.yml (the control plane pods do not come up completely - they >> come up, become ready and then are deleted and started over again in a >> cycle every 5 seconds or so). If I run deploy_cluster.yml again (without >> changing anything) the deploy seems to go better the second time. >> >> I am unable to enable metrics. First, the ansible installer seems to want >> to get the metrics images with the tag v3.10.0 which doesn't exist. I tried >> pulling them down, labeling latest as v3.10.0 and pushing them to my local >> registry, but the image for openshift/origin-metrics-schema-installer >> doesn't seem to exist with any label. >> >> Anyway, thanks again Alexander - this is significant progress even though >> I'm definitely not ready to move off of 3.9.0 yet >> >> On Tue, Sep 11, 2018 at 1:42 PM Peter Heitman <pe...@heitman.us> wrote: >> >>> Thanks for the reply. I was pinning the release only because I was >>> updating a working inventory from 3.9 and forgot that I had pinned that >>> release to avoid upgrading to 3.10. I've updated the inventory to set >>> openshift_release="3.10" and commented out openshift_image_tag and >>> openshift_pkg_version so that the ansible scripts will derive the correct >>> values. I have re-run the installer using a fresh version of the master and >>> minion VMs (CentOS 7.5 with docker installed). I get the same error. The >>> output of systemctl status origin-node on the master is: >>> >>> ● origin-node.service - OpenShift Node >>> Loaded: loaded (/etc/systemd/system/origin-node.service; enabled; >>> vendor preset: disabled) >>> Active: active (running) since Tue 2018-09-11 10:31:51 PDT; 3min 29s >>> ago >>> Docs: https://github.com/openshift/origin >>> Main PID: 21183 (hyperkube) >>> CGroup: /system.slice/origin-node.service >>> └─21183 /usr/bin/hyperkube kubelet --v=2 --address=0.0.0.0 >>> --allow-privileged=true --anonymous-auth=true >>> --authentication-token-webhook=true >>> --authentication-token-webhook-cache-ttl=5m --authorization-mode=Webhook >>> --authorization-webhook-cache-authorized-ttl=5m >>> --authorization-webhook-cache-unauthorized-ttl=5m >>> --bootstrap-kubeconfig=/etc/origin/node/bootstrap.kubeconfig >>> --cadvisor-port=0 --cert-dir=/etc/origin/node/certificates >>> --cgroup-driver=systemd --client-ca-file=/etc/origin/node/client-ca.crt >>> --cluster-dns=10.93.233.126 --cluster-domain=cluster.local >>> --container-runtime-endpoint=/var/run/dockershim.sock --containerized=false >>> --enable-controller-attach-detach=true >>> --experimental-dockershim-root-directory=/var/lib/dockershim >>> --fail-swap-on=false >>> --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true >>> --file-check-frequency=0s --healthz-bind-address= --healthz-port=0 >>> --host-ipc-sources=api --host-ipc-sources=file --host-network-sources=api >>> --host-network-sources=file --host-pid-sources=api --host-pid-sources=file >>> --hostname-override= --http-check-frequency=0s >>> --image-service-endpoint=/var/run/dockershim.sock >>> --iptables-masquerade-bit=0 --kubeconfig=/etc/origin/node/node.kubeconfig >>> --max-pods=250 --network-plugin=cni --node-ip= --pod-infra-container-image= >>> docker.io/openshift/origin-pod:v3.10.0 >>> --pod-manifest-path=/etc/origin/node/pods --port=10250 --read-only-port=0 >>> --register-node=true --root-dir=/var/lib/origin/openshift.local.volumes >>> --rotate-certificates=true --tls-cert-file= >>> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 >>> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 >>> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 >>> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 >>> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 >>> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 >>> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 >>> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 >>> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA >>> --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA >>> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA >>> --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA >>> --tls-cipher-suites=TLS_RSA_WITH_AES_128_GCM_SHA256 >>> --tls-cipher-suites=TLS_RSA_WITH_AES_256_GCM_SHA384 >>> --tls-cipher-suites=TLS_RSA_WITH_AES_128_CBC_SHA >>> --tls-cipher-suites=TLS_RSA_WITH_AES_256_CBC_SHA >>> --tls-min-version=VersionTLS12 --tls-private-key-file= >>> >>> Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:17.667696 21183 reflector.go:205] >>> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: >>> Failed to list *v1.Pod: Get >>> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >>> dial tcp 10.93.233.126:8443: getsockopt: connection refused >>> Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:17.668264 21183 reflector.go:205] >>> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: >>> Failed to list *v1.Node: Get >>> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >>> dial tcp 10.93.233.126:8443: getsockopt: connection refused >>> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: W0911 >>> 10:35:18.518516 21183 cni.go:171] Unable to update cni config: No >>> networks found in /etc/cni/net.d >>> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:18.518716 21183 kubelet.go:2143] Container runtime network not >>> ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: >>> network plugin is not ready: cni config uninitialized >>> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:18.667894 21183 reflector.go:205] >>> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: >>> Failed to list *v1.Service: Get >>> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&resourceVersion=0: >>> dial tcp 10.93.233.126:8443: getsockopt: connection refused >>> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:18.669169 21183 reflector.go:205] >>> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: >>> Failed to list *v1.Pod: Get >>> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >>> dial tcp 10.93.233.126:8443: getsockopt: connection refused >>> Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:18.670127 21183 reflector.go:205] >>> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: >>> Failed to list *v1.Node: Get >>> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >>> dial tcp 10.93.233.126:8443: getsockopt: connection refused >>> Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:19.669734 21183 reflector.go:205] >>> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: >>> Failed to list *v1.Service: Get >>> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&resourceVersion=0: >>> dial tcp 10.93.233.126:8443: getsockopt: connection refused >>> Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:19.670769 21183 reflector.go:205] >>> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: >>> Failed to list *v1.Pod: Get >>> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >>> dial tcp 10.93.233.126:8443: getsockopt: connection refused >>> Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 >>> 10:35:19.671644 21183 reflector.go:205] >>> github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: >>> Failed to list *v1.Node: Get >>> https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: >>> dial tcp 10.93.233.126:8443: getsockopt: connection refused >>> >>> >>> On Tue, Sep 11, 2018 at 10:41 AM Alexander Bartilla < >>> alexander.barti...@cloudwerkstatt.com> wrote: >>> >>>> Hi Peter, >>>> >>>> Is there a reason behind pinning the release, image_tag and pkg_version >>>> variables to this release version? I would recommend you use just 3.10, >>>> this will ensure that you get the latest version of Openshift installed >>>> >>>> Futhermore I found several bugreports with this issue: >>>> >>>> https://github.com/openshift/openshift-ansible/issues/7967 >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1568583 >>>> https://bugzilla.redhat.com/show_bug.cgi?id=1568450#c7 >>>> >>>> Some more logs from the node would help to troubleshoot the problem. >>>> >>>> Best regards, >>>> Alexander >>>> >>>> On Tue, Sep 11, 2018 at 3:50 PM, Peter Heitman <pe...@heitman.us> >>>> wrote: >>>> >>>>> I am attempting to use the openshift-ansible installer for 3.10 to >>>>> deploy openshift on 1 master and 3 minions. I am using the same inventory >>>>> I >>>>> have been using for 3.9 with the changes shown below. I'm consistently >>>>> hitting a problem with the control plane pods not appearing. Looking in to >>>>> it, it seems that the cni plugin is not being configured properly. From >>>>> systemctl status origin-node, I see the following: >>>>> >>>>> E0911 06:19:25.821170 18922 kubelet.go:2143] Container runtime >>>>> network not ready: NetworkReady=false reason:NetworkPluginNotReady >>>>> message:docker: network plugin is not ready: cni config uninitialized >>>>> >>>>> Is there something I need to add to my 3.10 inventory to address this? >>>>> Are there other workarounds? >>>>> >>>>> - openshift_release=v3.9.0 >>>>> + openshift_release=v3.10.0 >>>>> >>>>> - openshift_image_tag=v3.9.0 >>>>> - openshift_image_tag=v3.10.0 >>>>> + openshift_pkg_version=-3.10.0 >>>>> + openshift_pkg_version=-3.9.0 >>>>> >>>>> - openshift_metrics_image_version=v3.9 >>>>> + openshift_metrics_image_version=v3.10 >>>>> >>>>> - [masters] >>>>> - <master fqdn> openshift_node_labels="{' >>>>> node-role.kubernetes.io/master': 'true', ' >>>>> node-role.kubernetes.io/infra': 'true'}" openshift_schedulable=true >>>>> >>>>> + [masters] >>>>> + <master fqdn> >>>>> >>>>> + [masters:vars] >>>>> + #openshift_node_group_name="node-config-master" >>>>> + openshift_node_group_name="node-config-master-infra" >>>>> + openshift_schedulable=true >>>>> >>>>> - [compute-nodes] >>>>> - <minion1 fqdn> openshift_node_labels="{' >>>>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>>>> - <minion2 fqdn> openshift_node_labels="{' >>>>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>>>> - <minion3 fqdn> openshift_node_labels="{' >>>>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>>>> >>>>> + [compute-nodes] >>>>> + <minion1 fqdn> >>>>> + <minion2 fqdn> >>>>> + <minion3 fqdn> >>>>> >>>>> + [compute-nodes:vars] >>>>> + openshift_node_group_name="node-config-compute" >>>>> + openshift_schedulable=true >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> users@lists.openshift.redhat.com >>>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Alexander Bartilla >>>> >>>> IT-Consultant >>>> >>>> Cloudwerkstatt GmbH - Lassallestraße 7b >>>> <https://maps.google.com/?q=Lassallestra%C3%9Fe+7b&entry=gmail&source=g> >>>> – A-1020 Wien >>>> >>>> +43-660-8989058 <+43%20660%208989058> >>>> >>>> alexander.barti...@cloudwerkstatt.com >>>> >>>> >>>> *[image: id:image001.png@01D24B57.D1D08F70]* >>>> >>>> ------------------------------ >>>> >>>> Cloudwerkstatt GmbH - Lassallestraße 7b >>>> <https://maps.google.com/?q=Lassallestra%C3%9Fe+7b&entry=gmail&source=g> >>>> - A-1020 Wien - ATU68384759 - FN408516i - Handelsgericht Wien >>>> >>>> >>>> -- > > Alexander Bartilla > > IT-Consultant > > Cloudwerkstatt GmbH - Lassallestraße 7b > <https://maps.google.com/?q=Lassallestra%C3%9Fe+7b&entry=gmail&source=g> > – A-1020 Wien > > +43-660-8989058 <+43%20660%208989058> > > alexander.barti...@cloudwerkstatt.com > > > *[image: id:image001.png@01D24B57.D1D08F70]* > > ------------------------------ > > Cloudwerkstatt GmbH - Lassallestraße 7b > <https://maps.google.com/?q=Lassallestra%C3%9Fe+7b&entry=gmail&source=g> > - A-1020 Wien - ATU68384759 - FN408516i - Handelsgericht Wien > > >
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users