Thanks to Alexander, I found out that a major part of my problem is that my nodes have a poor internet connection and pulling images from docker.io is either slow or docker.io reports that the manifest is not found. Pulling the images locally, pushing them to a local registry and changing system_images_registry to my local registry helped a lot.
However, it seems to consistently fail the first time I run deploy_cluster.yml (the control plane pods do not come up completely - they come up, become ready and then are deleted and started over again in a cycle every 5 seconds or so). If I run deploy_cluster.yml again (without changing anything) the deploy seems to go better the second time. I am unable to enable metrics. First, the ansible installer seems to want to get the metrics images with the tag v3.10.0 which doesn't exist. I tried pulling them down, labeling latest as v3.10.0 and pushing them to my local registry, but the image for openshift/origin-metrics-schema-installer doesn't seem to exist with any label. Anyway, thanks again Alexander - this is significant progress even though I'm definitely not ready to move off of 3.9.0 yet On Tue, Sep 11, 2018 at 1:42 PM Peter Heitman <pe...@heitman.us> wrote: > Thanks for the reply. I was pinning the release only because I was > updating a working inventory from 3.9 and forgot that I had pinned that > release to avoid upgrading to 3.10. I've updated the inventory to set > openshift_release="3.10" and commented out openshift_image_tag and > openshift_pkg_version so that the ansible scripts will derive the correct > values. I have re-run the installer using a fresh version of the master and > minion VMs (CentOS 7.5 with docker installed). I get the same error. The > output of systemctl status origin-node on the master is: > > ● origin-node.service - OpenShift Node > Loaded: loaded (/etc/systemd/system/origin-node.service; enabled; > vendor preset: disabled) > Active: active (running) since Tue 2018-09-11 10:31:51 PDT; 3min 29s ago > Docs: https://github.com/openshift/origin > Main PID: 21183 (hyperkube) > CGroup: /system.slice/origin-node.service > └─21183 /usr/bin/hyperkube kubelet --v=2 --address=0.0.0.0 > --allow-privileged=true --anonymous-auth=true > --authentication-token-webhook=true > --authentication-token-webhook-cache-ttl=5m --authorization-mode=Webhook > --authorization-webhook-cache-authorized-ttl=5m > --authorization-webhook-cache-unauthorized-ttl=5m > --bootstrap-kubeconfig=/etc/origin/node/bootstrap.kubeconfig > --cadvisor-port=0 --cert-dir=/etc/origin/node/certificates > --cgroup-driver=systemd --client-ca-file=/etc/origin/node/client-ca.crt > --cluster-dns=10.93.233.126 --cluster-domain=cluster.local > --container-runtime-endpoint=/var/run/dockershim.sock --containerized=false > --enable-controller-attach-detach=true > --experimental-dockershim-root-directory=/var/lib/dockershim > --fail-swap-on=false > --feature-gates=RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true > --file-check-frequency=0s --healthz-bind-address= --healthz-port=0 > --host-ipc-sources=api --host-ipc-sources=file --host-network-sources=api > --host-network-sources=file --host-pid-sources=api --host-pid-sources=file > --hostname-override= --http-check-frequency=0s > --image-service-endpoint=/var/run/dockershim.sock > --iptables-masquerade-bit=0 --kubeconfig=/etc/origin/node/node.kubeconfig > --max-pods=250 --network-plugin=cni --node-ip= --pod-infra-container-image= > docker.io/openshift/origin-pod:v3.10.0 > --pod-manifest-path=/etc/origin/node/pods --port=10250 --read-only-port=0 > --register-node=true --root-dir=/var/lib/origin/openshift.local.volumes > --rotate-certificates=true --tls-cert-file= > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305 > --tls-cipher-suites=TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 > --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 > --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 > --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA > --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA > --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA > --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA > --tls-cipher-suites=TLS_RSA_WITH_AES_128_GCM_SHA256 > --tls-cipher-suites=TLS_RSA_WITH_AES_256_GCM_SHA384 > --tls-cipher-suites=TLS_RSA_WITH_AES_128_CBC_SHA > --tls-cipher-suites=TLS_RSA_WITH_AES_256_CBC_SHA > --tls-min-version=VersionTLS12 --tls-private-key-file= > > Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:17.667696 21183 reflector.go:205] > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: > Failed to list *v1.Pod: Get > https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: > dial tcp 10.93.233.126:8443: getsockopt: connection refused > Sep 11 10:35:17 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:17.668264 21183 reflector.go:205] > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: > Failed to list *v1.Node: Get > https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: > dial tcp 10.93.233.126:8443: getsockopt: connection refused > Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: W0911 > 10:35:18.518516 21183 cni.go:171] Unable to update cni config: No > networks found in /etc/cni/net.d > Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:18.518716 21183 kubelet.go:2143] Container runtime network not > ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: > network plugin is not ready: cni config uninitialized > Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:18.667894 21183 reflector.go:205] > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: > Failed to list *v1.Service: Get > https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&resourceVersion=0: > dial tcp 10.93.233.126:8443: getsockopt: connection refused > Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:18.669169 21183 reflector.go:205] > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: > Failed to list *v1.Pod: Get > https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: > dial tcp 10.93.233.126:8443: getsockopt: connection refused > Sep 11 10:35:18 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:18.670127 21183 reflector.go:205] > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: > Failed to list *v1.Node: Get > https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: > dial tcp 10.93.233.126:8443: getsockopt: connection refused > Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:19.669734 21183 reflector.go:205] > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:452: > Failed to list *v1.Service: Get > https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/services?limit=500&resourceVersion=0: > dial tcp 10.93.233.126:8443: getsockopt: connection refused > Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:19.670769 21183 reflector.go:205] > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: > Failed to list *v1.Pod: Get > https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/pods?fieldSelector=spec.nodeName%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: > dial tcp 10.93.233.126:8443: getsockopt: connection refused > Sep 11 10:35:19 ph67-dev-psh-oso310-master origin-node[21183]: E0911 > 10:35:19.671644 21183 reflector.go:205] > github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:461: > Failed to list *v1.Node: Get > https://ph67-dev-psh-oso310-master.pdx.hcl.com:8443/api/v1/nodes?fieldSelector=metadata.name%3Dph67-dev-psh-oso310-master&limit=500&resourceVersion=0: > dial tcp 10.93.233.126:8443: getsockopt: connection refused > > > On Tue, Sep 11, 2018 at 10:41 AM Alexander Bartilla < > alexander.barti...@cloudwerkstatt.com> wrote: > >> Hi Peter, >> >> Is there a reason behind pinning the release, image_tag and pkg_version >> variables to this release version? I would recommend you use just 3.10, >> this will ensure that you get the latest version of Openshift installed >> >> Futhermore I found several bugreports with this issue: >> >> https://github.com/openshift/openshift-ansible/issues/7967 >> https://bugzilla.redhat.com/show_bug.cgi?id=1568583 >> https://bugzilla.redhat.com/show_bug.cgi?id=1568450#c7 >> >> Some more logs from the node would help to troubleshoot the problem. >> >> Best regards, >> Alexander >> >> On Tue, Sep 11, 2018 at 3:50 PM, Peter Heitman <pe...@heitman.us> wrote: >> >>> I am attempting to use the openshift-ansible installer for 3.10 to >>> deploy openshift on 1 master and 3 minions. I am using the same inventory I >>> have been using for 3.9 with the changes shown below. I'm consistently >>> hitting a problem with the control plane pods not appearing. Looking in to >>> it, it seems that the cni plugin is not being configured properly. From >>> systemctl status origin-node, I see the following: >>> >>> E0911 06:19:25.821170 18922 kubelet.go:2143] Container runtime network >>> not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: >>> network plugin is not ready: cni config uninitialized >>> >>> Is there something I need to add to my 3.10 inventory to address this? >>> Are there other workarounds? >>> >>> - openshift_release=v3.9.0 >>> + openshift_release=v3.10.0 >>> >>> - openshift_image_tag=v3.9.0 >>> - openshift_image_tag=v3.10.0 >>> + openshift_pkg_version=-3.10.0 >>> + openshift_pkg_version=-3.9.0 >>> >>> - openshift_metrics_image_version=v3.9 >>> + openshift_metrics_image_version=v3.10 >>> >>> - [masters] >>> - <master fqdn> openshift_node_labels="{'node-role.kubernetes.io/master': >>> 'true', 'node-role.kubernetes.io/infra': 'true'}" >>> openshift_schedulable=true >>> >>> + [masters] >>> + <master fqdn> >>> >>> + [masters:vars] >>> + #openshift_node_group_name="node-config-master" >>> + openshift_node_group_name="node-config-master-infra" >>> + openshift_schedulable=true >>> >>> - [compute-nodes] >>> - <minion1 fqdn> openshift_node_labels="{' >>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>> - <minion2 fqdn> openshift_node_labels="{' >>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>> - <minion3 fqdn> openshift_node_labels="{' >>> node-role.kubernetes.io/compute': 'true'}" openshift_schedulable=true >>> >>> + [compute-nodes] >>> + <minion1 fqdn> >>> + <minion2 fqdn> >>> + <minion3 fqdn> >>> >>> + [compute-nodes:vars] >>> + openshift_node_group_name="node-config-compute" >>> + openshift_schedulable=true >>> >>> >>> _______________________________________________ >>> users mailing list >>> users@lists.openshift.redhat.com >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >>> >> >> >> -- >> >> Alexander Bartilla >> >> IT-Consultant >> >> Cloudwerkstatt GmbH - Lassallestraße 7b >> <https://maps.google.com/?q=Lassallestra%C3%9Fe+7b&entry=gmail&source=g> >> – A-1020 Wien >> >> +43-660-8989058 <+43%20660%208989058> >> >> alexander.barti...@cloudwerkstatt.com >> >> >> *[image: id:image001.png@01D24B57.D1D08F70]* >> >> ------------------------------ >> >> Cloudwerkstatt GmbH - Lassallestraße 7b >> <https://maps.google.com/?q=Lassallestra%C3%9Fe+7b&entry=gmail&source=g> >> - A-1020 Wien - ATU68384759 - FN408516i - Handelsgericht Wien >> >> >>
_______________________________________________ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
