Hi, I'm trying to install kubernetes on a vmware vm which in theory should be pretty easy. I have done several test installs using qemu and virtualbox and both of those works perfectly using this guide:
https://blog.alexellis.io/kubernetes-in-10-minutes/ (The only difference is that I'm installing kubernetes 1.9) and this way seems to be supported by several other documents and videos. Anyhow I have decided that I wanted to move my test into our vmware (6.5) environment but I have been trying to weeks now and I cannot figure out what is going wrong. I'm using # . /etc/os-release ; echo $VERSION 16.04.3 LTS (Xenial Xerus) # kubeadm version kubeadm version: &version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} and the command: # kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=10.78.0.7 --kubernetes-version stable-1.9 fails with the message: <quote> Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) - There is no internet connection, so the kubelet cannot pull the following control plane images: - gcr.io/google_containers/kube-apiserver-amd64:v1.9.2 - gcr.io/google_containers/kube-controller-manager-amd64:v1.9.2 - gcr.io/google_containers/kube-scheduler-amd64:v1.9.2 If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' couldn't initialize a Kubernetes cluster </quote> The VM has internet access and kubelet is running <quote> ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf Active: active (running) since Wed 2018-01-31 10:44:58 CET; 32min ago Docs: http://kubernetes.io/docs/ Main PID: 3039 (kubelet) Tasks: 14 Memory: 42.4M CPU: 27.121s CGroup: /system.slice/kubelet.service └─3039 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/n Jan 31 11:17:51 ramsley kubelet[3039]: E0131 11:17:51.131101 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:644 Jan 31 11:17:51 ramsley kubelet[3039]: E0131 11:17:51.131447 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&res Jan 31 11:17:51 ramsley kubelet[3039]: E0131 11:17:51.308097 3039 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "ramsley" not found Jan 31 11:17:52 ramsley kubelet[3039]: E0131 11:17:52.131562 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceV Jan 31 11:17:52 ramsley kubelet[3039]: E0131 11:17:52.133082 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:644 Jan 31 11:17:52 ramsley kubelet[3039]: E0131 11:17:52.134104 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&res Jan 31 11:17:53 ramsley kubelet[3039]: E0131 11:17:53.132590 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceV Jan 31 11:17:53 ramsley kubelet[3039]: E0131 11:17:53.133736 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:644 Jan 31 11:17:53 ramsley kubelet[3039]: E0131 11:17:53.134837 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&res Jan 31 11:17:53 ramsley kubelet[3039]: I0131 11:17:53.794945 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach </quote> the logs says: <quote> Jan 31 11:15:09 ramsley kubelet[3039]: E0131 11:15:09.032102 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:09 ramsley kubelet[3039]: E0131 11:15:09.092233 3039 event.go:209] Unable to write event: 'Patch https://10.78.0.7:6443/api/v1/namespaces/default/events/ramsley.150edbe4789d9039: dial tcp 10.78.0.7:6443: getsockopt: connection refused' (may retry after sleeping) Jan 31 11:15:09 ramsley kubelet[3039]: I0131 11:15:09.097406 3039 kuberuntime_manager.go:514] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.1.11 Command:[etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/etcd] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:etcd ReadOnly:false MountPath:/var/lib/etcd SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:2379,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it. Jan 31 11:15:09 ramsley kubelet[3039]: I0131 11:15:09.098368 3039 kuberuntime_manager.go:758] checking backoff for container "etcd" in pod "etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)" Jan 31 11:15:09 ramsley kubelet[3039]: I0131 11:15:09.098754 3039 kuberuntime_manager.go:768] Back-off 5m0s restarting failed container=etcd pod=etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320) Jan 31 11:15:09 ramsley kubelet[3039]: E0131 11:15:09.099063 3039 pod_workers.go:186] Error syncing pod 7278f85057e8bf5cb81c9f96d3b25320 ("etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)"), skipping: failed to "StartContainer" for "etcd" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=etcd pod=etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)" Jan 31 11:15:10 ramsley kubelet[3039]: E0131 11:15:10.031306 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:10 ramsley kubelet[3039]: E0131 11:15:10.032355 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:10 ramsley kubelet[3039]: E0131 11:15:10.033520 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:11 ramsley kubelet[3039]: E0131 11:15:11.032572 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:11 ramsley kubelet[3039]: E0131 11:15:11.034777 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:11 ramsley kubelet[3039]: E0131 11:15:11.035304 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:11 ramsley kubelet[3039]: E0131 11:15:11.295871 3039 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "ramsley" not found Jan 31 11:15:11 ramsley kubelet[3039]: I0131 11:15:11.794970 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 11:15:12 ramsley kubelet[3039]: E0131 11:15:12.033747 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:12 ramsley kubelet[3039]: E0131 11:15:12.035573 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:12 ramsley kubelet[3039]: E0131 11:15:12.036628 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:13 ramsley kubelet[3039]: E0131 11:15:13.034781 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:13 ramsley kubelet[3039]: E0131 11:15:13.036213 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:13 ramsley kubelet[3039]: E0131 11:15:13.037239 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:13 ramsley kubelet[3039]: I0131 11:15:13.833597 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 11:15:13 ramsley kubelet[3039]: I0131 11:15:13.836117 3039 kubelet_node_status.go:82] Attempting to register node ramsley Jan 31 11:15:13 ramsley kubelet[3039]: E0131 11:15:13.836732 3039 kubelet_node_status.go:106] Unable to register node "ramsley" with API server: Post https://10.78.0.7:6443/api/v1/nodes: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:14 ramsley kubelet[3039]: E0131 11:15:14.035782 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:14 ramsley kubelet[3039]: E0131 11:15:14.037031 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:14 ramsley kubelet[3039]: E0131 11:15:14.037925 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:15 ramsley kubelet[3039]: E0131 11:15:15.036998 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:15 ramsley kubelet[3039]: E0131 11:15:15.037786 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:15 ramsley kubelet[3039]: E0131 11:15:15.039447 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:16 ramsley kubelet[3039]: E0131 11:15:16.038248 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:16 ramsley kubelet[3039]: E0131 11:15:16.039922 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:16 ramsley kubelet[3039]: E0131 11:15:16.040682 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:17 ramsley kubelet[3039]: E0131 11:15:17.039472 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:17 ramsley kubelet[3039]: E0131 11:15:17.040714 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:17 ramsley kubelet[3039]: E0131 11:15:17.041720 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:18 ramsley kubelet[3039]: E0131 11:15:18.040553 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:18 ramsley kubelet[3039]: E0131 11:15:18.041354 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:18 ramsley kubelet[3039]: E0131 11:15:18.042369 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:19 ramsley kubelet[3039]: E0131 11:15:19.041627 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:19 ramsley kubelet[3039]: E0131 11:15:19.042629 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:19 ramsley kubelet[3039]: E0131 11:15:19.043758 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:19 ramsley kubelet[3039]: E0131 11:15:19.093221 3039 event.go:209] Unable to write event: 'Patch https://10.78.0.7:6443/api/v1/namespaces/default/events/ramsley.150edbe4789d9039: dial tcp 10.78.0.7:6443: getsockopt: connection refused' (may retry after sleeping) Jan 31 11:15:19 ramsley kubelet[3039]: E0131 11:15:19.093671 3039 event.go:144] Unable to write event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"ramsley.150edbe5a3257ec9", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"ramsley", UID:"ramsley", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"NodeHasSufficientMemory", Message:"Node ramsley status is now: NodeHasSufficientMemory", Source:v1.EventSource{Component:"kubelet", Host:"ramsley"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbe9481e7efc708c9, ext:5359525541, loc:(*time.Location)(0x58992c0)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbe9481e7efc708c9, ext:5359525541, loc:(*time.Location)(0x58992c0)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}' (retry limit exceeded!) Jan 31 11:15:19 ramsley kubelet[3039]: E0131 11:15:19.094825 3039 event.go:209] Unable to write event: 'Patch https://10.78.0.7:6443/api/v1/namespaces/default/events/ramsley.150edbe4789d9ad1: dial tcp 10.78.0.7:6443: getsockopt: connection refused' (may retry after sleeping) Jan 31 11:15:19 ramsley kubelet[3039]: I0131 11:15:19.794917 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 11:15:20 ramsley kubelet[3039]: E0131 11:15:20.042780 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:474: Failed to list *v1.Node: Get https://10.78.0.7:6443/api/v1/nodes?fieldSelector=metadata.name%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:20 ramsley kubelet[3039]: E0131 11:15:20.043597 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/kubelet.go:465: Failed to list *v1.Service: Get https://10.78.0.7:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:20 ramsley kubelet[3039]: E0131 11:15:20.044700 3039 reflector.go:205] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://10.78.0.7:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dramsley&limit=500&resourceVersion=0: dial tcp 10.78.0.7:6443: getsockopt: connection refused Jan 31 11:15:20 ramsley kubelet[3039]: I0131 11:15:20.117639 3039 kuberuntime_manager.go:514] Container {Name:kube-apiserver Image:gcr.io/google_containers/kube-apiserver-amd64:v1.9.2 Command:[kube-apiserver --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --service-cluster-ip-range=10.96.0.0/12 --service-account-key-file=/etc/kubernetes/pki/sa.pub --client-ca-file=/etc/kubernetes/pki/ca.crt --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --insecure-port=0 --enable-bootstrap-token-auth=true --allow-privileged=true --requestheader-group-headers=X-Remote-Group --requestheader-allowed-names=front-proxy-client --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --secure-port=6443 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota --requestheader-username-headers=X-Remote-User --advertise-address=10.78.0.7 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --authorization-mode=Node,RBAC --etcd-servers=http://127.0.0.1:2379] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[cpu:{i:{value:250 scale:-3} d:{Dec:<nil>} s:250m Format:DecimalSI}]} VolumeMounts:[{Name:k8s-certs ReadOnly:true MountPath:/etc/kubernetes/pki SubPath: MountPropagation:<nil>} {Name:ca-certs ReadOnly:true MountPath:/etc/ssl/certs SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/healthz,Port:6443,Host:10.78.0.7,Scheme:HTTPS,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePa Jan 31 11:15:20 ramsley kubelet[3039]: th:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it. Jan 31 11:15:20 ramsley kubelet[3039]: I0131 11:15:20.118940 3039 kuberuntime_manager.go:758] checking backoff for container "kube-apiserver" in pod "kube-apiserver-ramsley_kube-system(43959289de185d73b35d989a33228f0b)" Jan 31 11:15:20 ramsley kernel: aufs au_opts_verify:1597:dockerd[1812]: dirperm1 breaks the protection by the permission bits on the lower branch Jan 31 11:15:20 ramsley dockerd[1253]: time="2018-01-31T11:15:20.130618672+01:00" level=warning msg="Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/d221d5461f3cbe7b6d6fffbe72897cb3ba857b0e4d4c799e4c020e4606496a02-init: exec: \"auplink\": executable file not found in $PATH" Jan 31 11:15:20 ramsley kernel: aufs au_opts_verify:1597:dockerd[1812]: dirperm1 breaks the protection by the permission bits on the lower branch Jan 31 11:15:20 ramsley dockerd[1253]: time="2018-01-31T11:15:20.180530664+01:00" level=warning msg="Couldn't run auplink before unmount /var/lib/docker/aufs/mnt/d221d5461f3cbe7b6d6fffbe72897cb3ba857b0e4d4c799e4c020e4606496a02: exec: \"auplink\": executable file not found in $PATH" Jan 31 11:15:20 ramsley kernel: aufs au_opts_verify:1597:dockerd[1812]: dirperm1 breaks the protection by the permission bits on the lower branch Jan 31 11:15:20 ramsley dockerd[1253]: time="2018-01-31T11:15:20.408100180+01:00" level=warning msg="Unknown healthcheck type 'NONE' (expected 'CMD') in container 7605901afed28d062c69f8e4a12a4270a1b0710b2c980402fe84cfae7f117321" Jan 31 11:15:20 ramsley kubelet[3039]: I0131 11:15:20.435047 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 11:15:20 ramsley kubelet[3039]: I0131 11:15:20.837182 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 11:15:20 ramsley kubelet[3039]: I0131 11:15:20.839413 3039 kubelet_node_status.go:82] Attempting to register node ramsley Jan 31 11:15:21 ramsley kubelet[3039]: E0131 11:15:21.296541 3039 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "ramsley" not found Jan 31 11:15:23 ramsley kubelet[3039]: I0131 11:15:23.794879 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 11:15:24 ramsley kubelet[3039]: I0131 11:15:24.098988 3039 kuberuntime_manager.go:514] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.1.11 Command:[etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/etcd] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:etcd ReadOnly:false MountPath:/var/lib/etcd SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:2379,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it. Jan 31 11:15:24 ramsley kubelet[3039]: I0131 11:15:24.099991 3039 kuberuntime_manager.go:758] checking backoff for container "etcd" in pod "etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)" Jan 31 11:15:24 ramsley kubelet[3039]: I0131 11:15:24.100407 3039 kuberuntime_manager.go:768] Back-off 5m0s restarting failed container=etcd pod=etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320) Jan 31 11:15:24 ramsley kubelet[3039]: E0131 11:15:24.100707 3039 pod_workers.go:186] Error syncing pod 7278f85057e8bf5cb81c9f96d3b25320 ("etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)"), skipping: failed to "StartContainer" for "etcd" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=etcd pod=etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)" Jan 31 11:15:31 ramsley kubelet[3039]: E0131 11:15:31.297186 3039 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "ramsley" not found Jan 31 11:15:37 ramsley kubelet[3039]: I0131 11:15:37.794876 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach Jan 31 11:15:38 ramsley kubelet[3039]: I0131 11:15:38.097690 3039 kuberuntime_manager.go:514] Container {Name:etcd Image:gcr.io/google_containers/etcd-amd64:3.1.11 Command:[etcd --listen-client-urls=http://127.0.0.1:2379 --advertise-client-urls=http://127.0.0.1:2379 --data-dir=/var/lib/etcd] Args:[] WorkingDir: Ports:[] EnvFrom:[] Env:[] Resources:{Limits:map[] Requests:map[]} VolumeMounts:[{Name:etcd ReadOnly:false MountPath:/var/lib/etcd SubPath: MountPropagation:<nil>}] VolumeDevices:[] LivenessProbe:&Probe{Handler:Handler{Exec:nil,HTTPGet:&HTTPGetAction{Path:/health,Port:2379,Host:127.0.0.1,Scheme:HTTP,HTTPHeaders:[],},TCPSocket:nil,},InitialDelaySeconds:15,TimeoutSeconds:15,PeriodSeconds:10,SuccessThreshold:1,FailureThreshold:8,} ReadinessProbe:nil Lifecycle:nil TerminationMessagePath:/dev/termination-log TerminationMessagePolicy:File ImagePullPolicy:IfNotPresent SecurityContext:nil Stdin:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it. Jan 31 11:15:38 ramsley kubelet[3039]: I0131 11:15:38.098919 3039 kuberuntime_manager.go:758] checking backoff for container "etcd" in pod "etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)" Jan 31 11:15:38 ramsley kubelet[3039]: I0131 11:15:38.099397 3039 kuberuntime_manager.go:768] Back-off 5m0s restarting failed container=etcd pod=etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320) Jan 31 11:15:38 ramsley kubelet[3039]: E0131 11:15:38.099861 3039 pod_workers.go:186] Error syncing pod 7278f85057e8bf5cb81c9f96d3b25320 ("etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)"), skipping: failed to "StartContainer" for "etcd" with CrashLoopBackOff: "Back-off 5m0s restarting failed container=etcd pod=etcd-ramsley_kube-system(7278f85057e8bf5cb81c9f96d3b25320)" Jan 31 11:15:41 ramsley kubelet[3039]: E0131 11:15:41.297932 3039 eviction_manager.go:238] eviction manager: unexpected err: failed to get node info: node "ramsley" not found Jan 31 11:15:41 ramsley kubelet[3039]: I0131 11:15:41.377653 3039 kubelet_node_status.go:273] Setting node annotation to enable volume controller attach/detach </quote> I have tried comparing the vmware vm and the qemu images kernel modules in order to see if that makes a difference by loading (The qemu image has these loaded) 8021q veth vxlan ip_vs ip_set I have also tried to run it via strace but I don't see any errors that seems to stick out. In my crazy search for an answer I have even tried to disable mac filtering in vmware and allowed promiscuous mode on the vm but no luck Anyone got an idea on where to debug from here ? -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
