ah, so the problem is *not* communications within the cluster but talking out? This should completely be handled by a MASQUERADE rule in `iptables -t nat -L -n`, is that rule there and the same?
/MR On Thu, Jul 20, 2017 at 2:44 PM Jeremy Derr <[email protected]> wrote: > Unfortunately not. The only symptom is that any outgoing network > connections (and by outgoing, i mean to destinations outside of the > cluster) will eventually timeout. > > For instance, my service connects to a mysql server in AWS RDS. That > connection will timeout, unless I enable `hostNetworking: true` on the pod. > If i mark all of the "newer" nodes as unscheduleable and force this pod to > go to one of the older machines, it works fine even without hostNetworking. > > flannel's logs also do not impart any failure or error information. it > behaves like maybe there's a problem with how it set up iptables, but > comparing the table between a working and a non-working machine, i find no > difference. > > > On Thursday, July 20, 2017 at 3:03:54 AM UTC-5, Matthias Rampke wrote: > >> Are there any errors in either `kubectl describe pod` or the kubelet >> logs? Is the kubelet at the version you expect (it logs this on start)? >> 1.6.2 is pretty old, have you considered upgrading to 1.6.7? >> >> /MR >> >> On Wed, Jul 19, 2017, 22:02 Jeremy Derr <[email protected]> wrote: >> > We recently had an instance check failure on one of our k8s 1.6.2 nodes, >>> resulting in it being replaced by the AWS Autoscaler. The new node booted >>> and joined the cluster, but pods scheduled to it are unable to access >>> network resources outside of the cluster unless host network is enabled for >>> the pod. >>> >>> SSHing into the host OS, networking is properly configured and I can >>> perform any arbitrary network operation (apt-get update, curl ..., telnet, >>> ssh, etc etc) to any destination. >>> >>> Entering a pod that has hostNetworking: true enabled, this continues to >>> be the case: all network connections to any arbitrary destination on the >>> internet will be successful. >>> >>> Entering a pod that does not, however, I can only connect to other IPs >>> inside the cluster. >>> >>> $ kubectl get svc >>> NAME CLUSTER-IP EXTERNAL-IP >>> PORT(S) AGE >>> [ .. redacted .. ] >>> feature-flip 100.71.179.196 <nodes> >>> 9292:30800/TCP 27d >>> $ curl http://feature-flip:9292 >>> {"status": "ok"} >>> $ wget https://www.google.com/ >>> Connecting to www.google.com (172.217.7.228:80) >>> wget: can't connect to remote host (172.217.7.228): Connection timed >>> out >>> >>> On the same host, a pod with hostNetworking will succeed. I'm using >>> flannel networking, and the hosts that don't work are part of an EC2 ASG >>> with the hosts that do work. The base AMI and launchconfiguration are the >>> same. Comparing a variety of possible avenues of difference, I have found >>> no obvious difference between the working and non-working nodes. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Kubernetes user discussion and Q&A" group. >>> >> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To post to this group, send email to [email protected]. >> >> >>> Visit this group at https://groups.google.com/group/kubernetes-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
