This behavior puzzles me. API service is definitely a unique service, but this issue can be tough debug, and causes a problem if some traffic needs to be sent through a security device for monitoring.
Let me explain: - Assume there is no direct API connectivity between worker nodes and control plane nodes. All API access goes through a security device and/or external load balancer (say ELB) - One could tell kubelet to connect to api server through a load balancer / security device - and kubelet will seem to function fine. - kube-proxy can spin up and seem to be working fine - service tokens can't come through to nodes because kubelet tries to talk to the api server through the api's "advertised ip address", which defaults to the default route, which is shunted. - User can't figure out issue, results in (╯°□°）╯︵ ┻━┻) It happened to me, and the only fix for me was to set a valid advertise ip address and allow traffic to flow to it, despite me setting --master on kube-proxy Is there any reason why kube-proxy cannot have a mode (or default behavior) to route default/kubernetes traffic to the same endpoint _it_ is connecting to for api service? Otherwise we leave people with no other option than to have direct connectivity to api server node ports. Mike On Wednesday, May 31, 2017 at 12:38:53 PM UTC-4, reza....@gmail.com wrote: > On Wednesday, May 31, 2017 at 11:04:38 AM UTC-5, Tim Hockin wrote: > > This being the kubernetes Service, the value is coming from Endpoints, > > which is being written by your apiserver. By default, it chooses the > > interface with a default route. If that is wrong, look at the > > `--advertise-address` flag. > > > > On Wed, May 31, 2017 at 8:33 AM, <reza.ra...@gmail.com> wrote: > > > > > > Kubernetes version: 1.6.3 > > > > > > I have following interfaces on my vagrant machine. > > > > > > enp0s3 Link encap:Ethernet HWaddr 08:00:27:ee:32:98 > > > inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0 > > > ... > > > > > > enp0s8 Link encap:Ethernet HWaddr 08:00:27:88:a1:e8 > > > inet addr:10.0.15.11 Bcast:10.0.15.255 Mask:255.255.255.0 > > > ... > > > > > > I have deployed gcr.io/google_containers/kube-proxy-amd64:v1.6.3 on the > > > cluster. > > > > > > vagrant@node1:/vagrant/kubeadm$ kubectl get po -n kube-system > > > NAME READY STATUS > > > RESTARTS AGE > > > etcd-node1 1/1 Running > > > 0 11m > > > kube-apiserver-node1 1/1 Running 0 > > > 10m > > > kube-controller-manager-node1 1/1 Running 0 > > > 11m > > > kube-proxy-95pq2 1/1 Running 0 > > > 1m > > > kube-scheduler-node1 1/1 Running 0 > > > 11m > > > > > > > > > vagrant@node1:/vagrant/kubeadm$ sudo iptables-save |grep 443 > > > > > > -A KUBE-SEP-OGNOLD2JUSLFPOMZ -p tcp -m comment --comment > > > "default/kubernetes:https" -m recent --set --name > > > KUBE-SEP-OGNOLD2JUSLFPOMZ --mask 255.255.255.255 --rsource -m tcp -j DNAT > > > --to-destination 10.0.2.15:6443 > > > > > > -A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment > > > "default/kubernetes:https cluster IP" -m tcp --dport 443 -j > > > KUBE-SVC-NPX46M4PTMTKRN6Y > > > > > > --------- > > > > > > The problem the iptable rule says " DNAT --to-destination > > > 10.0.2.15:6443", it should be set to "DNAT --to-destination > > > 10.0.15.11:6443" -- the interface. > > > > > > is there anyway I can force to use 10.0.15.11 instead of 10.0.2.15. > > > > > > In 1.5.2 Kube version, the IP table rule uses 10.0.15.11. > > > > > > thanks in advance > > > > > > -Reza > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Kubernetes user discussion and Q&A" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to kubernetes-users+unsubscr...@googlegroups.com. > > > To post to this group, send email to kubernetes-users@googlegroups.com. > > > Visit this group at https://groups.google.com/group/kubernetes-users. > > > For more options, visit https://groups.google.com/d/optout. > > thank u very much. I was missing --advertise-address flag from > kube-apiserver.yaml on my kube 1.6.3 installation. > > -Reza -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
