Hi Mike, > service tokens can't come through to nodes because kubelet tries to talk to > the api server through the api's "advertised ip address", which defaults to > the default route, which is shunted.
This seems wrong. Kubelet has a master address that is *NOT* dependent on Services. If that is not being used in all cases, it seems to be a problem. > Is there any reason why kube-proxy cannot have a mode (or default behavior) > to route default/kubernetes traffic to the same endpoint _it_ is connecting > to for api service I'm not sure if/how this plays with things like HA masters, and it makes the "specialness" of the kubernetes Service leak even further. It seems strictly simpler to set the flag - that's what it was intended for.. On Tue, Jan 2, 2018 at 2:00 PM, <mvene...@gmail.com> wrote: > This behavior puzzles me. API service is definitely a unique service, but > this issue can be tough debug, and causes a problem if some traffic needs to > be sent through a security device for monitoring. > > Let me explain: > - Assume there is no direct API connectivity between worker nodes and > control plane nodes. All API access goes through a security device and/or > external load balancer (say ELB) > - One could tell kubelet to connect to api server through a load balancer / > security device - and kubelet will seem to function fine. > - kube-proxy can spin up and seem to be working fine > - service tokens can't come through to nodes because kubelet tries to talk > to the api server through the api's "advertised ip address", which defaults > to the default route, which is shunted. > - User can't figure out issue, results in (╯°□°）╯︵ ┻━┻) > > It happened to me, and the only fix for me was to set a valid advertise ip > address and allow traffic to flow to it, despite me setting --master on > kube-proxy > > Is there any reason why kube-proxy cannot have a mode (or default behavior) > to route default/kubernetes traffic to the same endpoint _it_ is connecting > to for api service? Otherwise we leave people with no other option than to > have direct connectivity to api server node ports. > > Mike > > > On Wednesday, May 31, 2017 at 12:38:53 PM UTC-4, reza....@gmail.com wrote: >> On Wednesday, May 31, 2017 at 11:04:38 AM UTC-5, Tim Hockin wrote: >> > This being the kubernetes Service, the value is coming from Endpoints, >> > which is being written by your apiserver. By default, it chooses the >> > interface with a default route. If that is wrong, look at the >> > `--advertise-address` flag. >> > >> > On Wed, May 31, 2017 at 8:33 AM, <reza.ra...@gmail.com> wrote: >> > > >> > > Kubernetes version: 1.6.3 >> > > >> > > I have following interfaces on my vagrant machine. >> > > >> > > enp0s3 Link encap:Ethernet HWaddr 08:00:27:ee:32:98 >> > > inet addr:10.0.2.15 Bcast:10.0.2.255 Mask:255.255.255.0 >> > > ... >> > > >> > > enp0s8 Link encap:Ethernet HWaddr 08:00:27:88:a1:e8 >> > > inet addr:10.0.15.11 Bcast:10.0.15.255 Mask:255.255.255.0 >> > > ... >> > > >> > > I have deployed gcr.io/google_containers/kube-proxy-amd64:v1.6.3 on the >> > > cluster. >> > > >> > > vagrant@node1:/vagrant/kubeadm$ kubectl get po -n kube-system >> > > NAME READY STATUS >> > > RESTARTS AGE >> > > etcd-node1 1/1 Running >> > > 0 11m >> > > kube-apiserver-node1 1/1 Running 0 >> > > 10m >> > > kube-controller-manager-node1 1/1 Running 0 >> > > 11m >> > > kube-proxy-95pq2 1/1 Running 0 >> > > 1m >> > > kube-scheduler-node1 1/1 Running 0 >> > > 11m >> > > >> > > >> > > vagrant@node1:/vagrant/kubeadm$ sudo iptables-save |grep 443 >> > > >> > > -A KUBE-SEP-OGNOLD2JUSLFPOMZ -p tcp -m comment --comment >> > > "default/kubernetes:https" -m recent --set --name >> > > KUBE-SEP-OGNOLD2JUSLFPOMZ --mask 255.255.255.255 --rsource -m tcp -j >> > > DNAT --to-destination 10.0.2.15:6443 >> > > >> > > -A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment >> > > "default/kubernetes:https cluster IP" -m tcp --dport 443 -j >> > > KUBE-SVC-NPX46M4PTMTKRN6Y >> > > >> > > --------- >> > > >> > > The problem the iptable rule says " DNAT --to-destination >> > > 10.0.2.15:6443", it should be set to "DNAT --to-destination >> > > 10.0.15.11:6443" -- the interface. >> > > >> > > is there anyway I can force to use 10.0.15.11 instead of 10.0.2.15. >> > > >> > > In 1.5.2 Kube version, the IP table rule uses 10.0.15.11. >> > > >> > > thanks in advance >> > > >> > > -Reza >> > > >> > > -- >> > > You received this message because you are subscribed to the Google >> > > Groups "Kubernetes user discussion and Q&A" group. >> > > To unsubscribe from this group and stop receiving emails from it, send >> > > an email to kubernetes-users+unsubscr...@googlegroups.com. >> > > To post to this group, send email to kubernetes-users@googlegroups.com. >> > > Visit this group at https://groups.google.com/group/kubernetes-users. >> > > For more options, visit https://groups.google.com/d/optout. >> >> thank u very much. I was missing --advertise-address flag from >> kube-apiserver.yaml on my kube 1.6.3 installation. >> >> -Reza > > -- > You received this message because you are subscribed to the Google Groups > "Kubernetes user discussion and Q&A" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to kubernetes-users+unsubscr...@googlegroups.com. > To post to this group, send email to kubernetes-users@googlegroups.com. > Visit this group at https://groups.google.com/group/kubernetes-users. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group. To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscr...@googlegroups.com. To post to this group, send email to kubernetes-users@googlegroups.com. Visit this group at https://groups.google.com/group/kubernetes-users. For more options, visit https://groups.google.com/d/optout.
