So you can reach one Service (DNS) but not another? I would start
with doing some tcpdump to see what packets are moving around.
On Mon, Apr 24, 2017 at 10:59 AM, Tim Hockin <thoc...@google.com> wrote:
> On Mon, Apr 24, 2017 at 8:59 AM, Rijie Song <rijie.s...@gmail.com> wrote:
>> Thanks for response, Tim.
>>
>> 1. What network driver are you using? kubenet? CNI + flannel? CNI +
>> weave? CNI + calico?
>>
>> CNI+flannel.
>>
>> flannel pod output on this particular node:
>>
>> [root@k8s manifests]# kubectl logs -f po/kube-flannel-ds-bn66x -n
>> kube-system -c kube-flannel
>> I0424 06:33:46.210053 1 kube.go:109] Waiting 10m0s for node controller
>> to sync
>> I0424 06:33:46.210339 1 kube.go:289] starting kube subnet manager
>> I0424 06:33:47.218505 1 kube.go:116] Node controller sync successful
>> I0424 06:33:47.218556 1 main.go:132] Installing signal handlers
>> I0424 06:33:47.218656 1 manager.go:136] Determining IP address of
>> default interface
>> I0424 06:33:47.219642 1 manager.go:149] Using interface with name
>> ens160 and address 172.172.10.32
>> I0424 06:33:47.219673 1 manager.go:166] Defaulting external address to
>> interface address (172.172.10.32)
>> I0424 06:33:47.517408 1 ipmasq.go:47] Adding iptables rule: -s
>> 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
>> I0424 06:33:47.524679 1 ipmasq.go:47] Adding iptables rule: -s
>> 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
>> I0424 06:33:47.530320 1 ipmasq.go:47] Adding iptables rule: ! -s
>> 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
>> I0424 06:33:47.535263 1 manager.go:250] Lease acquired: 172.10.2.0/24
>> I0424 06:33:47.536092 1 network.go:58] Watching for L3 misses
>> I0424 06:33:47.536117 1 network.go:66] Watching for new subnet leases
>>
>>
>> 2. What is your Service cluster IP range?
>>
>> - --service-cluster-ip-range=10.96.0.0/12
>
> /12 is a LARGE service range - it allows 1 million Service IPs in your
> cluster. That's unusual, and may cause problems elsewhere.
>
>> 3. Can you ping from the pod to its own Node IP?
>>
>> Yes.
>>
>> [root@tulip-saas-xnode-2720274701-3fmqd /]# ping 172.172.10.32
>> PING 172.172.10.32 (172.172.10.32) 56(84) bytes of data.
>> 64 bytes from 172.172.10.32: icmp_seq=1 ttl=64 time=0.108 ms
>> 64 bytes from 172.172.10.32: icmp_seq=2 ttl=64 time=0.076 ms
>>
>> 4. Can you ping from the pod to a different Node IP?
>>
>> No.
>>
>> [root@tulip-saas-xnode-2720274701-3fmqd /]# ping 172.172.10.31
>> PING 172.172.10.31 (172.172.10.31) 56(84) bytes of data.
>> ^C
>>
>>
>>
>> On Mon, Apr 24, 2017 at 11:20 PM, 'Tim Hockin' via Kubernetes user
>> discussion and Q&A <kubernetes-users@googlegroups.com> wrote:
>>>
>>> What network driver are you using? kubenet? CNI + flannel? CNI +
>>> weave? CNI + calico?
>>>
>>> What is your Service cluster IP range?
>>>
>>> Can you ping from the pod to its own Node IP?
>>>
>>> Can you ping from the pod to a different Node IP?
>>>
>>> On Mon, Apr 24, 2017 at 6:29 AM, Roger Song <rijie.s...@gmail.com> wrote:
>>> > Hi all,
>>> >
>>> > Kindly help me review this issue. Thanks!
>>> >
>>> > [ Description ]
>>> >
>>> > I am newbie to k8s, recently setup k8s cluster on top of CentOS 7.3 with
>>> > kubeadm 1.6.1.
>>> >
>>> > Master: k8s
>>> > Minions: host01, host02, host03
>>> >
>>> > In one of pods(po/tulip-saas-xnode), I tried to make connection from pod
>>> > to
>>> > external rds service(172.172.10.16:3306). That's the reason I setup
>>> > service&endpoint "tulip-saas-db2" manually, as follows:
>>> > ===========
>>> > # kubectl get service tulip-saas-db2 -o yaml
>>> > apiVersion: v1
>>> > kind: Service
>>> > metadata:
>>> > creationTimestamp: 2017-04-24T07:46:10Z
>>> > name: tulip-saas-db2
>>> > namespace: default
>>> > resourceVersion: "905529"
>>> > selfLink: /api/v1/namespaces/default/services/tulip-saas-db2
>>> > uid: 153b3520-28c2-11e7-a272-000c29235036
>>> > spec:
>>> > clusterIP: 10.111.128.117
>>> > ports:
>>> > - port: 3306
>>> > protocol: TCP
>>> > targetPort: 3306
>>> > sessionAffinity: None
>>> > type: ClusterIP
>>> > status:
>>> > loadBalancer: {}
>>> >
>>> > # kubectl get endpoints tulip-saas-db2 -o yaml
>>> > apiVersion: v1
>>> > kind: Endpoints
>>> > metadata:
>>> > creationTimestamp: 2017-04-24T07:46:10Z
>>> > name: tulip-saas-db2
>>> > namespace: default
>>> > resourceVersion: "905533"
>>> > selfLink: /api/v1/namespaces/default/endpoints/tulip-saas-db2
>>> > uid: 15552d0d-28c2-11e7-a272-000c29235036
>>> > subsets:
>>> > - addresses:
>>> > - ip: 172.172.10.16
>>> > ports:
>>> > - port: 3306
>>> > protocol: TCP
>>> >
>>> > ==========
>>> >
>>> >
>>> > I'm able to connect to the port in host02 OS via cluster IP.
>>> > ==========
>>> > [root@host02 .kube]# iptables-save | grep tulip-saas-db2
>>> > -A KUBE-SEP-TS2EMOGZXA7V27BD -s 172.172.10.16/32 -m comment --comment
>>> > "default/tulip-saas-db2:" -j KUBE-MARK-MASQ
>>> > -A KUBE-SEP-TS2EMOGZXA7V27BD -p tcp -m comment --comment
>>> > "default/tulip-saas-db2:" -m tcp -j DNAT --to-destination
>>> > 172.172.10.16:3306
>>> > -A KUBE-SERVICES ! -s 172.10.0.0/16 -d 10.111.128.117/32 -p tcp -m
>>> > comment
>>> > --comment "default/tulip-saas-db2: cluster IP" -m tcp --dport 3306 -j
>>> > KUBE-MARK-MASQ
>>> > -A KUBE-SERVICES -d 10.111.128.117/32 -p tcp -m comment --comment
>>> > "default/tulip-saas-db2: cluster IP" -m tcp --dport 3306 -j
>>> > KUBE-SVC-ASAFJW2B6372ZEVA
>>> > -A KUBE-SVC-ASAFJW2B6372ZEVA -m comment --comment
>>> > "default/tulip-saas-db2:"
>>> > -j KUBE-SEP-TS2EMOGZXA7V27BD
>>> > [root@host02 .kube]# telnet 10.111.128.117 3306
>>> > Trying 10.111.128.117...
>>> > Connected to 10.111.128.117.
>>> > Escape character is '^]'.
>>> > =====
>>> >
>>> > The pod is running on host02
>>> > ====
>>> > # kubectl get pod --all-namespaces -o wide | grep tulip
>>> > default tulip-saas-xnode-3216045024-ctctp 1/1
>>> > Running
>>> > 1 8h 172.10.2.22 host02.corp.mooit.net
>>> > ====
>>> >
>>> > Inside the pod, service name can be resolved. However, I'm not able to
>>> > connect to the port.
>>> > ===
>>> > [root@tulip-saas-xnode-3216045024-ctctp /]# nslookup tulip-saas-db2
>>> > Server: 10.96.0.10
>>> > Address: 10.96.0.10#53
>>> >
>>> > Name: tulip-saas-db2.default.svc.cluster.local
>>> > Address: 10.111.128.117
>>> >
>>> > [root@tulip-saas-xnode-3216045024-ctctp /]# telnet tulip-saas-db2 3306
>>> > Trying 10.111.128.117...
>>> > ^C
>>> > ===
>>> >
>>> > kube-proxy logs in host02 doesn't give any message. I tried to delete
>>> > the
>>> > pod, and let DS create it again, no lucky. No remarkable messages in
>>> > /var/log/message.
>>> > ======
>>> > [root@k8s manifests]# kubectl logs -f po/kube-proxy-p279k -n kube-system
>>> > I0424 12:37:24.220402 1 server.go:225] Using iptables Proxier.
>>> > I0424 12:37:24.301205 1 server.go:249] Tearing down userspace
>>> > rules.
>>> > I0424 12:37:24.433983 1 conntrack.go:81] Set sysctl
>>> > 'net/netfilter/nf_conntrack_max' to 131072
>>> > I0424 12:37:24.435683 1 conntrack.go:66] Setting conntrack
>>> > hashsize to
>>> > 32768
>>> > I0424 12:37:24.436164 1 conntrack.go:81] Set sysctl
>>> > 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
>>> > I0424 12:37:24.436217 1 conntrack.go:81] Set sysctl
>>> > 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
>>> > =======
>>> >
>>> > I tried to follow
>>> >
>>> > https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/,
>>> > but failed to understand in which layer the problem takes place.
>>> > firewalld
>>> > is disabled in all nodes.
>>> >
>>> > Had tried some RBAC stuff for kube-proxy account referring to
>>> > https://github.com/uruddarraju/kubernetes-rbac-policies
>>> >
>>> >
>>> > [ env ]
>>> > # kubeadm version
>>> > kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1",
>>> > GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e",
>>> > GitTreeState:"clean",
>>> > BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc",
>>> > Platform:"linux/amd64"}
>>> >
>>> > # kubectl get all --all-namespaces
>>> > NAMESPACE NAME READY
>>> > STATUS RESTARTS AGE
>>> > default po/busybox 1/1
>>> > Running 1 3h
>>> > default po/http-svc-zzj1q 1/1
>>> > Running 1 4d
>>> > default po/nginx-deployment-4234284026-04wb3 1/1
>>> > Running 2 3d
>>> > default po/nginx-deployment-4234284026-pdvml 1/1
>>> > Running 1 3d
>>> > default po/tulip-saas-xnode-3216045024-ctctp 1/1
>>> > Running 1 8h
>>> > kube-system po/default-http-backend-2198840601-7wdbk 1/1
>>> > Running 2 4d
>>> > kube-system po/etcd-k8s.corp.mooit.net 1/1
>>> > Running 11 7d
>>> > kube-system po/kube-apiserver-k8s.corp.mooit.net 1/1
>>> > Running 6 6d
>>> > kube-system po/kube-controller-manager-k8s.corp.mooit.net 1/1
>>> > Running 13 7d
>>> > kube-system po/kube-dns-3913472980-mtml5 3/3
>>> > Running 96 7d
>>> > kube-system po/kube-flannel-ds-57crg 2/2
>>> > Running 7 7d
>>> > kube-system po/kube-flannel-ds-bn66x 2/2
>>> > Running 4 7d
>>> > kube-system po/kube-flannel-ds-wxj4d 2/2
>>> > Running 3 7d
>>> > kube-system po/kube-flannel-ds-xk9wh 2/2
>>> > Running 56 7d
>>> > kube-system po/kube-proxy-mp6xr 1/1
>>> > Running 10 7d
>>> > kube-system po/kube-proxy-p279k 1/1
>>> > Running 0 7m
>>> > kube-system po/kube-proxy-qqdvd 1/1
>>> > Running 2 7d
>>> > kube-system po/kube-proxy-vjmnw 1/1
>>> > Running 1 7d
>>> > kube-system po/kube-scheduler-k8s.corp.mooit.net 1/1
>>> > Running 13 7d
>>> > kube-system po/kubernetes-dashboard-915795657-wf3fp 1/1
>>> > Running 2 6d
>>> > kube-system po/nginx-ingress-lb-0q6n8 1/1
>>> > Running 1 2d
>>> > kube-system po/nginx-ingress-lb-20km8 1/1
>>> > Running 2 2d
>>> > kube-system po/nginx-ingress-lb-fk7nd 1/1
>>> > Running 1 2d
>>> > kube-system po/nginx-ingress-lb-q0z4c 1/1
>>> > Running 1 2d
>>> >
>>> > NAMESPACE NAME DESIRED CURRENT READY AGE
>>> > default rc/http-svc 1 1 1 4d
>>> >
>>> > NAMESPACE NAME CLUSTER-IP EXTERNAL-IP
>>> > PORT(S) AGE
>>> > default svc/http-svc 10.109.111.193 <nodes>
>>> > 80:30301/TCP 4d
>>> > default svc/kubernetes 10.96.0.1 <none>
>>> > 443/TCP 7d
>>> > default svc/nginx-svc 10.105.48.156 <nodes>
>>> > 80:30302/TCP 3d
>>> > default svc/tulip-saas-db2 10.111.128.117 <none>
>>> > 3306/TCP 5h
>>> > default svc/tulip-saas-xnode 10.106.241.164 <nodes>
>>> > 80:30189/TCP 1d
>>> > kube-system svc/default-http-backend 10.98.17.92 <none>
>>> > 80/TCP 4d
>>> > kube-system svc/kube-dns 10.96.0.10 <none>
>>> > 53/UDP,53/TCP 7d
>>> > kube-system svc/kubernetes-dashboard 10.106.75.115 <nodes>
>>> > 80:32416/TCP 7d
>>> >
>>> > NAMESPACE NAME DESIRED CURRENT
>>> > UP-TO-DATE
>>> > AVAILABLE AGE
>>> > default deploy/nginx-deployment 2 2 2
>>> > 2 3d
>>> > default deploy/tulip-saas-xnode 1 1 1
>>> > 1 8h
>>> > kube-system deploy/default-http-backend 1 1 1
>>> > 1 4d
>>> > kube-system deploy/kube-dns 1 1 1
>>> > 1 7d
>>> > kube-system deploy/kubernetes-dashboard 1 1 1
>>> > 1 7d
>>> >
>>> > NAMESPACE NAME DESIRED CURRENT
>>> > READY
>>> > AGE
>>> > default rs/nginx-deployment-4234284026 2 2 2
>>> > 3d
>>> > default rs/tulip-saas-xnode-3216045024 1 1 1
>>> > 8h
>>> > kube-system rs/default-http-backend-2198840601 1 1 1
>>> > 4d
>>> > kube-system rs/kube-dns-3913472980 1 1 1
>>> > 7d
>>> > kube-system rs/kubernetes-dashboard-915795657 1 1 1
>>> > 6d
>>> > [root@k8s manifests]# kubectl get ds --all-namespaces
>>> > NAMESPACE NAME DESIRED CURRENT READY
>>> > UP-TO-DATE
>>> > AVAILABLE NODE-SELECTOR AGE
>>> > kube-system kube-flannel-ds 4 4 4 4
>>> > 4 beta.kubernetes.io/arch=amd64 7d
>>> > kube-system kube-proxy 4 4 4 4
>>> > 4 <none> 7d
>>> > kube-system nginx-ingress-lb 4 4 4 4
>>> > 4 <none> 2d
>>> >
>>> > # kubectl get ing --all-namespaces
>>> > NAMESPACE NAME HOSTS ADDRESS
>>> > PORTS AGE
>>> > default ng k8s.corp.mooit.net 172.172.10.23...
>>> > 80, 443 3d
>>> > default test-http k8s.corp.mooit.net 172.172.10.23...
>>> > 80, 443 3d
>>> > default tulip-saas-xnode xnode.svr.mooit.net 172.172.10.23...
>>> > 80, 443 1d
>>> > kube-system dashboard k8s.corp.mooit.net 172.172.10.23...
>>> > 80, 443 3d
>>> >
>>> >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "Kubernetes user discussion and Q&A" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send
>>> > an
>>> > email to kubernetes-users+unsubscr...@googlegroups.com.
>>> > To post to this group, send email to kubernetes-users@googlegroups.com.
>>> > Visit this group at https://groups.google.com/group/kubernetes-users.
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "Kubernetes user discussion and Q&A" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/kubernetes-users/zQB4eS5BaGs/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> kubernetes-users+unsubscr...@googlegroups.com.
>>> To post to this group, send email to kubernetes-users@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/kubernetes-users.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
>>
>> --
>> Regards
>> Rijie Song
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kubernetes-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to kubernetes-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
