On Mon, Apr 24, 2017 at 8:59 AM, Rijie Song <rijie.s...@gmail.com> wrote:
> Thanks for response, Tim.
>
> 1. What network driver are you using? kubenet? CNI + flannel? CNI +
> weave? CNI + calico?
>
> CNI+flannel.
I don't have first-hand up-to-date flannel notes...
> flannel pod output on this particular node:
>
> [root@k8s manifests]# kubectl logs -f po/kube-flannel-ds-bn66x -n
> kube-system -c kube-flannel
> I0424 06:33:46.210053 1 kube.go:109] Waiting 10m0s for node controller
> to sync
> I0424 06:33:46.210339 1 kube.go:289] starting kube subnet manager
> I0424 06:33:47.218505 1 kube.go:116] Node controller sync successful
> I0424 06:33:47.218556 1 main.go:132] Installing signal handlers
> I0424 06:33:47.218656 1 manager.go:136] Determining IP address of
> default interface
> I0424 06:33:47.219642 1 manager.go:149] Using interface with name
> ens160 and address 172.172.10.32
> I0424 06:33:47.219673 1 manager.go:166] Defaulting external address to
> interface address (172.172.10.32)
> I0424 06:33:47.517408 1 ipmasq.go:47] Adding iptables rule: -s
> 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
> I0424 06:33:47.524679 1 ipmasq.go:47] Adding iptables rule: -s
> 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
> I0424 06:33:47.530320 1 ipmasq.go:47] Adding iptables rule: ! -s
> 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
> I0424 06:33:47.535263 1 manager.go:250] Lease acquired: 172.10.2.0/24
> I0424 06:33:47.536092 1 network.go:58] Watching for L3 misses
> I0424 06:33:47.536117 1 network.go:66] Watching for new subnet leases
>
>
> 2. What is your Service cluster IP range?
>
> - --service-cluster-ip-range=10.96.0.0/12
>
> 3. Can you ping from the pod to its own Node IP?
>
> Yes.
>
> [root@tulip-saas-xnode-2720274701-3fmqd /]# ping 172.172.10.32
> PING 172.172.10.32 (172.172.10.32) 56(84) bytes of data.
> 64 bytes from 172.172.10.32: icmp_seq=1 ttl=64 time=0.108 ms
> 64 bytes from 172.172.10.32: icmp_seq=2 ttl=64 time=0.076 ms
>
> 4. Can you ping from the pod to a different Node IP?
>
> No.
can you ping from a pod to a different pod on the other node?
> [root@tulip-saas-xnode-2720274701-3fmqd /]# ping 172.172.10.31
> PING 172.172.10.31 (172.172.10.31) 56(84) bytes of data.
> ^C
>
>
>
> On Mon, Apr 24, 2017 at 11:20 PM, 'Tim Hockin' via Kubernetes user
> discussion and Q&A <kubernetes-users@googlegroups.com> wrote:
>>
>> What network driver are you using? kubenet? CNI + flannel? CNI +
>> weave? CNI + calico?
>>
>> What is your Service cluster IP range?
>>
>> Can you ping from the pod to its own Node IP?
>>
>> Can you ping from the pod to a different Node IP?
>>
>> On Mon, Apr 24, 2017 at 6:29 AM, Roger Song <rijie.s...@gmail.com> wrote:
>> > Hi all,
>> >
>> > Kindly help me review this issue. Thanks!
>> >
>> > [ Description ]
>> >
>> > I am newbie to k8s, recently setup k8s cluster on top of CentOS 7.3 with
>> > kubeadm 1.6.1.
>> >
>> > Master: k8s
>> > Minions: host01, host02, host03
>> >
>> > In one of pods(po/tulip-saas-xnode), I tried to make connection from pod
>> > to
>> > external rds service(172.172.10.16:3306). That's the reason I setup
>> > service&endpoint "tulip-saas-db2" manually, as follows:
>> > ===========
>> > # kubectl get service tulip-saas-db2 -o yaml
>> > apiVersion: v1
>> > kind: Service
>> > metadata:
>> > creationTimestamp: 2017-04-24T07:46:10Z
>> > name: tulip-saas-db2
>> > namespace: default
>> > resourceVersion: "905529"
>> > selfLink: /api/v1/namespaces/default/services/tulip-saas-db2
>> > uid: 153b3520-28c2-11e7-a272-000c29235036
>> > spec:
>> > clusterIP: 10.111.128.117
>> > ports:
>> > - port: 3306
>> > protocol: TCP
>> > targetPort: 3306
>> > sessionAffinity: None
>> > type: ClusterIP
>> > status:
>> > loadBalancer: {}
>> >
>> > # kubectl get endpoints tulip-saas-db2 -o yaml
>> > apiVersion: v1
>> > kind: Endpoints
>> > metadata:
>> > creationTimestamp: 2017-04-24T07:46:10Z
>> > name: tulip-saas-db2
>> > namespace: default
>> > resourceVersion: "905533"
>> > selfLink: /api/v1/namespaces/default/endpoints/tulip-saas-db2
>> > uid: 15552d0d-28c2-11e7-a272-000c29235036
>> > subsets:
>> > - addresses:
>> > - ip: 172.172.10.16
>> > ports:
>> > - port: 3306
>> > protocol: TCP
>> >
>> > ==========
>> >
>> >
>> > I'm able to connect to the port in host02 OS via cluster IP.
>> > ==========
>> > [root@host02 .kube]# iptables-save | grep tulip-saas-db2
>> > -A KUBE-SEP-TS2EMOGZXA7V27BD -s 172.172.10.16/32 -m comment --comment
>> > "default/tulip-saas-db2:" -j KUBE-MARK-MASQ
>> > -A KUBE-SEP-TS2EMOGZXA7V27BD -p tcp -m comment --comment
>> > "default/tulip-saas-db2:" -m tcp -j DNAT --to-destination
>> > 172.172.10.16:3306
>> > -A KUBE-SERVICES ! -s 172.10.0.0/16 -d 10.111.128.117/32 -p tcp -m
>> > comment
>> > --comment "default/tulip-saas-db2: cluster IP" -m tcp --dport 3306 -j
>> > KUBE-MARK-MASQ
>> > -A KUBE-SERVICES -d 10.111.128.117/32 -p tcp -m comment --comment
>> > "default/tulip-saas-db2: cluster IP" -m tcp --dport 3306 -j
>> > KUBE-SVC-ASAFJW2B6372ZEVA
>> > -A KUBE-SVC-ASAFJW2B6372ZEVA -m comment --comment
>> > "default/tulip-saas-db2:"
>> > -j KUBE-SEP-TS2EMOGZXA7V27BD
>> > [root@host02 .kube]# telnet 10.111.128.117 3306
>> > Trying 10.111.128.117...
>> > Connected to 10.111.128.117.
>> > Escape character is '^]'.
>> > =====
>> >
>> > The pod is running on host02
>> > ====
>> > # kubectl get pod --all-namespaces -o wide | grep tulip
>> > default tulip-saas-xnode-3216045024-ctctp 1/1
>> > Running
>> > 1 8h 172.10.2.22 host02.corp.mooit.net
>> > ====
>> >
>> > Inside the pod, service name can be resolved. However, I'm not able to
>> > connect to the port.
>> > ===
>> > [root@tulip-saas-xnode-3216045024-ctctp /]# nslookup tulip-saas-db2
>> > Server: 10.96.0.10
>> > Address: 10.96.0.10#53
>> >
>> > Name: tulip-saas-db2.default.svc.cluster.local
>> > Address: 10.111.128.117
>> >
>> > [root@tulip-saas-xnode-3216045024-ctctp /]# telnet tulip-saas-db2 3306
>> > Trying 10.111.128.117...
>> > ^C
>> > ===
>> >
>> > kube-proxy logs in host02 doesn't give any message. I tried to delete
>> > the
>> > pod, and let DS create it again, no lucky. No remarkable messages in
>> > /var/log/message.
>> > ======
>> > [root@k8s manifests]# kubectl logs -f po/kube-proxy-p279k -n kube-system
>> > I0424 12:37:24.220402 1 server.go:225] Using iptables Proxier.
>> > I0424 12:37:24.301205 1 server.go:249] Tearing down userspace
>> > rules.
>> > I0424 12:37:24.433983 1 conntrack.go:81] Set sysctl
>> > 'net/netfilter/nf_conntrack_max' to 131072
>> > I0424 12:37:24.435683 1 conntrack.go:66] Setting conntrack
>> > hashsize to
>> > 32768
>> > I0424 12:37:24.436164 1 conntrack.go:81] Set sysctl
>> > 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
>> > I0424 12:37:24.436217 1 conntrack.go:81] Set sysctl
>> > 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
>> > =======
>> >
>> > I tried to follow
>> >
>> > https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/,
>> > but failed to understand in which layer the problem takes place.
>> > firewalld
>> > is disabled in all nodes.
>> >
>> > Had tried some RBAC stuff for kube-proxy account referring to
>> > https://github.com/uruddarraju/kubernetes-rbac-policies
>> >
>> >
>> > [ env ]
>> > # kubeadm version
>> > kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1",
>> > GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e",
>> > GitTreeState:"clean",
>> > BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc",
>> > Platform:"linux/amd64"}
>> >
>> > # kubectl get all --all-namespaces
>> > NAMESPACE NAME READY
>> > STATUS RESTARTS AGE
>> > default po/busybox 1/1
>> > Running 1 3h
>> > default po/http-svc-zzj1q 1/1
>> > Running 1 4d
>> > default po/nginx-deployment-4234284026-04wb3 1/1
>> > Running 2 3d
>> > default po/nginx-deployment-4234284026-pdvml 1/1
>> > Running 1 3d
>> > default po/tulip-saas-xnode-3216045024-ctctp 1/1
>> > Running 1 8h
>> > kube-system po/default-http-backend-2198840601-7wdbk 1/1
>> > Running 2 4d
>> > kube-system po/etcd-k8s.corp.mooit.net 1/1
>> > Running 11 7d
>> > kube-system po/kube-apiserver-k8s.corp.mooit.net 1/1
>> > Running 6 6d
>> > kube-system po/kube-controller-manager-k8s.corp.mooit.net 1/1
>> > Running 13 7d
>> > kube-system po/kube-dns-3913472980-mtml5 3/3
>> > Running 96 7d
>> > kube-system po/kube-flannel-ds-57crg 2/2
>> > Running 7 7d
>> > kube-system po/kube-flannel-ds-bn66x 2/2
>> > Running 4 7d
>> > kube-system po/kube-flannel-ds-wxj4d 2/2
>> > Running 3 7d
>> > kube-system po/kube-flannel-ds-xk9wh 2/2
>> > Running 56 7d
>> > kube-system po/kube-proxy-mp6xr 1/1
>> > Running 10 7d
>> > kube-system po/kube-proxy-p279k 1/1
>> > Running 0 7m
>> > kube-system po/kube-proxy-qqdvd 1/1
>> > Running 2 7d
>> > kube-system po/kube-proxy-vjmnw 1/1
>> > Running 1 7d
>> > kube-system po/kube-scheduler-k8s.corp.mooit.net 1/1
>> > Running 13 7d
>> > kube-system po/kubernetes-dashboard-915795657-wf3fp 1/1
>> > Running 2 6d
>> > kube-system po/nginx-ingress-lb-0q6n8 1/1
>> > Running 1 2d
>> > kube-system po/nginx-ingress-lb-20km8 1/1
>> > Running 2 2d
>> > kube-system po/nginx-ingress-lb-fk7nd 1/1
>> > Running 1 2d
>> > kube-system po/nginx-ingress-lb-q0z4c 1/1
>> > Running 1 2d
>> >
>> > NAMESPACE NAME DESIRED CURRENT READY AGE
>> > default rc/http-svc 1 1 1 4d
>> >
>> > NAMESPACE NAME CLUSTER-IP EXTERNAL-IP
>> > PORT(S) AGE
>> > default svc/http-svc 10.109.111.193 <nodes>
>> > 80:30301/TCP 4d
>> > default svc/kubernetes 10.96.0.1 <none>
>> > 443/TCP 7d
>> > default svc/nginx-svc 10.105.48.156 <nodes>
>> > 80:30302/TCP 3d
>> > default svc/tulip-saas-db2 10.111.128.117 <none>
>> > 3306/TCP 5h
>> > default svc/tulip-saas-xnode 10.106.241.164 <nodes>
>> > 80:30189/TCP 1d
>> > kube-system svc/default-http-backend 10.98.17.92 <none>
>> > 80/TCP 4d
>> > kube-system svc/kube-dns 10.96.0.10 <none>
>> > 53/UDP,53/TCP 7d
>> > kube-system svc/kubernetes-dashboard 10.106.75.115 <nodes>
>> > 80:32416/TCP 7d
>> >
>> > NAMESPACE NAME DESIRED CURRENT
>> > UP-TO-DATE
>> > AVAILABLE AGE
>> > default deploy/nginx-deployment 2 2 2
>> > 2 3d
>> > default deploy/tulip-saas-xnode 1 1 1
>> > 1 8h
>> > kube-system deploy/default-http-backend 1 1 1
>> > 1 4d
>> > kube-system deploy/kube-dns 1 1 1
>> > 1 7d
>> > kube-system deploy/kubernetes-dashboard 1 1 1
>> > 1 7d
>> >
>> > NAMESPACE NAME DESIRED CURRENT
>> > READY
>> > AGE
>> > default rs/nginx-deployment-4234284026 2 2 2
>> > 3d
>> > default rs/tulip-saas-xnode-3216045024 1 1 1
>> > 8h
>> > kube-system rs/default-http-backend-2198840601 1 1 1
>> > 4d
>> > kube-system rs/kube-dns-3913472980 1 1 1
>> > 7d
>> > kube-system rs/kubernetes-dashboard-915795657 1 1 1
>> > 6d
>> > [root@k8s manifests]# kubectl get ds --all-namespaces
>> > NAMESPACE NAME DESIRED CURRENT READY
>> > UP-TO-DATE
>> > AVAILABLE NODE-SELECTOR AGE
>> > kube-system kube-flannel-ds 4 4 4 4
>> > 4 beta.kubernetes.io/arch=amd64 7d
>> > kube-system kube-proxy 4 4 4 4
>> > 4 <none> 7d
>> > kube-system nginx-ingress-lb 4 4 4 4
>> > 4 <none> 2d
>> >
>> > # kubectl get ing --all-namespaces
>> > NAMESPACE NAME HOSTS ADDRESS
>> > PORTS AGE
>> > default ng k8s.corp.mooit.net 172.172.10.23...
>> > 80, 443 3d
>> > default test-http k8s.corp.mooit.net 172.172.10.23...
>> > 80, 443 3d
>> > default tulip-saas-xnode xnode.svr.mooit.net 172.172.10.23...
>> > 80, 443 1d
>> > kube-system dashboard k8s.corp.mooit.net 172.172.10.23...
>> > 80, 443 3d
>> >
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "Kubernetes user discussion and Q&A" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an
>> > email to kubernetes-users+unsubscr...@googlegroups.com.
>> > To post to this group, send email to kubernetes-users@googlegroups.com.
>> > Visit this group at https://groups.google.com/group/kubernetes-users.
>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "Kubernetes user discussion and Q&A" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/kubernetes-users/zQB4eS5BaGs/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> kubernetes-users+unsubscr...@googlegroups.com.
>> To post to this group, send email to kubernetes-users@googlegroups.com.
>> Visit this group at https://groups.google.com/group/kubernetes-users.
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> Regards
> Rijie Song
>
> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-users+unsubscr...@googlegroups.com.
> To post to this group, send email to kubernetes-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
