Hi
I've encountered a pod network latency problem.
I run simple mysql query ('time mysql -e "..."') to outer
(not in k8s cluster) host and got following results:
1. From k8s-pod to outer host:
avg time: 60ms
worst time: 100ms
(So, running applications in pods has same latency when works to outer
resources)
2. From k8s-host (which contains pod above) to outer host:
avg time: 10ms
worst time: 20ms
Also:
- There are no any dependency on DNS: avg and worst times not changed
when mysql-cli connects to IP-address or hostname of outer host.
- There are enough free resources: typical k8s node have 56-cores CPU
with LA ~ 10 (CPU load only and no any I/O load), more than half free
memory,
and no threads/processes (user/kernel), which cosume up to 100% cpu time.
As a possible solution I divided CPUs between kernel and user processes
(via kernel parameter isolcpus and cgroups), but latency did not changes.
For now I perform more complex measurements, and trying to investigate
which subsytem (k8s, cni, iptables etc.) may leads to this behaviour.
Have you encountered such or similar behavior?
May you provide any tip for my investigations?
Would be very grateful for your help.
My k8s cluster:
kubelet version: 1.5.7
network type: calico, image: calico/cni:v1.5.5, quay.io/calico/node:v1.0.0
num of nodes: ~20
pods per node: ~30
OS: Debian GNU/Linux 8.6 (jessie)
Kernel: 4.9.0-0.bpo.2-amd64
Thanks!
--
You received this message because you are subscribed to the Google Groups
"Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to kubernetes-users+unsubscr...@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.
