On 6 Nov 2020, at 12:18, Jesper Dangaard Brouer wrote:
On Fri, 06 Nov 2020 10:18:10 +0100
"Thomas Rosenstein" <thomas.rosenst...@creamfinance.com> wrote:
I just tested 5.9.4 seems to also fix it partly, I have long
stretches where it looks good, and then some increases again. (3.10
Stock has them too, but not so high, rather 1-3 ms)
That you have long stretches where latency looks good is interesting
information. My theory is that your system have a periodic userspace
process that does a kernel syscall that takes too long, blocking
network card from processing packets. (Note it can also be a kernel
thread).
The weird part is, I first only updated router-02 and pinged to
router-04 (out of traffic flow), there I noticed these long stretches of
ok ping.
When I updated also router-03 and router-04, the old behaviour kind of
was back, this confused me.
Could this be related to netlink? I have gobgpd running on these
routers, which injects routes via netlink.
But the churn rate during the tests is very minimal, maybe 30 - 40
routes every second.
Otherwise we got: salt-minion, collectd, node_exporter, sshd
Another theory is the NIC HW does strange things, but it is not very
likely. E.g. delaying the packets before generating the IRQ
interrupt,
which hide it from my IRQ-to-softirq latency tool.
A question: What traffic control qdisc are you using on your system?
kernel 4+ uses pfifo, but there's no dropped packets
I have also tested with fq_codel, same behaviour and also no weirdness
in the packets queue itself
kernel 3.10 uses mq, and for the vlan interfaces noqueue
Here's the mail archive link for the question on lartc :
https://www.spinics.net/lists/lartc/msg23774.html
What you looked at the obvious case if any of your qdisc report a
large
backlog? (during the incidents)
as said above, nothing in qdiscs or reported
for example:
64 bytes from x.x.x.x: icmp_seq=10 ttl=64 time=0.169 ms
64 bytes from x.x.x.x: icmp_seq=11 ttl=64 time=5.53 ms
64 bytes from x.x.x.x: icmp_seq=12 ttl=64 time=9.44 ms
64 bytes from x.x.x.x: icmp_seq=13 ttl=64 time=0.167 ms
64 bytes from x.x.x.x: icmp_seq=14 ttl=64 time=3.88 ms
and then again:
64 bytes from x.x.x.x: icmp_seq=15 ttl=64 time=0.569 ms
64 bytes from x.x.x.x: icmp_seq=16 ttl=64 time=0.148 ms
64 bytes from x.x.x.x: icmp_seq=17 ttl=64 time=0.286 ms
64 bytes from x.x.x.x: icmp_seq=18 ttl=64 time=0.257 ms
64 bytes from x.x.x.x: icmp_seq=19 ttl=64 time=0.220 ms
These very low ping times tell me that you are measuring very close to
the target machine, which is good. Here on the bufferbloat list, we
are
always suspicious of network equipment being use in these kind of
setups. As experience tells us that this can be the cause of
bufferbloat latency.
yes, I'm just testing across two machines connected directly to the same
switch,
basically that's the best case scenario apart from direct connection.
I do also use a VLAN on this interface, so the pings go through the vlan
stack!
You mention some fs.com switches (your desc below signature), can you
tell us more?
It's a fs.com N5850-48S6Q
48 Port 10 Gbit + 6 port 40 Gbit
there are only 6 ports with 10 G in use, and 2 with 1 G, basically no
traffic
[...]
I have a feeling that maybe not all config options were correctly
moved
to the newer kernel.
Or there's a big bug somewhere ... (which would seem rather weird for
me
to be the first one to discover this)
I really appreciate that you report this. This is a periodic issue,
that often result in people not reporting this.
Even if we find this to be caused by some process running on your
system, or a bad config, this it is really important that we find the
root-cause.
I'll rebuild the 5.9 kernel on one of the 3.10 kernel and see if it
makes a difference ...
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
On Wed, 04 Nov 2020 16:23:12 +0100
Thomas Rosenstein via Bloat <bloat@lists.bufferbloat.net> wrote:
General Info:
Routers are connected between each other with 10G Mellanox Connect-X
cards via 10G SPF+ DAC cables via a 10G Switch from fs.com
Latency generally is around 0.18 ms between all routers (4).
Throughput is 9.4 Gbit/s with 0 retransmissions when tested with
iperf3.
2 of the 4 routers are connected upstream with a 1G connection
(separate
port, same network card)
All routers have the full internet routing tables, i.e. 80k entries
for
IPv6 and 830k entries for IPv4
Conntrack is disabled (-j NOTRACK)
Kernel 5.4.60 (custom)
2x Xeon X5670 @ 2.93 Ghz
96 GB RAM
_______________________________________________
Bloat mailing list
Bloat@lists.bufferbloat.net
https://lists.bufferbloat.net/listinfo/bloat
