On 5/16/2018 7:02 PM, Thomas Bellman wrote:
On 2018-05-16 18:06, Brian Rak wrote:
We've been trying to track down why our 5100's are dropping traffic
due to lack of buffer space, even with very low link utilization.
There's only 12 Mbyte of buffer space on the Trident II chip. If you
get 10 Gbit/s bursts simultaneously on two ports, contending for the
same output port, it will only take 10 ms to fill 12 Mbyte. (And of
those 12 Mbyte, 3 Mbyte is used for dedicated per-port buffers, so you
really only have ~9 Mbyte, so you would actually fill your buffers in
7.5-8 ms.)
Do you see any actual problems due to the dropped packets? Some people
would have you believe that TCP suffers horribly from a single dropped
packet, but reality is not quite that bad. So don't chase problems
that aren't there.
Our busiest ports have drop rates at about 1 in every 15'000 packets
(average over a few months), and so far we haven't noticed any TCP
performance problems related to that. (But I should note that most
of our traffic is long-distance, to and from sites at least several
milliseconds away from us, and often a 10-20 ms away.)
That said, for Trident II / Tomahawk level of buffer sizes, I think
it makes sense to configure them to have it all actually used, and
not wasted on the lossless queues.
You should probably also consider enabling cut-through forwarding, if
you haven't already done so. That should decrease the amount of buffer
space used, leaving more available for when contention happens.
/Bellman
A lot of what we run are game servers, which are very heavily UDP. They
can deal with some dropped packets, but it's not ideal.
We're not even doing 10gbit of traffic, so the buffers should last at
least a little bit.
Thanks for the tip about cut-through, we didn't have that enabled. Do
you happen to know if it works from a 10g port to a broken out 4x10g port?
It's annoying to be dropping packets with a bunch of unused buffer space.
_______________________________________________
juniper-nsp mailing list juniper-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/juniper-nsp
