There's no free lunch, but it seems like you and others want one. :-)
Increasing these values just increases the depth of the kernel's packet
queue for the sync processes to consume as able. It doesn't mean they're
able, and accordingly, request response time will go up.
A healthy system that is able to keep up with the load you're throwing
at it should show a receive queue at +/- 0 most of the time, maybe with
some ephemeral spikes but generally trending around 0. If packets are
stacking up in the RecvQ, it means the SIP worker processes aren't
available enough to consume them all in a timely fashion.
Leaning on async won't help here if the workload is largely CPU-bound.
If it's largely bound over waiting on network I/O from external
services, it merely deputises the problem of notifying you when there's
a response from those services to the kernel. But - vitally - it won't
get your requests processed faster, and setup latency is a very
important consideration in real-time communications, especially from the
perspective of interoperability with the synchronous/circuit-switched PSTN.
In short, async isn't magic, and neither is increasing the receive
queue. It's simple thermodynamics; there's only so much CPU available,
and depending on the nature of the workload, throughput becomes more a
linear function of available CPU hardware threads, or less, but slower,
if it's largely I/O-bound.
The metaphor of a balloon is appropriate. You're pushing the problem
around by squeezing one part of the balloon, causing another to enlarge.
Various parts of the balloon can be squeezed - async vs. sync, various
queues and buffers, etc. But the internal volume of air held by the
balloon is more or less the same. A little slack can be added into the
system through your rmem_max technique, as long as you're willing to
tolerate increased processing latency--and it will generate increased
latency; if it didn't, you wouldn't need to increase it--but ultimately,
you're just pushing the air around the balloon. A fixed amount of CPU
and memory is available to accommodate the large number of processes
that sleep on an external I/O-bound workload, and there are diminishing
returns from both internal OpenSIPS contention and context switching.
I'm not saying there aren't some local minima and maxima, but they
aren't as magnitudinal as folks think. It's not that Ubuntu Server is
mistuned, it's that you're abusing it. :-) You can't put the milk back
in the cow, although it's quite a spectacle ...
-- Alex
On 6/12/20 6:02 PM, Calvin Ellison wrote:
I noticed a way-too-small receive buffer value in the OpenSIPS startup
messages and it turns out that a fresh Ubuntu 18 Server install has
absolutely terrible sysctl defaults for high-performance networking. I
got my 8-core lab from less than 2,000 CPS up to 14,000 CPS using a
spread of all dips in non-async mode just by setting the following to
match "maxbuffer=16777216":
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
Does OpenSIPS have guidelines for sysclt and other OS parameters?
Async requires the TM module which adds additional overhead and
memory allocation.
According to with the docs:
"By requiring less processes to complete the same amount of work in the
same amount of time, process context switching is minimized and
overall CPU usage is improved. Less processes will also eat up less
system memory."
So which is it? When should async be used, and when should async not be
used? One can only invest so many hours in load testing combinations of
sync/async, the number of children, timer_partitions, etc. Some fuzzy
math based on CPU core count, SpecInt Rate, BogoMIPS, etc. would be a
great starting point.
Regards,
*Calvin Ellison*
Senior Voice Operations Engineer
calvin.elli...@voxox.com <mailto:calvin.elli...@voxox.com>
_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
--
Alex Balashov | Principal | Evariste Systems LLC
Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free)
Web: http://www.evaristesys.com/, http://www.csrpswitch.com/
_______________________________________________
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
