Somehow I think there is a confusion - the drop_requests stat has
nothing to do with the dropping on the socket buffers (net level).
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
https://www.siphub.com
On 19.04.2024 11:13, johan wrote:
Bogdan,
on the augmenting drop_requests,
drop is not used in that script. Neither is forward. Everything is
t_relay. There is no b2b.
The thing is that we observed drops on udp level.
We followed the recommendations of Ovidiu Sas's presentation of last
year in Houston:
- increase PKG mem
- increase SHM mem
- increase workers to 24 so that the queue empties faster.
- we checked the udp queues on linux level and we saw drops there.
=> we augmented them to 50 megs (sysctl -w
net.core.rmem_max=52428800 and sysctl -w
net.core.rmem_default=52428800) and the drops on OS level where gone.
Also worker and memory load are max 30 %.
Hence we thought that we were okay, but still drops on opensips
level. Net result was that this node in the system lost all
connection with the destination of the loadbalancer although it
received keep alive options responses from the loadbalancer
destination on its NIC (we could see that in a continuously running
tcpdump).
=> hence it seems that it is opensips's receive buffer that is too
small (as I read the description : "Returns the number of requests
dropped even before entering the script routing logic.", I thought
that this pointed to the receive buffer of opensips). All of this is
happening on a physical machine on which two other opensips instances
are running also. Interestingly enough the problem is only observed in
the instance that handles registrations and invites (1600 REG/ s and
300 INV /s).
Therefore we dived a bit deeper and came on this MAX_RECV_BUFFER_SIZE
262144 (which is the default udp queue size setting on linux). Could
this be related somehow ?
Secondly, what would the recommendation be for scaling a system like
this ?
On 18/04/2024 16:29, Bogdan-Andrei Iancu wrote:
The `drop_requests` statistic is incremented when:
* the request is dropped by a pre-script callback (like B2B when
there is no script execution for certain messages)
* the stateless `forward()` core function failed to send out something.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
https://www.siphub.com
On 18.04.2024 17:19, Johan De Clercq wrote:
No I don't.
what I find strange is that MAX_RECV_BUFFER_SIZE 262144 is the
default value of net.core.rmem_max and net.core.rmem_default.
Op do 18 apr 2024 om 16:02 schreef Ben Newlin <[email protected]>:
Are you calling drop() anywhere in your script?
https://www.opensips.org/Documentation/Script-CoreFunctions-3-4#toc13
Ben Newlin
*From: *Users <[email protected]> on behalf of
Johan De Clercq <[email protected]>
*Date: *Thursday, April 18, 2024 at 5:27 AM
*To: *OpenSIPS users mailling list <[email protected]>
*Subject: *Re: [OpenSIPS-Users] question on core statistics.
* EXTERNAL EMAIL - Please use caution with links and attachments *
------------------------------------------------------------------------
would it make sense to recompile with other flags ? And how do I
set them (I don't find these of menuconfig's compile options)?
Currently it has MAX_RECV_BUFFER_SIZE 262144 and BUF_SIZE 65535.
Can somebody explain also what both flags mean.
Op do 18 apr 2024 om 11:07 schreef Johan De Clercq
<[email protected]>:
would it make sense to recompile with other flags ?
Currently it has MAX_RECV_BUFFER_SIZE 262144 and BUF_SIZE
65535.
Can somebody explain also what both flags mean.
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP,
PKG_MALLOC, F_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144,
MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
Op do 18 apr 2024 om 10:32 schreef Johan De Clercq
<[email protected]>:
Guys,
I have an opensips instance running with 24 worker
children.
The worker load is very low.
UDP queues are on 50 megs.
when i query via the OS
cat /proc/net/udp
sl local_address rem_address st tx_queue rx_queue
tr tm->when retrnsmt uid timeout inode ref pointer drops
590: 03231D0A:13C4 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 413684019 2
ffff880074820bc0 0
591: 03231D0A:13C5 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 413766438 2
ffff880465e4a440 0
592: 03231D0A:13C6 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 412035865 2
ffff8803e5a56b80 0
934: 01231D0A:151C 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 26790 2 ffff88046c054840 0
935: 0201FFEF:151D 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 26787 2 ffff88046c054bc0 0
935: 01231D0A:151D 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 26791 2 ffff88046c0544c0 0
1972: 00000000:D92A 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 15506 2 ffff88046dce5040 0
5479: 00000000:E6DD 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 22811 2 ffff880465e4ab40 0
12075: AA0914AC:00A1 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 20572 2 ffff88086d020800 0
12075: 0100007F:00A1 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 20571 2 ffff88086d020b80 0
13320: 00000000:857E 00000000:0000 07 00000000:00000000
00:00000000 00000000 100 0 17515 2 ffff8800368ac780 0
15661: 00000000:CEA3 00000000:0000 07 00000000:00000000
00:00000000 00000000 0 0 15505 2 ffff8800368acb00 0
=> no drops
what worries me is that there are drop requests and
they go up when I query via the mi interface
opensipsctl fifo get_statistics drop_requests
core:drop_requests:: 198107
opensipsctl fifo get_statistics drop_requests
core:drop_requests:: 199157
opensipsctl_reg fifo get_statistics drop_requests
core:drop_requests:: 204116
I don't see any memory issue, also the processload is low.
so 3 questions:
- what exactly is drop_request.
- do I need to worry about this
- how can I make them go lower.
_______________________________________________
Users mailing list
[email protected]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[email protected]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
_______________________________________________
Users mailing list
[email protected]
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
