Re: [OpenSIPS-Users] Auto Scaling Event
Dear Alexey,
Thanks, I figured it out.
On Sat, Jul 16, 2022 at 3:15 PM Alexey Kazantsev via Users <
users@lists.opensips.org> wrote:
> Hi Ali,
>
> overall, you need to configure auto-scaling [1][2] and the event interface
> [3] usage.
> I never tried fetching exactly this event, but I think you haven’t
> configured event_route usage [4][5].
>
> Have a try something like this:
>
> event_route[E_CORE_PROC_AUTO_SCALE] {
> xlog(" … something you want to log … ");
> }
>
>
> [1]
> https://blog.opensips.org/2019/02/25/auto-process-scaling-a-cure-for-load-and-resources-concerns/
> [2]
> https://www.opensips.org/Documentation/Script-CoreParameters-3-3#auto_scaling_profile
> [3] https://www.opensips.org/Documentation/Interface-Events-3-3
> [4] https://www.opensips.org/Documentation/Tutorials-EventInterface#toc17
> [5] https://opensips.org/html/docs/modules/3.3.x/event_route
>
> ---
> BR, Alexey
> https://alexeyka.zantsev.com/
>
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] Auto Scaling Event
Hi Ali,
overall, you need to configure auto-scaling [1][2] and the event interface [3]
usage.
I never tried fetching exactly this event, but I think you haven’t configured
event_route usage [4][5].
Have a try something like this:
event_route[E_CORE_PROC_AUTO_SCALE] {
xlog(" … something you want to log … ");
}
[1]
https://blog.opensips.org/2019/02/25/auto-process-scaling-a-cure-for-load-and-resources-concerns/
[2]
https://www.opensips.org/Documentation/Script-CoreParameters-3-3#auto_scaling_profile
[3] https://www.opensips.org/Documentation/Interface-Events-3-3
[4] https://www.opensips.org/Documentation/Tutorials-EventInterface#toc17
[5] https://opensips.org/html/docs/modules/3.3.x/event_route
---
BR, Alexey
https://alexeyka.zantsev.com/
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
[OpenSIPS-Users] Auto Scaling Event
Dear All, Would any one please support me with an example on how to use E_CORE_PROC_AUTO_SCALE in the script. I want to get a value of group_load parameter of E_CORE_PROC_AUTO_SCALE after any upscale or downscale. Regards, Ali ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] auto scaling
Hello Bogdan, Yes, I will do pull today and test again auto scaling feature. Thank you. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "Slava Bendersky" Cc: "OpenSIPS users mailling list" Sent: Friday, May 10, 2019 5:02:48 AM Subject: Re: [OpenSIPS-Users] auto scaling Hi Slava, There were some fixes in the auto-scaling in the last weeks, could you confirm or not your issue against the latest 3.0 code ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/15/2019 12:47 PM, Slava Bendersky wrote: Hello Bogdan, Upscaling error is didn't show up any more in logs at least I didn't so it. Might need trigger bigger load. do you do auto-scaling only for TCP ? Our setup in most multi homed, as example public side TCP private side UDP Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ? Yes, that how is look like end result that opensips start blocking TCP connections. If specific process handle sip registration via tcp and opensips trying close to down scale end point need send re register in order do not loose registration. Other wise it cause stale connections. Just close properly connection not enough I think. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "OpenSIPS users mailling list" , "Slava Bendersky" Sent: Thursday, March 14, 2019 1:45:20 PM Subject: Re: [OpenSIPS-Users] auto scaling Slava, What about the upscaling error you originally reported ? is there any way to reproduce it ? do you do auto-scaling only for TCP (no UDP,timer) ? Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ? Is is a proper close of the connection ? or ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/14/2019 02:14 PM, Slava Bendersky wrote: Hello Bogdan, After we doing more testing with auto scaling we observed issue with TCP, where opensips try scale down the processes it tear down them and break TCP connections with end points and that cause blocked connection state where opensips stops process TCP connections. volga629 - Original Message - From: "Slava Bendersky" To: "Bogdan-Andrei Iancu" Cc: "OpenSIPS users mailling list" Sent: Wednesday, March 13, 2019 12:21:14 AM Subject: Re: [OpenSIPS-Users] auto scaling I will do pull today. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "Slava Bendersky" Cc: "OpenSIPS users mailling list" Sent: Tuesday, March 12, 2019 4:52:06 AM Subject: Re: [OpenSIPS-Users] auto scaling Slava, Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/12/2019 05:51 AM, Slava Bendersky wrote: Hello Bogdan, My question how to calculate correctly tcp_workers udp_workers to properly assign with in auto scaling profile. Which parameters need use as base ? As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers. It message is appear in log on first attempt to fork in scale up. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "OpenSIPS users mailling list" , "Slava Bendersky" Sent: Monday, March 11, 2019 4:44:57 AM Subject: Re: [OpenSIPS-Users] auto scaling Hi Slava, Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ? Also I just pushed a small commit to improve the date provided by the bug log. In regards to the "not clear how to calculate"...could you be specific ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/10/2019 05:12 PM, Slava Bendersky wrote: Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:f
Re: [OpenSIPS-Users] auto scaling
Hi Slava,
There were some fixes in the auto-scaling in the last weeks, could you
confirm or not your issue against the latest 3.0 code ?
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/15/2019 12:47 PM, Slava Bendersky wrote:
Hello Bogdan,
Upscaling error is didn't show up any more in logs at least I didn't
so it. Might need trigger bigger load.
*do you do auto-scaling only for TCP ?*
Our setup in most multi homed, as example public side TCP private
side UDP
*Now, about the TCP downscaling - I guess you see as terminated by
OpenSIPS the TCP conns anchored in the process that is terminated ?*
Yes, that how is look like end result that opensips start blocking TCP
connections. If specific process handle sip registration via tcp and
opensips trying close to down scale end point need send re register in
order do not loose registration. Other wise it cause stale
connections. Just close properly connection not enough I think.
volga629
*From: *"Bogdan-Andrei Iancu"
*To: *"OpenSIPS users mailling list" ,
"Slava Bendersky"
*Sent: *Thursday, March 14, 2019 1:45:20 PM
*Subject: *Re: [OpenSIPS-Users] auto scaling
Slava,
What about the upscaling error you originally reported ? is there any
way to reproduce it ? do you do auto-scaling only for TCP (no UDP,timer) ?
Now, about the TCP downscaling - I guess you see as terminated by
OpenSIPS the TCP conns anchored in the process that is terminated ? Is
is a proper close of the connection ? or ?
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/14/2019 02:14 PM, Slava Bendersky wrote:
Hello Bogdan,
After we doing more testing with auto scaling we observed issue
with TCP, where opensips try scale down the processes it tear down
them and break TCP connections with end points and that cause
blocked connection state where opensips stops process TCP connections.
volga629
*From: *"Slava Bendersky"
*To: *"Bogdan-Andrei Iancu"
*Cc: *"OpenSIPS users mailling list"
*Sent: *Wednesday, March 13, 2019 12:21:14 AM
*Subject: *Re: [OpenSIPS-Users] auto scaling
I will do pull today.
volga629
*From: *"Bogdan-Andrei Iancu"
*To: *"Slava Bendersky"
*Cc: *"OpenSIPS users mailling list"
*Sent: *Tuesday, March 12, 2019 4:52:06 AM
*Subject: *Re: [OpenSIPS-Users] auto scaling
Slava,
Have you pulled my change on the improved log (just update from
master) ? and post here the new output you get for the error.
Thanks,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/12/2019 05:51 AM, Slava Bendersky wrote:
Hello Bogdan,
My question how to calculate correctly
*tcp_workers*
*udp_workers*
*
*
to properly assign with in auto scaling profile. Which
parameters need use as base ?
As example if vm 4GB memory up what will be max number for
scale up processes and what should be value for tcp and udp
workers.
It message is appear in log on first attempt to fork in scale up.
volga629
*From: *"Bogdan-Andrei Iancu"
*To: *"OpenSIPS users mailling list"
, "Slava Bendersky"
*Sent: *Monday, March 11, 2019 4:44:57 AM
*Subject: *Re: [OpenSIPS-Users] auto scaling
Hi Slava,
Based on the OpenSIPS logs, did you get that error on the
first forking attempt or sometime later after some number of
new tcp processes forking ? More or less, my question is :
what was the process creation/termination history before
getting that bug log ?
Also I just pushed a small commit to improve the date provided
by the bug log.
In regards to the "not clear how to calculate"...could you be
specific ?
Best regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Hello Everyone,
Re: [OpenSIPS-Users] auto scaling
Hi Slava,
OK, I will need to run more tests on the TCP conns stuff - the intended
behavior is *NOT* to close the connections handled by the process to be
terminated.
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/15/2019 12:47 PM, Slava Bendersky wrote:
Hello Bogdan,
Upscaling error is didn't show up any more in logs at least I didn't
so it. Might need trigger bigger load.
*do you do auto-scaling only for TCP ?*
Our setup in most multi homed, as example public side TCP private
side UDP
*Now, about the TCP downscaling - I guess you see as terminated by
OpenSIPS the TCP conns anchored in the process that is terminated ?*
Yes, that how is look like end result that opensips start blocking TCP
connections. If specific process handle sip registration via tcp and
opensips trying close to down scale end point need send re register in
order do not loose registration. Other wise it cause stale
connections. Just close properly connection not enough I think.
volga629
*From: *"Bogdan-Andrei Iancu"
*To: *"OpenSIPS users mailling list" ,
"Slava Bendersky"
*Sent: *Thursday, March 14, 2019 1:45:20 PM
*Subject: *Re: [OpenSIPS-Users] auto scaling
Slava,
What about the upscaling error you originally reported ? is there any
way to reproduce it ? do you do auto-scaling only for TCP (no UDP,timer) ?
Now, about the TCP downscaling - I guess you see as terminated by
OpenSIPS the TCP conns anchored in the process that is terminated ? Is
is a proper close of the connection ? or ?
Regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/14/2019 02:14 PM, Slava Bendersky wrote:
Hello Bogdan,
After we doing more testing with auto scaling we observed issue
with TCP, where opensips try scale down the processes it tear down
them and break TCP connections with end points and that cause
blocked connection state where opensips stops process TCP connections.
volga629
*From: *"Slava Bendersky"
*To: *"Bogdan-Andrei Iancu"
*Cc: *"OpenSIPS users mailling list"
*Sent: *Wednesday, March 13, 2019 12:21:14 AM
*Subject: *Re: [OpenSIPS-Users] auto scaling
I will do pull today.
volga629
*From: *"Bogdan-Andrei Iancu"
*To: *"Slava Bendersky"
*Cc: *"OpenSIPS users mailling list"
*Sent: *Tuesday, March 12, 2019 4:52:06 AM
*Subject: *Re: [OpenSIPS-Users] auto scaling
Slava,
Have you pulled my change on the improved log (just update from
master) ? and post here the new output you get for the error.
Thanks,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/12/2019 05:51 AM, Slava Bendersky wrote:
Hello Bogdan,
My question how to calculate correctly
*tcp_workers*
*udp_workers*
*
*
to properly assign with in auto scaling profile. Which
parameters need use as base ?
As example if vm 4GB memory up what will be max number for
scale up processes and what should be value for tcp and udp
workers.
It message is appear in log on first attempt to fork in scale up.
volga629
*From: *"Bogdan-Andrei Iancu"
*To: *"OpenSIPS users mailling list"
, "Slava Bendersky"
*Sent: *Monday, March 11, 2019 4:44:57 AM
*Subject: *Re: [OpenSIPS-Users] auto scaling
Hi Slava,
Based on the OpenSIPS logs, did you get that error on the
first forking attempt or sometime later after some number of
new tcp processes forking ? More or less, my question is :
what was the process creation/termination history before
getting that bug log ?
Also I just pushed a small commit to improve the date provided
by the bug log.
In regards to the "not clear how to calculate"...could you be
specific ?
Best regards,
Bogdan-Andrei Iancu
OpenSIPS Founder and Developer
https://www.opensips-solutions.com
OpenSIPS Summit 2019
https://www.opensips.org/events/Summit-2019Amsterdam/
On 03/10/2019 05:12 PM, Slava Bendersky wrote:
Re: [OpenSIPS-Users] auto scaling
Hello Bogdan, Upscaling error is didn't show up any more in logs at least I didn't so it. Might need trigger bigger load. do you do auto-scaling only for TCP ? Our setup in most multi homed, as example public side TCP private side UDP Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ? Yes, that how is look like end result that opensips start blocking TCP connections. If specific process handle sip registration via tcp and opensips trying close to down scale end point need send re register in order do not loose registration. Other wise it cause stale connections. Just close properly connection not enough I think. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "OpenSIPS users mailling list" , "Slava Bendersky" Sent: Thursday, March 14, 2019 1:45:20 PM Subject: Re: [OpenSIPS-Users] auto scaling Slava, What about the upscaling error you originally reported ? is there any way to reproduce it ? do you do auto-scaling only for TCP (no UDP,timer) ? Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ? Is is a proper close of the connection ? or ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/14/2019 02:14 PM, Slava Bendersky wrote: Hello Bogdan, After we doing more testing with auto scaling we observed issue with TCP, where opensips try scale down the processes it tear down them and break TCP connections with end points and that cause blocked connection state where opensips stops process TCP connections. volga629 - Original Message - From: "Slava Bendersky" To: "Bogdan-Andrei Iancu" Cc: "OpenSIPS users mailling list" Sent: Wednesday, March 13, 2019 12:21:14 AM Subject: Re: [OpenSIPS-Users] auto scaling I will do pull today. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "Slava Bendersky" Cc: "OpenSIPS users mailling list" Sent: Tuesday, March 12, 2019 4:52:06 AM Subject: Re: [OpenSIPS-Users] auto scaling Slava, Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/12/2019 05:51 AM, Slava Bendersky wrote: Hello Bogdan, My question how to calculate correctly tcp_workers udp_workers to properly assign with in auto scaling profile. Which parameters need use as base ? As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers. It message is appear in log on first attempt to fork in scale up. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "OpenSIPS users mailling list" , "Slava Bendersky" Sent: Monday, March 11, 2019 4:44:57 AM Subject: Re: [OpenSIPS-Users] auto scaling Hi Slava, Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ? Also I just pushed a small commit to improve the date provided by the bug log. In regards to the "not clear how to calculate"...could you be specific ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/10/2019 05:12 PM, Slava Bendersky wrote: Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues Not clear how to calculate workers for tcp or udp tcp_workers=10 use_auto_scaling_profile PRF_SIP listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP [root@vprx00 ~]# opensips -V version: opensips 3.0.0-dev (x86_64/linux) flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_
Re: [OpenSIPS-Users] auto scaling
Slava, What about the upscaling error you originally reported ? is there any way to reproduce it ? do you do auto-scaling only for TCP (no UDP,timer) ? Now, about the TCP downscaling - I guess you see as terminated by OpenSIPS the TCP conns anchored in the process that is terminated ? Is is a proper close of the connection ? or ? Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/14/2019 02:14 PM, Slava Bendersky wrote: Hello Bogdan, After we doing more testing with auto scaling we observed issue with TCP, where opensips try scale down the processes it tear down them and break TCP connections with end points and that cause blocked connection state where opensips stops process TCP connections. volga629 *From: *"Slava Bendersky" *To: *"Bogdan-Andrei Iancu" *Cc: *"OpenSIPS users mailling list" *Sent: *Wednesday, March 13, 2019 12:21:14 AM *Subject: *Re: [OpenSIPS-Users] auto scaling I will do pull today. volga629 *From: *"Bogdan-Andrei Iancu" *To: *"Slava Bendersky" *Cc: *"OpenSIPS users mailling list" *Sent: *Tuesday, March 12, 2019 4:52:06 AM *Subject: *Re: [OpenSIPS-Users] auto scaling Slava, Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/12/2019 05:51 AM, Slava Bendersky wrote: Hello Bogdan, My question how to calculate correctly *tcp_workers* *udp_workers* * * to properly assign with in auto scaling profile. Which parameters need use as base ? As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers. It message is appear in log on first attempt to fork in scale up. volga629 *From: *"Bogdan-Andrei Iancu" *To: *"OpenSIPS users mailling list" , "Slava Bendersky" *Sent: *Monday, March 11, 2019 4:44:57 AM *Subject: *Re: [OpenSIPS-Users] auto scaling Hi Slava, Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ? Also I just pushed a small commit to improve the date provided by the bug log. In regards to the "not clear how to calculate"...could you be specific ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/10/2019 05:12 PM, Slava Bendersky wrote: Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues *Not clear how to calculate workers for tcp or udp* *tcp_workers=10* use_auto_scaling_profile PRF_SIP *listen=udp:lan_ip:5060* use_auto_scaling_profile PRF_SIP [root@vprx00 ~]# opensips -V version: opensips 3.0.0-dev (x86_64/linux) flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on 10:06:47 Mar 7 2019 with gcc 8 volga629 ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___
Re: [OpenSIPS-Users] auto scaling
Hello Bogdan, After we doing more testing with auto scaling we observed issue with TCP, where opensips try scale down the processes it tear down them and break TCP connections with end points and that cause blocked connection state where opensips stops process TCP connections. volga629 - Original Message - From: "Slava Bendersky" To: "Bogdan-Andrei Iancu" Cc: "OpenSIPS users mailling list" Sent: Wednesday, March 13, 2019 12:21:14 AM Subject: Re: [OpenSIPS-Users] auto scaling I will do pull today. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "Slava Bendersky" Cc: "OpenSIPS users mailling list" Sent: Tuesday, March 12, 2019 4:52:06 AM Subject: Re: [OpenSIPS-Users] auto scaling Slava, Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/12/2019 05:51 AM, Slava Bendersky wrote: Hello Bogdan, My question how to calculate correctly tcp_workers udp_workers to properly assign with in auto scaling profile. Which parameters need use as base ? As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers. It message is appear in log on first attempt to fork in scale up. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "OpenSIPS users mailling list" , "Slava Bendersky" Sent: Monday, March 11, 2019 4:44:57 AM Subject: Re: [OpenSIPS-Users] auto scaling Hi Slava, Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ? Also I just pushed a small commit to improve the date provided by the bug log. In regards to the "not clear how to calculate"...could you be specific ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/10/2019 05:12 PM, Slava Bendersky wrote: Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues Not clear how to calculate workers for tcp or udp tcp_workers=10 use_auto_scaling_profile PRF_SIP listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP [root@vprx00 ~]# opensips -V version: opensips 3.0.0-dev (x86_64/linux) flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on 10:06:47 Mar 7 2019 with gcc 8 volga629 ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] auto scaling
I will do pull today. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "Slava Bendersky" Cc: "OpenSIPS users mailling list" Sent: Tuesday, March 12, 2019 4:52:06 AM Subject: Re: [OpenSIPS-Users] auto scaling Slava, Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/12/2019 05:51 AM, Slava Bendersky wrote: Hello Bogdan, My question how to calculate correctly tcp_workers udp_workers to properly assign with in auto scaling profile. Which parameters need use as base ? As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers. It message is appear in log on first attempt to fork in scale up. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "OpenSIPS users mailling list" , "Slava Bendersky" Sent: Monday, March 11, 2019 4:44:57 AM Subject: Re: [OpenSIPS-Users] auto scaling Hi Slava, Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ? Also I just pushed a small commit to improve the date provided by the bug log. In regards to the "not clear how to calculate"...could you be specific ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/10/2019 05:12 PM, Slava Bendersky wrote: Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues Not clear how to calculate workers for tcp or udp tcp_workers=10 use_auto_scaling_profile PRF_SIP listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP [root@vprx00 ~]# opensips -V version: opensips 3.0.0-dev (x86_64/linux) flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on 10:06:47 Mar 7 2019 with gcc 8 volga629 ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] auto scaling
Slava, Have you pulled my change on the improved log (just update from master) ? and post here the new output you get for the error. Thanks, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/12/2019 05:51 AM, Slava Bendersky wrote: Hello Bogdan, My question how to calculate correctly *tcp_workers* *udp_workers* * * to properly assign with in auto scaling profile. Which parameters need use as base ? As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers. It message is appear in log on first attempt to fork in scale up. volga629 *From: *"Bogdan-Andrei Iancu" *To: *"OpenSIPS users mailling list" , "Slava Bendersky" *Sent: *Monday, March 11, 2019 4:44:57 AM *Subject: *Re: [OpenSIPS-Users] auto scaling Hi Slava, Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ? Also I just pushed a small commit to improve the date provided by the bug log. In regards to the "not clear how to calculate"...could you be specific ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/10/2019 05:12 PM, Slava Bendersky wrote: Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues *Not clear how to calculate workers for tcp or udp* *tcp_workers=10* use_auto_scaling_profile PRF_SIP *listen=udp:lan_ip:5060* use_auto_scaling_profile PRF_SIP [root@vprx00 ~]# opensips -V version: opensips 3.0.0-dev (x86_64/linux) flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on 10:06:47 Mar 7 2019 with gcc 8 volga629 ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] auto scaling
Hello Bogdan, My question how to calculate correctly tcp_workers udp_workers to properly assign with in auto scaling profile. Which parameters need use as base ? As example if vm 4GB memory up what will be max number for scale up processes and what should be value for tcp and udp workers. It message is appear in log on first attempt to fork in scale up. volga629 - Original Message - From: "Bogdan-Andrei Iancu" To: "OpenSIPS users mailling list" , "Slava Bendersky" Sent: Monday, March 11, 2019 4:44:57 AM Subject: Re: [OpenSIPS-Users] auto scaling Hi Slava, Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ? Also I just pushed a small commit to improve the date provided by the bug log. In regards to the "not clear how to calculate"...could you be specific ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/10/2019 05:12 PM, Slava Bendersky wrote: Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues Not clear how to calculate workers for tcp or udp tcp_workers=10 use_auto_scaling_profile PRF_SIP listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP [root@vprx00 ~]# opensips -V version: opensips 3.0.0-dev (x86_64/linux) flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on 10:06:47 Mar 7 2019 with gcc 8 volga629 ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
Re: [OpenSIPS-Users] auto scaling
Hi Slava, Based on the OpenSIPS logs, did you get that error on the first forking attempt or sometime later after some number of new tcp processes forking ? More or less, my question is : what was the process creation/termination history before getting that bug log ? Also I just pushed a small commit to improve the date provided by the bug log. In regards to the "not clear how to calculate"...could you be specific ? Best regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer https://www.opensips-solutions.com OpenSIPS Summit 2019 https://www.opensips.org/events/Summit-2019Amsterdam/ On 03/10/2019 05:12 PM, Slava Bendersky wrote: Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues *Not clear how to calculate workers for tcp or udp* *tcp_workers=10* use_auto_scaling_profile PRF_SIP *listen=udp:lan_ip:5060* use_auto_scaling_profile PRF_SIP [root@vprx00 ~]# opensips -V version: opensips 3.0.0-dev (x86_64/linux) flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on 10:06:47 Mar 7 2019 with gcc 8 volga629 ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
[OpenSIPS-Users] auto scaling
Hello Everyone, Trying test auto scaling feature with following config auto_scaling_profile = PRF_SIP scale up to 24 on 70% for 4 cycles within 5 scale down to 4 on 20% for 8 cycles On box 4096 Mb memory and 4 virtual CPU Getting in log CRITICAL:core:fork_dynamic_tcp_process: #012>>> trying to fork one more TCP worker but no free slots in the TCP table#012#012It seems you have hit a programming bug.#012Please help us make OpenSIPS better by reporting it at https://github.com/OpenSIPS/opensips/issues Not clear how to calculate workers for tcp or udp tcp_workers=10 use_auto_scaling_profile PRF_SIP listen=udp:lan_ip:5060 use_auto_scaling_profile PRF_SIP [root@vprx00 ~]# opensips -V version: opensips 3.0.0-dev (x86_64/linux) flags: STATS: On, SHM_EXTRA_STATS, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, QM_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. main.c compiled on 10:06:47 Mar 7 2019 with gcc 8 volga629 ___ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users
