Hi Bogdan, This has finally happened, OS is stuck again in 100% for one of its processes. Here's the output of load: command:
opensips-cli -x mi get_statistics load: { "load:load-proc-1": 0, "load:load1m-proc-1": 0, "load:load10m-proc-1": 0, "load:load-proc-2": 0, "load:load1m-proc-2": 0, "load:load10m-proc-2": 0, "load:load-proc-3": 0, "load:load1m-proc-3": 0, "load:load10m-proc-3": 0, "load:load-proc-4": 0, "load:load1m-proc-4": 0, "load:load10m-proc-4": 0, "load:load-proc-5": 0, "load:load1m-proc-5": 0, "load:load10m-proc-5": 8, "load:load-proc-6": 0, "load:load1m-proc-6": 0, "load:load10m-proc-6": 6, "load:load-proc-13": 0, "load:load1m-proc-13": 0, "load:load10m-proc-13": 0, "load:load-proc-14": 0, "load:load1m-proc-14": 0, "load:load10m-proc-14": 0, "load:load-proc-21": 0, "load:load1m-proc-21": 0, "load:load10m-proc-21": 0, "load:load-proc-22": 0, "load:load1m-proc-22": 0, "load:load10m-proc-22": 0, "load:load-proc-23": 0, "load:load1m-proc-23": 0, "load:load10m-proc-23": 0, "load:load-proc-24": 100, "load:load1m-proc-24": 100, "load:load10m-proc-24": 100, "load:load": 12, "load:load1m": 12, "load:load10m": 14, "load:load-all": 10, "load:load1m-all": 10, "load:load10m-all": 11, "load:processes_number": 13 } As you can see, process 24 is consuming 100% of time for more than a minute already Here's the output of process list, it's a UDP socket listener on internal interface that's stuck at 100% load: opensips-cli -x mi ps { "Processes": [ { "ID": 0, "PID": 5457, "Type": "attendant" }, { "ID": 1, "PID": 5463, "Type": "HTTPD 10.x.x.x:8888" }, { "ID": 2, "PID": 5464, "Type": "MI FIFO" }, { "ID": 3, "PID": 5465, "Type": "time_keeper" }, { "ID": 4, "PID": 5466, "Type": "timer" }, { "ID": 5, "PID": 5467, "Type": "SIP receiver udp:10.x.x.x:5060" }, { "ID": 6, "PID": 5470, "Type": "SIP receiver udp:10.x.x.x:5060" }, { "ID": 13, "PID": 5477, "Type": "SIP receiver udp:103.x.x.x:7060" }, { "ID": 14, "PID": 5478, "Type": "SIP receiver udp:103.x.x.x:7060" }, { "ID": 21, "PID": 5485, "Type": "TCP receiver" }, { "ID": 22, "PID": 5486, "Type": "Timer handler" }, { "ID": 23, "PID": 5487, "Type": "TCP main" }, { "ID": 24, "PID": 5759, "Type": "SIP receiver udp:10.x.x.x:5060" } ] } opensips -V version: opensips 3.2.8 (x86_64/linux) flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535 poll method support: poll, epoll, sigio_rt, select. git revision: d2496fed5 main.c compiled on 16:17:53 Aug 24 2022 with gcc 9 This time server has some load but still it's not heavy at all plus I'm using async requests for REST queries. This is my autoscaling section: # Scaling section auto_scaling_profile = PROFILE_UDP_PUB scale up to 16 on 70% for 4 cycles within 5 scale down to 2 on 20% for 5 cycles auto_scaling_profile = PROFILE_UDP_PRIV scale up to 16 on 70% for 4 cycles within 5 scale down to 2 on 20% for 5 cycles auto_scaling_profile = PROFILE_TCP scale up to 16 on 70% for 4 cycles within 5 scale down to 2 on 20% for 10 cycles And that's how I apply it to sockets, I'm not applying it to UDP workers at all: socket=udp:10.x.x.x:5060 use_auto_scaling_profile PROFILE_UDP_PRIV socket=udp:103.x.x.x:7060 use_auto_scaling_profile PROFILE_UDP_PUB tcp_workers = 1 use_auto_scaling_profile PROFILE_TCP I can't get this process unstuck until I restart OpenSIPS. Just to add - if I turn off auto scaling and enable 16 UDP and 16 TCP workers and just specify sockets without any parameters - load goes to 0, see graph attached, load was at 25% all the time until I restarted OpenSIPS in normal mode, then it's immediately 0: [image: image.png] Here's an output of load: opensips-cli -x mi get_statistics load: { "load:load-proc-1": 0, "load:load1m-proc-1": 0, "load:load10m-proc-1": 0, "load:load-proc-2": 0, "load:load1m-proc-2": 0, "load:load10m-proc-2": 0, "load:load-proc-3": 0, "load:load1m-proc-3": 0, "load:load10m-proc-3": 0, "load:load-proc-4": 0, "load:load1m-proc-4": 0, "load:load10m-proc-4": 0, "load:load-proc-5": 0, "load:load1m-proc-5": 0, "load:load10m-proc-5": 2, "load:load-proc-6": 0, "load:load1m-proc-6": 0, "load:load10m-proc-6": 0, "load:load-proc-7": 0, "load:load1m-proc-7": 0, "load:load10m-proc-7": 1, "load:load-proc-8": 0, "load:load1m-proc-8": 0, "load:load10m-proc-8": 1, "load:load-proc-9": 0, "load:load1m-proc-9": 0, "load:load10m-proc-9": 1, "load:load-proc-10": 0, "load:load1m-proc-10": 0, "load:load10m-proc-10": 0, "load:load-proc-11": 0, "load:load1m-proc-11": 0, "load:load10m-proc-11": 3, "load:load-proc-12": 0, "load:load1m-proc-12": 0, "load:load10m-proc-12": 2, "load:load-proc-13": 0, "load:load1m-proc-13": 0, "load:load10m-proc-13": 1, "load:load-proc-14": 0, "load:load1m-proc-14": 0, "load:load10m-proc-14": 3, "load:load-proc-15": 0, "load:load1m-proc-15": 0, "load:load10m-proc-15": 2, "load:load-proc-16": 0, "load:load1m-proc-16": 0, "load:load10m-proc-16": 1, "load:load-proc-17": 0, "load:load1m-proc-17": 0, "load:load10m-proc-17": 4, "load:load-proc-18": 0, "load:load1m-proc-18": 0, "load:load10m-proc-18": 2, "load:load-proc-19": 0, "load:load1m-proc-19": 0, "load:load10m-proc-19": 3, "load:load-proc-20": 0, "load:load1m-proc-20": 0, "load:load10m-proc-20": 2, "load:load-proc-21": 0, "load:load1m-proc-21": 0, "load:load10m-proc-21": 0, "load:load-proc-22": 0, "load:load1m-proc-22": 0, "load:load10m-proc-22": 0, "load:load-proc-23": 0, "load:load1m-proc-23": 0, "load:load10m-proc-23": 0, "load:load-proc-24": 0, "load:load1m-proc-24": 0, "load:load10m-proc-24": 0, "load:load-proc-25": 0, "load:load1m-proc-25": 0, "load:load10m-proc-25": 0, "load:load-proc-26": 0, "load:load1m-proc-26": 0, "load:load10m-proc-26": 0, "load:load-proc-27": 0, "load:load1m-proc-27": 0, "load:load10m-proc-27": 0, "load:load-proc-28": 0, "load:load1m-proc-28": 0, "load:load10m-proc-28": 0, "load:load-proc-29": 0, "load:load1m-proc-29": 0, "load:load10m-proc-29": 0, "load:load-proc-30": 0, "load:load1m-proc-30": 0, "load:load10m-proc-30": 0, "load:load-proc-31": 0, "load:load1m-proc-31": 0, "load:load10m-proc-31": 0, "load:load-proc-32": 0, "load:load1m-proc-32": 0, "load:load10m-proc-32": 0, "load:load-proc-33": 0, "load:load1m-proc-33": 0, "load:load10m-proc-33": 0, "load:load-proc-34": 0, "load:load1m-proc-34": 0, "load:load10m-proc-34": 0, "load:load-proc-35": 3, "load:load1m-proc-35": 0, "load:load10m-proc-35": 0, "load:load-proc-36": 0, "load:load1m-proc-36": 0, "load:load10m-proc-36": 0, "load:load-proc-37": 0, "load:load1m-proc-37": 0, "load:load10m-proc-37": 0, "load:load-proc-38": 0, "load:load1m-proc-38": 0, "load:load10m-proc-38": 0, "load:load-proc-39": 0, "load:load1m-proc-39": 0, "load:load10m-proc-39": 0, "load:load-proc-40": 0, "load:load1m-proc-40": 0, "load:load10m-proc-40": 0, "load:load-proc-41": 0, "load:load1m-proc-41": 0, "load:load10m-proc-41": 0, "load:load-proc-42": 0, "load:load1m-proc-42": 0, "load:load10m-proc-42": 0, "load:load-proc-43": 0, "load:load1m-proc-43": 0, "load:load10m-proc-43": 0, "load:load-proc-44": 0, "load:load1m-proc-44": 0, "load:load10m-proc-44": 0, "load:load-proc-45": 0, "load:load1m-proc-45": 0, "load:load10m-proc-45": 0, "load:load-proc-46": 0, "load:load1m-proc-46": 0, "load:load10m-proc-46": 0, "load:load-proc-47": 0, "load:load1m-proc-47": 0, "load:load10m-proc-47": 0, "load:load-proc-48": 0, "load:load1m-proc-48": 0, "load:load10m-proc-48": 0, "load:load-proc-49": 0, "load:load1m-proc-49": 0, "load:load10m-proc-49": 0, "load:load-proc-50": 0, "load:load1m-proc-50": 0, "load:load10m-proc-50": 0, "load:load-proc-51": 0, "load:load1m-proc-51": 0, "load:load10m-proc-51": 0, "load:load-proc-52": 0, "load:load1m-proc-52": 0, "load:load10m-proc-52": 0, "load:load-proc-53": 0, "load:load1m-proc-53": 0, "load:load10m-proc-53": 0, "load:load-proc-54": 0, "load:load1m-proc-54": 0, "load:load10m-proc-54": 0, "load:load": 0, "load:load1m": 0, "load:load10m": 0, "load:load-all": 0, "load:load1m-all": 0, "load:load10m-all": 0, "load:processes_number": 55 } Hope this is all the information you need! Thanks! Best regards, Yury. On Thu, Aug 25, 2022 at 8:24 PM Bogdan-Andrei Iancu <bog...@opensips.org> wrote: > Hi Yury, > > And when that scaling up happens, do you actually have traffic ? or your > OpenSIPS is idle ? > > Also, could you run `opensips-cli -x mi get_statistics load:` (not the > colon at the end). > > Regards, > > Bogdan-Andrei Iancu > > OpenSIPS Founder and Developer > https://www.opensips-solutions.com > OpenSIPS Summit 27-30 Sept 2022, Athens > https://www.opensips.org/events/Summit-2022Athens/ > > On 8/25/22 10:57 AM, Yury Kirsanov wrote: > > Hi all, > I've ran into a strange issue, if I enable autoscaler on OpenSIPS 3.2.x > (tried 5,6,7 and now 8) on a server without any load using 'socket' > statement like this: > > auto_scaling_profile = PROFILE_UDP_PRIV > scale up to 16 on 30% for 4 cycles within 5 > scale down to 2 on 10% for 5 cycles > > udp_workers=4 > > socket=udp:10.x.x.x:5060 use_auto_scaling_profile PROFILE_UDP_PRIV > > then after a while OpenSIPS load goes up to some high number, autoscaler > starts to open new processes up to a maximum number specified in profile > and them load stays at that number, for example: > > opensips-cli -x mi get_statistics load > { > "load:load": 60 > } > > It never changes and looks just 'stuck'. > > Any ideas why this is happening in my case? Or should I file a bug report? > Thanks. > > Regards, > Yury. > > _______________________________________________ > Users mailing > listUsers@lists.opensips.orghttp://lists.opensips.org/cgi-bin/mailman/listinfo/users > > >
_______________________________________________ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users