[SR-Users] Re: ERROR: run_top_route
Hi, no we don't need to change any code level. but in your script just remove the return with negative code. you probably have in your script failure_route[MY_FAILURE]{ return -5; // <-- you probably have this. remove this instruction return route[ANOTHER_ROUTE] // <-- or you have this flavour . remove the return instruction. } you should remove any return instruction present in faillure_route. its not useful to have a return instruction in a faillure_route. De : satyaprakash ch Envoyé : mercredi 11 octobre 2023 07:27 À : Patrick Karton Cc : Kamailio (SER) - Users Mailing List Objet : Re: [SR-Users] ERROR: run_top_route Hi, Thanks for the reply, What if we would get a negative value, Do we need to change any code level, Will you please suggest what we need to do to resolve this? Waiting for your reply. On Mon, Oct 9, 2023 at 11:45 AM satyaprakash ch mailto:chiramchetty.satyaprak...@gmail.com>> wrote: Hi, Thanks for the reply, What if we would get a negative value, Do we need to change any code level, Will you please suggest what we need to do to resolve this? On Wed, Sep 27, 2023 at 9:26 PM Patrick Karton mailto:patrickar...@hotmail.com>> wrote: Hi, Probably because you are returning negative value in this failure_route De : satyaprakash ch via sr-users mailto:sr-users@lists.kamailio.org>> Envoyé : mercredi 27 septembre 2023 06:45 À : Kamailio (SER) - Users Mailing List mailto:sr-users@lists.kamailio.org>> Cc : satyaprakash ch mailto:chiramchetty.satyaprak...@gmail.com>> Objet : [SR-Users] ERROR: run_top_route Hi, We are having an error in the Kamailio logs which we need to resolve this issue, ERROR is :: /usr/local/sbin/kamailio[10149]: ERROR: tm [t_reply.c:1081]: run_failure_handlers(): error running run_top_route for failure handler. We are getting this error at the time of the 3xx response, Can anyone help me on this? Thank you. __ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-le...@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
[SR-Users] Re: Using DMQ to sync the TM module.
Yes, but this is not helpful if trying to mitigate an outage of the node that actually holds the transaction state. Unfortunately, from a technical point of view, replicating TM state is not as straightforward as it sounds, or it would have been done already. It's not just a matter of serialising the transaction information and shipping it across and reconstituting it, timers and all. There are hooks deep into other runtime state, connection state, etc. which either aren't exposed for easy deserialisation/hydration, or are not possible to replicate altogether because it would require integration with the OS network stack (e.g. connection info). All to say, even if such a thing were developed, it would be an extremely flawed and imperfect process, and--perhaps I can take a bit of liberty on behalf of the core development team--the project doesn't like to support half-assed features. A more stateless, fault-tolerant design is, by comparison, a significantly lower engineering lift in most cases where replicating TM would be useful. -- Alex > On 11 Oct 2023, at 09:47, Michel Pelletier via sr-users > wrote: > > Hi, > > I agree. But the example puts me on the right track. It tells me that what > I need to do is bounce the reply off to the other nodes(s) if TM doesn't > recognize it. Having a DMQ option for the TM module would be great though. > > Cheers, > > Michel Pelletier > > > On Tue, Oct 10, 2023 at 4:22 PM Alex Balashov via sr-users > wrote: > I don't think the anycast example is going to get you out of this problem > entirely. > > > On 10 Oct 2023, at 17:22, Michel Pelletier via sr-users > > wrote: > > > > Many thanks. I am afraid I need stateful TM if only for the > > retransmissions and how to avoid them. The Anycast example will prove very > > useful. > > Cheers, > > > > Michel Pelletier > > > > > > On Tue, Oct 10, 2023 at 11:58 AM Alex Balashov via sr-users > > wrote: > > But I should add: do you actually need state? All replies can be routed > > back based on the content of SIP headers alone -- that is to say, > > statelessly. Most simple load balancers remain stateless for this very > > reason. > > > > > On 10 Oct 2023, at 13:09, Alex Balashov wrote: > > > > > > There is not. > > > > > >> On 10 Oct 2023, at 12:50, Michel Pelletier via sr-users > > >> wrote: > > >> > > >> Hi, > > >> > > >> I have 2 kamailio instances behind a load balancer. The problem I have > > >> is that the load balancer can only track TCP connections, but not UDP. > > >> So one Kamailio instance might send a request using UDP, while the > > >> corresponding UDP reply arrives on the other. This doesn't play well > > >> with the (stateful) TM module. Is there a way to synchronize the TM > > >> module accross Kamailio instances using DMQ? > > >> > > >> Cheers, > > >> > > >> Michel Pelletier > > >> __ > > >> Kamailio - Users Mailing List - Non Commercial Discussions > > >> To unsubscribe send an email to sr-users-le...@lists.kamailio.org > > >> Important: keep the mailing list in the recipients, do not reply only to > > >> the sender! > > >> Edit mailing list options or unsubscribe: > > > > > > -- > > > Alex Balashov > > > Principal Consultant > > > Evariste Systems LLC > > > Web: https://evaristesys.com > > > Tel: +1-706-510-6800 > > > > > > > -- > > Alex Balashov > > Principal Consultant > > Evariste Systems LLC > > Web: https://evaristesys.com > > Tel: +1-706-510-6800 > > > > __ > > Kamailio - Users Mailing List - Non Commercial Discussions > > To unsubscribe send an email to sr-users-le...@lists.kamailio.org > > Important: keep the mailing list in the recipients, do not reply only to > > the sender! > > Edit mailing list options or unsubscribe: > > __ > > Kamailio - Users Mailing List - Non Commercial Discussions > > To unsubscribe send an email to sr-users-le...@lists.kamailio.org > > Important: keep the mailing list in the recipients, do not reply only to > > the sender! > > Edit mailing list options or unsubscribe: > > -- > Alex Balashov > Principal Consultant > Evariste Systems LLC > Web: https://evaristesys.com > Tel: +1-706-510-6800 > > __ > Kamailio - Users Mailing List - Non Commercial Discussions > To unsubscribe send an email to sr-users-le...@lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the > sender! > Edit mailing list options or unsubscribe: > __ > Kamailio - Users Mailing List - Non Commercial Discussions > To unsubscribe send an email to sr-users-le...@lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to the > sender! > Edit mailing list options or unsubscribe: -- A
[SR-Users] Re: Using DMQ to sync the TM module.
Hi, I agree. But the example puts me on the right track. It tells me that what I need to do is bounce the reply off to the other nodes(s) if TM doesn't recognize it. Having a DMQ option for the TM module would be great though. Cheers, Michel Pelletier On Tue, Oct 10, 2023 at 4:22 PM Alex Balashov via sr-users < sr-users@lists.kamailio.org> wrote: > I don't think the anycast example is going to get you out of this problem > entirely. > > > On 10 Oct 2023, at 17:22, Michel Pelletier via sr-users < > sr-users@lists.kamailio.org> wrote: > > > > Many thanks. I am afraid I need stateful TM if only for the > retransmissions and how to avoid them. The Anycast example will prove very > useful. > > Cheers, > > > > Michel Pelletier > > > > > > On Tue, Oct 10, 2023 at 11:58 AM Alex Balashov via sr-users < > sr-users@lists.kamailio.org> wrote: > > But I should add: do you actually need state? All replies can be routed > back based on the content of SIP headers alone -- that is to say, > statelessly. Most simple load balancers remain stateless for this very > reason. > > > > > On 10 Oct 2023, at 13:09, Alex Balashov > wrote: > > > > > > There is not. > > > > > >> On 10 Oct 2023, at 12:50, Michel Pelletier via sr-users < > sr-users@lists.kamailio.org> wrote: > > >> > > >> Hi, > > >> > > >> I have 2 kamailio instances behind a load balancer. The problem I > have is that the load balancer can only track TCP connections, but not > UDP. So one Kamailio instance might send a request using UDP, while the > corresponding UDP reply arrives on the other. This doesn't play well with > the (stateful) TM module. Is there a way to synchronize the TM module > accross Kamailio instances using DMQ? > > >> > > >> Cheers, > > >> > > >> Michel Pelletier > > >> __ > > >> Kamailio - Users Mailing List - Non Commercial Discussions > > >> To unsubscribe send an email to sr-users-le...@lists.kamailio.org > > >> Important: keep the mailing list in the recipients, do not reply only > to the sender! > > >> Edit mailing list options or unsubscribe: > > > > > > -- > > > Alex Balashov > > > Principal Consultant > > > Evariste Systems LLC > > > Web: https://evaristesys.com > > > Tel: +1-706-510-6800 > > > > > > > -- > > Alex Balashov > > Principal Consultant > > Evariste Systems LLC > > Web: https://evaristesys.com > > Tel: +1-706-510-6800 > > > > __ > > Kamailio - Users Mailing List - Non Commercial Discussions > > To unsubscribe send an email to sr-users-le...@lists.kamailio.org > > Important: keep the mailing list in the recipients, do not reply only to > the sender! > > Edit mailing list options or unsubscribe: > > __ > > Kamailio - Users Mailing List - Non Commercial Discussions > > To unsubscribe send an email to sr-users-le...@lists.kamailio.org > > Important: keep the mailing list in the recipients, do not reply only to > the sender! > > Edit mailing list options or unsubscribe: > > -- > Alex Balashov > Principal Consultant > Evariste Systems LLC > Web: https://evaristesys.com > Tel: +1-706-510-6800 > > __ > Kamailio - Users Mailing List - Non Commercial Discussions > To unsubscribe send an email to sr-users-le...@lists.kamailio.org > Important: keep the mailing list in the recipients, do not reply only to > the sender! > Edit mailing list options or unsubscribe: > __ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-le...@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
[SR-Users] Re: Using DMQ to sync the TM module.
It would seem the best solution is to use something that can use the SIP protocol for making routing decisions - for example, Kamailio acting as a stateless load balancer, either as a replacement for your load balancer or between your layer4 load balancer (just doing pure round-robin of received packets) and your stateful SIP endpoint. A simple deterministic algorithm (like modulo over a hash of the called) will ensure that a CANCEL will reach the same stateful SIP endpoint as the INVITE. -Original Message- From: Alex Balashov via sr-users Sent: Tuesday, October 10, 2023 12:31 PM To: Kamailio (SER) - Users Mailing List Cc: Alex Balashov Subject: [SR-Users] Re: Using DMQ to sync the TM module. CAUTION: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. But I should add: do you actually need state? All replies can be routed back based on the content of SIP headers alone -- that is to say, statelessly. Most simple load balancers remain stateless for this very reason. > On 10 Oct 2023, at 13:09, Alex Balashov wrote: > > There is not. > >> On 10 Oct 2023, at 12:50, Michel Pelletier via sr-users >> wrote: >> >> Hi, >> >> I have 2 kamailio instances behind a load balancer. The problem I have is >> that the load balancer can only track TCP connections, but not UDP. So one >> Kamailio instance might send a request using UDP, while the corresponding >> UDP reply arrives on the other. This doesn't play well with the (stateful) >> TM module. Is there a way to synchronize the TM module accross Kamailio >> instances using DMQ? >> >> Cheers, >> >> Michel Pelletier >> __ >> Kamailio - Users Mailing List - Non Commercial Discussions To >> unsubscribe send an email to sr-users-le...@lists.kamailio.org >> Important: keep the mailing list in the recipients, do not reply only to the >> sender! >> Edit mailing list options or unsubscribe: > > -- > Alex Balashov > Principal Consultant > Evariste Systems LLC > Web: > https://evar/ > istesys.com%2F&data=05%7C01%7Cbkaufman%40bcmone.com%7Ce635413347124b00 > 36fc08dbc9b7857f%7Cafc1818e7b6848568913201b9396c4fc%7C1%7C0%7C63832556 > 2303100216%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzI > iLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=6GfRy1N%2BIYEob% > 2FXoMnrfBnWUYQePN6SJVCRYuV6czPQ%3D&reserved=0 > Tel: +1-706-510-6800 > -- Alex Balashov Principal Consultant Evariste Systems LLC Web: https://evaristesys.com/ Tel: +1-706-510-6800 __ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-le...@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: __ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-le...@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
[SR-Users] Re: 503(service unavailable)
Thank you very much for your quick response! I am considering the option of switching to 5.7, but I would like to understand this error now and fix it. Most of the clients are behind NAT, but Kamailio and RTPEngiene have a white IP, and billing to whom Kamailio contacts to receive a quota and further billing are located in the same subnet. The database is on the same server as Kamailio. There is one more point that I did not mention above, before the 2nd accident occurs: cdp[api_process.c:110] api_callback(): Recived diameter response outside of threshold (500) - [520-800]. CDP config: Vendor_Id="10415" Product_Name="CDiameterPeer" AcceptUnknownPeers="0" DropUnknownOnDisconnect="0" Tc="30" Workers="10" QueueLength="2" ConnectTimeout="5" TransactionTimeout="5" SessionsHashSize="128" DefaultAuthSessionTimeout="60" MaxAuthSessionTimeout="300" > Are these parameters enough for cps 70-100? I tried "kamctl trap" several times but couldn't figure it out. The main question that worries me is why, when the workers are full, the service freezes and does not recover on its own? Only restarting the service helps. __ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-le...@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
[SR-Users] Re: rtpproxy -- timeout_socket
Hi Maksym, Thanks, I'll try it out and let you know.__ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-le...@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
[SR-Users] Re: 503(service unavailable)
Hello, Kamailio 5.5.x series are no longer maintained, you should upgrade to a supported version, for example 5.7.x series. Then, it seems there is a problem with transmission, either the targets of tcp/tls connections do not accept them, or the corresponding connections get stuck. If you look at kamailio core cookbook, there are some parameters to tune the size of outgoing tcp buffers, but probably getting the queue filled up is the effect of another cause and trying to handle better the effect is not going to help much in long term, just prolong a bit more. You should figure out the cause, maybe you have end points behind the nat and try to open connections towards them? You can also look at the output generated by "kamctl trap" (it requires gdb) to see what each kamailio process is doing. You can also increase the debug level when such log messages appear to get more details in the syslog (there is a rpc command to change debug level at runtime without restart). Cheers, Daniel On 11.10.23 11:59, Masud via sr-users wrote: > Hello! > It often happens to me that the service begins to refuse service (503) and so > far I have > not been able to understand the exact reason. Typically, before clients start > receiving > 503, many before them receive a 408 code. As soon as the service starts > responding with > 503, it does not recover on its own, you have to restart. There is no way to > understand > from the logs what led to this (nothing unusual). My server is powerful > enough for the > number of users of my service (what I mean is that there is no problem with > resources). > For example, 64 cores, 125GB of DDR4 RAM, 2TB of disk, of which 1TB SSD for > the database, > 10Gigabit channel, this is only a server for Kamailio for Rtpengines separate > servers. > Please help me figure out what causes a service failure!? > > OS: Debian 11, 64 cores, 125GB of DDR4 RAM; DB: MariaDB 10.5.15; > > kamailio -v > version: kamailio 5.5.3 (x86_64/linux) 473cef > flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, > USE_MCAST, > DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, > DBG_SR_MEMORY, > USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, > USE_NAPTR, > USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED > ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, > BUF_SIZE 65535, > DEFAULT PKG_SIZE 8MB > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > > udp_children: 30; tcp_children: 34; TLS: YES; async_workers: 64. > Incoming calls are sent via push notifications( Federico Cabiddu method: > https://www.voztovoice.org/sites/default/files/KamilioWorld2015%20-Federico…). > > NetBridging(for SIP and RTPEngine). > ims_charging for billing (integration with our billing system using the > Diameter > protocol). > > In the logs exactly at the moment when the service began to refuse, the > recording disappears completely for 7 > minutes, absolutely no entries, after which the recording of the TCP queue > full begins. > > Oct 3 18:38:07 sip1-life3 kamailio[3380553]: CRITICAL: {2 3691 INVITE > 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: > msg_send_buffer(): > tcp_send failed > Oct 3 18:38:09 sip1-life3 kamailio[3380539]: CRITICAL: {2 3691 INVITE > 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: > msg_send_buffer(): > tcp_send failed > Oct 3 18:38:12 sip1-life3 kamailio[3380534]: CRITICAL: {2 30106 INVITE > 032f8650-1d90-49ed-82f1-c50ccbef191d} tm [../../core/forward.h:292]: > msg_send_buffer(): > tcp_send failed > Oct 3 18:45:14 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: > [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, > 285 requests > queued (total handled 2899418) > Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: > [core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, > 286 requests > queued (total handled 2964063) > Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: > [core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, > 286 requests > queued (total handled 2923484) > Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: > [core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, > 286 requests > queued (total handled 2911633) > Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: > [core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, > 286 requests > queued (total handled 2907132) > Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: > [core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, > 286 requests > queued (total handled 2903281) > Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: > [core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, > 286 requests > queu
[SR-Users] Kamailio Developers Meeting, Nov 7-8, 2023, in Dusseldorf
Hello, Kamailio SIP Server project is organizing another meeting of its developers and community members during November 7-8, 2023, hosted again by sipgate.de in Dusseldorf, Germany. The event is intended to facilitate the interaction between Kamailio developers and contributors in order to offer a convenient environment for working together on several topics of high interest for the project, including writing code for Kamailio and its tools, improving documentation, or discuss about future development. Everyone from the community is welcome to join, developer or user interested in helping the project. Please note we have a limited capacity of seats in the meeting room, the main policy for accepting participants being first come first server. Also, very important to be aware that this is not an event to learn how to use Kamailio. More details about the event, the venue, how to register, are available at: * https://www.kamailio.org/w/developers-meeting/ Looking forward to those two intensive hacking Kamailio days in Dusseldorf! Cheers, Daniel -- Daniel-Constantin Mierla (@ asipto.com) twitter.com/miconda -- linkedin.com/in/miconda Kamailio Consultancy and Development Services Kamailio Advanced Training - Online - Nov 14-16, 2023 -- asipto.com __ Kamailio - Users Mailing List - Non Commercial Discussions To unsubscribe send an email to sr-users-le...@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe:
[SR-Users] Wrong RTP endpoint announced by Kamailio when forwarding calls, but looks correct from RTPEngine
Hi all, I have the following setup: Phone <-> private net using TCP transport <-> (172.30.0.156) Kamailio v5.6.3 + RTPEngine v9.3.1 (22.33.177.156) <-> public net using UDP transport <-> Asterisk (22.33.178.77) Phone is registered on Kamailio and forward calls on no answer or when ext is unregistered, to Asterisk's custom IVR (nodejs app + CouchDB), using failure_route() [1]. The problem is that when the none answer the phone and the call is redirected to Asterisk, Kamailio announce its private IP address in the SDP connection endpoint, hence there is no audio [2]. From RTPEngine debug logs, it seems that RTPEngine replies with the correct SDP connection endpoint IP [3]. See also RTPEngine interfaces definition [4]. What am I missing? Why Kamailio is offering its private IP address in the SDP connection, instead of its public address when forwarding the call? If comment "route(SDP_MANAGE_IN)" in route[IN], Kamailio uses the correct IP endpoint address in the SDP offer when forwarding the call to Asterisk. But of course this breaks audio when someone answer the phone. It seems that Kamailio is storing the initial SDP offer settings and does not update it upon RTPEngine's request, even though I have added "replace-session-connection" in route[SDP_OFFER_IVR]. Thanks for any help! leo [1] route[IN] { if (!lookup('location')) { t_on_reply("SDP_MANAGE"); route(TOIVR); } else { xlog("L_INFO", "Route(IN) Req $mi $rm From <$fu> To <$tu> RURI <$ru>\n"); record_route(); t_on_reply("SDP_MANAGE"); route(SDP_MANAGE_IN); t_on_failure("TOIVR"); t_relay(); exit; } } route[SDP_MANAGE] { if (has_body("application/sdp")) { rtpengine_manage(); } } route[SDP_MANAGE_IN] { if (has_body("application/sdp")) { rtpengine_manage("direction=public direction=private via-branch=1"); } } ... route[SDP_OFFER_IVR] { if (has_body("application/sdp")) { rtpengine_offer("direction=public direction=public replace-session-connection via-branch=1"); } } failure_route[TOIVR] { if (t_is_canceled()) { exit; } append_branch(); route(SDP_OFFER_IVR); # using t_relay_to so that UDP is used with the correct IP address if (!t_relay_to("xh.voip.net")) { xlog("L_INFO", "No answer - Failure Route(TOIVR) Req $mi $rm >From <$fu> To <$tu> RURI <$ru> failed\n"); t_reply("500", "Unable to route"); } else { xlog("L_INFO", "No answer - Failure Route(TOIVR) Req $mi $rm >From <$fu> To <$tu> RURI <$ru>\n"); } } [2] U 2023/10/11 09:05:43.797393 22.33.177.156:5060 -> 22.33.176.8:5060 #3 SIP/2.0 180 Ringing. Via: SIP/2.0/UDP 22.33.176.8;branch=z9hG4bK2cb2.e559c4a326678f69ae9b4656be7bc36c.0. Via: SIP/2.0/UDP 22.33.176.20;branch=z9hG4bK2cb2.2920561c845ee5279051359f40cb7c0f.0. Via: SIP/2.0/UDP 44.55.73.157:5060;branch=z9hG4bK0cBdccef20c0d8bfefb. Record-Route: . Record-Route: . Record-Route: . Record-Route: . From: "Unavailable" ;tag=gK0c1bb12e. To: ;tag=660096088. Call-ID: 474758100_116775912@44.55.73.157. CSeq: 704680 INVITE. Contact: . Allow: INVITE, INFO, PRACK, ACK, BYE, CANCEL, OPTIONS, NOTIFY, REGISTER, SUBSCRIBE, REFER, PUBLISH, UPDATE, MESSAGE. User-Agent: Yealink SIP-T31P 124.86.0.40. Allow-Events: talk,hold,conference,refer,check-sync. Content-Length: 0. . U 2023/10/11 09:06:03.751585 22.33.177.156:5060 -> 22.33.178.77:5060 #4 INVITE sip:+1855533@172.30.96.2:11815;transport=TCP SIP/2.0. Record-Route: . Record-Route: . Record-Route: . Via: SIP/2.0/UDP 22.33.177.156;branch=z9hG4bK2cb2.afb825e30e294872c72c5e21ef26f3f3.1. Via: SIP/2.0/UDP 22.33.176.8;branch=z9hG4bK2cb2.e559c4a326678f69ae9b4656be7bc36c.0. Via: SIP/2.0/UDP 22.33.176.20;branch=z9hG4bK2cb2.2920561c845ee5279051359f40cb7c0f.0. Via: SIP/2.0/UDP 44.55.73.157:5060;branch=z9hG4bK0cBdccef20c0d8bfefb. f: "Unavailable" ;tag=gK0c1bb12e. t: . i: 474758100_116775912@44.55.73.157. CSeq: 704680 INVITE. Max-Forwards: 32. Allow: INVITE,ACK,CANCEL,BYE,REGISTER,REFER,INFO,SUBSCRIBE,NOTIFY,PRACK,UPDATE,OPTIONS,MESSAGE,PUBLISH. Accept: application/sdp, application/isup, application/dtmf, application/dtmf-relay, multipart/mixed. m: "Unavailable" . Remote-Party-ID: "Unavailable" ;privacy=off. k: timer,100rel,precondition. Session-Expires: 1800. Min-SE: 90. l: 604. Content-Disposition: session; handling=required. c: application/sdp. . v=0. o=Sonus_UAC 525653 764118 IN IP4 44.55.73.157. s=SIP Media Capabilities. c=IN IP4 172.30.0.156. t=0 0. m=audio 30030 RTP/AVP 0 18 101. a=maxptime:20. a=rtpmap:0 PCMU/8000. a=rtpmap:18 G729/8000. a=fmtp:18 annexb=n [3] Oct 11 09:06:03 voipsipr7 daemon.info rtpengine[35061]: INFO: [474758100_116775912@44.55.73.157]: [control] Received command 'offer' from 127.0.0.1:39942 Oct 11 09:06:03 voipsipr7 daemon.debug rtpengine[35061]: DEBUG: [474758100_116775912@44.55.73.157]: [control] Dump for 'offer' from 127.0.0.1:39942: { "supports": [ "load limit" ], "sdp": "v=0^M o=Sonus_UA
[SR-Users] 503(service unavailable)
Hello! It often happens to me that the service begins to refuse service (503) and so far I have not been able to understand the exact reason. Typically, before clients start receiving 503, many before them receive a 408 code. As soon as the service starts responding with 503, it does not recover on its own, you have to restart. There is no way to understand from the logs what led to this (nothing unusual). My server is powerful enough for the number of users of my service (what I mean is that there is no problem with resources). For example, 64 cores, 125GB of DDR4 RAM, 2TB of disk, of which 1TB SSD for the database, 10Gigabit channel, this is only a server for Kamailio for Rtpengines separate servers. Please help me figure out what causes a service failure!? OS: Debian 11, 64 cores, 125GB of DDR4 RAM; DB: MariaDB 10.5.15; kamailio -v version: kamailio 5.5.3 (x86_64/linux) 473cef flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. udp_children: 30; tcp_children: 34; TLS: YES; async_workers: 64. Incoming calls are sent via push notifications( Federico Cabiddu method: https://www.voztovoice.org/sites/default/files/KamilioWorld2015%20-Federico…). NetBridging(for SIP and RTPEngine). ims_charging for billing (integration with our billing system using the Diameter protocol). In the logs exactly at the moment when the service began to refuse, the recording disappears completely for 7 minutes, absolutely no entries, after which the recording of the TCP queue full begins. Oct 3 18:38:07 sip1-life3 kamailio[3380553]: CRITICAL: {2 3691 INVITE 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:38:09 sip1-life3 kamailio[3380539]: CRITICAL: {2 3691 INVITE 672699d1-c76b-4153-b72a-fe647913a9e9} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:38:12 sip1-life3 kamailio[3380534]: CRITICAL: {2 30106 INVITE 032f8650-1d90-49ed-82f1-c50ccbef191d} tm [../../core/forward.h:292]: msg_send_buffer(): tcp_send failed Oct 3 18:45:14 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 285 requests queued (total handled 2899418) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 1, socket 204: queue full, 286 requests queued (total handled 2964063) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 4, socket 210: queue full, 286 requests queued (total handled 2923484) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 6, socket 214: queue full, 286 requests queued (total handled 2911633) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 7, socket 216: queue full, 286 requests queued (total handled 2907132) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 8, socket 218: queue full, 286 requests queued (total handled 2903281) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 9, socket 220: queue full, 286 requests queued (total handled 2903438) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 11, socket 224: queue full, 286 requests queued (total handled 2899419) Oct 3 18:45:15 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 12, socket 226: queue full, 286 requests queued (total handled 2897842) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 16, socket 234: queue full, 286 requests queued (total handled 2895429) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 17, socket 236: queue full, 286 requests queued (total handled 2895303) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 22, socket 246: queue full, 286 requests queued (total handled 2891873) Oct 3 18:45:16 sip1-life3 /usr/local/sbin/kamailio[3380778]: CRITICAL: [core/tcp_main.c:4170]: send2child(): tcp child 26, socket 254: queue full, 286 requests queued (total handled 2892619) Oct 3 18:45:16 sip1-life3 /usr/local/sbin