Consider scaling out instead of scaling up. Now that you know the apparent limit of a single node for your use case, put it behind a Kamailio dispatcher.
On Thu, Dec 19, 2024 at 7:36 AM Sergio Charrua via sr-users <[email protected]> wrote: > > Hello! > > for this ST/SH project, we are using Kamailio 5.8.4 in stateless mode, making > HTTP requests to a REST API (Java Spring) that will reply with a JSON object, > and then Kamailio replaces the Contact header and sends a SIP 300 Multiple > Choice reply to the SBC that sent the initial INVITE. > > Kamailio is running on a VM with 4vCPUs and 4GB RAM and 2 NICs. > On NIC ens224 there is a Virtual IP managed by Keepalived. > Kamailio is listening on port 5060 on ens224 and set with 32 children process > (tried with 8, 16, 24 and also 64, but with this value it was a lot worst). > > We are doing load performance tests on our PreProd environment. With the SBC > we are sending 450 to 600 CAPS to Kamailio, and what we noticed is that above > 450 CAPS, and after less than 1 minute, Kamailio only replies SIP 100 to some > INVITEs. We also could figure out that Kamailio is not receiving all the > INVITES, despite having proved that ALL invites are sent to the server. > It seems to me that, for some reason, the OS is somehow not able to deliver > *all* SIP packets to Kamailio, because: > - sngrep does capture all the SIP packets and shows the flows > - though the flow shows SIP invite coming from SBC to Kamailio Server, there > is no SIP 100 replied back to SBC, even though this is the 1st thing kamailio > does, so, for me, Kamailio did not received the SIP Invite > - picking a call with reinvites, which after some seconds, SBC cancels, if I > copy the Call-ID value and grep on kamailio's logs, I just do not find any > content! > > I have read some "network tuning" articles, including Linux Tune Network > Stack (Buffers Size) To Increase Networking Performance - nixCraft and Tuning > Kamailio for high throughput and performance | Evariste Systems Blog and when > running the following command I get: > > $ ss -4 -n -l | grep 5060 > udp UNCONN 0 0 10.242.17.125:5060 0.0.0.0:* > udp UNCONN 324224 0 10.242.17.146:5060 0.0.0.0:* > > According to the article, the 3rd column should be as near 0 as possible, > ideally 0. > > While I am waiting for some sysadmin with root permissions to be available > and modify some of the network parameters on this VM, I wonder if anyone has > some tips to share on how to solve this behaviour. I'm not sure, either, what > parameters to change or if they should be changed! The goal is to have a > minimum 750 CAPS. > > Lastly, some parts of the code: > > Kamailio.cfg: >> >> ### LOG Levels: 3=DBG, 2=INFO, 1=NOTICE, 0=WARN, -1=ERR >> debug=1 >> log_stderror=no >> rundir="/tmp" >> memdbg=5 >> memlog=5 >> log_facility=LOG_LOCAL0 >> # configure the prefix for all log messages >> log_prefix_mode = 1 >> log_prefix="{$mt $hdr(CSeq) $ci} " >> /* number of SIP routing processes */ >> children=32 # ------------- ALREADY TRIED WITH 8,16,24... no results..... >> /* uncomment the next line to disable TCP (default on) */ >> disable_tcp=no >> /* uncomment the next line to disable the auto discovery of local aliases >> * based on reverse DNS on IPs (default on) */ >> # auto_aliases=no >> /* add local domain aliases */ >> # alias="sip.mydomain.com" > > > ####### Custom Parameters ######### > > /* These parameters can be modified runtime via RPC interface > * - see the documentation of 'cfg_rpc' module. > * > * Format: group.id = value 'desc' description > * Access: $sel(cfg_get.group.id) or @cfg_get.group.id */ > > ####### Modules Section ######## > > /* set paths to location of modules */ > loadmodule "db_mysql.so" > loadmodule "db_cluster.so" > loadmodule "http_client.so" > loadmodule "jsonrpcs.so" > loadmodule "kex.so" > loadmodule "corex.so" > loadmodule "sl.so" > loadmodule "pv.so" > loadmodule "maxfwd.so" > loadmodule "textops.so" > loadmodule "xlog.so" > loadmodule "sanity.so" > loadmodule "jansson.so" > loadmodule "snmpstats.so" > loadmodule "file_out.so" > loadmodule "ctl.so" > loadmodule "permissions.so" > loadmodule "xhttp.so" > loadmodule "xhttp_rpc.so" > > ####### Other Args and Env Vars ######### > > /* listen addresses */ > include_file "listen.cfg" > include_file "db_conn.cfg" > > # ----------------- setting module-specific parameters -------------- > # --- DB Cluster params --- > # minimum requirement is to have DBURL1 defined > #!ifdef DBURL1 > modparam("db_cluster", "connection" , DBURL1) > #!endif > #!ifdef DBURL2 > modparam("db_cluster", "connection" , DBURL2) > #!endif > #!ifdef DBURL3 > modparam("db_cluster", "connection" , DBURL3) > #!endif > #!ifdef DBURL4 > modparam("db_cluster", "connection" , DBURL4) > #!endif > #!ifdef DBURL5 > modparam("db_cluster", "connection" , DBURL5) > #!endif > modparam("db_cluster", "cluster", DBCLUSTER) > > # --- Permissions params --- > modparam("permissions", "db_url", "cluster://k1") > modparam("permissions", "db_mode", 1) > #modparam("permissions", "reload_delta", 30) > > > # ----- jsonrpcs params ----- > modparam("jsonrpcs", "pretty_format", 1) > /* set the path to RPC fifo control file */ > modparam("jsonrpcs", "fifo_name", "/tmp/kamailio_rpc.fifo") > /* set the path to RPC unix socket control file */ > modparam("jsonrpcs", "dgram_socket", "/tmp/kamailio_rpc.sock") > > modparam("jsonrpcs", "transport", 0) > # ----- ctl params ----- > /* set the path to RPC unix socket control file */ > modparam("ctl", "binrpc", "unix:/tmp/kamailio_ctl") > > # ----- http_async_client params ----- > modparam("http_client", "query_result", 0) > modparam("http_client", "keep_connections", 0) > modparam("http_client", "connection_timeout",2) > modparam("http_client", "timeout_mode",2) > modparam("http_client", "config_file", > "/usr/local/etc/kamailio/http_client.cfg") > # ---- SNMP Stats params ---- > modparam("snmpstats", "sipEntityType", "proxyServer") > modparam("snmpstats", "sipEntityType", "redirectServer") > modparam("snmpstats", "sipEntityType", "other") > modparam("snmpstats", "snmpgetPath", "/usr/bin/") > modparam("snmpstats", "MsgQueueMinorThreshold", 20) > modparam("snmpstats", "MsgQueueMajorThreshold", 100) > modparam("snmpstats", "dlg_minor_threshold", 20) > modparam("snmpstats", "dlg_major_threshold", 100) > modparam("snmpstats", "snmpCommunity", "kamailio") > > # ---- File_Out params ---- > modparam("file_out", "base_folder", "/data/sabire002/log/kamailio/") > modparam("file_out", "file", > "name=stsh_requests;interval=1440;extension=.log") > modparam("file_out", "file", "name=cdr;interval=1440;extension=.log") > modparam("file_out", "file", "name=http;interval=1440;extension=.log") > ####### Routing Logic ######## > > include_file "includes/reqinit.cfg" > include_file "includes/handle_options.cfg" > include_file "includes/handle_cancel.cfg" > include_file "includes/handle_stir_shaken_stateless.cfg" > include_file "includes/handle_http_rpc.cfg" > > request_route { > $avp(START_TIME)=$utimef(%Y-%m-%d %H:%M:%S); > $avp(GROUPID) = allow_address_group($si, $sp); > > if ( $avp(GROUPID) == 100 || !allow_address_group($si, $sp) ) { > xlog("L_INFO", "INIT - $si:$sp is not in the allowed ACL Group ID > !\n"); > #sl_reply("401", "Address not allowed"); > exit; > }; > > if (is_method("ACK") ) { #&& t_check_trans() ){ > exit; > } > > if (is_method("INVITE")) { > > file_out("cdr","$rm|$ft|$tt|$ci|$rs|$rr|$Ts|$avp(START_TIME)|$fU|$fd|$si|$tU|$rU|$rd|$utimef(%Y-%m-%d > %H:%M:%S)"); > send_reply("100","Trying"); > } > > route(HANDLE_OPTIONS); > > route(REQINIT); > > xlog("L_INFO"," ********** Route START ***********"); > # log the basic info regarding this call > xlog("L_INFO","=================================================== \n"); > xlog("L_INFO","New SIP message $rm with call-ID $ci received $pr request > $rm $ou source $si:$sp from $fu to $tu \n"); > xlog("L_INFO","=================================================== \n"); > > route(HANDLE_STIRSHAKEN); > > route(HANDLE_CANCEL); > > if (method == "INVITE"){ > route(RELAY); > } > } > > route[RELAY] { > # Sends a 300 Multiple Choices back to the proxy that requested the routing > lookup > xlog("L_INFO","RELAY - send reply \n"); > > file_out("cdr","RELAY|$ft|$tt|$ci|$rs|$rr|$Ts|$avp(START_TIME)|$fU|$fd|$si|$tU|$rU|$rd|$utimef(%Y-%m-%d > %H:%M:%S)"); > send_reply("300", "Multiple Choices"); > exit; > } > > As for the HANDLE_STIRSHAKEN route, where the main process is: > > route[HANDLE_STIRSHAKEN] > { > > if (!is_method("INVITE")) { > return; > } > > $xavp(requestTime) = $utimef(%s); > $var(post) = $null; // resets and makes sure the $var(post) variable is > null before usage > > [.... get some header values ...] > > $var(res) = http_connect("api1", "/stsh", "application/json", "$var(post)", > "$var(result)"); > jansson_xdecode($var(result), "json"); > if ( $xavp(json=>sip-response-code) == 300 ) > { > remove_hf("Contact"); > append_to_reply("Contact: $xavp(json=>contact)\r\n"); > } > else > { > # add error message to reply > > sl_reply("$xavp(json=>sip-response-code)","$xavp(json=>sip-response-text)"); > exit; > } > } > > > > > Atenciosamente / Kind Regards / Cordialement / Un saludo, > > > Sérgio Charrua > > > __________________________________________________________ > Kamailio - Users Mailing List - Non Commercial Discussions -- > [email protected] > To unsubscribe send an email to [email protected] > Important: keep the mailing list in the recipients, do not reply only to the > sender! __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions -- [email protected] To unsubscribe send an email to [email protected] Important: keep the mailing list in the recipients, do not reply only to the sender!
