Yeah, I agree with that. And of course, my proposed in-script solution might generate some very undesirable TCP/TLS bottlenecks.
> On Dec 6, 2022, at 10:14 AM, Daniel-Constantin Mierla <mico...@gmail.com> > wrote: > > I did get it that the complain was about Kamailio processing style, but > overall is about getting the SIP traffic from A to B. With or without > proxy in the middle, B has to be able to deal with packets our of order. > It can be a SIP proxy that changes the order or it can be IP routers on > different paths that lead to different order, for B is irrelevant. The > SIP proxy may send in the order of receiving, still B can get them out > of order. > > It is like implementing a TCP stack and then report dysfunctionality to > ISPs/carriers when the IP packets come out of order because IP packets > went on different paths or IP routers made different routing decisions. > > Daniel > > On 06.12.22 14:10, Alex Balashov wrote: >> I suspect that what Jawaid meant by "naive" was that, in Kamailio, there is >> not a central distributor thread which aims to buffer, serialise and >> otherwise order packets prior to distribution into worker processes or >> threads, as is common in many other multiprocess systems. >> >> I agree that "naive" is perhaps not the best choice of vocabulary for it, >> and of course there would be performance downsides to such a centralised >> pipeline. But I think that's what was probably meant. >> >> -- Alex >> >>> On Dec 6, 2022, at 4:15 AM, Daniel-Constantin Mierla <mico...@gmail.com> >>> wrote: >>> >>> Hello, >>> actually your expectation that packets should come in order is "naive", >>> just think about how internet is constructed and IP routing works. In the >>> past it was easy to reproduce on mobile networks scenarios when sending >>> CANCEL very quickly after the INVITE resulted in CANCEL arriving first at >>> the proxy, then the INVITE. >>> Probably you don't get it in your lab environment where you have sipp on >>> the same system as the sip server or one network segment away, but over the >>> internet the packets can get in different order because of network >>> transmission (different IP routing paths). It is the reason in some cases >>> there are jitter buffers and sequence numbers (e.g., in RTP and SIP >>> (CSeq)). In other words, the protocols like RTP or SIP were designed to >>> deal with these out-of-order packets. >>> Ans here is a wikipedia short article enumerating what can cause out of >>> order: >>> - https://en.wikipedia.org/wiki/Out-of-order_delivery >>> That said, if you missed in the message from mailing list archive that you >>> linked to, there is a global parameter that should reduce the risk of >>> sending out of order sip packets to the minimum that can be done from SIP >>> processing point of view based on call-id, but there are still chances that >>> on multi-cpu systems the packets read from the network can get to be >>> processed in different order because of how read on udp sockets is done by >>> kernel/libc and how cpu scheduler allocate cycles to running application >>> processes. >>> Cheers, >>> Daniel >>> On 05.12.22 19:34, Jawaid Bazyar wrote: >>>> Hi, >>>> I have experienced out of order packet processing when testing a simple >>>> Kamailio config. >>>> The configuration is as follows, basically: >>>> request_route{ >>>> record_route(); >>>> enum_query(); >>>> xlog("INVITE ENUM query - To URI $tU"); >>>> forward(); >>>> } >>>> I saw this thread from 2020: >>>> https://www.mail-archive.com/sr-users@lists.kamailio.org/msg11786.html >>>> The issue seems to be that kamailio process scheduling is naïve – i.e., >>>> incoming SIP packets are processed without regard to whether packets >>>> received before this one, are currently being processed. This means any >>>> packets after the initial INVITE that require more processing, can get >>>> reordered. In my test lab I have: >>>> SIPp – UAC >>>> Kamailio Proxy >>>> SIPp – UAS >>>> The proxy uses enum NAPR lookups to route calls to +13038151000 to the UAS. >>>> Now, if I do SIPp UAC o SIPp UAS directly, I have no problems – no out of >>>> order packets. >>>> It is only when I introduce Kamailio in the middle that I get OOO packets. >>>> See the images attached: uac-to-proxy, proxy, and proxy-to-uas. >>>> In this example, Kamailio OOO causes SIPp UAC to fail to “generate audio” >>>> – because UAC does not see packets in the correct order, I never turns on >>>> the simulated audio. Calls that have in-order dialogs, work fine, and SIPP >>>> UAC “pauses” 10 seconds to simulate a phone call. >>>> So far, the error rate runs from 1/1000 to around 1/200 – pretty bad. >>>> In the thread, a few things were suggested. >>>> Have fewer kamailio processes than cores: >>>> Did not resolve issue. >>>> Try route_locks_size = 256 >>>> Did not resolve issue. Though, it did alter it somewhat. >>>> But, it is not clear to me how this works – would this setting restrict >>>> the number of open calls to 256? >>>> Have only one kamailio process: (“children=1”) >>>> This works. “Works”, I should say, because this would >>>> greatly restrict total platform scalability to a point where it is >>>> probably useless for my application. >>>> I saw the same issue discussed in the OpenSIPS mailing list from 2010, >>>> and the response was “this is not a bug”. >>>> Well, I respectfully beg to differ with both OpenSIPS and Kamailio – it IS >>>> a bug. I don’t think a proxy should reorder packets involved in a call in >>>> a non-deterministic way. In the Kamailio list thread, Alex Balashov >>>> discusses the contortions he has to go through to avoid repercussions from >>>> this issue. >>>> Kamailio as-is probably works fine for relatively low-volume operations. >>>> And a lot of the feedback is “why are out of order packets a problem?” OK, >>>> sure, in the very specific example given in the 2020 thread, maybe who >>>> cares. But in my thinking, there is absolutely nothing preventing Kamailio >>>> from generating much more serious OOO scenarios that would cause calls to >>>> fail. In my example, Kamailio OOO causes SIPp to fail to “generate audio”. >>>> Who knows how other SIP stacks will behave? >>>> But the more one will try to scale Kamailio, the more significantly this >>>> out of order processing issue will become. >>>> The solution to this seems very simple and straightforward – put packets >>>> to be processed into a queue PER Call-ID, or something along those lines. >>>> In short, the parallelism should be by call, not by packet. >>>> What say ye? Have I misunderstood something here? >>>> Cheers, >>>> Jawaid >>>> >>>> __________________________________________________________ >>>> Kamailio - Users Mailing List - Non Commercial Discussions >>>> sr-users@lists.kamailio.org >>>> Important: keep the mailing list in the recipients, do not reply only to >>>> the sender! >>>> Edit mailing list options or unsubscribe: >>>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >>>> >>> -- >>> Daniel-Constantin Mierla -- www.asipto.com >>> www.twitter.com/miconda -- www.linkedin.com/in/miconda >>> __________________________________________________________ >>> Kamailio - Users Mailing List - Non Commercial Discussions >>> sr-users@lists.kamailio.org >>> Important: keep the mailing list in the recipients, do not reply only to >>> the sender! >>> Edit mailing list options or unsubscribe: >>> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users >> >> -- >> Alex Balashov | Principal | Evariste Systems LLC >> >> Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) >> Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ >> >> >> __________________________________________________________ >> Kamailio - Users Mailing List - Non Commercial Discussions >> sr-users@lists.kamailio.org >> Important: keep the mailing list in the recipients, do not reply only to the >> sender! >> Edit mailing list options or unsubscribe: >> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > > -- > Daniel-Constantin Mierla -- www.asipto.com > www.twitter.com/miconda -- www.linkedin.com/in/miconda > -- Alex Balashov | Principal | Evariste Systems LLC Tel: +1-706-510-6800 / +1-800-250-5920 (toll-free) Web: http://www.evaristesys.com/, http://www.csrpswitch.com/ __________________________________________________________ Kamailio - Users Mailing List - Non Commercial Discussions sr-users@lists.kamailio.org Important: keep the mailing list in the recipients, do not reply only to the sender! Edit mailing list options or unsubscribe: https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
