Hi, Alex,
It's 16 virtual cores (8 physical plus HyperThreading) and 48GB of RAM.
Luis
On 4/10/20 8:02 AM, Alex Balashov wrote:
Luis,
I wonder, how many CPU cores/available hardware threads (taking into
account HyperThreading and all that—so just the number of CPUs in
/proc/cpuinfo) are available here? It almost sounds like something
which would occur with maybe 1 or 2 CPUs being contended over. Perhaps
the simplest thing for this kind of call volume is to run Kamailio on
a host with 8 or 12+ CPU cores?
— Alex
—
Sent from my iPad
On Apr 10, 2020, at 2:20 AM, Luis Rojas G. <luis.ro...@sixbell.com>
wrote:
Hello,
I have a lot of experience developing mutithreaded applications, and
I don't see it so unlikely at all that a process loses cpu just after
recvfrom(). It's just as probable as to lose it just before, or when
writing on a cache or just before of after sendto(). If there are
many messages going through, some of them will fall in this scenario.
if I try sending a burst of 100 messages, I see two or three
presenting the scenario.
Just forward() with a single process does not give the capacity. I'm
getting almost 1000caps. More than that and start getting errores,
retransmissions, etc. And this is just one way. I need to receive the
call to go back to the network (our application is a B2BUA), so I
will be down to 500caps, with a simple scenario, with no reliable
responses, reinvites, updates, etc. I will end up having as many
standalone kamailio processes as the current servers I do have now.
I really think the simplest way would be to add a small delay to 200
OK. Very small, like 10ms, should be enough. Simple and it should
work. As Alex Balashov commented he did for the case with ACK-Re-Invite.
I have to figure out how to make async_ms_sleep() work in reply_route().
Thanks for all the comments and ideas
Best regards,
Luis
. On 4/9/20 12:17 PM, Daniel-Constantin Mierla wrote:
mico...@gmail.com appears similar to someone who previously sent you
email, but may not be that person. Learn why this could be a risk
<http://aka.ms/LearnAboutSenderIdentification>
Feedback <http://aka.ms/SafetyTipsFeedback>
Hello,
then the overtaking is in between reading from the socket and
getting to parsing the call-id value -- the cpu is lost by first
reader after recvfrom() and the second process get enough cpu time
to go ahead further. I haven't encountered this case, but as I said
previously, it is very unlikely, but still possible. I added the
route_locks_size because in the past I had cases when processing of
some messages took longer executing config (e.g., due to
authentication, accounting, ..) and I needed to be sure they are
processed in the order they enter config execution.
Then the option is to see if a single process with stateless sending
out (using forward()) gives the capacity, if you don't do any other
complex processing. Or if you do more complex processing, use a
dispatcher process with forwarding to local host or in a similar
manner try to use mqueue+rtimer for dispatching using shared memory
queues.
Of course, it is open source and there is also the C coding way, to
add a synchronizing mechanism to protect against parallel execution
of the code from recvfrom() till call-id lock is acquired.
Cheers,
Daniel
On 09.04.20 17:37, Luis Rojas G. wrote:
Hello,
Well, it did not work at all. Exactly same behavior, with random
out of order messages.
Best regards,
Luis
On 4/9/20 11:28 AM, Daniel-Constantin Mierla wrote:
mico...@gmail.com appears similar to someone who previously sent
you email, but may not be that person. Learn why this could be a
risk <http://aka.ms/LearnAboutSenderIdentification>
Feedback <http://aka.ms/SafetyTipsFeedback>
Hello,
the sip messages belonging to the same dialog have the same value
for Call-Id header. The locking is done based on hashing the
Call-Id, so it doesn't need at all the dialog module for its purpose.
Cheers,
Daniel
On 09.04.20 14:19, Luis Rojas G. wrote:
Hello, Daniel,
I am not so sure. I first tried adding that parameter, but it did
not work at all. Same behavior. Then I read the documentation
more carefully :
https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size
route_locks_size
Set the number of mutex locks to be used for synchronizing the
execution of messages sharing the same Call-Id. In other words,
enables Kamailio to execute the config script sequentially for
the requests and replies received *within the same dialog* – a
new message received *within the same dialog* waits until the
previous one is routed out.
Locks to execute sequentially messages belonging to same dialog.
How will Kamailio be aware that messages belong to same dialog,
without the dialog module?. With just stateless proxy it has no
idea about dialogs, it just forwards messages. I guess that's why
just adding that parameter did not work.
Am I wrong?
Luis
On 4/9/20 3:47 AM, Daniel-Constantin Mierla wrote:
Hello,
On 08.04.20 23:03, Luis Rojas G. wrote:
Hello, Daniel,
I looked into that parameter, but I need to use with the
dialog module, and I'm pretty afraid to use that.
who said or where is written than you need to load the dialog
module? You definitely don't.
Cheers,
Daniel
I was looking more into the stateless proxy, because I need to
process a lot of traffic.
My target is 4200CAPS. with duration between 90s and 210. Let's
say, 150 seconds. That would mean 630.000 simultaneous dialogs.
I don't think the solution can go that way.
it would really help me to be able to use completely stateless
proxy plus Async in reply_route(), to introduce an artificial
delay before forwarding 200 OK to Invite.. As someone
mentioned, it would help me on request_route(), for race
conditions between ACK and Re-Invite.
Any idea why Async is not allowed in reply_route()?
Best regards,
Luis
On 4/8/20 1:07 PM, Daniel-Constantin Mierla wrote:
Hello,
you have to keep in mind that Kamailio is a SIP packet router,
not a telephony engine. If 180 and 200 replies are part of a
call is not something that Kamailio recognize at its core. Its
main goal is to route out as fast as possible what is
received, by executing the configuration file script. Now, a
matter of your configuration file, processing of some SIP
messages can take longer than processing other. And the
processing is done in parallel, a matter of children parameter
(and tcp_children, sctp_children).
With that in mind, a way to try to cope better with the issue
you face is to set route_locks_size parameter, see:
*
https://www.kamailio.org/wiki/cookbooks/devel/core#route_locks_size
<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.kamailio.org%2Fwiki%2Fcookbooks%2Fdevel%2Fcore%23route_locks_size&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477633477&sdata=P5gWkVhKbY0EJj6J5jpUZYHL09xAARlgOpvgETTZJGo%3D&reserved=0>
Probably is what you look for.
But if you want more tight constraints, like when receiving a
180 after a 200ok and not route it out, you have to make the
logic in configuration file by combining modules such as
dialog or htable (as already suggested).
Cheers,
Daniel
On 08.04.20 16:04, Luis Rojas G. wrote:
Hi, Henning,
No need to be ironic. As I mentioned on my first post, I
tried stateful proxy and I observed the same behavior.
/"I tried using stateful proxy and I obtained the same result."/
The asynchronous sleep seems promising. I will look into it.
Thanks,
Luis
On 4/8/20 9:30 AM, Henning Westerholt wrote:
Hi Luis,
I see. Well, you want to use Kamailio as a stateless proxy,
on the other hand it should do things that are inherently
stateful. 😉
As mentioned, have a look to the dialog module to track the
state of dialogs that you process. This will not work in a
stateless mode, though.
You can also use the htable module to just store some data
about the processed messages in a shared memory table and
use this to enforce your ordering. There is also the option
to do an asynchronous sleep (with the async) module on the
message that you want to delay but still processing other
messages during it.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskalatan.de%2Fblog%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477643469&sdata=7wVK%2BNcbz0LZBJKlnUgvt0XqOYLQh2xZoasaU0q29qI%3D&reserved=0>
Kamailio services – https://gilawa.com
<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgilawa.com%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477643469&sdata=CHNJCNYzwqEkgdLXXyfKcv8McuEutS4PPhCX5Oe9FGE%3D&reserved=0>
*From:* Luis Rojas G. <luis.ro...@sixbell.com>
*Sent:* Wednesday, April 8, 2020 3:00 PM
*To:* Henning Westerholt <h...@skalatan.de>; Kamailio (SER) -
Users Mailing List <sr-users@lists.kamailio.org>
*Subject:* Re: [SR-Users] Kamailio propagates 180 and 200 OK
OUT OF ORDER
Hello, Henning,
I am worried about this scenario, because it's a symptom of
what may happen in other cases. For instance, I've seen that
this operator usually sends re-invites immediate after
sending ACK. This may create race conditions like 3.1.5 of
RFC5407
https://tools.ietf.org/html/rfc5407#page-22
<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc5407%23page-22&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477653469&sdata=57Iwh49x3vUdl5541kBc1mvD50u4w5wQQCovMcvE030%3D&reserved=0>
I'd understand that one happens because of packet loss, as
it's in UDP's nature, but in this case it would be
artificially created by Kamailio. if there was no problem at
network level (packet loss, packets following different path
on the network and arriving out of order), why Kamailio
creates it?
I'd expect that the shared memory is used precisely for
this. If an instance of kamailio receives a 200 OK, it could
check on the shm and say "hey, another instance is
processing a 180 for this call. Let's wait for it to finish"
(*). I know there could still be a problem, the instance
processing the 180 undergoes a context switch just after it
receives the message, but before writing to shm, but it
would greatly reduce the chance.
In our applications we use a SIP stack that always sends
messages to the application in the same order it receives
them, even though is multi-threaded and messages from the
network are received by different threads. So, they really
syncronize between them. Why Kamailio instances don't?
I am evaluating kamailio to use it as a dispatcher to
balance load against our several Application Servers, to
present to the operator just a couple of entrance points to
our platform (they don't want to establish connections to
each one of our servers). This operator is very difficult to
deal with. I am sure they will complain something like "why
are you sending messages out of order? Fix that". The
operator will be able to see traces and check that messages
entered the Kamailio nodes in order and left out of order.
They will not accept it.
(*) Not really "wait", as it would introduce a delay in
processing all messages. it should be like putting it on a
queue, continue processing other messages, and go back to
the queue later.
Well, thanks for your answer.
Luis
On 4/8/20 3:01 AM, Henning Westerholt wrote:
Hello Luis,
as the 1xx responses are usually send unreliable (unless
you use PRACK), you should not make any assumption on
the order or even the arrival of this messages. It can
also happens on a network level, if send by UDP.
Can you elaborate why you think this re-ordering is a
problem for you?
One idea to enforce some ordering would be to use the
dialog module in combination with reply routes and the
textops(x) module.
About the shared memory question – Kamailio implement
its own memory manager (private memory and shared memory
pool).
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fskalatan.de%2Fblog%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477663459&sdata=kUCddL3S%2FQx8%2B1eFbrlSJZy3GSj2OmLHdnXncvdDKfo%3D&reserved=0>
Kamailio services – https://gilawa.com
<https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgilawa.com%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477663459&sdata=sZ3pmtBa9noTCpm6mp5P1Mzelc%2Byxz1XFOB0tvPHRCI%3D&reserved=0>
*From:* sr-users <sr-users-boun...@lists.kamailio.org>
<mailto:sr-users-boun...@lists.kamailio.org> *On Behalf
Of *Luis Rojas G.
*Sent:* Tuesday, April 7, 2020 10:43 PM
*To:* sr-users@lists.kamailio.org
<mailto:sr-users@lists.kamailio.org>
*Subject:* [SR-Users] Kamailio propagates 180 and 200 OK
OUT OF ORDER
Good day,
I am testing the dispatcher module, using Kamailio as
stateless proxy. I have a pool of UAC (scripts in SIPP)
and a pool of UAS (also scripts in SIPP) for the
destinations. Kamailio version is kamailio-5.3.3-4.1.x86_64.
Problem I have is, if UAS responds 180 and 200 OK to
Invite immediately, sometimes they are propagated out of
order. 200 OK before 180, like this :
<image001.png>
UAS is 172.30.4.195:5061. UAC is 172.30.4.195:5080.
Kamailio is 192.168.253.4:5070
Difference between 180 and 200 is just about 50
microseconds.
My guess is that both messages are received by different
instances of Kamailio, and then because of context
switches, even though the 180 is received before, that
process ends after the processing of 200. However, I had
the idea that in order to avoid these problems the
kamailio processes synchronized with each other using a
shared memory. I tried using stateful proxy and I
obtained the same result.
By the way, anyone has any idea about how Kamailio's
share memory is implemented? It clearly does not use the
typical system calls shmget(), shmat(), because they are
not shown by ipcs command.
Before posting here I googled, but I couldn't find
anything related to this. I can't believe I am the only
one who ever had this problem, so I guess I am doing
something wrong...
Please, any help. I'm really stuck on this.
Thanks.
--
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.ro...@sixbell.com
http://www.sixbell.com
<https://nam02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sixbell.com%2F&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477673456&sdata=FPhaFAUYu%2BEg8%2B6R4%2BgpvgG12f3MQJ%2FFJFvhTT2ITP0%3D&reserved=0>
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.ro...@sixbell.com
http://www.sixbell.com
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla --www.asipto.com
www.twitter.com/miconda --www.linkedin.com/in/miconda
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.ro...@sixbell.com
http://www.sixbell.com
--
Daniel-Constantin Mierla --www.asipto.com
www.twitter.com/miconda --www.linkedin.com/in/miconda
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.ro...@sixbell.com
http://www.sixbell.com
--
Daniel-Constantin Mierla --www.asipto.com
www.twitter.com/miconda --www.linkedin.com/in/miconda
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.ro...@sixbell.com
http://www.sixbell.com
--
Daniel-Constantin Mierla --www.asipto.com
www.twitter.com/miconda --www.linkedin.com/in/miconda
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.ro...@sixbell.com
http://www.sixbell.com
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.kamailio.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fsr-users&data=02%7C01%7C%7C839e2eada7ca4f6f254908d7dd47ba45%7Cab4a33c2b5614f798601bc921698ad08%7C0%7C0%7C637221172477823379&sdata=9SJPhjaKPNJyIVJJy6Ag43u1GYl6RbDn2faPAwbHXn4%3D&reserved=0
--
Luis Rojas
Software Architect
Sixbell
Los Leones 1200
Providencia
Santiago, Chile
Phone: (+56-2) 22001288
mailto:luis.ro...@sixbell.com
http://www.sixbell.com
_______________________________________________
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users