I mentioned that, but did not change the topic of the issue... Latest 5.0 from
git being used here, not 5.1. Wanted to find this issue before upgrading.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues
What version are you running now there?
If the issue is reproducible with a lot of calls via sipp, then it sounds like
not being related to t_replicate() of registrations...
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/ka
The server was running over christmas with no issues (but no calls either).
Registrations doesn't seem to be the major problem (and thus not replication).
I can force a memory issue by running a lot of calls with SIPP.
Another observation is that the timer process runs up to 99% of the CPU it's
Finally have a day where it's possible to run som tests. Will try to fire up
GDB :-)
Building latest 5.0 from git.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-345624236__
This is still a problem, even though we managed to push it back a little. After
a few weeks of service, kamailio stopped working properly because it was out of
memory. Had to restart quickly because of other dependencies (important tests
of devices) but will try to isolate the problem with small
Have you tried with gdb script? That should work with any version.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-330154626___
Kamailio (S
Sorry, can't disturb the system until middle of week. Will test then.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-330149090___
Kamailio
I will try installing latest branch. This is the current stats, something is
definitely keeping transactions open.
```
# kamcmd tm.stats
{
current: 5060
waiting: 5055
total: 5065
total_local: 0
rpl_received: 5415
rpl_generated: 266
rpl_sent
Any chance that you can test with latest master branch? I pushed some new
features to list transactions via rpc as well as trigger a clean via rpc.
Alternative would be to backport last patches from tm module in your clone.
I will also add an option to do hard clean on time, but it would be good
I couldn't find a reason after looking around, likely something is triggered at
some point, in certain conditions.
As a workaround, I am considering to add a hard cleanup on timer to the
transactions that leave too long after lifetime, with a log message to gather
some useful information for tr
Did you get the chance to run the gdb script from my previous comment?
Is the shm memory usage still growing without t_replicate?
I master I added a rpc command that prints some details of the transactions in
memory, that should make it easier in the future to see what's there.
--
You are rece
There is something fishy going on here. I removed t_replicate and the server
has been running for six days without getting into trouble.
However when I run kamcmd tm.stats there are 767 waiting transactions, not
freed. It's a very basic configuration using the dialog module, the dialog
event ro
Here is a gdb script snippet to print transaction structures from memory --
hopefully works ok, not tested here yet:
```
set $i=0
while($ientries[$i]
set $tcell = _tm_table->entries[$i].next_c
while($ehead!=(void*)$tcell)
p $tcell
p *$tcell
set $tcell = $tcell->next_c;
end
set $i = $i + 1
end
`
replication stops. Can't find anything unusual in the syslogs. Can attach GDB,
but have no clue on how to look at the transaction structures. Need some help
:-)
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamail
Quite strange ... any relevant log messages in syslog around the time when it
goes astray? Are any new registrations still replicated (sent out) and replied?
Would you be able to attach with gdb to one of the kamailio processes and look
at the transaction structures?
--
You are receiving this
Started test 12:42, all seems quiet and current goes up and down around 20. At
13:14 it begins to climb again, so around 30 minutes quiet. We have two-minute
registration timeouts, so at the time it starts running astray we have
processed 6762 transactions. All over UDP.
--
You are receiving t
After 30 minutes, everything looks fine... So it takes a while before the TM
queue starts to build
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326936424_
After three hours I have 14726 transactions waiting using around 300 accounts
for REGISTER.
I will restart now and see how fast I notice a new queue.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/12
Could you spot if it appears from first replication or starts after a while?
Are you having any test scenario that could be reused to reproduce it? As wrote
previously, I tested with default config + t_replicate_tcp() for each register
and worked.
--
You are receiving this because you commente
No strange tm modparams:
```
# - tm params -
# auto-discard branches from previous serial forking leg
modparam("tm", "failure_reply_mode", 3)
# default retransmission timeout: 30sec
modparam("tm", "fr_timer", 3)
# default invite retransmission timeout after 1xx: 120sec
modparam("tm", "
After a bit mor testing of t_replicate_udp I am sorry to say that we still have
a problem with that. Firewall is open. We see the same locked transactions and
replication has stopped, the other side has an empty usrloc. Kamailio from the
5.0 branch of git before the 5.0.3 release.
Version:
```
#
Latest release as snapshot/tarball 5.0.2, or latest branch 5.0 at some point in
time?
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326302491___
Latest release. Can't access parameters until monday.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-326301700___
Kamailio (SER) - Develop
I tried to reproduce using master, but all ok. Based on default kamailio.cfg, I
have added t_replicate_tcp:
```
# Handle SIP registrations
route[REGISTRAR] {
if (!is_method("REGISTER")) return;
if(isflagset(FLT_NATS)) {
setbflag(FLB_NATB);
#!ifdef WITH_NATSIPPING
So connections are no ok, but the transactions still live forever, right?
Do you know if the replicated requests were replied? Or the replicated
registrations were handled in the location table of the pair server?
--
You are receiving this because you commented.
Reply to this email directly or
After fixing the firewall we still have issues. I see the TCP keepalives
coming, but replication stopped, no SIP messages flowing and tm.stats has a big
pile of open transactions, no more shm memory available.
Current and waiting seems to follow each other. No calls during the night, so
this is
@oej - were those transactions counted in `current` field of the stats?
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325574130___
Kamail
Note for the future: Maybe an event_route for failed replications would be a
good thing.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325332824
Regardless of the TCP connection, it seems like the transactions should time
out anyway...
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325332490__
Not iptables
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325331558___
Kamailio (SER) - Development Mailing List
sr-dev@lists.kamailio.o
If FW was iptables - it doesn't send neither FIN nor RST to existing
connections.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325330627___
What happened was that the firewall closed the TCP connection after a while,
Kamailio continued sending but got dropped.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325330425___
I bet the problem was with established connections been dropped with a
firewall. TCP takes ages to timeout on Linux.
--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325330403
If transaction relies on tcp stack than it could depend on whether firewall
dropped or rejected connections in your initial case. As for "UDP with a
random port" it was most probably icmp rejected.
2017-08-28 14:30 GMT+03:00 Olle E. Johansson :
> Changed to t_replicate_udp with a random port and
Changed to t_replicate_udp with a random port and it seems to not behave in the
same way. I see that transactions are freed after a timeout.
```
# kamcmd tm.stats
{
current: 119
waiting: 17
total: 462
total_local: 0
rpl_received: 0
rpl_generated: 36
I can check with UDP next week.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomment-325139566___
Kamailio (SER) - Developm
Needs to be investigated, indeed. By chance, did you test also with udp? If
yes, same behaviour, or is just for tcp?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/1220#issuecomme
One server replicates to others with t_replicate_tcp. The connection was by
mistake blocked by a firewall and after a few hours the memory of Kamailio ran
out, with over 4 open transactions in tm.stats on a server with around 300
registrations and no calls since restart.
t_replicate transac
38 matches
Mail list logo
