Daniel-Constantin Mierla writes:
> Try to install debugging symbols for libmariadb and maybe there will be
> more hints in the backtace of what it does internally, which can improve
> troubleshooting by searching on the web for similar cases.
I did that already earlier when I reported about simil
On 12.03.20 09:09, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>>> DB access works fine from K worker processes during the time when
>>> insert is stuck in timer process.
>> Are new records inserted in acc table? I think there are some
>> tools/commands for mysql to inspect the state
Daniel-Constantin Mierla writes:
> > DB access works fine from K worker processes during the time when
> > insert is stuck in timer process.
>
> Are new records inserted in acc table? I think there are some
> tools/commands for mysql to inspect the state a database table, check
> acc table and mi
On 11.03.20 20:12, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> If it happens periodically, maybe you can track why: try to identify
>> apps accessing the database for back up, cdr generation, etc ... as well
>> as infrastructure maintenance operations (vm backup snapshot).
> DB ac
Daniel-Constantin Mierla writes:
> If it happens periodically, maybe you can track why: try to identify
> apps accessing the database for back up, cdr generation, etc ... as well
> as infrastructure maintenance operations (vm backup snapshot).
DB access works fine from K worker processes during t
On 11.03.20 09:04, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> It seems to be the case of a retransmission timeout:
>>
>> #17 0x7f7dc04d4aca in acc_onreply (t=0x7f7d9e3b0650, req=0x7f7d9e357650,
>> reply=0x, code=408) at acc_logic.c:604
>>
>> Code is 408 and t
Daniel-Constantin Mierla writes:
> It seems to be the case of a retransmission timeout:
>
> #17 0x7f7dc04d4aca in acc_onreply (t=0x7f7d9e3b0650, req=0x7f7d9e357650,
> reply=0x, code=408) at acc_logic.c:604
>
> Code is 408 and the reply is faked value. This case is happening
: [SR-Users] out of shm without any visible reason
On 10.03.20 22:54, Juha Heinanen wrote:
> Regarding db_mysql timeout_interval, it has its default value 2, which
> means 6 seconds. The insert was hanging in the timer process much
> longer and no error messages related to abort
On 10.03.20 22:54, Juha Heinanen wrote:
> Regarding db_mysql timeout_interval, it has its default value 2, which
> means 6 seconds. The insert was hanging in the timer process much
> longer and no error messages related to abort appear in syslog. How is
> that possible?
I can't say why it happe
On 10.03.20 20:30, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> There is no async-insert done by acc with db_cluster, it is always
>> standard (sync) insert. You would have to track why the mysql server (or
>> the client library) is blocking from time to time, I don't think it is
>
Regarding db_mysql timeout_interval, it has its default value 2, which
means 6 seconds. The insert was hanging in the timer process much
longer and no error messages related to abort appear in syslog. How is
that possible?
-- Juha
___
Kamailio (SER) -
Daniel-Constantin Mierla writes:
> There is no async-insert done by acc with db_cluster, it is always
> standard (sync) insert. You would have to track why the mysql server (or
> the client library) is blocking from time to time, I don't think it is
> something that kamailio can do.
OK, thanks fo
Daniel-Constantin Mierla writes:
> The process is stuck in mysql client library.
>
> The async insert is not implemented in the db_cluster module, so at this
> moment, if you want it, you have to use acc directly with db_mysql
> module.
Then how is it possible that most of the time accounting us
On 10.03.20 20:07, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> The process is stuck in mysql client library.
>>
>> The async insert is not implemented in the db_cluster module, so at this
>> moment, if you want it, you have to use acc directly with db_mysql
>> module.
> Then how i
– https://skalatan.de/blog/
Kamailio services – https://gilawa.com
-Original Message-
From: sr-users On Behalf Of Juha Heinanen
Sent: Tuesday, March 10, 2020 4:41 AM
To: mico...@gmail.com
Cc: Kamailio (SER) - Users Mailing List
Subject: Re: [SR-Users] out of shm without any visible reason
The process is stuck in mysql client library.
The async insert is not implemented in the db_cluster module, so at this
moment, if you want it, you have to use acc directly with db_mysql module.
Cheers,
Daniel
On 10.03.20 04:41, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> If you
Daniel-Constantin Mierla writes:
> If you can reproduce it, watch what the timer processes do during that
> time frame. Get the list of processes with 'kamctl ps', then when the
> issue is exposed, grab the backtraces of all processes with:
>
> kamctl trap
>
> A file is created with the backtrac
Daniel-Constantin Mierla writes:
> It is ok, just add the documentation for the new function as well.
Done, Juha
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
It is ok, just add the documentation for the new function as well.
Cheers,
Daniel
On 05.03.20 23:48, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> The function can be added, should be easy to extract the code from the
>> rpc command function.
> How about the tm.c patch below? Or s
Daniel-Constantin Mierla writes:
> The function can be added, should be easy to extract the code from the
> rpc command function.
How about the tm.c patch below? Or should the function be placed in
some other tm/.c file and if so, which one?
-- Juha
---
*** /us
Daniel-Constantin Mierla writes:
> There was an unsafe list iteration - can you try with:
>
> -
> https://github.com/kamailio/kamailio/commit/325a45e846faae3e5dfa333727d5fab294e44dac
>
> If all ok, you can backport.
Thanks for the patch. The crash has only happened once, so it is not
easy to
Daniel-Constantin Mierla writes:
> If you can reproduce it, watch what the timer processes do during that
> time frame. Get the list of processes with 'kamctl ps', then when the
> issue is exposed, grab the backtraces of all processes with:
>
> kamctl trap
>
> A file is created with the backtrac
There was an unsafe list iteration - can you try with:
-
https://github.com/kamailio/kamailio/commit/325a45e846faae3e5dfa333727d5fab294e44dac
If all ok, you can backport.
Cheers,
Daniel
On 04.03.20 22:18, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> If you can reproduce it, wa
On 03.03.20 21:16, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> The rpc command tm.clean is like a last resort option when dealing with
>> an unexpected situation that messed up the timer process, otherwise the
>> transactions should be cleaned as they are expired or terminated bec
Daniel-Constantin Mierla writes:
> The rpc command tm.clean is like a last resort option when dealing with
> an unexpected situation that messed up the timer process, otherwise the
> transactions should be cleaned as they are expired or terminated because
> of final response sent upstream, with a
If you can reproduce it, watch what the timer processes do during that
time frame. Get the list of processes with 'kamctl ps', then when the
issue is exposed, grab the backtraces of all processes with:
kamctl trap
A file is created with the backtraces taken with gdb. Sent it over here
to analyze,
On 03.03.20 02:24, Juha Heinanen wrote:
> Daniel-Constantin Mierla writes:
>
>> The $stat(name) can be used for any internal statistic. I think it takes
>> only the name of statistic, not the group, so something like
>> $stat(free_size).
> Thanks, free_size and the others in
>
> core.shmmem
> {
>
Looks like something odd happened there. Is the timer process somehow
stuck? Any jump of the system clock? Virtual machine frozen somehow? Or
the transactions remained hooked in some callback, not releasing the
reference counter.
The rpc command tm.clean is like a last resort option when dealing w
Here is some more details about the shm usage. Usage was steady at
about 10 MB until time 17:05. Then it usage started to steadily grow
during 15 min period and 17:21 hit 80 MB at which point tm.clean was
executed. After that usage dropped straight back to 10 MB level.
How is that possible, i.e
Daniel-Constantin Mierla writes:
> The $stat(name) can be used for any internal statistic. I think it takes
> only the name of statistic, not the group, so something like
> $stat(free_size).
Thanks, free_size and the others in
core.shmmem
{
total: 67108864
free: 58274976
What kind of delay there is on cleaning of an expired transaction? I'm
asking, since tm.clean released lots of shm.
-- Juha
___
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On 01.03.20 21:53, Juha Heinanen wrote:
> John Petrini writes:
>
>> The times we've seen this is when transactions are waiting on something so
>> they pile up consuming shared memory. Do you have any database lookups or
>> calls out to external services or scripts?
> John,
>
> Thanks for your repl
Maybe not the nicest solution, but you can use (from inside the routing
script) the jsonrpc_exec() to get stats and grab the shmem usage, and build
your logic there... if this do that...
On Sun, Mar 1, 2020 at 12:54 Juha Heinanen wrote:
> John Petrini writes:
>
> > The times we've seen this is
John Petrini writes:
> The times we've seen this is when transactions are waiting on something so
> they pile up consuming shared memory. Do you have any database lookups or
> calls out to external services or scripts?
John,
Thanks for your reply. Yes, there are MySQL operations both during
req
K reported during about 90 sec period that it is out of shared memory:
Feb 28 09:47:28 rox1 /usr/bin/sip-proxy[19725]: ERROR: tm
[t_hooks.c:136]: insert_tmcb(): out of shm. mem
Feb 28 09:47:28 rox1 /usr/bin/sip-proxy[19725]: ERROR: acc
[acc_logic.c:394]: acc_onreq(): cannot register additional c
The times we've seen this is when transactions are waiting on something so
they pile up consuming shared memory. Do you have any database lookups or
calls out to external services or scripts?
Long timeouts can also contribute if something stops responding because
transactions are waiting for a lon
36 matches
Mail list logo
