subject:"\"Re\\\: \\\[OpenSIPS\\\-Users\\\] OpenSIPS core dumps\""

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-10-14 Thread thrillerbee

I have this info from dmesg:

[1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]

[2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp
7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]

On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee  wrote:

> When OpenSIPS crashes, three corefiles are generated that are 2.1GB in
> size.  How do I use these files to understand what's causing the crash?
>
> Thanks.
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-10-15 Thread Anca Vamanu


Hi,

You need to inspect them with gdb, run: gdb path_to_opensips_executable 
path_to_corefile, and then run 'bt full' and send the output.


Regards,

--
Anca Vamanu
www.voice-system.ro



On 10/14/2010 10:12 PM, thrillerbee wrote:

I have this info from dmesg:

[1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
[1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 
7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]


[2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp 
7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp 
7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
[2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp 
7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]


On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee > wrote:


When OpenSIPS crashes, three corefiles are generated that are
2.1GB in size.  How do I use these files to understand what's
causing the crash?

Thanks.


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-10-22 Thread Bogdan-Andrei Iancu

Any chance with the backtraces ?

Regards,
Bogdan

Anca Vamanu wrote:
> Hi,
>
> You need to inspect them with gdb, run: gdb 
> path_to_opensips_executable path_to_corefile, and then run 'bt full' 
> and send the output.
>
> Regards,
> -- 
> Anca Vamanu
> www.voice-system.ro
>
>
> On 10/14/2010 10:12 PM, thrillerbee wrote:
>> I have this info from dmesg:
>>
>> [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp 
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp 
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp 
>> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>>
>> [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp 
>> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp 
>> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp 
>> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>>
>> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee > > wrote:
>>
>> When OpenSIPS crashes, three corefiles are generated that are
>> 2.1GB in size.  How do I use these files to understand what's
>> causing the crash?
>>
>> Thanks.
>>
>>
> 
>
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>   


-- 
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-01 Thread thrillerbee

Bogdan,

Yes,  I've attached 2 to my response - one was a crash from 10/29.  The
other occurred a few minutes ago.

Thanks.

On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu <
bog...@voice-system.ro> wrote:

> Any chance with the backtraces ?
>
> Regards,
> Bogdan
>
> Anca Vamanu wrote:
> > Hi,
> >
> > You need to inspect them with gdb, run: gdb
> > path_to_opensips_executable path_to_corefile, and then run 'bt full'
> > and send the output.
> >
> > Regards,
> > --
> > Anca Vamanu
> > www.voice-system.ro
> >
> >
> > On 10/14/2010 10:12 PM, thrillerbee wrote:
> >> I have this info from dmesg:
> >>
> >> [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> >> [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> >> [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> >>
> >> [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp
> >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> >> [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp
> >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> >> [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp
> >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> >>
> >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee  >> > wrote:
> >>
> >> When OpenSIPS crashes, three corefiles are generated that are
> >> 2.1GB in size.  How do I use these files to understand what's
> >> causing the crash?
> >>
> >> Thanks.
> >>
> >>
> > 
> >
> > ___
> > Users mailing list
> > Users@lists.opensips.org
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >
>
>
> --
> Bogdan-Andrei Iancu
> OpenSIPS Bootcamp
> 15 - 19 November 2010, Edison, New Jersey, USA
> www.voice-system.ro
>
>
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
Core was generated by `/usr/local/sbin/opensips -P 
/var/run/opensips/opensips.pid -m 512 -u root -g ro'.
Program terminated with signal 11, Segmentation fault.
[New process 22242]
#0  0x7f66e354821f in flat_db_insert (h=0x7ea710, k=0x7f66e2288a60, 
v=0x7f66e2288d20, n=19) at flatstore.c:165
165 f = CON_FILE(h);
(gdb) bt full
#0  0x7f66e354821f in flat_db_insert (h=0x7ea710, k=0x7f66e2288a60, 
v=0x7f66e2288d20, n=19) at flatstore.c:165
f = 
i = 
l = 
p = 
__FUNCTION__ = "flat_db_insert"
#1  0x7f66e206f7ef in acc_db_request (rq=0x7f66c34266b8, rpl=) at acc.c:364
m = 19
n = 
i = 
my_ps = (db_ps_t) 0x0
__FUNCTION__ = "acc_db_request"
#2  0x7f66e207469e in tmcb_func (t=, type=, ps=) at acc_logic.c:386
No locals.
#3  0x7f66e2efd4a2 in run_trans_callbacks (type=256, trans=0x7f66c314eba0, 
req=0x7f66c34266b8, rpl=0x7eaac8, code=200) at t_hooks.c:208
cbp = (struct tm_callback *) 0x7f66c1892ac8
backup = (struct usr_avp **) 0x771f48
trans_backup = (struct cell *) 0x7f66c314eba0
__FUNCTION__ = "run_trans_callbacks"
#4  0x7f66e2efd5c7 in run_trans_callbacks_locked (type=256, 
trans=0x7f66c314eba0, req=0x7f66c34266b8, rpl=0x7eaac8, code=200) at 
t_hooks.c:254
No locals.
#5  0x7f66e2f1aa38 in relay_reply (t=0x7f66c314eba0, p_msg=, branch=1, msg_status=200, cancel_bitmap=0x7fff3aa31ca8) at t_reply.c:1257
relay = 1
save_clone = 0
buf = 0x11846d0 "SIP/2.0 200 OK\r\nRecord-Route: 
\r\nRecord-Route: 
\r\nVia: SIP/2.0/UDP 
24.121.80.36;branch=z9hG4bK13cd.038ec486.0\r\nV"...
res_len = 685
relayed_code = 200
relayed_msg = (struct sip_msg *) 0x7eaac8
bm = {to_tag_val = {s = 0x1 , len = 8301256}}
totag_retr = 
reply_status = RPS_COMPLETED
cb_s = {
  s = 0x11846d0 "SIP/2.0 200 OK\r\nRecord-Route: 
\r\nRecord-Route: 
\r\nVia: SIP/2.0/UDP 
24.121.80.36;branch=z9hG4bK13cd.038ec486.0\r\nV"..., len = 685}
text = {s = 0x16c , len = 746}
__FUNCTION__ = "relay_reply"
#6  0x7f66e2f1b548 in reply_received (p_msg=0x7eaac8) at t_reply.c:1502
last_uac_status = 
branch = 1
reply_status = 
timer = 
cancel_bitmap = 0
t = (struct cell *) 0x7f66c314eba0
backup_list = 
has_reply_route = 3222688728
__FUNCTION__ = "reply_received"
#7  0x0042519d in forward_reply (msg=0x7eaac8) at forward.c:559
new_buf = 
to = 
new_len = 
mod = (struct sr_module *) 0x78ddb8
---Type  to continue, or q  to quit---
proto = 
id = 
send_sock = 
len =

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-01 Thread thrillerbee

Bogdan,

It crashed again tonight.  I've attached the backtrace.

Thanks.

On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee  wrote:

> Bogdan,
>
> Yes,  I've attached 2 to my response - one was a crash from 10/29.  The
> other occurred a few minutes ago.
>
> Thanks.
>
> On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu <
> bog...@voice-system.ro> wrote:
>
>> Any chance with the backtraces ?
>>
>> Regards,
>> Bogdan
>>
>> Anca Vamanu wrote:
>> > Hi,
>> >
>> > You need to inspect them with gdb, run: gdb
>> > path_to_opensips_executable path_to_corefile, and then run 'bt full'
>> > and send the output.
>> >
>> > Regards,
>> > --
>> > Anca Vamanu
>> > www.voice-system.ro
>> >
>> >
>> > On 10/14/2010 10:12 PM, thrillerbee wrote:
>> >> I have this info from dmesg:
>> >>
>> >> [1985853.285221] opensips[30865]: segfault at 10 ip 7f43899ce21f sp
>> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> >> [1985856.379671] opensips[30858]: segfault at 10 ip 7f43899ce21f sp
>> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> >> [1985896.961279] opensips[30868]: segfault at 10 ip 7f43899ce21f sp
>> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> >>
>> >> [2000131.245512] opensips[17672]: segfault at 10 ip 7fd0f21fb21f sp
>> >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> >> [2000161.735962] opensips[17668]: segfault at 10 ip 7fd0f21fb21f sp
>> >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> >> [2000167.299402] opensips[17670]: segfault at 10 ip 7fd0f21fb21f sp
>> >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> >>
>> >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee > >> > wrote:
>> >>
>> >> When OpenSIPS crashes, three corefiles are generated that are
>> >> 2.1GB in size.  How do I use these files to understand what's
>> >> causing the crash?
>> >>
>> >> Thanks.
>> >>
>> >>
>> > 
>> >
>> > ___
>> > Users mailing list
>> > Users@lists.opensips.org
>> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>> >
>>
>>
>> --
>> Bogdan-Andrei Iancu
>> OpenSIPS Bootcamp
>> 15 - 19 November 2010, Edison, New Jersey, USA
>> www.voice-system.ro
>>
>>
>> ___
>> Users mailing list
>> Users@lists.opensips.org
>> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>>
>
>
Core was generated by `/usr/local/sbin/opensips -P 
/var/run/opensips/opensips.pid -m 512 -u root -g ro'.
Program terminated with signal 11, Segmentation fault.
[New process 26978]
#0  0x7f726cb9b21f in flat_db_insert (h=0x7f09a8, k=0x7f726b8dba60, 
v=0x7f726b8dbd20, n=19) at flatstore.c:165
165 f = CON_FILE(h);
(gdb) bt full
#0  0x7f726cb9b21f in flat_db_insert (h=0x7f09a8, k=0x7f726b8dba60, 
v=0x7f726b8dbd20, n=19) at flatstore.c:165
f = 
i = 
l = 
p = 
__FUNCTION__ = "flat_db_insert"
#1  0x7f726b6c27ef in acc_db_request (rq=0x7f726c77f160, rpl=) at acc.c:364
m = 19
n = 
i = 
my_ps = (db_ps_t) 0x0
__FUNCTION__ = "acc_db_request"
#2  0x7f726b6c7e28 in tmcb_func (t=0x7f724cea4fd8, type=, ps=0x4ccf5229) at acc_logic.c:319
No locals.
#3  0x7f726c5504a2 in run_trans_callbacks (type=64, trans=0x7f724cea4fd8, 
req=0x7f726c77f160, rpl=0x7f0d60, code=404) at t_hooks.c:208
cbp = (struct tm_callback *) 0x7f724ab5c220
backup = (struct usr_avp **) 0x7f724cea6808
trans_backup = (struct cell *) 0x7f724cea4fd8
__FUNCTION__ = "run_trans_callbacks"
#4  0x7f726c56c9b4 in t_should_relay_response (Trans=0x7f724cea4fd8, 
new_code=, branch=1, should_store=0x7083fa18, 
should_relay=0x7083fa1c, cancel_bitmap=0x7083fae8, reply=0x7f0d60) 
at t_reply.c:611
branch_cnt = 2
inv_through = 
do_cancel = 
__FUNCTION__ = "t_should_relay_response"
#5  0x7f726c56cfe2 in relay_reply (t=0x7f09a8, p_msg=0x7f0d60, 
branch=1288655401, msg_status=404, cancel_bitmap=0x7083fae8)
at t_reply.c:1133
relay = 
save_clone = 
buf = 
res_len = 0
relayed_code = 404
relayed_msg = 
bm = {to_tag_val = {s = 0x1 , len = 8326496}}
totag_retr = 
reply_status = 
cb_s = {
  s = 0x76202d "From: \"Office\" 
;tag=Uvyr85XQ8FNvj\r\nTo: 
;tag=B2B.2064.650\r\nCall-ID: 
91b97e04-f697-4827-bc02-d76b56b579b5\r\nCSeq: 3991724 INVITE\r\nContact"..., 
len = 8187368}
text = {
  s = 0x76206e "To: 
;tag=B2B.2064.650\r\nCall-ID: 
91b97e04-f697-4827-bc02-d76b56b579b5\r\nCSeq: 3991724 INVITE\r\nContact: 
\r\nServer: StarView Solutions SIP Proxy\r\nCo"..., len 
= 59}
__FUNCTION__ = "relay_reply"
#6  0x7f726c56e548 in reply_received (p_msg=0x7f0d60) at t_reply.c:1502
last_uac_status = 
branch = 1
reply_status =

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-02 Thread Bogdan-Andrei Iancu

Hi,

in frame 0, could you print:
h
h->tail
((struct flat_con*)(h->tail))->file

Also, before crash, do you see in the logs any errors from the 
db_flatstore module ?

Regards,
Bogdan

thrillerbee wrote:
> Bogdan,
>
> It crashed again tonight.  I've attached the backtrace.
>
> Thanks.
>
> On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee  > wrote:
>
> Bogdan,
>
> Yes,  I've attached 2 to my response - one was a crash from 10/29.
>  The other occurred a few minutes ago.
>
> Thanks.
>
> On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
> mailto:bog...@voice-system.ro>> wrote:
>
> Any chance with the backtraces ?
>
> Regards,
> Bogdan
>
> Anca Vamanu wrote:
> > Hi,
> >
> > You need to inspect them with gdb, run: gdb
> > path_to_opensips_executable path_to_corefile, and then run
> 'bt full'
> > and send the output.
> >
> > Regards,
> > --
> > Anca Vamanu
> > www.voice-system.ro 
> >
> >
> > On 10/14/2010 10:12 PM, thrillerbee wrote:
> >> I have this info from dmesg:
> >>
> >> [1985853.285221] opensips[30865]: segfault at 10 ip
> 7f43899ce21f sp
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> >> [1985856.379671] opensips[30858]: segfault at 10 ip
> 7f43899ce21f sp
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> >> [1985896.961279] opensips[30868]: segfault at 10 ip
> 7f43899ce21f sp
> >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> >>
> >> [2000131.245512] opensips[17672]: segfault at 10 ip
> 7fd0f21fb21f sp
> >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> >> [2000161.735962] opensips[17668]: segfault at 10 ip
> 7fd0f21fb21f sp
> >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> >> [2000167.299402] opensips[17670]: segfault at 10 ip
> 7fd0f21fb21f sp
> >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> >>
> >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
> mailto:thriller...@gmail.com>
> >>  >> wrote:
> >>
> >> When OpenSIPS crashes, three corefiles are generated
> that are
> >> 2.1GB in size.  How do I use these files to understand
> what's
> >> causing the crash?
> >>
> >> Thanks.
> >>
> >>
> >
> 
> 
> >
> > ___
> > Users mailing list
> > Users@lists.opensips.org 
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >
>
>
> --
> Bogdan-Andrei Iancu
> OpenSIPS Bootcamp
> 15 - 19 November 2010, Edison, New Jersey, USA
> www.voice-system.ro 
>
>
> ___
> Users mailing list
> Users@lists.opensips.org 
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
>
> 
>
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users


-- 
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-02 Thread thrillerbee

Bogdan,

Below is the info requested:
(gdb) frame 0
#0  0x7f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60,
v=0x7f5198732d20, n=19) at flatstore.c:165
165 f = CON_FILE(h);
(gdb) print h
$1 = (const db_con_t *) 0x7f0978
(gdb) print h->tail
$2 = 0
(gdb) print ((struct flat_con*)(h->tail))->file
Cannot access memory at address 0x10

Before each of the crashes yesterday, I saw these in the logs:
Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at
10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
db_flatstore.so[7f51999ef000+5000]
Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at
10 ip 7f726cb9b21f sp 7083f6f0 error 4 in
db_flatstore.so[7f726cb98000+5000]

Are there any compiler flags I should use for debugging?
(gdb) info locals
f = 
i = 
l = 
p = 
__FUNCTION__ = "flat_db_insert"

Thanks again.


On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
wrote:

> Hi,
>
> in frame 0, could you print:
>h
>h->tail
>((struct flat_con*)(h->tail))->file
>
> Also, before crash, do you see in the logs any errors from the
> db_flatstore module ?
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
> > Bogdan,
> >
> > It crashed again tonight.  I've attached the backtrace.
> >
> > Thanks.
> >
> > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee  > > wrote:
> >
> > Bogdan,
> >
> > Yes,  I've attached 2 to my response - one was a crash from 10/29.
> >  The other occurred a few minutes ago.
> >
> > Thanks.
> >
> > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
> > mailto:bog...@voice-system.ro>> wrote:
> >
> > Any chance with the backtraces ?
> >
> > Regards,
> > Bogdan
> >
> > Anca Vamanu wrote:
> > > Hi,
> > >
> > > You need to inspect them with gdb, run: gdb
> > > path_to_opensips_executable path_to_corefile, and then run
> > 'bt full'
> > > and send the output.
> > >
> > > Regards,
> > > --
> > > Anca Vamanu
> > > www.voice-system.ro 
> > >
> > >
> > > On 10/14/2010 10:12 PM, thrillerbee wrote:
> > >> I have this info from dmesg:
> > >>
> > >> [1985853.285221] opensips[30865]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> > >> [1985856.379671] opensips[30858]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> > >> [1985896.961279] opensips[30868]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
> > >>
> > >> [2000131.245512] opensips[17672]: segfault at 10 ip
> > 7fd0f21fb21f sp
> > >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> > >> [2000161.735962] opensips[17668]: segfault at 10 ip
> > 7fd0f21fb21f sp
> > >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> > >> [2000167.299402] opensips[17670]: segfault at 10 ip
> > 7fd0f21fb21f sp
> > >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
> > >>
> > >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
> > mailto:thriller...@gmail.com>
> > >>  > >> wrote:
> > >>
> > >> When OpenSIPS crashes, three corefiles are generated
> > that are
> > >> 2.1GB in size.  How do I use these files to understand
> > what's
> > >> causing the crash?
> > >>
> > >> Thanks.
> > >>
> > >>
> > >
> >
> 
> > >
> > > ___
> > > Users mailing list
> > > Users@lists.opensips.org 
> > > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> > >
> >
> >
> > --
> > Bogdan-Andrei Iancu
> > OpenSIPS Bootcamp
> > 15 - 19 November 2010, Edison, New Jersey, USA
> > www.voice-system.ro 
> >
> >
> > ___
> > Users mailing list
> > Users@lists.opensips.org 
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> >
> >
> >
> > 
> >
> > ___
> > Users mailing list
> > Users@lists.opensips.org
> > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
> --
> Bogdan-Andrei Iancu
> OpenSIPS Bootcamp
> 15 - 19 November 2010, Edison, New Jersey, USA
> www

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-02 Thread thrillerbee

Bogdan,

One more detail that may help - I added another OpenSIPS proxy in parallel
with this one (& load balancing between the two) and now both OpenSIPS
proxies crash at the same time (within a couple seconds).

I can provide more core dumps if it will help.

Thanks.


On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee  wrote:

> Bogdan,
>
> Below is the info requested:
> (gdb) frame 0
> #0  0x7f51999f221f in flat_db_insert (h=0x7f0978, k=0x7f5198732a60,
> v=0x7f5198732d20, n=19) at flatstore.c:165
> 165 f = CON_FILE(h);
> (gdb) print h
> $1 = (const db_con_t *) 0x7f0978
> (gdb) print h->tail
> $2 = 0
> (gdb) print ((struct flat_con*)(h->tail))->file
> Cannot access memory at address 0x10
>
> Before each of the crashes yesterday, I saw these in the logs:
> Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]: segfault at
> 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> db_flatstore.so[7f51999ef000+5000]
> Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]: segfault at
> 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in
> db_flatstore.so[7f726cb98000+5000]
>
> Are there any compiler flags I should use for debugging?
> (gdb) info locals
> f = 
> i = 
> l = 
> p = 
> __FUNCTION__ = "flat_db_insert"
>
> Thanks again.
>
>
> On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu <
> bog...@voice-system.ro> wrote:
>
>> Hi,
>>
>> in frame 0, could you print:
>>h
>>h->tail
>>((struct flat_con*)(h->tail))->file
>>
>> Also, before crash, do you see in the logs any errors from the
>> db_flatstore module ?
>>
>> Regards,
>> Bogdan
>>
>> thrillerbee wrote:
>> > Bogdan,
>> >
>> > It crashed again tonight.  I've attached the backtrace.
>> >
>> > Thanks.
>> >
>> > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee > > > wrote:
>> >
>> > Bogdan,
>> >
>> > Yes,  I've attached 2 to my response - one was a crash from 10/29.
>> >  The other occurred a few minutes ago.
>> >
>> > Thanks.
>> >
>> > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
>> > mailto:bog...@voice-system.ro>> wrote:
>> >
>> > Any chance with the backtraces ?
>> >
>> > Regards,
>> > Bogdan
>> >
>> > Anca Vamanu wrote:
>> > > Hi,
>> > >
>> > > You need to inspect them with gdb, run: gdb
>> > > path_to_opensips_executable path_to_corefile, and then run
>> > 'bt full'
>> > > and send the output.
>> > >
>> > > Regards,
>> > > --
>> > > Anca Vamanu
>> > > www.voice-system.ro 
>> > >
>> > >
>> > > On 10/14/2010 10:12 PM, thrillerbee wrote:
>> > >> I have this info from dmesg:
>> > >>
>> > >> [1985853.285221] opensips[30865]: segfault at 10 ip
>> > 7f43899ce21f sp
>> > >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> > >> [1985856.379671] opensips[30858]: segfault at 10 ip
>> > 7f43899ce21f sp
>> > >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> > >> [1985896.961279] opensips[30868]: segfault at 10 ip
>> > 7f43899ce21f sp
>> > >> 7fff8de1cf40 error 4 in db_flatstore.so[7f43899cb000+5000]
>> > >>
>> > >> [2000131.245512] opensips[17672]: segfault at 10 ip
>> > 7fd0f21fb21f sp
>> > >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> > >> [2000161.735962] opensips[17668]: segfault at 10 ip
>> > 7fd0f21fb21f sp
>> > >> 7fff3a3b4e40 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> > >> [2000167.299402] opensips[17670]: segfault at 10 ip
>> > 7fd0f21fb21f sp
>> > >> 7fff3a3b4f00 error 4 in db_flatstore.so[7fd0f21f8000+5000]
>> > >>
>> > >> On Thu, Oct 14, 2010 at 1:57 PM, thrillerbee
>> > mailto:thriller...@gmail.com>
>> > >> > > >> wrote:
>> > >>
>> > >> When OpenSIPS crashes, three corefiles are generated
>> > that are
>> > >> 2.1GB in size.  How do I use these files to understand
>> > what's
>> > >> causing the crash?
>> > >>
>> > >> Thanks.
>> > >>
>> > >>
>> > >
>> >
>> 
>> > >
>> > > ___
>> > > Users mailing list
>> > > Users@lists.opensips.org 
>> > > http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>> > >
>> >
>> >
>> > --
>> > Bogdan-Andrei Iancu
>> > OpenSIPS Bootcamp
>> > 15 - 19 November 2010, Edison, New Jersey, USA
>> > www.voice-system.ro 
>> >
>> >
>> > ___
>> >

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-03 Thread Bogdan-Andrei Iancu

I got some clue about what is happening - as you are using flatstore for 
acc, the acc module does not check the success of the "use_table" DB 
operation - in 90% of the cases (for mysql, postgres, etc) this function 
cannot fail, but it seams that for flatstore can. And if it fails, the 
h->tail is set to NULL, leading to crash.

Now, before the crash itself, do you see any other ERROR messages (even 
long before the crash) related to flatstore module ? Try:
grep "ERROR" opensips_log_file | grep "flatstore"

Regards,
Bogdan

thrillerbee wrote:
> Bogdan,
>
> One more detail that may help - I added another OpenSIPS proxy in 
> parallel with this one (& load balancing between the two) and now both 
> OpenSIPS proxies crash at the same time (within a couple seconds).
>
> I can provide more core dumps if it will help.
>
> Thanks.
>
>
> On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee  > wrote:
>
> Bogdan,
>
> Below is the info requested:
> (gdb) frame 0
> #0  0x7f51999f221f in flat_db_insert (h=0x7f0978,
> k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
> 165 f = CON_FILE(h);
> (gdb) print h
> $1 = (const db_con_t *) 0x7f0978
> (gdb) print h->tail
> $2 = 0
> (gdb) print ((struct flat_con*)(h->tail))->file
> Cannot access memory at address 0x10
>
> Before each of the crashes yesterday, I saw these in the logs:
> Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
> segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> db_flatstore.so[7f51999ef000+5000]
> Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
> segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in
> db_flatstore.so[7f726cb98000+5000]
>
> Are there any compiler flags I should use for debugging?
> (gdb) info locals
> f = 
> i = 
> l = 
> p = 
> __FUNCTION__ = "flat_db_insert"
>
> Thanks again.
>
>
> On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> mailto:bog...@voice-system.ro>> wrote:
>
> Hi,
>
> in frame 0, could you print:
>h
>h->tail
>((struct flat_con*)(h->tail))->file
>
> Also, before crash, do you see in the logs any errors from the
> db_flatstore module ?
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
> > Bogdan,
> >
> > It crashed again tonight.  I've attached the backtrace.
> >
> > Thanks.
> >
> > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
> mailto:thriller...@gmail.com>
> >  >> wrote:
> >
> > Bogdan,
> >
> > Yes,  I've attached 2 to my response - one was a crash
> from 10/29.
> >  The other occurred a few minutes ago.
> >
> > Thanks.
> >
> > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
> > mailto:bog...@voice-system.ro>
>  >> wrote:
> >
> > Any chance with the backtraces ?
> >
> > Regards,
> > Bogdan
> >
> > Anca Vamanu wrote:
> > > Hi,
> > >
> > > You need to inspect them with gdb, run: gdb
> > > path_to_opensips_executable path_to_corefile, and
> then run
> > 'bt full'
> > > and send the output.
> > >
> > > Regards,
> > > --
> > > Anca Vamanu
> > > www.voice-system.ro 
> 
> > >
> > >
> > > On 10/14/2010 10:12 PM, thrillerbee wrote:
> > >> I have this info from dmesg:
> > >>
> > >> [1985853.285221] opensips[30865]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in
> db_flatstore.so[7f43899cb000+5000]
> > >> [1985856.379671] opensips[30858]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in
> db_flatstore.so[7f43899cb000+5000]
> > >> [1985896.961279] opensips[30868]: segfault at 10 ip
> > 7f43899ce21f sp
> > >> 7fff8de1cf40 error 4 in
> db_flatstore.so[7f43899cb000+5000]
> > >>
> > >> [2000131.245512] opensips[17672]: segfault at 10 ip
> > 7fd0f21fb21f sp
> > >> 7fff3a3b4f00 error 4 in
> db_flatstore.so[7fd0f21f8000+5000]
> > >> [2000161.735962] opensips[17668]: segfault at 10 ip
> > 7fd0f21fb21

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-04 Thread thrillerbee

Bogdan,

I am not seeing any other errors in the logs.  Is there anything else I can
look at?  Both proxies are crashing everyday.

Thanks.

On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
wrote:

> I got some clue about what is happening - as you are using flatstore for
> acc, the acc module does not check the success of the "use_table" DB
> operation - in 90% of the cases (for mysql, postgres, etc) this function
> cannot fail, but it seams that for flatstore can. And if it fails, the
> h->tail is set to NULL, leading to crash.
>
> Now, before the crash itself, do you see any other ERROR messages (even
> long before the crash) related to flatstore module ? Try:
>grep "ERROR" opensips_log_file | grep "flatstore"
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
> > Bogdan,
> >
> > One more detail that may help - I added another OpenSIPS proxy in
> > parallel with this one (& load balancing between the two) and now both
> > OpenSIPS proxies crash at the same time (within a couple seconds).
> >
> > I can provide more core dumps if it will help.
> >
> > Thanks.
> >
> >
> > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee  > > wrote:
> >
> > Bogdan,
> >
> > Below is the info requested:
> > (gdb) frame 0
> > #0  0x7f51999f221f in flat_db_insert (h=0x7f0978,
> > k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
> > 165 f = CON_FILE(h);
> > (gdb) print h
> > $1 = (const db_con_t *) 0x7f0978
> > (gdb) print h->tail
> > $2 = 0
> > (gdb) print ((struct flat_con*)(h->tail))->file
> > Cannot access memory at address 0x10
> >
> > Before each of the crashes yesterday, I saw these in the logs:
> > Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
> > segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> > db_flatstore.so[7f51999ef000+5000]
> > Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
> > segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in
> > db_flatstore.so[7f726cb98000+5000]
> >
> > Are there any compiler flags I should use for debugging?
> > (gdb) info locals
> > f = 
> > i = 
> > l = 
> > p = 
> > __FUNCTION__ = "flat_db_insert"
> >
> > Thanks again.
> >
> >
> > On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> > mailto:bog...@voice-system.ro>> wrote:
> >
> > Hi,
> >
> > in frame 0, could you print:
> >h
> >h->tail
> >((struct flat_con*)(h->tail))->file
> >
> > Also, before crash, do you see in the logs any errors from the
> > db_flatstore module ?
> >
> > Regards,
> > Bogdan
> >
> > thrillerbee wrote:
> > > Bogdan,
> > >
> > > It crashed again tonight.  I've attached the backtrace.
> > >
> > > Thanks.
> > >
> > > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
> > mailto:thriller...@gmail.com>
> > >  > >> wrote:
> > >
> > > Bogdan,
> > >
> > > Yes,  I've attached 2 to my response - one was a crash
> > from 10/29.
> > >  The other occurred a few minutes ago.
> > >
> > > Thanks.
> > >
> > > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
> > > mailto:bog...@voice-system.ro>
> >  > >> wrote:
> > >
> > > Any chance with the backtraces ?
> > >
> > > Regards,
> > > Bogdan
> > >
> > > Anca Vamanu wrote:
> > > > Hi,
> > > >
> > > > You need to inspect them with gdb, run: gdb
> > > > path_to_opensips_executable path_to_corefile, and
> > then run
> > > 'bt full'
> > > > and send the output.
> > > >
> > > > Regards,
> > > > --
> > > > Anca Vamanu
> > > > www.voice-system.ro 
> > 
> > > >
> > > >
> > > > On 10/14/2010 10:12 PM, thrillerbee wrote:
> > > >> I have this info from dmesg:
> > > >>
> > > >> [1985853.285221] opensips[30865]: segfault at 10 ip
> > > 7f43899ce21f sp
> > > >> 7fff8de1cf40 error 4 in
> > db_flatstore.so[7f43899cb000+5000]
> > > >> [1985856.379671] opensips[30858]: segfault at 10 ip
> > > 7f43899ce21f sp
> > > >> 7fff8de1cf40 error 4 in
> > db_flatstore.so[7f43899cb000+5000]
> > > >> [1985896.961279] opensips[30868]: segfault at 10 ip

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-08 Thread Bogdan-Andrei Iancu

Hi,

strange if you do not have any errors :(

I just made a fix on both trunk and 1.6 to extend some checks in 
flatstore and prevent crashing (even if the DB op will not be executed).

Could you update from SVN and see if stops crashing ?

Regards,
Bogdan


thrillerbee wrote:
> Bogdan,
>
> I am not seeing any other errors in the logs.  Is there anything else 
> I can look at?  Both proxies are crashing everyday.
>
> Thanks.
>
> On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu 
> mailto:bog...@voice-system.ro>> wrote:
>
> I got some clue about what is happening - as you are using
> flatstore for
> acc, the acc module does not check the success of the "use_table" DB
> operation - in 90% of the cases (for mysql, postgres, etc) this
> function
> cannot fail, but it seams that for flatstore can. And if it fails, the
> h->tail is set to NULL, leading to crash.
>
> Now, before the crash itself, do you see any other ERROR messages
> (even
> long before the crash) related to flatstore module ? Try:
>grep "ERROR" opensips_log_file | grep "flatstore"
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
> > Bogdan,
> >
> > One more detail that may help - I added another OpenSIPS proxy in
> > parallel with this one (& load balancing between the two) and
> now both
> > OpenSIPS proxies crash at the same time (within a couple seconds).
> >
> > I can provide more core dumps if it will help.
> >
> > Thanks.
> >
> >
> > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
> mailto:thriller...@gmail.com>
> > >>
> wrote:
> >
> > Bogdan,
> >
> > Below is the info requested:
> > (gdb) frame 0
> > #0  0x7f51999f221f in flat_db_insert (h=0x7f0978,
> > k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
> > 165 f = CON_FILE(h);
> > (gdb) print h
> > $1 = (const db_con_t *) 0x7f0978
> > (gdb) print h->tail
> > $2 = 0
> > (gdb) print ((struct flat_con*)(h->tail))->file
> > Cannot access memory at address 0x10
> >
> > Before each of the crashes yesterday, I saw these in the logs:
> > Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
> > segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> > db_flatstore.so[7f51999ef000+5000]
> > Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
> > segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in
> > db_flatstore.so[7f726cb98000+5000]
> >
> > Are there any compiler flags I should use for debugging?
> > (gdb) info locals
> > f = 
> > i = 
> > l = 
> > p = 
> > __FUNCTION__ = "flat_db_insert"
> >
> > Thanks again.
> >
> >
> > On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> > mailto:bog...@voice-system.ro>
> >>
> wrote:
> >
> > Hi,
> >
> > in frame 0, could you print:
> >h
> >h->tail
> >((struct flat_con*)(h->tail))->file
> >
> > Also, before crash, do you see in the logs any errors
> from the
> > db_flatstore module ?
> >
> > Regards,
> > Bogdan
> >
> > thrillerbee wrote:
> > > Bogdan,
> > >
> > > It crashed again tonight.  I've attached the backtrace.
> > >
> > > Thanks.
> > >
> > > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
> > mailto:thriller...@gmail.com>
> >
> > >  
> >   > >
> > > Bogdan,
> > >
> > > Yes,  I've attached 2 to my response - one was a crash
> > from 10/29.
> > >  The other occurred a few minutes ago.
> > >
> > > Thanks.
> > >
> > > On Fri, Oct 22, 2010 at 11:41 AM, Bogdan-Andrei Iancu
> > >    >
> >  
> >   > >
> > > Any chance with the backtraces ?
> > >
> > > Regards,
> > > Bogdan
> > >
> > > Anca Vamanu

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-09 Thread thrillerbee

Bogdan,

It seems the issue is with 'opensipsctl fifo' - it's very sensitive to
simultaneous calls.  Basically, I've combined all my scripts to prevent
'opensipsctl fifo' from being called too frequently and that seems (so far)
to have mitigated the issue.  Is there anything one should know about how
(not) to use *opensipsctl*?

Thanks.

On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
wrote:

> Hi,
>
> strange if you do not have any errors :(
>
> I just made a fix on both trunk and 1.6 to extend some checks in
> flatstore and prevent crashing (even if the DB op will not be executed).
>
> Could you update from SVN and see if stops crashing ?
>
> Regards,
> Bogdan
>
>
> thrillerbee wrote:
> > Bogdan,
> >
> > I am not seeing any other errors in the logs.  Is there anything else
> > I can look at?  Both proxies are crashing everyday.
> >
> > Thanks.
> >
> > On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
> > mailto:bog...@voice-system.ro>> wrote:
> >
> > I got some clue about what is happening - as you are using
> > flatstore for
> > acc, the acc module does not check the success of the "use_table" DB
> > operation - in 90% of the cases (for mysql, postgres, etc) this
> > function
> > cannot fail, but it seams that for flatstore can. And if it fails,
> the
> > h->tail is set to NULL, leading to crash.
> >
> > Now, before the crash itself, do you see any other ERROR messages
> > (even
> > long before the crash) related to flatstore module ? Try:
> >grep "ERROR" opensips_log_file | grep "flatstore"
> >
> > Regards,
> > Bogdan
> >
> > thrillerbee wrote:
> > > Bogdan,
> > >
> > > One more detail that may help - I added another OpenSIPS proxy in
> > > parallel with this one (& load balancing between the two) and
> > now both
> > > OpenSIPS proxies crash at the same time (within a couple seconds).
> > >
> > > I can provide more core dumps if it will help.
> > >
> > > Thanks.
> > >
> > >
> > > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
> > mailto:thriller...@gmail.com>
> > > >>
> > wrote:
> > >
> > > Bogdan,
> > >
> > > Below is the info requested:
> > > (gdb) frame 0
> > > #0  0x7f51999f221f in flat_db_insert (h=0x7f0978,
> > > k=0x7f5198732a60, v=0x7f5198732d20, n=19) at flatstore.c:165
> > > 165 f = CON_FILE(h);
> > > (gdb) print h
> > > $1 = (const db_con_t *) 0x7f0978
> > > (gdb) print h->tail
> > > $2 = 0
> > > (gdb) print ((struct flat_con*)(h->tail))->file
> > > Cannot access memory at address 0x10
> > >
> > > Before each of the crashes yesterday, I saw these in the logs:
> > > Nov  1 14:17:40 core1 kernel: [4287745.452111] opensips[22141]:
> > > segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> > > db_flatstore.so[7f51999ef000+5000]
> > > Nov  1 23:52:58 core1 kernel: [4348562.990735] opensips[26978]:
> > > segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in
> > > db_flatstore.so[7f726cb98000+5000]
> > >
> > > Are there any compiler flags I should use for debugging?
> > > (gdb) info locals
> > > f = 
> > > i = 
> > > l = 
> > > p = 
> > > __FUNCTION__ = "flat_db_insert"
> > >
> > > Thanks again.
> > >
> > >
> > > On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> > > mailto:bog...@voice-system.ro>
> > >>
> > wrote:
> > >
> > > Hi,
> > >
> > > in frame 0, could you print:
> > >h
> > >h->tail
> > >((struct flat_con*)(h->tail))->file
> > >
> > > Also, before crash, do you see in the logs any errors
> > from the
> > > db_flatstore module ?
> > >
> > > Regards,
> > > Bogdan
> > >
> > > thrillerbee wrote:
> > > > Bogdan,
> > > >
> > > > It crashed again tonight.  I've attached the backtrace.
> > > >
> > > > Thanks.
> > > >
> > > > On Mon, Nov 1, 2010 at 9:32 AM, thrillerbee
> > > mailto:thriller...@gmail.com>
> > >
> > > >  > 
> > >  >  > > >
> > > > Bogdan,
> > > >
> > > > Yes,  I've attached 2 to my response - one was a
> crash
> > > from 10/29.
> > > >  The other occurred a few minutes ago.
> > >

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-10 Thread thrillerbee

Bogdan,

Well, I spoke too soon - it's not just an issue with the opensipsctl fifo
calls - looks more like a memory leak.  It crashed again today, but I did
get some errors in the syslog this time right before the crash:
Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at
10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
db_flatstore.so[7fa7ff749000+5000]
To be thorough, I've attached the backtrace & output from print commands
(although they're the same as before).

To answer your question, yes - I do use the flat_rotate MI command.

Thanks again.

On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu  wrote:

> Hi,
>
> opensipsctl takes care that each command takes a separate fifo reply, so
> here it should be no problem. But the problem may be when comes with sending
> multiple commands (via FIFO) in the same time - this translates into
> parallel writes to the same file and depends on the atomicity of the write
> op.
>
> But in the worst case, a mixture at the FIFO level may lead to bogus
> command and not in any kind of crashDo you use the "flat_rotate" MI
> command ?
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
>
>> Bogdan,
>>
>> It seems the issue is with 'opensipsctl fifo' - it's very sensitive to
>> simultaneous calls.  Basically, I've combined all my scripts to prevent
>> 'opensipsctl fifo' from being called too frequently and that seems (so far)
>> to have mitigated the issue.  Is there anything one should know about how
>> (not) to use /opensipsctl/?
>>
>> Thanks.
>>
>> On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu <
>> bog...@voice-system.ro > wrote:
>>
>>Hi,
>>
>>strange if you do not have any errors :(
>>
>>I just made a fix on both trunk and 1.6 to extend some checks in
>>flatstore and prevent crashing (even if the DB op will not be
>>executed).
>>
>>Could you update from SVN and see if stops crashing ?
>>
>>Regards,
>>Bogdan
>>
>
Core was generated by `/usr/local/sbin/opensips -P 
/var/run/opensips/opensips.pid -m 512 -u root -g ro'.
Program terminated with signal 11, Segmentation fault.
[New process 27044]
#0  0x7fa7ff74c21f in flat_db_insert (h=0x7efb38, k=0x7fa7fe48ca60, 
v=0x7fa7fe48cd20, n=19) at flatstore.c:165
165 f = CON_FILE(h);
(gdb) bt full
#0  0x7fa7ff74c21f in flat_db_insert (h=0x7efb38, k=0x7fa7fe48ca60, 
v=0x7fa7fe48cd20, n=19) at flatstore.c:165
f = 
i = 
l = 
p = 
__FUNCTION__ = "flat_db_insert"
#1  0x7fa7fe2737ef in acc_db_request (rq=0x7fa7de27a978, rpl=) at acc.c:364
m = 19
n = 
i = 
my_ps = (db_ps_t) 0x0
__FUNCTION__ = "acc_db_request"
#2  0x7fa7fe27869e in tmcb_func (t=, type=, ps=) at acc_logic.c:386
No locals.
#3  0x7fa7ff1014a2 in run_trans_callbacks (type=256, trans=0x7fa7df8a6e00, 
req=0x7fa7de27a978, rpl=0x7efef0, code=200) at t_hooks.c:208
cbp = (struct tm_callback *) 0x7fa7dc84f6f0
backup = (struct usr_avp **) 0x771f48
trans_backup = (struct cell *) 0x7fa7df8a6e00
__FUNCTION__ = "run_trans_callbacks"
#4  0x7fa7ff1015c7 in run_trans_callbacks_locked (type=256, 
trans=0x7fa7df8a6e00, req=0x7fa7de27a978, rpl=0x7efef0, code=200) at 
t_hooks.c:254
No locals.
#5  0x7fa7ff11ea38 in relay_reply (t=0x7fa7df8a6e00, p_msg=, branch=1, msg_status=200, cancel_bitmap=0x7fffdc101ad8) at t_reply.c:1257
relay = 1
save_clone = 0
buf = 0x1184390 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 
24.121.80.36;branch=z9hG4bKadcb.35a7dc2.0\r\nVia: SIP/2.0/UDP 
184.106.205.223;branch=z9hG4bKadcb.be5e7083.0\r\nVia: SIP/2.0/UDP 
216.18.222.3;branch=z9hG4bKadcb.18d6b484.0\r"...
res_len = 1391
relayed_code = 200
relayed_msg = (struct sip_msg *) 0x7efef0
bm = {to_tag_val = {s = 0x1 , len = 8322800}}
totag_retr = 
reply_status = RPS_COMPLETED
cb_s = {
  s = 0x1184390 "SIP/2.0 200 OK\r\nVia: SIP/2.0/UDP 
24.121.80.36;branch=z9hG4bKadcb.35a7dc2.0\r\nVia: SIP/2.0/UDP 
184.106.205.223;branch=z9hG4bKadcb.be5e7083.0\r\nVia: SIP/2.0/UDP 
216.18.222.3;branch=z9hG4bKadcb.18d6b484.0\r"..., len = 1391}
text = {s = 0x18f , len = 1452}
__FUNCTION__ = "relay_reply"
#6  0x7fa7ff11f548 in reply_received (p_msg=0x7efef0) at t_reply.c:1502
last_uac_status = 
branch = 1
reply_status = 
timer = 
cancel_bitmap = 0
t = (struct cell *) 0x7fa7df8a6e00
backup_list = 
has_reply_route = 3692375232
__FUNCTION__ = "reply_received"
#7  0x0042519d in forward_reply (msg=0x7efef0) at forward.c:559
new_buf = 
to = 
new_len = 
mod = (struct sr_module *) 0x78dfe0
proto = 
id = 
send_sock = 
len = 
__FUNCTION__ = "forward_reply"
#8  0x004

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-10 Thread thrillerbee

My other proxy crashed as well with these ERRORs in the syslog:

Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
ERROR:db_flatstore:get_name: pkg memory allocation failure
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
ERROR:db_flatstore:flat_reopen_connection: failed to get_name
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
ERROR:db_flatstore:flat_db_insert: uninitialized connection
...
Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]:
ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault at
10 ip 7f3db577e21f sp 7fffa260d640 error 4 in
db_flatstore.so[7f3db577b000+5000]

On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee  wrote:

> Bogdan,
>
> Well, I spoke too soon - it's not just an issue with the opensipsctl fifo
> calls - looks more like a memory leak.  It crashed again today, but I did
> get some errors in the syslog this time right before the crash:
>  Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
> ERROR:db_flatstore:new_flat_id: no pkg memory left
> Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at
> 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
> db_flatstore.so[7fa7ff749000+5000]
> To be thorough, I've attached the backtrace & output from print commands
> (although they're the same as before).
>
> To answer your question, yes - I do use the flat_rotate MI command.
>
> Thanks again.
>
> On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu <
> bog...@voice-system.ro> wrote:
>
>> Hi,
>>
>> opensipsctl takes care that each command takes a separate fifo reply, so
>> here it should be no problem. But the problem may be when comes with sending
>> multiple commands (via FIFO) in the same time - this translates into
>> parallel writes to the same file and depends on the atomicity of the write
>> op.
>>
>> But in the worst case, a mixture at the FIFO level may lead to bogus
>> command and not in any kind of crashDo you use the "flat_rotate" MI
>> command ?
>>
>> Regards,
>> Bogdan
>>
>> thrillerbee wrote:
>>
>>> Bogdan,
>>>
>>> It seems the issue is with 'opensipsctl fifo' - it's very sensitive to
>>> simultaneous calls.  Basically, I've combined all my scripts to prevent
>>> 'opensipsctl fifo' from being called too frequently and that seems (so far)
>>> to have mitigated the issue.  Is there anything one should know about how
>>> (not) to use /opensipsctl/?
>>>
>>> Thanks.
>>>
>>> On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu <
>>> bog...@voice-system.ro > wrote:
>>>
>>>Hi,
>>>
>>>strange if you do not have any errors :(
>>>
>>>I just made a fix on both trunk and 1.6 to extend some checks in
>>>flatstore and prevent crashing (even if the DB op will not be
>>>executed).
>>>
>>>Could you update from SVN and see if stops crashing ?
>>>
>>>Regards,
>>>Bogdan
>>>
>>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-11 Thread Bogdan-Andrei Iancu

This last crash had the same bt as the one from previous email ? the 
prev fix I made reports this:

 ERROR:db_flatstore:flat_db_insert: uninitialized connection
and does not crash, so this time the crash may be in a different place.

Now, it seams after all that the root of your problem is the mem 
exhaustion. To see what  causes this  (if a mem leak or simply not 
enough mem), there is a doc - 
http://www.opensips.org/Resources/DocsTsMem (it is for shm, but the same 
for pkg).


Maybe, first you should simply try to increase the pkg mem ( in config.h 
you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to 
recompile and reinstall after that.


If more pkg mem does not solve the problem, I can help you with using 
the memory debugger to see what is going on there.


Regards,
Bogdan


thrillerbee wrote:

My other proxy crashed as well with these ERRORs in the syslog:

Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: 
ERROR:db_flatstore:get_name: pkg memory allocation failure
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: 
ERROR:db_flatstore:flat_reopen_connection: failed to get_name
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: 
ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]: 
ERROR:db_flatstore:flat_db_insert: uninitialized connection

...
Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]: 
ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: 
ERROR:db_flatstore:flat_db_insert: uninitialized connection
Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]: 
ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: 
segfault at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in 
db_flatstore.so[7f3db577b000+5000]


On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee > wrote:


Bogdan,

Well, I spoke too soon - it's not just an issue with the
opensipsctl fifo calls - looks more like a memory leak.  It
crashed again today, but I did get some errors in the syslog this
time right before the crash:
Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]:
segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
db_flatstore.so[7fa7ff749000+5000]
To be thorough, I've attached the backtrace & output from print
commands (although they're the same as before).

To answer your question, yes - I do use the flat_rotate MI command.

Thanks again.

On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu
mailto:bog...@voice-system.ro>> wrote:

Hi,

opensipsctl takes care that each command takes a separate fifo
reply, so here it should be no problem. But the problem may be
when comes with sending multiple commands (via FIFO) in the
same time - this translates into parallel writes to the same
file and depends on the atomicity of the write op.

But in the worst case, a mixture at the FIFO level may lead to
bogus command and not in any kind of crashDo you use the
"flat_rotate" MI command ?

Regards,
Bogdan

thrillerbee wrote:

Bogdan,

It seems the issue is with 'opensipsctl fifo' - it's very
sensitive to simultaneous calls.  Basically, I've combined
all my scripts to prevent 'opensipsctl fifo' from being
called too frequently and that seems (so far) to have
mitigated the issue.  Is there anything one should know
about how (not) to use /opensipsctl/?

Thanks.

On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
mailto:bog...@voice-system.ro>
>> wrote:

   Hi,

   strange if you do not have any errors :(

   I just made a fix on both trunk and 1.6 to extend some
checks in
   flatstore and prevent crashing (even if the DB op will
not be
   executed).

   Could you update from SVN and see if stops crashing ?

   Regards,
   Bogdan




___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users
  



--
Bogdan-Andrei Iancu
OpenSIPS Bootcamp
15 - 19 November 2010, Edison, New Jersey, USA
www.voice-system.ro


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-11 Thread thrillerbee

Bogdan,

I had already increased the PKG_MEM_POOL_SIZE by *10.  I'm making the
changes suggested in the link you provided to try to narrow down the memory
issue.  I'll send over as soon as I have another crash with mem info.

Thanks.

On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu  wrote:

> This last crash had the same bt as the one from previous email ? the prev
> fix I made reports this:
>
> ERROR:db_flatstore:flat_db_insert: uninitialized connection
> and does not crash, so this time the crash may be in a different place.
>
> Now, it seams after all that the root of your problem is the mem
> exhaustion. To see what  causes this  (if a mem leak or simply not enough
> mem), there is a doc - http://www.opensips.org/Resources/DocsTsMem (it is
> for shm, but the same for pkg).
>
> Maybe, first you should simply try to increase the pkg mem ( in config.h
> you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile
> and reinstall after that.
>
> If more pkg mem does not solve the problem, I can help you with using the
> memory debugger to see what is going on there.
>
> Regards,
> Bogdan
>
>
> thrillerbee wrote:
>
>> My other proxy crashed as well with these ERRORs in the syslog:
>>
>> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
>> ERROR:db_flatstore:get_name: pkg memory allocation failure
>> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
>> ERROR:db_flatstore:flat_reopen_connection: failed to get_name
>> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>> ...
>> Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]:
>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>> Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>> Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
>> ERROR:db_flatstore:new_flat_id: no pkg memory left
>> Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault
>> at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in
>> db_flatstore.so[7f3db577b000+5000]
>>
>> On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee > thriller...@gmail.com>> wrote:
>>
>>Bogdan,
>>
>>Well, I spoke too soon - it's not just an issue with the
>>opensipsctl fifo calls - looks more like a memory leak.  It
>>crashed again today, but I did get some errors in the syslog this
>>time right before the crash:
>>Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
>>ERROR:db_flatstore:new_flat_id: no pkg memory left
>>Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]:
>>segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
>>db_flatstore.so[7fa7ff749000+5000]
>>To be thorough, I've attached the backtrace & output from print
>>commands (although they're the same as before).
>>
>>To answer your question, yes - I do use the flat_rotate MI command.
>>
>>Thanks again.
>>
>>On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu
>>mailto:bog...@voice-system.ro>> wrote:
>>
>>Hi,
>>
>>opensipsctl takes care that each command takes a separate fifo
>>reply, so here it should be no problem. But the problem may be
>>when comes with sending multiple commands (via FIFO) in the
>>same time - this translates into parallel writes to the same
>>file and depends on the atomicity of the write op.
>>
>>But in the worst case, a mixture at the FIFO level may lead to
>>bogus command and not in any kind of crashDo you use the
>>"flat_rotate" MI command ?
>>
>>Regards,
>>Bogdan
>>
>>thrillerbee wrote:
>>
>>Bogdan,
>>
>>It seems the issue is with 'opensipsctl fifo' - it's very
>>sensitive to simultaneous calls.  Basically, I've combined
>>all my scripts to prevent 'opensipsctl fifo' from being
>>called too frequently and that seems (so far) to have
>>mitigated the issue.  Is there anything one should know
>>about how (not) to use /opensipsctl/?
>>
>>Thanks.
>>
>>On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
>>mailto:bog...@voice-system.ro>
>>>>> wrote:
>>
>>   Hi,
>>
>>   strange if you do not have any errors :(
>>
>>   I just made a fix on both trunk and 1.6 to extend some
>>checks in
>>   flatstore and prevent crashing (even if the DB op will
>>not be
>>   executed).
>>
>>   Could you update from SVN and see if stops crashing ?
>>
>>   Regards,
>>   Bogdan
>>
>>
>> --

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-13 Thread thrillerbee

Bogdan,

Since I made those changes, it doesn't core dump anymore - it just runs out
of memory and stops processing packets.  It happened on both of my proxies -
the mem dump can be downloaded here:
http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz
http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz

Thanks.

On Thu, Nov 11, 2010 at 12:43 PM, thrillerbee  wrote:

> Bogdan,
>
> I had already increased the PKG_MEM_POOL_SIZE by *10.  I'm making the
> changes suggested in the link you provided to try to narrow down the memory
> issue.  I'll send over as soon as I have another crash with mem info.
>
> Thanks.
>
>
> On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu <
> bog...@voice-system.ro> wrote:
>
>> This last crash had the same bt as the one from previous email ? the prev
>> fix I made reports this:
>>
>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>> and does not crash, so this time the crash may be in a different place.
>>
>> Now, it seams after all that the root of your problem is the mem
>> exhaustion. To see what  causes this  (if a mem leak or simply not enough
>> mem), there is a doc - http://www.opensips.org/Resources/DocsTsMem (it is
>> for shm, but the same for pkg).
>>
>> Maybe, first you should simply try to increase the pkg mem ( in config.h
>> you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile
>> and reinstall after that.
>>
>> If more pkg mem does not solve the problem, I can help you with using the
>> memory debugger to see what is going on there.
>>
>> Regards,
>> Bogdan
>>
>>
>> thrillerbee wrote:
>>
>>> My other proxy crashed as well with these ERRORs in the syslog:
>>>
>>> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
>>> ERROR:db_flatstore:get_name: pkg memory allocation failure
>>> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
>>> ERROR:db_flatstore:flat_reopen_connection: failed to get_name
>>> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
>>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>>> Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
>>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>>> ...
>>> Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]:
>>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>>> Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
>>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>>> Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
>>> ERROR:db_flatstore:new_flat_id: no pkg memory left
>>> Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault
>>> at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in
>>> db_flatstore.so[7f3db577b000+5000]
>>>
>>> On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee >> thriller...@gmail.com>> wrote:
>>>
>>>Bogdan,
>>>
>>>Well, I spoke too soon - it's not just an issue with the
>>>opensipsctl fifo calls - looks more like a memory leak.  It
>>>crashed again today, but I did get some errors in the syslog this
>>>time right before the crash:
>>>Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
>>>ERROR:db_flatstore:new_flat_id: no pkg memory left
>>>Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]:
>>>segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
>>>db_flatstore.so[7fa7ff749000+5000]
>>>To be thorough, I've attached the backtrace & output from print
>>>commands (although they're the same as before).
>>>
>>>To answer your question, yes - I do use the flat_rotate MI command.
>>>
>>>Thanks again.
>>>
>>>On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu
>>>mailto:bog...@voice-system.ro>> wrote:
>>>
>>>Hi,
>>>
>>>opensipsctl takes care that each command takes a separate fifo
>>>reply, so here it should be no problem. But the problem may be
>>>when comes with sending multiple commands (via FIFO) in the
>>>same time - this translates into parallel writes to the same
>>>file and depends on the atomicity of the write op.
>>>
>>>But in the worst case, a mixture at the FIFO level may lead to
>>>bogus command and not in any kind of crashDo you use the
>>>"flat_rotate" MI command ?
>>>
>>>Regards,
>>>Bogdan
>>>
>>>thrillerbee wrote:
>>>
>>>Bogdan,
>>>
>>>It seems the issue is with 'opensipsctl fifo' - it's very
>>>sensitive to simultaneous calls.  Basically, I've combined
>>>all my scripts to prevent 'opensipsctl fifo' from being
>>>called too frequently and that seems (so far) to have
>>>mitigated the issue.  Is there anything one should know
>>>about how (not) to use /opensipsctl/?
>>>
>>>Thanks.
>>>
>>>On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu
>>>mailto:bog...@voice-system.ro>
>>>>>

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-16 Thread Bogdan-Andrei Iancu


Hi,

opensipsctl takes care that each command takes a separate fifo reply, so 
here it should be no problem. But the problem may be when comes with 
sending multiple commands (via FIFO) in the same time - this translates 
into parallel writes to the same file and depends on the atomicity of 
the write op.


But in the worst case, a mixture at the FIFO level may lead to bogus 
command and not in any kind of crashDo you use the "flat_rotate" MI 
command ?


Regards,
Bogdan

thrillerbee wrote:

Bogdan,

It seems the issue is with 'opensipsctl fifo' - it's very sensitive to 
simultaneous calls.  Basically, I've combined all my scripts to 
prevent 'opensipsctl fifo' from being called too frequently and that 
seems (so far) to have mitigated the issue.  Is there anything one 
should know about how (not) to use /opensipsctl/?


Thanks.

On Mon, Nov 8, 2010 at 6:07 AM, Bogdan-Andrei Iancu 
mailto:bog...@voice-system.ro>> wrote:


Hi,

strange if you do not have any errors :(

I just made a fix on both trunk and 1.6 to extend some checks in
flatstore and prevent crashing (even if the DB op will not be
executed).

Could you update from SVN and see if stops crashing ?

Regards,
Bogdan


thrillerbee wrote:
> Bogdan,
>
> I am not seeing any other errors in the logs.  Is there anything
else
> I can look at?  Both proxies are crashing everyday.
>
> Thanks.
>
> On Wed, Nov 3, 2010 at 4:21 AM, Bogdan-Andrei Iancu
> mailto:bog...@voice-system.ro>
>>
wrote:
>
> I got some clue about what is happening - as you are using
> flatstore for
> acc, the acc module does not check the success of the
"use_table" DB
> operation - in 90% of the cases (for mysql, postgres, etc) this
> function
> cannot fail, but it seams that for flatstore can. And if it
fails, the
> h->tail is set to NULL, leading to crash.
>
> Now, before the crash itself, do you see any other ERROR
messages
> (even
> long before the crash) related to flatstore module ? Try:
>grep "ERROR" opensips_log_file | grep "flatstore"
>
> Regards,
> Bogdan
>
> thrillerbee wrote:
> > Bogdan,
> >
> > One more detail that may help - I added another OpenSIPS
proxy in
> > parallel with this one (& load balancing between the two) and
> now both
> > OpenSIPS proxies crash at the same time (within a couple
seconds).
> >
> > I can provide more core dumps if it will help.
> >
> > Thanks.
> >
> >
> > On Tue, Nov 2, 2010 at 9:02 AM, thrillerbee
> mailto:thriller...@gmail.com>
>
> >   wrote:
> >
> > Bogdan,
> >
> > Below is the info requested:
> > (gdb) frame 0
> > #0  0x7f51999f221f in flat_db_insert (h=0x7f0978,
> > k=0x7f5198732a60, v=0x7f5198732d20, n=19) at
flatstore.c:165
> > 165 f = CON_FILE(h);
> > (gdb) print h
> > $1 = (const db_con_t *) 0x7f0978
> > (gdb) print h->tail
> > $2 = 0
> > (gdb) print ((struct flat_con*)(h->tail))->file
> > Cannot access memory at address 0x10
> >
> > Before each of the crashes yesterday, I saw these in
the logs:
> > Nov  1 14:17:40 core1 kernel: [4287745.452111]
opensips[22141]:
> > segfault at 10 ip 7f51999f221f sp 7fffbcd8d510 error 4 in
> > db_flatstore.so[7f51999ef000+5000]
> > Nov  1 23:52:58 core1 kernel: [4348562.990735]
opensips[26978]:
> > segfault at 10 ip 7f726cb9b21f sp 7083f6f0 error 4 in
> > db_flatstore.so[7f726cb98000+5000]
> >
> > Are there any compiler flags I should use for debugging?
> > (gdb) info locals
> > f = 
> > i = 
> > l = 
> > p = 
> > __FUNCTION__ = "flat_db_insert"
> >
> > Thanks again.
> >
> >
> > On Tue, Nov 2, 2010 at 4:18 AM, Bogdan-Andrei Iancu
> > mailto:bog...@voice-system.ro> >
>   wrote:
> >
> > Hi,
> >
> > in frame 0, could you print:
> >h
> >h->tail
>

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-16 Thread John Khvatov

Hello.

On 10.11.2010, at 19:19, thrillerbee wrote:
> Well, I spoke too soon - it's not just an issue with the opensipsctl fifo 
> calls - looks more like a memory leak.  It crashed again today, but I did get 
> some errors in the syslog this time right before the crash:
> Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: 
> ERROR:db_flatstore:new_flat_id: no pkg memory left
> Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 
> 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in 
> db_flatstore.so[7fa7ff749000+5000]
> To be thorough, I've attached the backtrace & output from print commands 
> (although they're the same as before).

I have the same issue. OpenSIPS crashes at db_flatstore.so due to lack of 
memory (according resident memory usage graph). Recently I removed the use of 
the t_newtran() function and localstore module in opensips.cfg and crash did 
not happens for 11 days. Earlier crash happens every two-tree days.

Do you use t_newtran() in your config?

-- 
WBR, John Khvatov

___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-16 Thread thrillerbee

John,

I appreciate the suggestion, but, unfortunately, I don't use t_newtran().
 What led you to remove that?

Thanks,
Ryan

On Tue, Nov 16, 2010 at 9:08 AM, John Khvatov  wrote:

> Hello.
>
> On 10.11.2010, at 19:19, thrillerbee wrote:
>
> Well, I spoke too soon - it's not just an issue with the opensipsctl fifo
> calls - looks more like a memory leak.  It crashed again today, but I did
> get some errors in the syslog this time right before the crash:
>  Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
> ERROR:db_flatstore:new_flat_id: no pkg memory left
> Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at
> 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
> db_flatstore.so[7fa7ff749000+5000]
> To be thorough, I've attached the backtrace & output from print commands
> (although they're the same as before).
>
>
> I have the same issue. OpenSIPS crashes at db_flatstore.so due to lack of
> memory (according resident memory usage graph). Recently I removed the use
> of the t_newtran() function and localstore module in opensips.cfg and crash
> did not happens for 11 days. Earlier crash happens every two-tree days.
>
> Do you use t_newtran() in your config?
>
> --
> WBR, John Khvatov
>
>
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
>
>
___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-19 Thread thrillerbee

Bogdan,

Are these memlogs useful?  Please let me know if I can provide any other
info.

Thanks again.

On Sat, Nov 13, 2010 at 4:20 PM, thrillerbee  wrote:

> Bogdan,
>
> Since I made those changes, it doesn't core dump anymore - it just runs out
> of memory and stops processing packets.  It happened on both of my proxies -
> the mem dump can be downloaded here:
> http://www.starviewconnect.com/tmp/core1_mem_dump_20101112.gz
> http://www.starviewconnect.com/tmp/core2_mem_dump_20101112.gz
>
> Thanks.
>
> On Thu, Nov 11, 2010 at 12:43 PM, thrillerbee wrote:
>
>> Bogdan,
>>
>> I had already increased the PKG_MEM_POOL_SIZE by *10.  I'm making the
>> changes suggested in the link you provided to try to narrow down the memory
>> issue.  I'll send over as soon as I have another crash with mem info.
>>
>> Thanks.
>>
>>
>> On Thu, Nov 11, 2010 at 5:10 AM, Bogdan-Andrei Iancu <
>> bog...@voice-system.ro> wrote:
>>
>>> This last crash had the same bt as the one from previous email ? the prev
>>> fix I made reports this:
>>>
>>> ERROR:db_flatstore:flat_db_insert: uninitialized connection
>>> and does not crash, so this time the crash may be in a different place.
>>>
>>> Now, it seams after all that the root of your problem is the mem
>>> exhaustion. To see what  causes this  (if a mem leak or simply not enough
>>> mem), there is a doc - http://www.opensips.org/Resources/DocsTsMem (it
>>> is for shm, but the same for pkg).
>>>
>>> Maybe, first you should simply try to increase the pkg mem ( in config.h
>>> you have PKG_MEM_POOL_SIZE that you can increase) - you NEED to recompile
>>> and reinstall after that.
>>>
>>> If more pkg mem does not solve the problem, I can help you with using the
>>> memory debugger to see what is going on there.
>>>
>>> Regards,
>>> Bogdan
>>>
>>>
>>> thrillerbee wrote:
>>>
 My other proxy crashed as well with these ERRORs in the syslog:

 Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
 ERROR:db_flatstore:get_name: pkg memory allocation failure
 Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
 ERROR:db_flatstore:flat_reopen_connection: failed to get_name
 Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
 ERROR:db_flatstore:flat_db_insert: uninitialized connection
 Nov 10 22:01:02 core2 /usr/local/sbin/opensips[22959]:
 ERROR:db_flatstore:flat_db_insert: uninitialized connection
 ...
 Nov 10 22:01:21 core2 /usr/local/sbin/opensips[22959]:
 ERROR:db_flatstore:flat_db_insert: uninitialized connection
 Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
 ERROR:db_flatstore:flat_db_insert: uninitialized connection
 Nov 10 22:01:22 core2 /usr/local/sbin/opensips[22959]:
 ERROR:db_flatstore:new_flat_id: no pkg memory left
 Nov 10 22:01:22 core2 kernel: [4297088.404734] opensips[22959]: segfault
 at 10 ip 7f3db577e21f sp 7fffa260d640 error 4 in
 db_flatstore.so[7f3db577b000+5000]

 On Wed, Nov 10, 2010 at 10:19 AM, thrillerbee 
 >>> thriller...@gmail.com>> wrote:

Bogdan,

Well, I spoke too soon - it's not just an issue with the
opensipsctl fifo calls - looks more like a memory leak.  It
crashed again today, but I did get some errors in the syslog this
time right before the crash:
Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]:
ERROR:db_flatstore:new_flat_id: no pkg memory left
Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]:
segfault at 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in
db_flatstore.so[7fa7ff749000+5000]
To be thorough, I've attached the backtrace & output from print
commands (although they're the same as before).

To answer your question, yes - I do use the flat_rotate MI command.

Thanks again.

On Wed, Nov 10, 2010 at 4:04 AM, Bogdan-Andrei Iancu
mailto:bog...@voice-system.ro>> wrote:

Hi,

opensipsctl takes care that each command takes a separate fifo
reply, so here it should be no problem. But the problem may be
when comes with sending multiple commands (via FIFO) in the
same time - this translates into parallel writes to the same
file and depends on the atomicity of the write op.

But in the worst case, a mixture at the FIFO level may lead to
bogus command and not in any kind of crashDo you use the
"flat_rotate" MI command ?

Regards,
Bogdan

thrillerbee wrote:

Bogdan,

It seems the issue is with 'opensipsctl fifo' - it's very
sensitive to simultaneous calls.  Basically, I've combined
all my scripts to prevent 'opensipsctl fifo' from being
called too frequently and that seems (so far) to have
mitigated the issue.  Is there a

Re: [OpenSIPS-Users] OpenSIPS core dumps

2010-11-25 Thread John Khvatov

Hi Ryan,

On 16.11.2010, at 18:17, thrillerbee wrote:

> John,
> 
> I appreciate the suggestion, but, unfortunately, I don't use t_newtran().  
> What led you to remove that?

Hm, sorry…

I was removing some functions from my opensips.cfg, then running various stress 
tests (REGISTER, INVITE etc.) and watching resident memory usage graph.

Starting with 6.11.2010 (12:00) opensips working without t_newtrans() and 
everything fine:
http://dev.sgu.ru/pub/without_tnewtrans.png (kKb == MB)

Btw, now I use localcache for handling sip-request retransmits. With localcache 
resident memory usage also increases, but OpenSIPS does not crash…

> On Tue, Nov 16, 2010 at 9:08 AM, John Khvatov  wrote:
> Hello.
> 
> On 10.11.2010, at 19:19, thrillerbee wrote:
>> Well, I spoke too soon - it's not just an issue with the opensipsctl fifo 
>> calls - looks more like a memory leak.  It crashed again today, but I did 
>> get some errors in the syslog this time right before the crash:
>> Nov 10 15:42:32 core1 /usr/local/sbin/opensips[27044]: 
>> ERROR:db_flatstore:new_flat_id: no pkg memory left
>> Nov 10 15:42:32 core1 kernel: [5508366.582447] opensips[27044]: segfault at 
>> 10 ip 7fa7ff74c21f sp 7fffdc101700 error 4 in 
>> db_flatstore.so[7fa7ff749000+5000]
>> To be thorough, I've attached the backtrace & output from print commands 
>> (although they're the same as before).
> 
> I have the same issue. OpenSIPS crashes at db_flatstore.so due to lack of 
> memory (according resident memory usage graph). Recently I removed the use of 
> the t_newtran() function and localstore module in opensips.cfg and crash did 
> not happens for 11 days. Earlier crash happens every two-tree days.
> 
> Do you use t_newtran() in your config?
> 
> -- 
> WBR, John Khvatov
> 
> 
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users
> 
> 
> ___
> Users mailing list
> Users@lists.opensips.org
> http://lists.opensips.org/cgi-bin/mailman/listinfo/users

-- 
WBR, John Khvatov


___
Users mailing list
Users@lists.opensips.org
http://lists.opensips.org/cgi-bin/mailman/listinfo/users

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

Re: [OpenSIPS-Users] OpenSIPS core dumps

22 matches

Site Navigation

Mail list logo

Footer information