Hi Bogdan, When I try to apply patch, I see all the code are already available in 2.2. So I did not add any patch. But did a code addition for 'REST_PUT' request in rest_client.c, rest_methods.c, rest_methohs.h Tried to perform load test. Observed 2 issues.
1) Tried to load 10,000 calls - But route[resume_http] is called only for 9985 calls. Every time approximately 10-20 calls, route[resume_http] is not called. But if I see the tcpdump, I am seeing 10,000 HTTP request and 10,000 HTTP 200 OK responses. When printing the response in resume_http for every call-id, 10-20 calls response is not printed - which means resume is not called for these calls. Am not filtering any response code. route[resume_http] { xlog("L_INFO","Resp $rc in HTTP PUT!\n"); xlog("L_INFO","route[relay] The content received from SM for $rm: [callId=$ci] : $var(body) in HTTP PUT\n"); ..................... } For first few calls, $rc is 1(ASYNC_DONE), for others $rc is -5(ASYNC_SYNC) 2) Tried to load more than 10,000 calls. When it reaches 25,000 calls after that some point opensips crashes. I have two core dump generated as of now. Coredump1: -------------- (gdb) bt #0 0x00007f8defacc840 in Curl_num_addresses () from /lib64/libcurl.so.4 #1 0x00007f8defaf09ee in Curl_connecthost () from /lib64/libcurl.so.4 #2 0x00007f8defae34ff in Curl_setup_conn () from /lib64/libcurl.so.4 #3 0x00007f8defae370c in Curl_connect () from /lib64/libcurl.so.4 #4 0x00007f8defaf32c0 in multi_runsingle () from /lib64/libcurl.so.4 #5 0x00007f8defaf4121 in curl_multi_perform () from /lib64/libcurl.so.4 #6 0x00007f8defd2b65b in start_async_http_req (msg=msg@entry=0x7f8df3740fc0, method=method@entry=REST_CLIENT_PUT, url=0x7f8df36ceb88 "http://test.comcast.net/RTCGSessionManager/rest/tel/session/createroom?", req_body=<optimized out>, req_ctype=<optimized out>, out_handle=out_handle@entry=0x7f8df3743300, body=body@entry=0x7f8df3743308, ctype=0x7f8df3743318) at rest_methods.c:265 #7 0x00007f8defd31b56 in w_async_rest_put (msg=0x7f8df3740fc0, resume_f=0x7ffd4afd8c90, resume_param=0x7ffd4afd8c88, gp_url=<optimized out>, gp_body=<optimized out>, gp_ctype=<optimized out>, body_pv=0x7f8df36f85a8 "N", ctype_pv=0x7f8df36f8700 "N", code_pv=0x7f8df36f88c0 "N") at rest_client.c:544 #8 0x00007f8df1031841 in t_handle_async (msg=0x7f8df3740fc0, a=0x7f8df36cf0c8, resume_route=2) at async.c:240 #9 0x0000000000424056 in do_action (a=0x7f8df36cf2d0, msg=0x7f8df3740fc0) at action.c:1863 #10 0x000000000041c330 in run_action_list (a=0x7f8df36cf2d0, msg=0x7f8df3740fc0) at action.c:172 #11 0x0000000000420637 in do_action (a=0x7f8df36cf3f8, msg=0x7f8df3740fc0) at action.c:1108 #12 0x000000000041c330 in run_action_list (a=0x7f8df36ce568, msg=0x7f8df3740fc0) at action.c:172 #13 0x000000000041c1fd in run_actions (a=0x7f8df36ce568, msg=0x7f8df3740fc0) at action.c:137 #14 0x000000000041ed55 in do_action (a=0x7f8df36ca080, msg=0x7f8df3740fc0) at action.c:745 #15 0x000000000041c330 in run_action_list (a=0x7f8df36c9e78, msg=0x7f8df3740fc0) at action.c:172 #16 0x0000000000420637 in do_action (a=0x7f8df36ca1a8, msg=0x7f8df3740fc0) at action.c:1108 #17 0x000000000041c330 in run_action_list (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:172 #18 0x000000000041c1fd in run_actions (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:137 #19 0x000000000041c3fd in run_top_route (a=0x7f8df36c2560, msg=0x7f8df3740fc0) at action.c:204 #20 0x000000000042bb64 in receive_msg ( buf=0x7eb340 <buf.8031> "INVITE sip:++12001000004@10.0.0.1:5060 SIP/2.0\r\nTo: sut <sip:+19086774567@10.0.0.0.1:5060;user=phone>\r\nFrom: sipp <sip:sipp@10.0.0.2:5060;user=phone>;tag=26939SIPpTag0014404\r\nCall-ID: 14"..., len=837, rcv_info=0x7ffd4afdae00, existing_context=0x0) at receive.c:208 #21 0x0000000000521838 in udp_read_req (si=0x7f8df36bda50, bytes_read=0x7ffd4afdaec8) at net/proto_udp/proto_udp.c:192 #22 0x000000000050d05b in handle_io (fm=0x7f8df371e128, idx=0, event_type=1) at net/net_udp.c:259 #23 0x000000000050ba2b in io_wait_loop_epoll (h=0x80be40 <_worker_io>, t=1, repeat=0) at net/../io_wait_loop.h:221 #24 0x000000000050d3db in udp_rcv_loop (si=0x7f8df36bda50) at net/net_udp.c:311 #25 0x000000000050d973 in udp_start_processes (chd_rank=0x7d8128 <chd_rank.10725>, startup_done=0x0) at net/net_udp.c:375 #26 0x0000000000495241 in main_loop () at main.c:671 #27 0x0000000000497cec in main (argc=3, argv=0x7ffd4afdb1c8) at main.c:1271 Coredump2: -------------- (gdb) bt #0 0x00000000004affeb in qm_detach_free (qm=0x7f5aa3ff4010, frag=0x7f5aa40c50f0) at mem/q_malloc.c:281 #1 0x00000000004b1233 in qm_free (qm=0x7f5aa3ff4010, p=0x7f5aa40c50b0, file=0x7f5aa069fbf5 "rest_client.c", func=0x7f5aa06a01fd <__FUNCTION__.8581> "osips_free", line=205) at mem/q_malloc.c:494 #2 0x00007f5aa04774f6 in curl_thread_create_thunk () from /lib64/libcurl.so.4 #3 0x00007f5a9edb9dc5 in start_thread () from /lib64/libpthread.so.0 #4 0x00007f5aa42eb28d in clone () from /lib64/libc.so.6 Regards, Agalya From: Bogdan-Andrei Iancu [mailto:bog...@opensips.org] Sent: Wednesday, October 05, 2016 3:58 PM To: Ramachandran, Agalya (Contractor) <agalya_ramachand...@comcast.com>; OpenSIPS users mailling list <users@lists.opensips.org> Subject: Re: opensips crash when doing load test That's a good news :) Try to add one patch at a time (incrementally) and let's see when it starts to crash again. Thanks and regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 22:55, Ramachandran, Agalya (Contractor) wrote: Hi Bogdan, With 2.2 version, am not seeing any crash. All looks good. Regards, Agalya From: users-boun...@lists.opensips.org<mailto:users-boun...@lists.opensips.org> [mailto:users-boun...@lists.opensips.org] On Behalf Of Ramachandran, Agalya (Contractor) Sent: Wednesday, October 05, 2016 10:50 AM To: Bogdan-Andrei Iancu <bog...@opensips.org><mailto:bog...@opensips.org>; OpenSIPS users mailling list <users@lists.opensips.org><mailto:users@lists.opensips.org> Subject: Re: [OpenSIPS-Users] opensips crash when doing load test Sure. I will test and share the results by end of day or tomorrow. Regards, Agalya From: Bogdan-Andrei Iancu [mailto:bog...@opensips.org] Sent: Wednesday, October 05, 2016 10:47 AM To: Ramachandran, Agalya (Contractor) <agalya_ramachand...@comcast.com<mailto:agalya_ramachand...@comcast.com>>; OpenSIPS users mailling list <users@lists.opensips.org<mailto:users@lists.opensips.org>> Subject: Re: opensips crash when doing load test Yes, that is ok (without the REST) - just to be sure which patch introduces the bug. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 05.10.2016 17:45, Ramachandran, Agalya (Contractor) wrote: Hi Bogdan, I have observed one more thing. In the same source code, when opensips acts as pure proxy no HTTP call is made opensips is not crashing. I suspect the crashes may be due to the new async fix provided. I can do load test in 2.2, but it will be just pure-proxy load test and not with the REST queries made. If that is OK, I can test and share the results. Regards, Agalya From: Bogdan-Andrei Iancu [mailto:bog...@opensips.org] Sent: Wednesday, October 05, 2016 9:46 AM To: Ramachandran, Agalya (Contractor) <agalya_ramachand...@comcast.com><mailto:agalya_ramachand...@comcast.com>; OpenSIPS users mailling list <users@lists.opensips.org><mailto:users@lists.opensips.org> Subject: Re: opensips crash when doing load test Hi Agalya, That is really weird, as that field (my_T) is initialized (with the address of the transaction) when the transaction is created (for all the possible branches) and never reset or set again. To avoid any regression due the patches you applied, could you please run your load test with the stock 2.2 code from git ? And see if the crash occurs again. Regards, Bogdan-Andrei Iancu OpenSIPS Founder and Developer http://www.opensips-solutions.com On 04.10.2016 19:51, Ramachandran, Agalya (Contractor) wrote: Hi Bogdan, Please find the values. Any common folder you have access for the outside world to put files? If yes, share me the details. I can upload the core file there. (gdb) print r_buf $1 = (struct retr_buf *) 0x7fded2484098 (gdb) print *r_buf $2 = {activ_type = 0, buffer = { s = 0x7fded0ca79f0 "INVITE sip:++12001000004@10.0.0.1:5060<mailto:sip:++12001000004@10.0.0.1:5060> SIP/2.0\r\nRecord-Route: <sip:96.119.240.35;lr>\r\nTo: sut <sip:+12452123456@10.0.0.1:5060;user=phone><mailto:sip:+12452123456@10.0.0.1:5060;user=phone>\r\nFrom: sipp <sip:sipp@10.0.0.2:5060;user=phone<mailto:sip:sipp@10.0.0.2:5060;user=phone>"..., len = 968}, dst = {proto = 1, proto_reserved1 = 0, to = {s = {sa_family = 2, sa_data = "\023\304`w\347\375\000\000\000\000\000\000\000"}, sin = { sin_family = 2, sin_port = 50195, sin_addr = {s_addr = 4259805024}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = { sin6_family = 2, sin6_port = 50195, sin6_flowinfo = 4259805024, sin6_addr = {__in6_u = { __u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, send_sock = 0x7fded841dcb0}, retr_timer = {next_tl = 0x7fded1af3518, prev_tl = 0x7fded15c8ae0, ld_tl = 0x0, time_out = 169500000, timer_list = 0x7fded0acbc48, deleted = 1, set = 0}, fr_timer = {next_tl = 0x0, prev_tl = 0x0, ld_tl = 0x0, time_out = 165, timer_list = 0x7fded8005b00 <detached_timer>, deleted = 0, set = 0}, retr_list = RT_T2, my_T = 0x0, branch = 0} Regards, Agalya From: Bogdan-Andrei Iancu [mailto:bog...@opensips.org] Sent: Tuesday, October 04, 2016 11:52 AM To: Ramachandran, Agalya (Contractor) <agalya_ramachand...@comcast.com><mailto:agalya_ramachand...@comcast.com>; OpenSIPS users mailling list <users@lists.opensips.org><mailto:users@lists.opensips.org> Subject: Re: opensips crash when doing load test Could you also print r_buf and *r_buf ?
_______________________________________________ Users mailing list Users@lists.opensips.org http://lists.opensips.org/cgi-bin/mailman/listinfo/users