Hi, Please reply, I am not able to debug what the issue is. Thanks
On Fri, Mar 16, 2018 at 11:51 PM, Sunil Kumar <skgola1...@gmail.com> wrote: > Hi, > PFA the sprout_current log file. Please guide me the solution, I am newbie > to clearwater :-) > > Thanks, > Sunil > > On Fri, Mar 16, 2018 at 11:07 PM, Sunil Kumar <skgola1...@gmail.com> > wrote: > >> Hi Michael, >> Thanks for replying. When I am running single call it giving some errors >> like: >> >> *[]ubuntu@stress:~$ /usr/share/clearwater/bin/run_stress iind.intel.com >> <http://iind.intel.com> 1 2* >> Starting initial registration, will take 0 seconds >> Initial registration succeeded >> Starting test >> Test complete >> Traceback (most recent call last): >> File "/usr/share/clearwater/bin/run_stress", line 340, in <module> >> with open(CALLER_STATS) as f: >> IOError: [Errno 2] No such file or directory: >> '/var/log/clearwater-sip-stress/11218_caller_stats.log' >> >> >> *[]ubuntu@stress:~$ cat >> /var/log/clearwater-sip-stress/11218_re_reg_stats.log* >> StartTime;LastResetTime;CurrentTime;ElapsedTime(P);ElapsedTi >> me(C);TargetRate;CallRate(P);CallRate(C);IncomingCall(P); >> IncomingCall(C);OutgoingCall(P);OutgoingCall(C);TotalCallCr >> eated;CurrentCall;SuccessfulCall(P);SuccessfulCall(C); >> FailedCall(P);FailedCall(C);FailedCannotSendMessage(P);Faile >> dCannotSendMessage(C);FailedMaxUDPRetrans(P);FailedMaxUDPRet >> rans(C);FailedTcpConnect(P);FailedTcpConnect(C);FailedTcpClo >> sed(P);FailedTcpClosed(C);FailedUnexpectedMessage(P);FailedU >> nexpectedMessage(C);FailedCallRejected(P);FailedCallRejected >> (C);FailedCmdNotSent(P);FailedCmdNotSent(C);FailedRegexpDoes >> ntMatch(P);FailedRegexpDoesntMatch(C);FailedRegexpShouldntMa >> tch(P);FailedRegexpShouldntMatch(C);FailedRegexpHdrNotFound( >> P);FailedRegexpHdrNotFound(C);FailedOutboundCongestion(P);Fa >> iledOutboundCongestion(C);FailedTimeoutOnRecv(P);FailedTimeo >> utOnRecv(C);FailedTimeoutOnSend(P);FailedTimeoutOnSend(C);Ou >> tOfCallMsgs(P);OutOfCallMsgs(C);DeadCallMsgs(P); >> DeadCallMsgs(C);Retransmissions(P);Retransmissions(C);AutoAn >> swered(P);AutoAnswered(C);Warnings(P);Warnings(C);FatalE >> rrors(P);FatalErrors(C);WatchdogMajor(P);WatchdogMajor(C);Wa >> tchdogMinor(P);WatchdogMinor(C);ResponseTime1(P);ResponseTi >> me1(C);ResponseTime1StDev(P);ResponseTime1StDev(C);CallLengt >> h(P);CallLength(C);CallLengthStDev(P);CallLengthStDev(C);Res >> ponseTimeRepartition1;ResponseTimeRepartition1_<2;ResponseTi >> meRepartition1_<10;ResponseTimeRepartition1_<20;ResponseTime >> Repartition1_<50;ResponseTimeRepartition1_<100;ResponseTimeR >> epartition1_<200;ResponseTimeRepartition1_<500;ResponseTimeR >> epartition1_<1000;ResponseTimeRepartition1_< >> 2000;ResponseTimeRepartition1_>=2000; >> 2018-03-17 06:43:11.763066 1521249191.763066;2018-03-17 >> 06:43:11.763066 1521249191.763066;2018-03-17 06:43:11.770581 >> 1521249191.770581;00:00:00;00:00:00;0.000555556;0;0;0;0;0;0; >> 0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0; >> 0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;00:00:00:000000;00:00:00 >> :000000;00:00:00:000000;00:00:00:000000;00:00:00:000000;00: >> 00:00:000000;00:00:00:000000;00:00:00:000000;;0;0;0;0;0;0;0;0;0;0; >> 2018-03-17 06:43:11.763066 1521249191.763066;2018-03-17 >> 06:43:11.770864 1521249191.770864;2018-03-17 06:43:11.772711 >> 1521249191.772711;00:00:00;00:00:00;0.000555556;0;0;0;0;0;0; >> 0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0; >> 0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;00:00:00:000000;00:00:00 >> :000000;00:00:00:000000;00:00:00:000000;00:00:00:000000;00: >> 00:00:000000;00:00:00:000000;00:00:00:000000;;0;0;0;0;0;0;0;0;0;0; >> 2018-03-17 06:43:11.763066 1521249191.763066;2018-03-17 >> 06:43:11.772796 1521249191.772796;2018-03-17 06:43:11.772860 >> 1521249191.772860;00:00:00;00:00:00;0.000555556;0;0;0;0;0;0; >> 0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0; >> 0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;00:00:00:000000;00:00:00 >> :000000;00:00:00:000000;00:00:00:000000;00:00:00:000000;00: >> 00:00:000000;00:00:00:000000;00:00:00:000000;;0;0;0;0;0;0;0;0;0;0; >> >> >> >> *[sprout]ubuntu@sprout:/var/log/sprout$ tail -30 sprout_current.txt* >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug >> thread_dispatcher.cpp:183: Request latency so far = 102us >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug pjsip: sip_endpoint.c >> Distributing rdata to modules: Request msg OPTIONS/cseq=370860 >> (rdata0x7f7548081478) >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug uri_classifier.cpp:139: >> home domain: false, local_to_node: true, is_gruu: false, >> enforce_user_phone: false, prefer_sip: true, treat_number_as_phone: false >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug uri_classifier.cpp:172: >> Classified URI as 3 >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug pjsip: endpoint >> Response msg 200/OPTIONS/cseq=370860 (tdta0x7f7548382670) created >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Verbose >> common_sip_processing.cpp:103: TX 282 bytes Response msg >> 200/OPTIONS/cseq=370860 (tdta0x7f7548382670) to TCP 10.224.61.22:51202: >> --start msg-- >> >> SIP/2.0 200 OK >> Via: SIP/2.0/TCP 10.224.61.22;rport=51202;recei >> ved=10.224.61.22;branch=z9hG4bK-370860 >> Call-ID: poll-sip-370860 >> From: "poll-sip" <sip:poll-sip@10.224.61.22>;tag=370860 >> To: <sip:poll-sip@10.224.61.22>;tag=z9hG4bK-370860 >> CSeq: 370860 OPTIONS >> Content-Length: 0 >> >> >> --end msg-- >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug >> common_sip_processing.cpp:275: Skipping SAS logging for OPTIONS response >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug pjsip: tdta0x7f754838 >> Destroying txdata Response msg 200/OPTIONS/cseq=370860 (tdta0x7f7548382670) >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug >> thread_dispatcher.cpp:270: Worker thread completed processing message >> 0x7f7548081478 >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug >> thread_dispatcher.cpp:284: Request latency = 278us >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug >> event_statistic_accumulator.cpp:32: Accumulate 278 for 0xf627a8 >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug >> event_statistic_accumulator.cpp:32: Accumulate 278 for 0xf62820 >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug load_monitor.cpp:341: >> Not recalculating rate as we haven't processed 20 requests yet (only 18). >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug utils.cpp:878: Removed >> IOHook 0x7f75a6face30 to stack. There are now 0 hooks >> 17-03-2018 01:18:51.121 UTC [7f75a6fad700] Debug >> thread_dispatcher.cpp:158: Attempting to process queue element >> 17-03-2018 01:18:53.122 UTC [7f754feff700] Verbose pjsip: tcps0x7f754842 >> TCP connection closed >> 17-03-2018 01:18:53.122 UTC [7f754feff700] Debug >> connection_tracker.cpp:67: Connection 0x7f75484260d8 has been destroyed >> 17-03-2018 01:18:53.122 UTC [7f754feff700] Verbose pjsip: tcps0x7f754842 >> TCP transport destroyed with reason 70016: End of file (PJ_EEOF) >> >> >> Thanks and Regards, >> Sunil >> >> >> >> On Fri, Mar 16, 2018 at 10:50 PM, Michael Duppré < >> michael.dup...@metaswitch.com> wrote: >> >>> Hello Sunil, >>> >>> >>> >>> It sounds like you have two different problems: >>> >>> 1. Running `/usr/share/clearwater/bin/run_stress <home_domain> 1800 >>> 2` doesn’t succeed making any calls >>> - The good news is that the initial REGISTERs that the tool sends >>> out are working fine as you can see in the output `Initial >>> registration >>> succeeded`! >>> - To make the calls work, I would suggest just running one call >>> with the tool initially and once this works go back to running 1800 >>> calls. >>> To debug this, you should turn on debug logs on Sprout (see >>> http://clearwater.readthedocs.io/en/stable/Troubleshooting_a >>> nd_Recovery.html#sprout >>> >>> <http://clearwater.readthedocs.io/en/stable/Troubleshooting_and_Recovery.html#sprout>), >>> run the tool again with one call. Then have a look at the logs of the >>> tool >>> and the Sprout logs and find out why Sprout is sending back a 503 >>> response. >>> 2. Running `/usr/share/clearwater/bin/run_stress <home_domain> 1800 >>> 10 --icscf-target TARGET=sprout.{domain}:5052 --scscf-target >>> TARGET=sprout.{domain}:5054` crashes >>> - You probably don’t need to provide more arguments when running >>> the tool as it should work with the defaults. So I suggest getting >>> the tool >>> work without any additional arguments and only use these if you have a >>> specific need for additional configuration. >>> - The error ` Unknown remote host 'TARGET=sprout.<domain>` says, >>> the tool wasn’t able to lookup this domain. In fact, you need to >>> replace >>> the word `TARGET` with your full sprout domain. If for example >>> `my_ims_domain` is your domain, you would need to run the command as >>> follows: >>> - `/usr/share/clearwater/bin/run_stress <home_domain> 1800 10 >>> --icscf-target sprout.my_ims_domain:5052 --scscf-target >>> sprout.my_ims_domain:5054` >>> >>> >>> >>> Hope this helps, good luck getting the calls to work! >>> >>> >>> >>> Kind regards, >>> >>> Michael >>> >>> >>> >>> >>> >>> *From:* Clearwater [mailto:clearwater-bounces@lis >>> ts.projectclearwater.org] *On Behalf Of *Pushpendra >>> *Sent:* 14 March 2018 12:05 >>> *To:* Bennett Allen <bennett.al...@metaswitch.com>; >>> clearwater@lists.projectclearwater.org >>> *Subject:* [Project Clearwater] stress testing >>> >>> >>> >>> Hi All, >>> >>> I need to ask few questions regarding stress testing for manual >>> installation of clearwater. I am following http://clearwater.re >>> adthedocs.io/en/stable/Clearwater_stress_testing.html for stress >>> testing. >>> >>> I have set the* local_ip of this node* only in >>> * /etc/clearwater/local_config. *I also set *reg_max_expires=1800* in >>> all the other node in *shared_config *after that I created the 50000 >>> no. in vellum node as given in doc >>> >>> https://github.com/Metaswitch/crest/blob/dev/docs/Bulk-Prov >>> isioning%20Numbers.md?utf8=%E2%9C%93 >>> >>> and then I install the debian package *sudo apt-get install >>> clearwater-sip-stress-careonly *on new node. >>> >>> when I run >>> >>> /usr/share/clearwater/bin/run_stress <home_domain> 1800 2 >>> >>> Starting initial registration, will take 22 seconds >>> >>> Initial registration succeeded >>> >>> Starting test >>> >>> Test complete >>> >>> >>> >>> Elapsed time: 00:02:00 >>> >>> Start: 2018-03-14 18:06:06.635672 >>> >>> End: 2018-03-14 18:08:06.714723 >>> >>> >>> >>> Total calls: 39 >>> >>> Successful calls: 0 (0.0%) >>> >>> Failed calls: 39 (100.0%) >>> >>> Unfinished calls: 0 >>> >>> >>> >>> Retransmissions: 0 >>> >>> >>> >>> Average time from INVITE to 180 Ringing: 0.0ms >>> >>> # of calls with 0-2ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 2-10ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 10-20ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 20-50ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 50-100ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 100-200ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 200-500ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 500-1000ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 1000-2000ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> # of calls with 2000+ms from INVITE to 180 Ringing: 0 (0.0%) >>> >>> Failed: call success rate 0.0% is lower than target 100.0%! >>> >>> >>> >>> Total re-REGISTERs: 120 >>> >>> Successful re-REGISTERs: 120 (100.0%) >>> >>> Failed re-REGISTERS: 0 (0.0%) >>> >>> >>> >>> REGISTER retransmissions: 0 >>> >>> >>> >>> Average time from REGISTER to 200 OK: 30.0ms >>> >>> >>> >>> Log files at /var/log/clearwater-sip-stress/13921_* >>> >>> >>> >>> 1. I have not mention new node's IP in any other node of clearwater, how >>> the other node know about it? >>> >>> >>> >>> My question is, *Do I need to mention the IP of new node in any other >>> node of clearwater?* >>> >>> >>> >>> >>> >>> when I run - >>> >>> []ubuntu@stress:~$ /usr/share/clearwater/bin/run_stress <home_domain> >>> 1800 10 --icscf-target TARGET=sprout.{domain}:5052 --scscf-target >>> TARGET=sprout.{domain}:5054 >>> >>> >>> >>> Starting initial registration, will take 22 seconds >>> >>> Initial registration failed - see >>> /var/log/clearwater-sip-stress/14115_initial_reg_errors.log >>> for details of the errors >>> >>> >>> >>> >>> >>> >>> >>> *[]ubuntu@stress:/var/log/clearwater-sip-stress$ tail -30 >>> 14115_initial_reg_errors.log* >>> >>> sipp: The following events occured: >>> >>> 2018-03-14 18:11:06.637370 1521031266.637370: Unknown remote host >>> 'TARGET=sprout.<domain>' (Name or service not known, Inappropriate ioctl >>> for device). >>> >>> Use 'sipp -h' for details. >>> >>> >>> >>> >>> >>> *[]ubuntu@stress:/var/log/clearwater-sip-stress$ cat >>> 9923_caller_errors.log* >>> >>> 2018-03-14 00:32:45.274812 1520967765.274812: Aborting call on >>> unexpected message for Call-Id '54-9937@127.0.1.1': while expecting >>> '183' (index 2), received 'SIP/2.0 503 Service Unavailable >>> >>> Via: SIP/2.0/TCP 127.0.1.1:34015;received=10.22 >>> 4.61.13;branch=z9hG4bK-9937-54-0 >>> >>> Record-Route: <sip:scscf.sprout. <home_domain> >>> ;transport=TCP;lr;billing-role=charge-term> >>> >>> Record-Route: <sip:scscf.sprout. <home_domain> >>> ;transport=TCP;lr;billing-role=charge-orig> >>> >>> Call-ID: 54-9937@127.0.1.1 >>> >>> From: <sip:2010000252@<home_domain>>;tag=9937SIPpTag0054 >>> >>> To: <sip:2010000647@ <home_domain> >;tag=z9hG4bKPjJVqopmpeih7pUo >>> xJrYOG7CO4-Bm69ozI >>> >>> CSeq: 1 INVITE >>> >>> P-Charging-Vector: icid-value="9937SIPpTag0054";o >>> rig-ioi=<home_domain>;term-ioi=<home_domain> >>> >>> P-Charging-Function-Addresses: ccf=0.0.0.0 >>> >>> Content-Length: 0 >>> >>> >>> >>> '. >>> >>> >>> >>> >>> >>> >>> >>> Regards, >>> >>> >>> >>> >>> >>> >> >> >
_______________________________________________ Clearwater mailing list Clearwater@lists.projectclearwater.org http://lists.projectclearwater.org/mailman/listinfo/clearwater_lists.projectclearwater.org
