I'm running performance tests and noticing that under load I am getting increasingly large amounts of time spent in the SEDA queues.
For example below are some lines from my log when the app is under load (20 concurrent threads). The first is from one Route (Get360List) that then call the calls the 3 routes via multicast in parallel (GetPolicyList, GetFeeList, GetClaimList). There is a ~600ms delay between the multicast being called and the Consumers on those SEDA queues picking up the messages. I have 30 concurrent consumers set for each SEDA endpoint, so I would have thought that there would be no shortage of consumers to pull messages from these queues. Are there any other settings that I should be looking at to reduce this delay? 2011-01-10 10:57:02,249 [a://get360Start] INFO Aon_360Request -(ID-aon43098-1487-1294617268515-0-3526) Started Single Client AONZBAD2571 for Get360List 2011-01-10 10:57:02,827 [PolicyListStart] INFO Aon_GetPolicyListRoute -(ID-aon43098-1487-1294617268515-0-3526) Starting Single-Client for GetPolicyList 2011-01-10 10:57:02,827 [PolicyListStart] INFO ExecProducer -(ID-aon43098-1487-1294617268515-0-3526) Executing ExecCommand [args=[cbslocal, AONZBAD2571PDTrue], executable=C:/workspaces/CRMS/MockCBS/GetPolicyList.bat, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=true] 2011-01-10 10:57:02,827 [PolicyListStart] INFO GetOsCmdRouteBuilder$1 -(ID-aon43098-1487-1294617268515-0-3526) Calling command: C:/workspaces/CRMS/MockCBS/GetPolicyList.bat cbslocal AONZBAD2571PDTrue 2011-01-10 10:57:02,827 [getFeeListStart] INFO Aon_GetFeeListRoute -(ID-aon43098-1487-1294617268515-0-3526) Starting Single-Client for GetFeeList 2011-01-10 10:57:02,827 [tClaimListStart] INFO Aon_GetClaimListRoute -(ID-aon43098-1487-1294617268515-0-3526) Starting Single-Client for GetClaimList 2011-01-10 10:57:02,827 [tClaimListStart] INFO GetOsCmdRouteBuilder$1 -(ID-aon43098-1487-1294617268515-0-3526) Calling command: C:/workspaces/CRMS/MockCBS/GetClaimList.bat cbslocal AONZBAD2571TrueD 2011-01-10 10:57:03,015 [PolicyListStart] INFO ExecProducer -(ID-aon43098-1487-1294617268515-0-3526) The command ExecCommand [args=[cbslocal, AONZBAD2571PDTrue], executable=C:/workspaces/CRMS/MockCBS/GetPolicyList.bat, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=true] had exit value 0 2011-01-10 10:57:03,499 [PolicyListStart] INFO Aon_GetPolicyListRoute -(ID-aon43098-1487-1294617268515-0-3526) Finished processing single Client for GetPolicyList 2011-01-10 10:57:03,593 [getFeeListStart] INFO GetOsCmdRouteBuilder$1 -(ID-aon43098-1487-1294617268515-0-3526) Calling command: C:/workspaces/CRMS/MockCBS/GetFeeList.bat cbslocal AONZBAD2571TrueD 2011-01-10 10:57:03,593 [tClaimListStart] INFO ExecProducer -(ID-aon43098-1487-1294617268515-0-3526) Executing ExecCommand [args=[cbslocal, AONZBAD2571TrueD], executable=C:/workspaces/CRMS/MockCBS/GetClaimList.bat, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=true] 2011-01-10 10:57:03,718 [getFeeListStart] INFO ExecProducer -(ID-aon43098-1487-1294617268515-0-3526) Executing ExecCommand [args=[cbslocal, AONZBAD2571TrueD], executable=C:/workspaces/CRMS/MockCBS/GetFeeList.bat, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=true] 2011-01-10 10:57:03,796 [tClaimListStart] INFO ExecProducer -(ID-aon43098-1487-1294617268515-0-3526) The command ExecCommand [args=[cbslocal, AONZBAD2571TrueD], executable=C:/workspaces/CRMS/MockCBS/GetClaimList.bat, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=true] had exit value 0 2011-01-10 10:57:03,874 [getFeeListStart] INFO ExecProducer -(ID-aon43098-1487-1294617268515-0-3526) The command ExecCommand [args=[cbslocal, AONZBAD2571TrueD], executable=C:/workspaces/CRMS/MockCBS/GetFeeList.bat, timeout=9223372036854775807, outFile=null, workingDir=null, useStderrOnEmptyStdout=true] had exit value 0 2011-01-10 10:57:04,608 [getFeeListStart] INFO Aon_GetFeeListRoute -(ID-aon43098-1487-1294617268515-0-3526) Finished processing single Client for GetFeeList 2011-01-10 10:57:04,655 [tClaimListStart] INFO Aon_GetClaimListRoute -(ID-aon43098-1487-1294617268515-0-3526) Finished processing single Client for GetClaimList 2011-01-10 10:57:05,155 [a://get360Start] INFO Aon_360Request -(ID-aon43098-1487-1294617268515-0-3526) Finished processing Single Client AONZBAD2571 for Get360List Thanks, Damian. Damian Harvey | Aon Australia/Pacific Level 31, 201 Kent Street Sydney NSW 2000 t: +61292537000 | f: +61292537952 e: damian.har...@aon.com.au<mailto:damian.har...@aon.com.au> Please consider the environment before printing this email. ________________________________ This communication (and any attachments) is directed in confidence to the addressee(s) listed above, and may not otherwise be distributed, copied or used. The contents of this communication may also be subject to privilege, and all rights to that privilege are expressly claimed and not waived. If you have received this communication in error, please notify us by reply e-mail or by telephone and delete this communication (and any attachments) without making a copy. Before opening or using attachments, you should check them for viruses and defects. We do not accept liability in connection with computer virus, data corruption, delay, interruption, unauthorised access or unauthorised amendment.
