[jira] Commented: (SM-1002) Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost
[ https://issues.apache.org/activemq/browse/SM-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39957 ] Thomas Termin commented on SM-1002: --- Bruce, The problem seems to be in the ProviderProcessor: You can test this with the following scenario for example: httpclient - consumer - lwcontainer1 - provider - NETWORK - consumer - lwcontainer2(with sleep of maybe 10sec) The httpclient has to send more requests in a loop then you have max connections per host. What happens is all threads (higher then max connections per host) are on lock to get a connection. If the first answer comes back there are no more threads to bring the answer back. And then you have the hang/deadlock. The http connection is closed after the done is initiated for the exchange. If the answer is not processed the done state will never initiated and the connection will not be closed and all other threads wait forever. To solve this there should be something like if the threads are don't get a connection, they should go back to the pool and the request should be queued. If there is a connection available the next request from the queue can be processed. Any thoughts on this? Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost -- Key: SM-1002 URL: https://issues.apache.org/activemq/browse/SM-1002 Project: ServiceMix Issue Type: Bug Components: servicemix-core Environment: Unix, Java 1.5 Reporter: Ruwan Linton When I am doing a performance test I have sent more than 40 concurrent requests from a single host to servicemix, and the maxConnectionsPerHost was 40. It is acceptable the failing to respond when it exceeds 40 (maxConnectionsPerHost) concurrent requests. But afterward, SM hangs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1002) Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost
[ https://issues.apache.org/activemq/browse/SM-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39958 ] Guillaume Nodet commented on SM-1002: - This is a good idea. The only problem is to make sure that if a connection is freed, a job will be automatically dequeued. So this is really a list and we need to manage it ourselves. Another option is to remove the limitation on the max number of connections and use the thread pool to control this. As there will be no more connections than the number of threads at any time. Well, I guess for the newer endpoints this is wrong because they are asynchronous, but ... Btw, I think the total number of connections should be set very high in any case, as this is more a system limitation. And we need to move the thread pool configuration on a per endpoint basis instead of per component imho. So what if we put both limits to a very high number in the default configuration ? Is there any drawbacks? Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost -- Key: SM-1002 URL: https://issues.apache.org/activemq/browse/SM-1002 Project: ServiceMix Issue Type: Bug Components: servicemix-core Environment: Unix, Java 1.5 Reporter: Ruwan Linton When I am doing a performance test I have sent more than 40 concurrent requests from a single host to servicemix, and the maxConnectionsPerHost was 40. It is acceptable the failing to respond when it exceeds 40 (maxConnectionsPerHost) concurrent requests. But afterward, SM hangs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1002) Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost
[ https://issues.apache.org/activemq/browse/SM-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39959 ] Thomas Termin commented on SM-1002: --- So if I understand you correctly then you will limit the number of connections with the number of threads. I will try this out. But you have to remove the opportunity for the user to change the values for max connections and max connections per host. Because If you don't do that the user can again configure a deadlock scenario, right? Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost -- Key: SM-1002 URL: https://issues.apache.org/activemq/browse/SM-1002 Project: ServiceMix Issue Type: Bug Components: servicemix-core Environment: Unix, Java 1.5 Reporter: Ruwan Linton When I am doing a performance test I have sent more than 40 concurrent requests from a single host to servicemix, and the maxConnectionsPerHost was 40. It is acceptable the failing to respond when it exceeds 40 (maxConnectionsPerHost) concurrent requests. But afterward, SM hangs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (SM-1002) Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost
So what if we put both limits to a very high number in the default configuration ? Is there any drawbacks? The risk there is that you are just increasing the size of the bang when it goes wrong. The system must degrade gracefully under heavy load in a production environment. If it has operating modes in which the container can hang then there is significant risk of message loss. -- Terry
[jira] Commented: (SM-1002) Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost
[ https://issues.apache.org/activemq/browse/SM-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39961 ] Guillaume Nodet commented on SM-1002: - I think that if you allow more connections that the number of threads available, you can not have any deadlock. I would not remove the way to impose a maximum of connections per host if the default settings are changed in a way that fix the problem (just setting them to Integer.MAX should work). If the user need to have a max number of connections, he could do so, knowing that the number of threads should be lower (or unbounded, in which case all threads would wait for a free connection, but this would not be very interesting) Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost -- Key: SM-1002 URL: https://issues.apache.org/activemq/browse/SM-1002 Project: ServiceMix Issue Type: Bug Components: servicemix-core Environment: Unix, Java 1.5 Reporter: Ruwan Linton When I am doing a performance test I have sent more than 40 concurrent requests from a single host to servicemix, and the maxConnectionsPerHost was 40. It is acceptable the failing to respond when it exceeds 40 (maxConnectionsPerHost) concurrent requests. But afterward, SM hangs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (SM-1002) Service mix hangs when the # concurrent requests exceeds the maxConnectionsPerHost
This solution is intended for the provider site not for the consumer site. So what I mean is to limit the outgoing connections from servicemix (for example to an external web service) over the provider endpoint with the thread limit. It doesn't depend on the consumer site. Do you know what I mean or did I get something wrong? You still have potentially the same problem as if you allow outgoing connections to propagate uncontrollably you will start to saturate the network layer leading to dropped packets, escalating retries and eventual bottleneck backing up to the sending threads. Since this can also disrupt other systems on the network it can turn into a self-inflicted DOS Attack. The system must play nicely with others in a production environment where scarce resources need to be shared (often in order of priority). -- Terry
[jira] Assigned: (SM-1033) WSDL-First example fails due to missing defaultMep attribute in wsdl-first-http-su's xbean.xml
[ https://issues.apache.org/activemq/browse/SM-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Snyder reassigned SM-1033: Assignee: Bruce Snyder WSDL-First example fails due to missing defaultMep attribute in wsdl-first-http-su's xbean.xml --- Key: SM-1033 URL: https://issues.apache.org/activemq/browse/SM-1033 Project: ServiceMix Issue Type: Bug Affects Versions: 3.2 Reporter: Bruce Snyder Assignee: Bruce Snyder Reported via the user mailing list here: http://www.nabble.com/WSDL-First-Sample-Failing-tf4193170s12049.html#a11924666 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1033) WSDL-First example fails due to missing defaultMep attribute in wsdl-first-http-su's xbean.xml
[ https://issues.apache.org/activemq/browse/SM-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39965 ] Bruce Snyder commented on SM-1033: -- Committed fix: http://svn.apache.org/viewvc?view=revrevision=566880 WSDL-First example fails due to missing defaultMep attribute in wsdl-first-http-su's xbean.xml --- Key: SM-1033 URL: https://issues.apache.org/activemq/browse/SM-1033 Project: ServiceMix Issue Type: Bug Affects Versions: 3.2 Reporter: Bruce Snyder Assignee: Bruce Snyder Fix For: 3.2 Reported via the user mailing list here: http://www.nabble.com/WSDL-First-Sample-Failing-tf4193170s12049.html#a11924666 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SM-1033) WSDL-First example fails due to missing defaultMep attribute in wsdl-first-http-su's xbean.xml
[ https://issues.apache.org/activemq/browse/SM-1033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Snyder resolved SM-1033. -- Resolution: Fixed Fix Version/s: 3.2 WSDL-First example fails due to missing defaultMep attribute in wsdl-first-http-su's xbean.xml --- Key: SM-1033 URL: https://issues.apache.org/activemq/browse/SM-1033 Project: ServiceMix Issue Type: Bug Affects Versions: 3.2 Reporter: Bruce Snyder Assignee: Bruce Snyder Fix For: 3.2 Reported via the user mailing list here: http://www.nabble.com/WSDL-First-Sample-Failing-tf4193170s12049.html#a11924666 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (SM-1027) InstallationContext.getInstallationDescriptorExtension() not returning all extensions
[ https://issues.apache.org/activemq/browse/SM-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chad Gallemoare updated SM-1027: Attachment: Sm-1027.zip I've attached the component.xml file and the ComponentTest that I modified to make the test fail to use as a diff for what is in the baseline. InstallationContext.getInstallationDescriptorExtension() not returning all extensions -- Key: SM-1027 URL: https://issues.apache.org/activemq/browse/SM-1027 Project: ServiceMix Issue Type: Bug Components: servicemix-core Affects Versions: incubation Reporter: Chad Gallemoare Attachments: Sm-1027.zip During the Bootsrap init(InstallationContext installContext) you should be able to call installContext.getInstallationDescriptorExtension() and receive a DocumentFragment back of all the extensions in the jbi.xml. For example: component . Component info . !-- Installation Description Extensions -- observer:Observertrue/observer:Observer config:Configuration config:Locationlocalhost/config:Location config:Port/config:Port /config:Configuration /component Currently when you call getInstallationDescriptorExtension during Bootstrap init(), you only get back the following extensions: observer:Observertrue/observer:Observer The rest of the extensions are being left off. I checked out the source code and verified that what I thought I was seeing was correct. I modified the ComponentTest (under /core/servicemix-core) to compare the expected output to the actual output, also modified the component.xml used in the test to include a extension like the one above. Once I did this the unit test failed, only returning me the first extension listed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
