Hello Asankha, At the moment we are running on Windows XP, so we'll have to do some research about how to increase the open file limit on Windows.
We did some further testing on this issue, and found that the "Too many open files" error only seems to occur when hot deployment is set to true in axis2.xml, and it occurs after ~400 iterations of loading the configurationContext. Also, even with the patch we put into the org.apache.axis2.deployment.util.Utils class, another problem (OutOfMemoryError) occurs after ~2826 iterations - so it's not really a permanent fix. When hot deployment is set to false however, no problems seem to occur even after 100000 iterations. I suspect this might be related to the issue raised previously in this user forum regarding cleanup of AxisConfigurators (see http://mail-archives.apache.org/mod_mbox/ws-axis-user/200706.mbox/[EMAIL PROTECTED] ). Also, when we substituted the nightly build from 3rd July into the dependencies for the test (leaving hotdeployment as true), we get the original "Too many open files" error after 400 iterations. I've created a issue in JIRA for this ( https://issues.apache.org/jira/browse/AXIS2-2899) and attached the configuration/modules we're using to the issue. Thanks, Greg "Asankha C. Perera" <[EMAIL PROTECTED]> 02/07/2007 10:03 PM Please respond to axis-user@ws.apache.org To axis-user@ws.apache.org cc Subject Re: [Axis2] Temporary files/ Stress testing with Axis2 1.2 [SEC=UNCLASSIFIED] Reference Expires Greg Are you running the system on Windows or Linux? The OS decides the number of files a process can open. If you are using a Unix system, try to increase the number of files as follows. This is not a solution for a possible Axis2 issue, but something to make sure before you run a typical load test. You may be able to find alternatives for Windows on Google. Maybe http://coldfused.blogspot.com/2007/02/mystery-of-too-many-open-files.html would be of interest too. /etc/sysctl.conf fs.file-max = 2097152 /etc/security/limits.conf * soft nofile 4096 * hard nofile 65535 asankha Davanum Srinivas wrote: > Greg, > > Please create an issue in JIRA and upload your test ASAP. we are in > the middle of perf testing. this would be a great scenario to fix for > 1.3 release. > > thx, > dims > > On 7/2/07, [EMAIL PROTECTED] > <[EMAIL PROTECTED]> wrote: >> >> Hello, >> >> Our team is currently using Axis2 1.2 in our application and we are >> running >> some performance and stress tests with JMeter. >> While running these stress tests we have come across two issues that >> we hope >> someone can help us with. >> >> The first issue relates to temporary files. In our stress testing we >> were >> using the URLBasedAxisConfigurator to load .mar files on the client >> side, >> but got the error below with 1 user and 1000 iterations: >> >> java.lang.RuntimeException: java.io.IOException: Too many open files;; >> at >> org.apache.axis2.deployment.util.Utils.getURLsForAllJars(Utils.java(Compiled >> >> Code));; >> at >> org.apache.axis2.deployment.util.Utils.createClassLoader(Utils.java:684);; >> >> at >> au.gov.centrelink.jfs.proxy.axis2.ClientURLBasedAxisConfigurator.loadRepositoryFromURL(ClientURLBasedAxisConfigurator.java(Compiled >> >> Code));; >> at >> org.apache.axis2.deployment.URLBasedAxisConfigurator.getAxisConfiguration(URLBasedAxisConfigurator.java:72);; >> >> at >> org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:60);; >> >> at >> au.gov.centrelink.jfs.proxy.ProxyFactory.createStub(ProxyFactory.java:699);; >> >> at >> au.gov.centrelink.jfs.proxy.ProxyFactory.createProxy(ProxyFactory.java:440);; >> >> at >> au.gov.centrelink.jfs.proxy.ProxyFactory.createProxy(ProxyFactory.java:395);; >> >> at >> au.gov.centrelink.jsf.pst.core.CDCCustomerEmailAddressTester2.execute(CDCCustomerEmailAddressTester2.java:64);; >> >> at >> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:421);; >> >> at >> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:226);; >> >> at >> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164);; >> >> at >> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415);; >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:763);; >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:856);; >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1572);; >> >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1521);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:113);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:82);; >> >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:759);; >> >> at >> com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java:334);; >> >> at >> org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1056);; >> >> at >> org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java:388);; >> >> at >> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:231);; >> >> at >> org.apache.struts.action.ActionServlet.process(ActionServlet.java:1164);; >> >> at >> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:415);; >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:763);; >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:856);; >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1572);; >> >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1521);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:136);; >> >> at >> au.gov.centrelink.itsecurity.acs.web.SecureRequestFilter.doFilter(SecureRequestFilter.java:60);; >> >> at >> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:142);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:121);; >> >> at >> au.gov.centrelink.jaf.liser.api.WebRequestLiser.doFilter(WebRequestLiser.java:267);; >> >> at >> au.gov.centrelink.jaf.interceptor.core.JAFFilter.doFilter(JAFFilter.java:99);; >> >> at >> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:142);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:121);; >> >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:82);; >> >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:759);; >> >> at >> com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:89);; >> >> at >> com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1924);; >> >> at >> com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:98);; >> >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:472);; >> >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:411);; >> >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:288);; >> >> at >> com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminaters(NewConnectionInitialReadCallback.java:207);; >> >> at >> com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:109);; >> >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java:566);; >> >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java:619);; >> >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java:952);; >> >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java:1039);; >> >> at >> com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1471);; >> >> Caused by: java.io.IOException: Too many open files;; >> at >> java.io.WinNTFileSystem.createFileExclusively(Native >> Method);; >> at java.io.File.checkAndCreate(File.java(Compiled Code));; >> at java.io.File.createTempFile(File.java(Compiled Code));; >> at java.io.File.createTempFile(File.java(Inlined Compiled >> Code));; >> at >> org.apache.axis2.deployment.util.Utils.createTempFile(Utils.java(Compiled >> >> Code));; ... 53 more;; ]] >> >> >> At that time we were loading the axis2 Configuration context each >> time the >> Stub was instantiated. Looking at the code for the method >> getURLsForAllJars() in >> org.apache.axis2.deployment.util.Utils, it looks like a >> temp file is always created for the URL that is passed in. Is it >> possible >> to change the method so it checks whether the URL uses the 'file' >> protocol, >> and if so, read from the file directly, instead of creating a temp >> file and >> reading from it? When we made this change we found that the "Too >> many open >> files" problem seemed to go away. Although we have switched to >> caching the >> configuration context rather than loading it every time, we still >> think the >> above change would be a good one to make. >> >> >> Our other issue is a WstxUnexpectedCharException that seems to occur >> intermittently - it has occurred only once in our last two runs. >> We were running JMeter with 20 concurrent users, each of which makes >> 10,000 >> requests (200,000 requests made in total) to a web application. The web >> application acts as the web service client and makes service calls to a >> servlet. The servlet returns essentially the same SOAP response for >> every >> request it receives (the only change is in the text content of an xml >> element in the SOAP body of the response, which changes slightly each >> time). >> In the last two runs, we got one instance of the following >> exception, which >> seems to happen in the web app while parsing the SOAP response: >> >> [29/06/07 13:27:14:469 EST] 0000004a SystemErr R >> au.gov.centrelink.jfs.proxy.envlookup.EnvironmentLookupException: >> RemoteException occurred while doing an environment lookup for CRN >> 200393036S >> at >> au.gov.centrelink.jfs.proxy.envlookup.impl.DefaultCustomerEnvironmentLookupImpl.findEnvironment(DefaultCustomerEnvironmentLookupImpl.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient.doRouting(AbstractCustomerServiceClient.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.CGLIB$doRouting$0(<generated>(Inlined >> >> Compiled Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46$$FastClassByCGLIB$$183db92b.invoke(<generated>(Compiled >> >> Code)) >> at >> net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java(Inlined >> Compiled Code)) >> at >> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.doRouting(<generated>(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractCustomerServiceClient$$EnhancerByCGLIB$$cdba8c46.retrieveEmailAddress(<generated>(Compiled >> >> Code)) >> at >> au.gov.centrelink.jsf.pst.core.CDCCustomerEmailAddressTester2.execute(CDCCustomerEmailAddressTester2.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java(Inlined >> >> Compiled Code)) >> at >> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined >> >> Compiled Code)) >> at >> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled >> >> Code)) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled >> Code)) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.webapp.WebAppRequestDispatcher.forward(WebAppRequestDispatcher.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.RequestProcessor.processForwardConfig(RequestProcessor.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled >> >> Code)) >> at >> org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined >> >> Compiled Code)) >> at >> org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled >> >> Code)) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled >> Code)) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.itsecurity.acs.web.SecureRequestFilter.doFilter(SecureRequestFilter.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jaf.liser.api.WebRequestLiser.doFilter(WebRequestLiser.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jaf.interceptor.core.JAFFilter.doFilter(JAFFilter.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled >> >> Code)) >> at >> com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled >> >> Code)) >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled >> >> Code)) >> at >> com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled >> >> Code)) >> at >> com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java(Compiled >> >> Code)) >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled >> >> Code)) >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled >> >> Code)) >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled >> >> Code)) >> at >> com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled >> >> Code)) >> at >> com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled >> Code)) >> Caused by: org.apache.axis2.AxisFault: >> com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected >> character 'E' (code 69) in prolog; expected '<' >> at [row,col {unknown-source}]: [1,1] >> at >> org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java(Compiled >> >> Code)) >> at >> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java(Compiled >> >> Code)) >> at >> org.apache.axis2.description.OutInAxisOperationClient.execute(OutInAxisOperation.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.isislocation.service.IsisLocationStub.envLookup(IsisLocationStub.java(Compiled >> >> Code)) >> at >> sun.reflect.GeneratedMethodAccessor40.invoke(Unknown >> Source) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled >> >> Code)) >> at java.lang.reflect.Method.invoke(Method.java(Compiled Code)) >> at >> au.gov.centrelink.jfs.proxy.ServiceClientMetaClass.intercept(ServiceClientMetaClass.java(Compiled >> >> Code)) >> at >> au.gov.centrelink.jfs.proxy.impl.AbstractDirectServiceClient$$EnhancerByCGLIB$$b007db1b.envLookup(<generated>(Compiled >> >> Code)) >> ... 52 more >> Caused by: org.apache.axiom.om.OMException: >> com.ctc.wstx.exc.WstxUnexpectedCharException: Unexpected >> character 'E' (code 69) in prolog; expected '<' >> at [row,col {unknown-source}]: [1,1] >> at >> org.apache.axiom.om.impl.builder.StAXOMBuilder.next(StAXOMBuilder.java(Compiled >> >> Code)) >> at >> org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.getSOAPEnvelope(StAXSOAPModelBuilder.java(Inlined >> >> Compiled Code)) >> at >> org.apache.axiom.soap.impl.builder.StAXSOAPModelBuilder.<init>(StAXSOAPModelBuilder.java(Inlined >> >> Compiled Code)) >> at >> org.apache.axis2.builder.BuilderUtil.getSOAPBuilder(BuilderUtil.java(Inlined >> >> Compiled Code)) >> at >> org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java(Compiled >> >> Code)) >> ... 61 more >> Caused by: com.ctc.wstx.exc.WstxUnexpectedCharException: >> Unexpected character 'E' (code 69) in prolog; expected '<' >> at [row,col {unknown-source}]: [1,1] >> at >> com.ctc.wstx.sr.StreamScanner.throwUnexpectedChar(StreamScanner.java:623) >> >> at >> com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java(Compiled >> >> Code)) >> at >> com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java(Compiled >> Code)) >> ... 66 more >> >> >> It is strange that this exception occurs only once given that the >> servlet >> returns essentially the same SOAP response every time. >> Has anyone encountered this error before in a stress testing >> scenario, or >> has any idea what might cause it to occur intermittently? >> >> Thanks for your ongoing assistance, >> >> Greg********************************************************************** >> >> IMPORTANT: This e-mail is intended for the use of the addressee and may >> contain information that is confidential, commercially valuable or >> subject >> to legal or parliamentary privilege. If you are not the intended >> recipient >> you are notified that any review, re-transmission, disclosure, use or >> dissemination of this communication is strictly prohibited by several >> Commonwealth Acts of Parliament. If you have received this >> communication in >> error please notify the sender immediately and delete all copies of this >> transmission together with any attachments. >> ********************************************************************** > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] ********************************************************************** IMPORTANT: This e-mail is intended for the use of the addressee and may contain information that is confidential, commercially valuable or subject to legal or parliamentary privilege. If you are not the intended recipient you are notified that any review, re-transmission, disclosure, use or dissemination of this communication is strictly prohibited by several Commonwealth Acts of Parliament. If you have received this communication in error please notify the sender immediately and delete all copies of this transmission together with any attachments. **********************************************************************
