Re: Hot update: Too Many Open Files
Hello Martin, Forgive me, but I am having a hard time figuring out what your are responding to. Must be the long weekend... What exactly do you intend these to be applied to? The temporary files? Personally, I still think we are dealing with a leak of class loaders (aren't being cleaned up when they are no longer in use). In the case of Steve, it is likely a similar issue. The only difference is that when he shuts down, there is still no cleanup of the class loaders. -Tom On 8/29/08 6:36 PM, Martin Gainty [EMAIL PROTECTED] wrote: set sticky bit off so other users (other than root) can access chmod chmod [OPTION]... MODE[,MODE]... FILE... http://linux.die.net/man/1/chmod chown --from=CURRENT_OWNER:CURRENT_GROUP file http://linux.die.net/man/1/chown if the file is located on linux extended file use chattr [ -RV ] [ -v version ] [ mode ] files... http://linux.die.net/man/1/chattr Martin __ Disclaimer and confidentiality note Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission. From: [EMAIL PROTECTED] To: axis-user@ws.apache.org Date: Fri, 29 Aug 2008 21:04:25 -0400 Subject: Re: Hot update: Too Many Open Files Hello Steve, We are not running under VMWare. I would say that our situation is similar, but not the same. In your previous messages you say: ³A new set gets generated each time I restart Tomcat. On my Windows XP system, these are deleted each time Tomcat stops, but not on our Linux (CentOS) systems.² In our case, we are trying to deploy without restarting Tomcat. We have the following set in axis2.xml: parameter name=hotdeploymenttrue/parameter parameter name=hotupdatetrue/parameter When we copy the updated aar file into the WEB-INF/services directory of the axis2 webapp, temporary files are created in the work directory. The next time we update aar file, more files are created in the work directory. When I shutdown tomcat, the files ARE cleaned up. But, if I don't shutdown tomcat, file handles are left open for every one of these temporary files! So, soon, we run out of file handles. As I mentioned in my original message, we want to do this for staging and we are not looking for this to be a production solution. As far as I can tell, we should be able to shutdown tomcat, deploy the aar, and restart tomcat in production without the problems you are seeing. We may just need to resort to the same solution in production. It would seem, given our joint experience, that there is an issue with the the temporary files used for deployment and update. Given what Deepal says: Yes , Axis2 creates temp files from your services and modules , and create a class loader from that. In that way we can ensure better performance. This help us a lot when we have service aar or mar with third party library inside it. It would appear that there is an issue with the class loader that is created. In the update scenario, I wonder if the class loaders are leaking (old class loaders not going away) after a service is updated. Current evidence for this is that my files do eventually get cleaned up and I assume that the class loader is doing this once it finally goes away. But, there may be some management process that is responsible for this. I have not looked into the code. Anyone with insight into this? Thanks, -Tom On 8/29/08 2:22 PM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Thomas, This is very similar to two threads I started, one in May (cached archive files not deleted) and one in July (Axis2 work files not deleted). I did not here reports from anyone else experiencing this. Particularly interesting because our environment is similar: Tomcat 5.5.17 CentOS 4.3 Axis2 1.3 Are you by any chance running your OS under VMWare?? Anyway, the solution (or workaround) is to deploy your web services and modules as an exploded directory structure rather than an AAR or MAR archive. One of the posts in the second thread has more details. - Steve __ Steve Gruverman, Programmer IntelliCare, Inc. | A Medco Health Solutions Company 500 Southborough Drive | South Portland ME 04106 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] Be the filmmaker you always wanted to be—learn how to burn a DVD with Windows®. Make your smash hit http://clk.atdmt.com/MRT/go/108588797/direct/01/
Hot update: Too Many Open Files
Hey all, Our project just started utilizing Axis2 for our web service framework and we are running into an issue with Hot Update. We recognize that this is not recommended for production, but would like to use it in our staging environment that will be managed through a continuous integration platform. The problem is after a few days, we started getting the following when the aar file was updated: 8/27|13:52:57 [Timer-3] INFO :ServiceDeployer - Deploying Web service: version-1.4.aar - file:/var/tomcat5/webapps/axis2/WEB-INF/services/ve rsion-1.4.aar 8/27|13:52:57 [Timer-3] WARN :Utils - Exception extracting jars into temporary directory : java.io.IOException: Too many open files : switch ing to alternate class loading mechanism 8/27|13:52:59 [Timer-3] ERROR:ServiceDeployer - The tc_web_service_1.aar service, which is not valid, caused The following error occurred du ring schema generation: Class Not found : com.sersol.fedsearch.dta.server.service.TCWSServer org.apache.axis2.deployment.DeploymentException: The following error occurred during schema generation: Class Not found : com.sersol.fedsearc We are currently running Axis2 1.4 under Tomcat 5.5.26 on CentOS 5.2 (Linux version 2.6.18-92.1.6.el5 ([EMAIL PROTECTED]) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jun 25 13:45:47 EDT 2008). If I look at the open files, the majority are under the work folder in the Tomcat directory structure. Example: /var/cache/tomcat5/work/Catalina/localhost/axis2/_axis2/axis21612jaxen-1.1.1 .jar Every time I do an update, the number of open files under the '_axis2' directory grows: # lsof | grep _axis2 | wc -l 232 # lsof | grep _axis2 | wc -l 424 # lsof | grep _axis2 | wc -l 616 I don't see anyone else reporting this issue. Is this a problem running under Tomcat? If this has been reported elsewhere that I have missed, please direct me to that discussion. Otherwise, if you have any insight, I'd love to hear it. Thanks, -Tom - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hot update: Too Many Open Files
Thomas, This is very similar to two threads I started, one in May (cached archive files not deleted) and one in July (Axis2 work files not deleted). I did not here reports from anyone else experiencing this. Particularly interesting because our environment is similar: Tomcat 5.5.17 CentOS 4.3 Axis2 1.3 Are you by any chance running your OS under VMWare?? Anyway, the solution (or workaround) is to deploy your web services and modules as an exploded directory structure rather than an AAR or MAR archive. One of the posts in the second thread has more details. - Steve __ Steve Gruverman, Programmer IntelliCare, Inc. | A Medco Health Solutions Company 500 Southborough Drive | South Portland ME 04106 Talbott, Thomas [EMAIL PROTECTED] wrote on 08/29/2008 02:10:32 PM: [image removed] Hot update: Too Many Open Files Talbott, Thomas to: Axis2 08/29/2008 02:11 PM Please respond to axis-user@ws.apache.org Hey all, Our project just started utilizing Axis2 for our web service framework and we are running into an issue with Hot Update. We recognize that this is not recommended for production, but would like to use it in our staging environment that will be managed through a continuous integration platform. The problem is after a few days, we started getting the following when the aar file was updated: 8/27|13:52:57 [Timer-3] INFO :ServiceDeployer - Deploying Web service: version-1.4.aar - file:/var/tomcat5/webapps/axis2/WEB-INF/services/ve rsion-1.4.aar 8/27|13:52:57 [Timer-3] WARN :Utils - Exception extracting jars into temporary directory : java.io.IOException: Too many open files : switch ing to alternate class loading mechanism 8/27|13:52:59 [Timer-3] ERROR:ServiceDeployer - The tc_web_service_1.aar service, which is not valid, caused The following error occurred du ring schema generation: Class Not found : com.sersol.fedsearch.dta.server.service.TCWSServer org.apache.axis2.deployment.DeploymentException: The following error occurred during schema generation: Class Not found : com.sersol.fedsearc We are currently running Axis2 1.4 under Tomcat 5.5.26 on CentOS 5.2 (Linux version 2.6.18-92.1.6.el5 ([EMAIL PROTECTED]) (gcc version 4.1.2 20071124 (Red Hat 4.1.2-42)) #1 SMP Wed Jun 25 13:45:47 EDT 2008). If I look at the open files, the majority are under the work folder in the Tomcat directory structure. Example: /var/cache/tomcat5/work/Catalina/localhost/axis2/_axis2/axis21612jaxen-1.1.1 .jar Every time I do an update, the number of open files under the '_axis2' directory grows: # lsof | grep _axis2 | wc -l 232 # lsof | grep _axis2 | wc -l 424 # lsof | grep _axis2 | wc -l 616 I don't see anyone else reporting this issue. Is this a problem running under Tomcat? If this has been reported elsewhere that I have missed, please direct me to that discussion. Otherwise, if you have any insight, I'd love to hear it. Thanks, -Tom - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Hot update: Too Many Open Files
Hello Steve, We are not running under VMWare. I would say that our situation is similar, but not the same. In your previous messages you say: ³A new set gets generated each time I restart Tomcat. On my Windows XP system, these are deleted each time Tomcat stops, but not on our Linux (CentOS) systems.² In our case, we are trying to deploy without restarting Tomcat. We have the following set in axis2.xml: parameter name=hotdeploymenttrue/parameter parameter name=hotupdatetrue/parameter When we copy the updated aar file into the WEB-INF/services directory of the axis2 webapp, temporary files are created in the work directory. The next time we update aar file, more files are created in the work directory. When I shutdown tomcat, the files ARE cleaned up. But, if I don't shutdown tomcat, file handles are left open for every one of these temporary files! So, soon, we run out of file handles. As I mentioned in my original message, we want to do this for staging and we are not looking for this to be a production solution. As far as I can tell, we should be able to shutdown tomcat, deploy the aar, and restart tomcat in production without the problems you are seeing. We may just need to resort to the same solution in production. It would seem, given our joint experience, that there is an issue with the the temporary files used for deployment and update. Given what Deepal says: Yes , Axis2 creates temp files from your services and modules , and create a class loader from that. In that way we can ensure better performance. This help us a lot when we have service aar or mar with third party library inside it. It would appear that there is an issue with the class loader that is created. In the update scenario, I wonder if the class loaders are leaking (old class loaders not going away) after a service is updated. Current evidence for this is that my files do eventually get cleaned up and I assume that the class loader is doing this once it finally goes away. But, there may be some management process that is responsible for this. I have not looked into the code. Anyone with insight into this? Thanks, -Tom On 8/29/08 2:22 PM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Thomas, This is very similar to two threads I started, one in May (cached archive files not deleted) and one in July (Axis2 work files not deleted). I did not here reports from anyone else experiencing this. Particularly interesting because our environment is similar: Tomcat 5.5.17 CentOS 4.3 Axis2 1.3 Are you by any chance running your OS under VMWare?? Anyway, the solution (or workaround) is to deploy your web services and modules as an exploded directory structure rather than an AAR or MAR archive. One of the posts in the second thread has more details. - Steve __ Steve Gruverman, Programmer IntelliCare, Inc. | A Medco Health Solutions Company 500 Southborough Drive | South Portland ME 04106 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Hot update: Too Many Open Files
set sticky bit off so other users (other than root) can access chmod chmod [OPTION]... MODE[,MODE]... FILE... http://linux.die.net/man/1/chmod chown --from=CURRENT_OWNER:CURRENT_GROUP file http://linux.die.net/man/1/chown if the file is located on linux extended file use chattr [ -RV ] [ -v version ] [ mode ] files... http://linux.die.net/man/1/chattr Martin __ Disclaimer and confidentiality note Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission. From: [EMAIL PROTECTED] To: axis-user@ws.apache.org Date: Fri, 29 Aug 2008 21:04:25 -0400 Subject: Re: Hot update: Too Many Open Files Hello Steve, We are not running under VMWare. I would say that our situation is similar, but not the same. In your previous messages you say: ³A new set gets generated each time I restart Tomcat. On my Windows XP system, these are deleted each time Tomcat stops, but not on our Linux (CentOS) systems.² In our case, we are trying to deploy without restarting Tomcat. We have the following set in axis2.xml: parameter name=hotdeploymenttrue/parameter parameter name=hotupdatetrue/parameter When we copy the updated aar file into the WEB-INF/services directory of the axis2 webapp, temporary files are created in the work directory. The next time we update aar file, more files are created in the work directory. When I shutdown tomcat, the files ARE cleaned up. But, if I don't shutdown tomcat, file handles are left open for every one of these temporary files! So, soon, we run out of file handles. As I mentioned in my original message, we want to do this for staging and we are not looking for this to be a production solution. As far as I can tell, we should be able to shutdown tomcat, deploy the aar, and restart tomcat in production without the problems you are seeing. We may just need to resort to the same solution in production. It would seem, given our joint experience, that there is an issue with the the temporary files used for deployment and update. Given what Deepal says: Yes , Axis2 creates temp files from your services and modules , and create a class loader from that. In that way we can ensure better performance. This help us a lot when we have service aar or mar with third party library inside it. It would appear that there is an issue with the class loader that is created. In the update scenario, I wonder if the class loaders are leaking (old class loaders not going away) after a service is updated. Current evidence for this is that my files do eventually get cleaned up and I assume that the class loader is doing this once it finally goes away. But, there may be some management process that is responsible for this. I have not looked into the code. Anyone with insight into this? Thanks, -Tom On 8/29/08 2:22 PM, [EMAIL PROTECTED] [EMAIL PROTECTED] wrote: Thomas, This is very similar to two threads I started, one in May (cached archive files not deleted) and one in July (Axis2 work files not deleted). I did not here reports from anyone else experiencing this. Particularly interesting because our environment is similar: Tomcat 5.5.17 CentOS 4.3 Axis2 1.3 Are you by any chance running your OS under VMWare?? Anyway, the solution (or workaround) is to deploy your web services and modules as an exploded directory structure rather than an AAR or MAR archive. One of the posts in the second thread has more details. - Steve __ Steve Gruverman, Programmer IntelliCare, Inc. | A Medco Health Solutions Company 500 Southborough Drive | South Portland ME 04106 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] _ Be the filmmaker you always wanted to be—learn how to burn a DVD with Windows®. http://clk.atdmt.com/MRT/go/108588797/direct/01/
