Re: [JBoss-user] Memory Leak problem
Please see my rather detailed reply to David Ward [EMAIL PROTECTED] in a post I sent a few minutes ago. I include the entire startup script as part of that message, explaining the use of the standard Debian utility start-stop-daemon which is at issue here. -- Mike On 2001-06-21 at 14:57 -0700, Scott M Stark wrote: How are you starting the server process? The only way I can see doing this is redirecting the stderr descriptor to a file as parting of the startup process and then sending the vm a SIGQUIT signal. - Original Message - From: Michael Bilow [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, June 21, 2001 1:13 PM Subject: Re: [JBoss-user] Memory Leak problem In our installation (Linux), jBoss runs as a server process, which means it has no console. In fact, it runs under the privilege of the jboss system user, who cannot even log in. All ownership and permission information is tied to this system user, so running under any other user would be prohibitively difficult. It is ordinary to set up something like jBoss this way, since otherwise it would be tied to the user console and die when the user logged out, which is obviously undesirable in real production. It is possible to rig up a test environment where a console is set up by a shell running as the jboss user and then jBoss is started through the nohup utility or some similar method to redirect stdout and stderr to disk files, but this is a pretty awkward configuration and I think it would require modifying the run.sh file. Is there any other way to get dump information to go to a file instead of to the usually non-existent stderr? This would be very helpful on Unix. -- Mike On 2001-06-21 at 08:41 -0700, Scott M Stark wrote: Obtain a thread dump of the vm in this situation to see where the threads are via a Ctrl-\ or SIGQUIT. This gets dumped to stderr so you need to redirect this or have a huge console buffer or logging. ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
JA == Jim Archer [EMAIL PROTECTED] writes: JA We have seen this behavior as well. If we have an error in the JA matching of our bean interface and our remote interface (a JA different parameter list or exception list, for example) and my JA client calls into the app (I think and call to any bean will do JA it, but I'm not sure and the client is a servlet in a WAR file in JA the EAR) jBoss 2.2.1_Tomcat 3.2.1 will start launching threads. It JA will continue to do so until the kernel thread limit (we have it JA set right now to 256 threads) is hit, then it will become JA uncontrolable. Yep. *sigh* Nasty. But does it really have to do this? JA We have never seen this happen of the EJB JAR has no errors of the JA type I described. It was a concern to us as well, but soince it JA only happens when we have a bug, we decided to just be carefull. JA I would be interested to know if you find the cause of this on JA your system, especially if it is different than thr cause on JA mine. Its easy to miss this error if you have many beans, as the JA verifier messages scroll off the window quickly. Yes. This happens when the file to be deployed does not pass the verifier test, has a method missing on it's home interface etc... etc... etc... I can confirm your experiences. However, I am seriously concerned about this feature for JBoss in a production environment. -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
--On Thursday, June 21, 2001 8:24 AM +0100 Nicolai P Guba [EMAIL PROTECTED] wrote: Yep. *sigh* Nasty. But does it really have to do this? Well, I would expect its just a bug. I agree its a serious one. Yes. This happens when the file to be deployed does not pass the verifier test, has a method missing on it's home interface etc... etc... etc... I can confirm your experiences. However, I am seriously concerned about this feature for JBoss in a production environment. Well, maybe now that you and I have verified it and know how to reproduce it someone will fix it. Danch, do you want me to write this up on SourceForge bug list, or is this a known issue? Jim I shall be telling this with a sigh Somewhere ages and ages hence: Two roads diverged in a wood, and I - I took the one less traveled by, And that has made all the difference. - Robert Frost, 1916 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
JA == Jim Archer [EMAIL PROTECTED] writes: JA Well, maybe now that you and I have verified it and know how to JA reproduce it someone will fix it. JA Danch, do you want me to write this up on SourceForge bug list, or JA is this a known issue? Hmmm, on deploying a valid jar I also got this exception (unexpectedly) then the same problem with running out of processes occured... [Container factory] java.lang.IllegalStateException: The session is closed [Container factory] at org.jbossmq.SpySession.createMessage(SpySession.java:132) [Container factory] at org.jboss.ejb.plugins.AbstractInstanceCache.createMessage(AbstractInstanceCache.java:129) [Container factory] at org.jboss.ejb.plugins.AbstractInstanceCache.logPassivation(AbstractInstanceCache.java:478) [Container factory] at org.jboss.ejb.plugins.AbstractInstanceCache$1.execute(AbstractInstanceCache.java:673) [Container factory] at org.jboss.util.WorkerQueue$QueueLoop.flush(WorkerQueue.java:218) [Container factory] at org.jboss.util.WorkerQueue$QueueLoop.run(WorkerQueue.java:194) [Container factory] at java.lang.Thread.run(Thread.java:484) Hope this helps a bit more in order to pin the problem -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
Jim Archer wrote: Well, maybe now that you and I have verified it and know how to reproduce it someone will fix it. Danch, do you want me to write this up on SourceForge bug list, or is this a known issue? Please put it out there as a bug. -danch ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
Obtain a thread dump of the vm in this situation to see where the threads are via a Ctrl-\ or SIGQUIT. This gets dumped to stderr so you need to redirect this or have a huge console buffer or logging. - Original Message - From: Nicolai P Guba [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, June 21, 2001 6:51 AM Subject: Re: [JBoss-user] Memory Leak problem JA == Jim Archer [EMAIL PROTECTED] writes: JA Well, maybe now that you and I have verified it and know how to JA reproduce it someone will fix it. JA Danch, do you want me to write this up on SourceForge bug list, or JA is this a known issue? Hmmm, on deploying a valid jar I also got this exception (unexpectedly) then the same problem with running out of processes occured... [Container factory] java.lang.IllegalStateException: The session is closed [Container factory] at org.jbossmq.SpySession.createMessage(SpySession.java:132) [Container factory] at org.jboss.ejb.plugins.AbstractInstanceCache.createMessage(AbstractInstanceCache.java:129) [Container factory] at org.jboss.ejb.plugins.AbstractInstanceCache.logPassivation(AbstractInstanceCache.java:478) [Container factory] at org.jboss.ejb.plugins.AbstractInstanceCache$1.execute(AbstractInstanceCache.java:673) [Container factory] at org.jboss.util.WorkerQueue$QueueLoop.flush(WorkerQueue.java:218) [Container factory] at org.jboss.util.WorkerQueue$QueueLoop.run(WorkerQueue.java:194) [Container factory] at java.lang.Thread.run(Thread.java:484) Hope this helps a bit more in order to pin the problem -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
SMS == Scott M Stark Scott writes: SMS Obtain a thread dump of the vm in this situation to see where the SMS threads are via a Ctrl-\ or SIGQUIT. This gets dumped to stderr SMS so you need to redirect this or have a huge console buffer or SMS logging. Ok. Will do as soon as thir reappears. cheers -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
In our installation (Linux), jBoss runs as a server process, which means it has no console. In fact, it runs under the privilege of the jboss system user, who cannot even log in. All ownership and permission information is tied to this system user, so running under any other user would be prohibitively difficult. It is ordinary to set up something like jBoss this way, since otherwise it would be tied to the user console and die when the user logged out, which is obviously undesirable in real production. It is possible to rig up a test environment where a console is set up by a shell running as the jboss user and then jBoss is started through the nohup utility or some similar method to redirect stdout and stderr to disk files, but this is a pretty awkward configuration and I think it would require modifying the run.sh file. Is there any other way to get dump information to go to a file instead of to the usually non-existent stderr? This would be very helpful on Unix. -- Mike On 2001-06-21 at 08:41 -0700, Scott M Stark wrote: Obtain a thread dump of the vm in this situation to see where the threads are via a Ctrl-\ or SIGQUIT. This gets dumped to stderr so you need to redirect this or have a huge console buffer or logging. ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
How are you starting the server process? The only way I can see doing this is redirecting the stderr descriptor to a file as parting of the startup process and then sending the vm a SIGQUIT signal. - Original Message - From: Michael Bilow [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Thursday, June 21, 2001 1:13 PM Subject: Re: [JBoss-user] Memory Leak problem In our installation (Linux), jBoss runs as a server process, which means it has no console. In fact, it runs under the privilege of the jboss system user, who cannot even log in. All ownership and permission information is tied to this system user, so running under any other user would be prohibitively difficult. It is ordinary to set up something like jBoss this way, since otherwise it would be tied to the user console and die when the user logged out, which is obviously undesirable in real production. It is possible to rig up a test environment where a console is set up by a shell running as the jboss user and then jBoss is started through the nohup utility or some similar method to redirect stdout and stderr to disk files, but this is a pretty awkward configuration and I think it would require modifying the run.sh file. Is there any other way to get dump information to go to a file instead of to the usually non-existent stderr? This would be very helpful on Unix. -- Mike On 2001-06-21 at 08:41 -0700, Scott M Stark wrote: Obtain a thread dump of the vm in this situation to see where the threads are via a Ctrl-\ or SIGQUIT. This gets dumped to stderr so you need to redirect this or have a huge console buffer or logging. ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
BTW, this isn't literally a memory leak - it's a thread leak and Java VMs don't have a separate exception/error for that. Not that leaking one resource is any better than leaking another... danch (Dan Christopherson) wrote: Nicolai P Guba wrote: Hello I noticed that on deploying entities several times, the server starts throwing an OutOfMemory Exception and has spanned too many processes for the OS (Linux) to handle so that there are not enough resources left for a fork to shut down the server gracefully (or reboot). Just for the record, those are threads, not processes. Not that the Linux kernel really differentiates or anything... Has anybody experienced similar problems? Could it be that I am not freeing some resurces or other problem with my BMP Entities? I haven't seen this. How many times do you redeploy? Are you doing anything unusual in your entities? Do you close database connections, statements, resultsets? What DB are you using? -danch ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
d == danch (Dan Christopherson) [EMAIL PROTECTED] writes: d BTW, this isn't literally a memory leak - it's a thread leak and d Java VMs don't have a separate exception/error for that. d Not that leaking one resource is any better than leaking another... I know that I ran out of processes since the box couldn't even fork the process required to shutdown or even reboot the server. Arrrgh. What I noticed is that on deploy there are somethimes IllegalStateExceptions. I have a look at how I manage resources. I close all db connections after use, as well as results. However, I didn't close PreparedStatements because there is some kind of proxy/cache in Jboss that barfs at me as I try this. H,... What worries me a bit is that I am thinking of deploying this stuff in the near future... I'll try and give more info when it occurs again. Maybe somebody can make some more sense of the logs than myself. -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
Re: [JBoss-user] Memory Leak problem
We have seen this behavior as well. If we have an error in the matching of our bean interface and our remote interface (a different parameter list or exception list, for example) and my client calls into the app (I think and call to any bean will do it, but I'm not sure and the client is a servlet in a WAR file in the EAR) jBoss 2.2.1_Tomcat 3.2.1 will start launching threads. It will continue to do so until the kernel thread limit (we have it set right now to 256 threads) is hit, then it will become uncontrolable. We run on Debian Linux with the 2.2 kernel (we'll be moving to the 2.4 kernel soon). We don't let jBoss run as root, of course, so it dossen;t take down the server. When this happens, we execute kill -9 against java and wait forthe sockets and other resources to be recovered by the kernel. We have never seen this happen of the EJB JAR has no errors of the type I described. It was a concern to us as well, but soince it only happens when we have a bug, we decided to just be carefull. I would be interested to know if you find the cause of this on your system, especially if it is different than thr cause on mine. Its easy to miss this error if you have many beans, as the verifier messages scroll off the window quickly. Jim --On Wednesday, June 20, 2001 6:46 PM +0100 Nicolai P Guba [EMAIL PROTECTED] wrote: d == danch (Dan Christopherson) [EMAIL PROTECTED] writes: d BTW, this isn't literally a memory leak - it's a thread leak and d Java VMs don't have a separate exception/error for that. d Not that leaking one resource is any better than leaking another... I know that I ran out of processes since the box couldn't even fork the process required to shutdown or even reboot the server. Arrrgh. What I noticed is that on deploy there are somethimes IllegalStateExceptions. I have a look at how I manage resources. I close all db connections after use, as well as results. However, I didn't close PreparedStatements because there is some kind of proxy/cache in Jboss that barfs at me as I try this. H,... What worries me a bit is that I am thinking of deploying this stuff in the near future... I'll try and give more info when it occurs again. Maybe somebody can make some more sense of the logs than myself. -- Nicolai P Gubahttp://www.gnu.org http://www.frontwire.com mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] GSM: +44 (0)7909 960 751 DDI: +44 (0)20 7368 9708 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user I shall be telling this with a sigh Somewhere ages and ages hence: Two roads diverged in a wood, and I - I took the one less traveled by, And that has made all the difference. - Robert Frost, 1916 ___ JBoss-user mailing list [EMAIL PROTECTED] http://lists.sourceforge.net/lists/listinfo/jboss-user
