RE: Fatal error: Cleaner terminated abnormally
I wanted to let you know it worked. The system.exit does get trapped. java.lang.Error: Cleaner terminated abnormally at sun.misc.Cleaner$1.run(Cleaner.java:130) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.Cleaner.clean(Cleaner.java:127) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:124) Caused by: java.lang.Error: java.io.IOException: Broken pipe at sun.nio.ch.Util$SelectorWrapper$Closer.run(Util.java:97) at sun.misc.Cleaner.clean(Cleaner.java:125) ... 1 more Caused by: java.io.IOException: Broken pipe at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:242) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:170) at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:92) at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:91) at sun.nio.ch.Util$SelectorWrapper$Closer.run(Util.java:95) ... 2 more Exception in thread "Reference Handler" java.lang.SecurityException: Can't call System.exit() at com.emessenger.web.CustomSecurityManager.checkExit(CustomSecurityManager .java:22) at java.lang.Runtime.exit(Runtime.java:88) at java.lang.System.exit(System.java:906) at sun.misc.Cleaner$1.run(Cleaner.java:132) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.Cleaner.clean(Cleaner.java:127) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:124) - What we see is, GC breaks and class unloading starts after the first full GC: [Unloading class sun.reflect.GeneratedMethodAccessor100] [Unloading class sun.reflect.GeneratedConstructorAccessor76] [Unloading class sun.reflect.GeneratedConstructorAccessor80] [Unloading class sun.reflect.GeneratedConstructorAccessor77] [Unloading class sun.reflect.GeneratedMethodAccessor95] [Unloading class sun.reflect.GeneratedMethodAccessor98] [Unloading class sun.reflect.GeneratedConstructorAccessor78] [Unloading class sun.reflect.GeneratedMethodAccessor106] [Unloading class sun.reflect.GeneratedMethodAccessor91] [Unloading class sun.reflect.GeneratedMethodAccessor105] [Unloading class sun.reflect.GeneratedMethodAccessor85] - And then Full GC madness begins: [GC 4080947K(4177280K), 0.0603370 secs] [GC 4090738K(4177280K), 0.0683390 secs] [Full GC 4177280K->4081390K(4177280K), 16.2954960 secs] [GC 4081673K(4177280K), 0.0607990 secs] [GC 4097803K(4177280K), 0.0739870 secs] [Full GC 4177279K->4081951K(4177280K), 16.2857450 secs] [GC 4082100K(4177280K), 0.0614000 secs] [GC 4101581K(4177280K), 0.0814330 secs] [Full GC 4177279K->4082845K(4177280K), 16.2079870 secs] [GC 4084452K(4177280K), 0.0628080 secs] [GC 4106928K(4177280K), 0.0835720 secs] [Full GC 4177279K->4083187K(4177280K), 16.3403530 secs] [GC 4084203K(4177280K), 0.0627750 secs] [GC 4101856K(4177280K), 0.0737540 secs] [Full GC 4177278K->4083998K(4177280K), 16.2605530 secs] [GC 4084493K(4177280K), 0.0632620 secs] [GC 4107486K(4177280K), 0.0804700 secs] [Full GC 4177278K->4084298K(4177280K), 16.3931240 secs] [GC 4084500K(4177280K), 0.0633480 secs] [Full GC 4177279K->4085842K(4177280K), 16.4017970 secs] [GC 409K(4177280K), 0.0702090 secs] [GC 4127816K(4177280K), 0.1089220 secs] - But it's still better then an instant shutdown. -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: donderdag 19 februari 2009 16:22 To: Tomcat Users List Subject: RE: Fatal error: Cleaner terminated abnormally > From: Taylan Develioglu [mailto:tdevelio...@ebuddy.com] > Subject: Re: Fatal error: Cleaner terminated abnormally > > By trapping the exit call using security manager we hope to prevent > Tomcat from closing down on a cleaner termination. This is not likely to work, since the Cleaner is running this code as a privileged operation; if regular applications could trap those, I think there would be some serious security holes. > Not sure what the side effects would be to keep running > after a cleaner terminates (any idea). The thread doing the System.exit() call is the reference handler; the JVM will not function properly if it's not running. The exception should have been logged and ignored, not result in JVM termination, but I suspect it will be difficult to convince Sun of that at this point. > I forgot to say thanks for the response guys. Especially > yours Chris, it was very helpful. Odd, because Chris didn't participate in this thread... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. ---
Re: Fatal error: Cleaner terminated abnormally
This is bad news, but it was a longshot to begin with. I submitted a bug report which is under review now. and apologies for the name mixup. Chuck is obviously a much prettier name :) Caldarale, Charles R wrote: >> From: Taylan Develioglu [mailto:tdevelio...@ebuddy.com] >> Subject: Re: Fatal error: Cleaner terminated abnormally >> >> By trapping the exit call using security manager we hope to prevent >> Tomcat from closing down on a cleaner termination. >> > > This is not likely to work, since the Cleaner is running this code as a > privileged operation; if regular applications could trap those, I think there > would be some serious security holes. > > >> Not sure what the side effects would be to keep running >> after a cleaner terminates (any idea). >> > > The thread doing the System.exit() call is the reference handler; the JVM > will not function properly if it's not running. The exception should have > been logged and ignored, not result in JVM termination, but I suspect it will > be difficult to convince Sun of that at this point. > > >> I forgot to say thanks for the response guys. Especially >> yours Chris, it was very helpful. >> > > Odd, because Chris didn't participate in this thread... > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Fatal error: Cleaner terminated abnormally
> From: Taylan Develioglu [mailto:tdevelio...@ebuddy.com] > Subject: Re: Fatal error: Cleaner terminated abnormally > > By trapping the exit call using security manager we hope to prevent > Tomcat from closing down on a cleaner termination. This is not likely to work, since the Cleaner is running this code as a privileged operation; if regular applications could trap those, I think there would be some serious security holes. > Not sure what the side effects would be to keep running > after a cleaner terminates (any idea). The thread doing the System.exit() call is the reference handler; the JVM will not function properly if it's not running. The exception should have been logged and ignored, not result in JVM termination, but I suspect it will be difficult to convince Sun of that at this point. > I forgot to say thanks for the response guys. Especially > yours Chris, it was very helpful. Odd, because Chris didn't participate in this thread... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fatal error: Cleaner terminated abnormally
We're trying a workaround now. By trapping the exit call using security manager we hope to prevent Tomcat from closing down on a cleaner termination. Not sure what the side effects would be to keep running after a cleaner terminates (any idea). Keeping fingers crossed. I forgot to say thanks for the response guys. Especially yours Chris, it was very helpful. Rgds, T - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Fatal error: Cleaner terminated abnormally
I found bug 4938372, but it didn't seem related to me at the time. There's a post dated 2007, from Alan Bateman, indicating they'd try putting the fix in a java 6 update. I'll submit a bug report and in the meanwhile explore other options such as native/apr then. -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: dinsdag 17 februari 2009 23:46 To: Tomcat Users List Subject: RE: Fatal error: Cleaner terminated abnormally > From: Filip Hanik - Dev Lists [mailto:devli...@hanik.com] > Subject: Re: Fatal error: Cleaner terminated abnormally > > search the sun database, some results there > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521677 It's somewhat related, but I don't think it will cover the case reported here, which looked like a simple socket closure rather than anything to do with memory mapping of files. I think a new bug submission is in order (preferably by the OP). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Fatal error: Cleaner terminated abnormally
Yes, 64-bit hotspot server vm. -Original Message- From: Mark Thomas [mailto:ma...@apache.org] Sent: dinsdag 17 februari 2009 16:23 To: Tomcat Users List Subject: Re: Fatal error: Cleaner terminated abnormally Taylan Develioglu wrote: > Following the gc trail, it looks like an oom situation (maybe a mem leak > in our application, our heapsize is 4GB), is it normal behavior for > tomcat to destroy itsself like this? Are you on a 64-bit JVM? If not, the process heap is limited to 4GB so the Java object heap (set with Xmx) needs to allow for this. I would use 3.5GB as a starting point. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Fatal error: Cleaner terminated abnormally
Sadly there is no mention of a fix related to NIO in the 6u12 release notes. This comes as kind of a bummer, as we were hoping to make a comet implementation soon. The native/apr connector looks like it could be a replacement for NIO for us, but after searching I could not find anything conclusive about the scalability and performance compared to NIO. Opinions on native vs nio in discussions I have found seem to be divided. I'm also not sure if the native/apr implementation is completely separate from the NIO api. Does anyone know of any downsides/pitfalls I should look out for when using native/apr ? As always, any comment is appreciated. - Taylan -Original Message- From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com] Sent: dinsdag 17 februari 2009 16:36 To: Tomcat Users List Subject: RE: Fatal error: Cleaner terminated abnormally > From: Taylan Develioglu [mailto:tdevelio...@ebuddy.com] > Subject: Fatal error: Cleaner terminated abnormally > > Lately we've been experiencing a fatal error, related to gc, > with Tomcat that causes it to stop and unload It's not really a GC problem - rather a silly bug in NIO. You might try the standard HTTP connector to avoid the problem. Sun seems to be continually fixing NIO, so there may be something for this in 6u12, if you want to keep using the NIO connector. > I'm still unclear on what is causing the cleaner to terminate The Cleaner terminates if the run() method of the registered object throws *any* kind of exception - and then takes the entire JVM down with it, via a System.exit() call (bloody brilliant, that one). In this case, the NIO Selector Closer object didn't like the fact that its peer had gone away, and puked. Not quite as robust as one might hope. - Chuck - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Fatal error: Cleaner terminated abnormally
> From: Filip Hanik - Dev Lists [mailto:devli...@hanik.com] > Subject: Re: Fatal error: Cleaner terminated abnormally > > search the sun database, some results there > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521677 It's somewhat related, but I don't think it will cover the case reported here, which looked like a simple socket closure rather than anything to do with memory mapping of files. I think a new bug submission is in order (preferably by the OP). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fatal error: Cleaner terminated abnormally
search the sun database, some results there http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521677 supposedly fixed in Java 7 :( Filip Taylan Develioglu wrote: Hi Guys, Our application is a servlet running in a container in Tomcat standalone. It uses the following NIO connector definition: Lately we've been experiencing a fatal error, related to gc, with Tomcat that causes it to stop and unload, which I hoped you could give some advice for. I'm still unclear on what is causing the cleaner to terminate, but I guess that's more of a question for the java forums (I cannot find anything related to tomcat when I cross reference) Following the gc trail, it looks like an oom situation (maybe a mem leak in our application, our heapsize is 4GB), is it normal behavior for tomcat to destroy itsself like this? Has anyone experienced a similar problem before? What are usual causes for Tomcat to stop like this? *Any* advice or feedback is welcome. Either way, thanks in advance. Debian 4.0 Tomcat 6.0.18 Sun jdk 1.6.0.11 We use the following java options: OPTS=" -verbose:gc -Dsun.net.inetaddr.ttl=60 -Dfile.encoding=UTF-8 -Djava.io.tmpdir=$TMP_DIR -Djava.library.path=/usr/local/lib -Djava.endorsed.dirs=$CATALINA_BASE/endorsed -Dcatalina.base=$CATALINA_BASE -Dcatalina.home=$CATALINA_HOME -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file="$CATALINA_BASE/conf/logging.properties" -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalMode -Xms4096M -Xmx4096M -Xss128k -XX:PermSize=256M -XX:MaxPermSize=256M --- catalina.out snippet [GC 4052829K->3924296K(4177280K), 0.0519680 secs] [GC 4060616K->3924100K(4177280K), 0.1517880 secs] [GC 4060420K->3926867K(4177280K), 0.0883940 secs] [GC 4062488K->3931589K(4177280K), 0.1008470 secs] [GC 4067906K->3935097K(4177280K), 0.0931530 secs] [GC 4071417K->3934946K(4177280K), 0.0787300 secs] [GC 4029027K(4177280K), 0.1941170 secs] java.lang.Error: Cleaner terminated abnormally at sun.misc.Cleaner$1.run(Cleaner.java:130) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.Cleaner.clean(Cleaner.java:127) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:124) Caused by: java.lang.Error: java.io.IOException: Broken pipe at sun.nio.ch.Util$SelectorWrapper$Closer.run(Util.java:97) at sun.misc.Cleaner.clean(Cleaner.java:125) ... 1 more Caused by: java.io.IOException: Broken pipe at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:242) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:170) at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:92) at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:91) at sun.nio.ch.Util$SelectorWrapper$Closer.run(Util.java:95) ... 2 more Feb 17, 2009 12:10:38 AM org.apache.coyote.http11.Http11NioProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 Feb 17, 2009 12:10:38 AM org.apache.coyote.http11.Http11AprProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-443 Feb 17, 2009 12:10:38 AM org.apache.coyote.ajp.AjpAprProtocol pause INFO: Pausing Coyote AJP/1.3 on ajp-8009 [GC 4071265K->3937784K(4177280K), 0.0921220 secs] Feb 17, 2009 12:10:39 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina Feb 17, 2009 12:10:39 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 28,017 instance(s) to be deallocated Feb 17, 2009 12:10:41 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 27,669 instance(s) to be deallocated Feb 17, 2009 12:10:42 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 27,666 instance(s) to be deallocated Feb 17, 2009 12:10:43 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 3 instance(s) to be deallocated Feb 17, 2009 12:10:44 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 3 instance(s) to be deallocated Feb 17, 2009 12:10:45 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 3 instance(s) to be deallocated 360358820 [SocketConnectorIoProcessor-0.0] null org.apache.mina.common.support.DefaultExceptionMonitor - Unexpected exception. java.lang.NullPointerException at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:225) at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:208) at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:210) at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198) at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45) at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485) at org.apache.mina.util.NamePreservingRunnabl
RE: Fatal error: Cleaner terminated abnormally
> From: Taylan Develioglu [mailto:tdevelio...@ebuddy.com] > Subject: Fatal error: Cleaner terminated abnormally > > Lately we've been experiencing a fatal error, related to gc, > with Tomcat that causes it to stop and unload It's not really a GC problem - rather a silly bug in NIO. You might try the standard HTTP connector to avoid the problem. Sun seems to be continually fixing NIO, so there may be something for this in 6u12, if you want to keep using the NIO connector. > I'm still unclear on what is causing the cleaner to terminate The Cleaner terminates if the run() method of the registered object throws *any* kind of exception - and then takes the entire JVM down with it, via a System.exit() call (bloody brilliant, that one). In this case, the NIO Selector Closer object didn't like the fact that its peer had gone away, and puked. Not quite as robust as one might hope. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fatal error: Cleaner terminated abnormally
Taylan Develioglu wrote: > Following the gc trail, it looks like an oom situation (maybe a mem leak > in our application, our heapsize is 4GB), is it normal behavior for > tomcat to destroy itsself like this? Are you on a 64-bit JVM? If not, the process heap is limited to 4GB so the Java object heap (set with Xmx) needs to allow for this. I would use 3.5GB as a starting point. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Fatal error: Cleaner terminated abnormally
Hi Guys, Our application is a servlet running in a container in Tomcat standalone. It uses the following NIO connector definition: Lately we've been experiencing a fatal error, related to gc, with Tomcat that causes it to stop and unload, which I hoped you could give some advice for. I'm still unclear on what is causing the cleaner to terminate, but I guess that's more of a question for the java forums (I cannot find anything related to tomcat when I cross reference) Following the gc trail, it looks like an oom situation (maybe a mem leak in our application, our heapsize is 4GB), is it normal behavior for tomcat to destroy itsself like this? Has anyone experienced a similar problem before? What are usual causes for Tomcat to stop like this? *Any* advice or feedback is welcome. Either way, thanks in advance. Debian 4.0 Tomcat 6.0.18 Sun jdk 1.6.0.11 We use the following java options: OPTS=" -verbose:gc -Dsun.net.inetaddr.ttl=60 -Dfile.encoding=UTF-8 -Djava.io.tmpdir=$TMP_DIR -Djava.library.path=/usr/local/lib -Djava.endorsed.dirs=$CATALINA_BASE/endorsed -Dcatalina.base=$CATALINA_BASE -Dcatalina.home=$CATALINA_HOME -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file="$CATALINA_BASE/conf/logging.properties" -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalMode -Xms4096M -Xmx4096M -Xss128k -XX:PermSize=256M -XX:MaxPermSize=256M --- catalina.out snippet [GC 4052829K->3924296K(4177280K), 0.0519680 secs] [GC 4060616K->3924100K(4177280K), 0.1517880 secs] [GC 4060420K->3926867K(4177280K), 0.0883940 secs] [GC 4062488K->3931589K(4177280K), 0.1008470 secs] [GC 4067906K->3935097K(4177280K), 0.0931530 secs] [GC 4071417K->3934946K(4177280K), 0.0787300 secs] [GC 4029027K(4177280K), 0.1941170 secs] java.lang.Error: Cleaner terminated abnormally at sun.misc.Cleaner$1.run(Cleaner.java:130) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.Cleaner.clean(Cleaner.java:127) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:124) Caused by: java.lang.Error: java.io.IOException: Broken pipe at sun.nio.ch.Util$SelectorWrapper$Closer.run(Util.java:97) at sun.misc.Cleaner.clean(Cleaner.java:125) ... 1 more Caused by: java.io.IOException: Broken pipe at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:242) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:170) at sun.nio.ch.SelectorImpl.implCloseSelector(SelectorImpl.java:92) at java.nio.channels.spi.AbstractSelector.close(AbstractSelector.java:91) at sun.nio.ch.Util$SelectorWrapper$Closer.run(Util.java:95) ... 2 more Feb 17, 2009 12:10:38 AM org.apache.coyote.http11.Http11NioProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 Feb 17, 2009 12:10:38 AM org.apache.coyote.http11.Http11AprProtocol pause INFO: Pausing Coyote HTTP/1.1 on http-443 Feb 17, 2009 12:10:38 AM org.apache.coyote.ajp.AjpAprProtocol pause INFO: Pausing Coyote AJP/1.3 on ajp-8009 [GC 4071265K->3937784K(4177280K), 0.0921220 secs] Feb 17, 2009 12:10:39 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina Feb 17, 2009 12:10:39 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 28,017 instance(s) to be deallocated Feb 17, 2009 12:10:41 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 27,669 instance(s) to be deallocated Feb 17, 2009 12:10:42 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 27,666 instance(s) to be deallocated Feb 17, 2009 12:10:43 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 3 instance(s) to be deallocated Feb 17, 2009 12:10:44 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 3 instance(s) to be deallocated Feb 17, 2009 12:10:45 AM org.apache.catalina.core.StandardWrapper unload INFO: Waiting for 3 instance(s) to be deallocated 360358820 [SocketConnectorIoProcessor-0.0] null org.apache.mina.common.support.DefaultExceptionMonitor - Unexpected exception. java.lang.NullPointerException at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:225) at org.apache.mina.common.ByteBuffer.allocate(ByteBuffer.java:208) at org.apache.mina.transport.socket.nio.SocketIoProcessor.read(SocketIoProcessor.java:210) at org.apache.mina.transport.socket.nio.SocketIoProcessor.process(SocketIoProcessor.java:198) at org.apache.mina.transport.socket.nio.SocketIoProcessor.access$400(SocketIoProcessor.java:45) at org.apache.mina.transport.socket.nio.SocketIoProcessor$Worker.run(SocketIoProcessor.java:485) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:51) at java.lang.Thread.run(Thread.java:619) 360359827 [SocketConnectorIoProcessor-0.0] null org.apache.mina.common.support.Defa