[Resin-interest] hmux broken pipe
when running resin in load-balance cluster with a web-tier(one server) and a app-tier(two server) my app-tier console display the error [09-12-15 00:14:34.369] {server-192.168.1.5:6801-34} com.caucho.java.JavaCompiler compileInt Compiling _jsp/_person/_Index__jsp.java [09-12-15 00:14:34.723] {server-192.168.1.5:6801-34} com.caucho.java.JavaCompiler compileInt Compiling _jsp/_person/_Header__jsp.java [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31} com.caucho.server.cluster.HmuxQueue dispatch java.net.SocketException: Write failed: Broken pipe [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at jrockit.net.SocketNativeIO.writeBytesPinned(Native Method) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at jrockit.net.SocketNativeIO.socketWrite(SocketNativeIO.java:73) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at java.net.SocketOutputStream.write(SocketOutputStream.java:137) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at com.caucho.vfs.TcpStream.write(TcpStream.java:150) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at com.caucho.vfs.WriteStream.flush(WriteStream.java:367) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at com.caucho.server.cluster.ClusterStream.writeYield(ClusterStream.java:626) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at com.caucho.server.cluster.HmuxQueue.dispatch(HmuxQueue.java:132) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at com.caucho.hemp.broker.HempMemoryQueue.consumeQueue(HempMemoryQueue.java:415) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at com.caucho.hemp.broker.HempMemoryQueue.run(HempMemoryQueue.java:473) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at com.caucho.util.ThreadPool$PoolThread.runTasks(ThreadPool.java:901) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at com.caucho.util.ThreadPool$PoolThread.run(ThreadPool.java:866) [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31} I'm using resin 4.0.2.s20091202 snapshot on CentOS 5.2 x86 linux compiled with JNI, with a evaluation PRO lisence. JDK: jrockit 1.6.0_14 x86 I thinks maybe caused by directive. Any ideas? -Wesley ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] hmux broken pipe
Wesley Wu wrote: > when running resin in load-balance cluster with a web-tier(one server) > and a app-tier(two server) > Thanks. I've filed this. It looks like a socket timeout issue. -- Scott > my app-tier console display the error > > [09-12-15 00:14:34.369] {server-192.168.1.5:6801-34} > com.caucho.java.JavaCompiler compileInt Compiling > _jsp/_person/_Index__jsp.java > [09-12-15 00:14:34.723] {server-192.168.1.5:6801-34} > com.caucho.java.JavaCompiler compileInt Compiling > _jsp/_person/_Header__jsp.java > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31} > com.caucho.server.cluster.HmuxQueue dispatch java.net.SocketException: > Write failed: Broken pipe > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > jrockit.net.SocketNativeIO.writeBytesPinned(Native Method) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > jrockit.net.SocketNativeIO.socketWrite(SocketNativeIO.java:73) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > java.net.SocketOutputStream.write(SocketOutputStream.java:137) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > com.caucho.vfs.TcpStream.write(TcpStream.java:150) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > com.caucho.vfs.WriteStream.flush(WriteStream.java:367) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > com.caucho.server.cluster.ClusterStream.writeYield(ClusterStream.java:626) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > com.caucho.server.cluster.HmuxQueue.dispatch(HmuxQueue.java:132) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > com.caucho.hemp.broker.HempMemoryQueue.consumeQueue(HempMemoryQueue.java:415) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > com.caucho.hemp.broker.HempMemoryQueue.run(HempMemoryQueue.java:473) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > com.caucho.util.ThreadPool$PoolThread.runTasks(ThreadPool.java:901) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31}at > com.caucho.util.ThreadPool$PoolThread.run(ThreadPool.java:866) > [09-12-15 00:14:53.096] {hmtp-aaa-to-aaa-31} > > I'm using resin 4.0.2.s20091202 snapshot on CentOS 5.2 x86 linux > compiled with JNI, with a evaluation PRO lisence. > > JDK: jrockit 1.6.0_14 x86 > > I thinks maybe caused by directive. > > Any ideas? > > -Wesley > > > ___ > resin-interest mailing list > resin-interest@caucho.com > http://maillist.caucho.com/mailman/listinfo/resin-interest > > ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Potential class loader leaks
Scott, can you tell me whether you've been able to reproduce this scenario? Mattias Jiderhamn wrote (2009-12-11 08:34): Scott Ferguson wrote (2009-12-11 00:23): Mattias Jiderhamn wrote: So I've spent another day hunting that lo(ooo)ng standing PermGen memory leak in our application and/or Resin. I made a new discovery which "shouldn't" be an issue, but could potentially fix problems. From my investigation it seems that whenever the application is reloaded, a reference to the old com.caucho.loader.EnvironmentClassLoader is kept in com.caucho.hessian.io.SerializerFactory._defaultFactory. I agree that in theory this shouldn't happen, since the ClassLoader is a weak reference key, /however/ if I add some code to the application shutdown, explicitly removing the EnvironmentClassLoader from _defaultFactory using reflection, the garbage collector is able to unload these classes. I changed this in 4.0.2 (with some more changes in 4.0.3). Even though the key is a weak reference, the value is a strong reference. Indeed, this has been fixed in the latest snapshot! Good job! Sorry for not testing against that... I should mention though, that there is still a minimum of two EnvironmentClassLoaders for the given application after reloading at least once. The former one seem to stick around somehow. We have discussed this before, Scott; how references are kept inside com.caucho.server.dispatch.Invocation, at least in a low traffic (dev / debugging) environment. On a web-app change the Invocation cache is cleared, so there shouldn't be any old references there. Could you please double check that? For example, what happens to the 503:ed requests received during redeployment? I have an examplifying YourKit (freely available EAP) snapshot I could send you if it helps. Here is what I'm getting, which seems repeatable over and over in my environment: Startup: 9k loaded classes, no unloaded classes, 49 MB PermGen First redeployment: 15k loaded classes, no unloaded classes, 77 MB PermGen = one extra instance of the app with 6k classes loaded. Assumed a total of 2 classloaders Second redeployment: 15k loaded classes, 6k unloaded classes, 77 MB PermGen = initial/first(?) instance GC:ed and a new one loaded. Still 2 classloaders. Third redeployment: 21k loaded classes, 6k unloaded classes, 105 MB PermGen = one extra instance of the app with 6k classes loaded, nothing GC:ed. Assumed a total of 3 classloaders Fourth redeployment: 21k loaded classes, 12k unloaded classes, 105 MB PermGen = second(?) instance GC:ed and a new one loaded. Still 3 classloaders. (Fifth redeployment: 15k loaded classes, 24k unloaded classes, 77 MB PermGen = third and fourth(?) instance GC:ed and a new one loaded. 2 classloaders. Haven't repeated this test enough to build a thesis) Not sure if this a real problem, but ideally the class loader of the previous version should be available for garbage collection before the classes of the new version are loaded ('cause there is no turning back anyway, is there...?). In the past this could be a problem with JNI because a JNI library can only be loaded in one classloader (that may have been changed but it always was a JDK restriction.) Are you saying this shouldn't be a issue in 4.0.2/snapshot running on JDK 1.6? -- ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest