[xmlblaster] server hangs when changing the size maximum of the DispathcWorkerPool
Hi Marcel, I'm using your last fix in order to avoid dead lock. But I encountered a new problem. I changed my server configuration in order to have less threads. So I added the following lines: dispatch/callback/maximumPoolSize=10 dispatch/callback/minimumPoolSize=5 But therefore I was adding some clients to my server, suddenly, I was not more able to connect to my server. It seems to be hung. Below it the dump of the threads (Unfortunately the Dispathworker 8 and 9 are missing ) Thanks in advance for your answer, Xavier. "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-7" daemon prio=10 tid=0x010fb400 nid=0xbb3 waiting for monitor entry [0xd308f000..0xd308fb70] java.lang.Thread.State: BLOCKED (on object monitor) at org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo ol.java:103) - waiting to lock <0xdae67640> (a org.xmlBlaster.util.dispatch.DispatchWorkerPool) at org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM anager.java:722) - locked <0xdf5776b0> (a org.xmlBlaster.util.dispatch.DispatchManager) at org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp atchManager.java:674) at org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D ispatchManager.java:888) at org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Thread.java:619) "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-6" daemon prio=10 tid=0x0015a000 nid=0x199 waiting for monitor entry [0xd49ff000..0xd49ff870] java.lang.Thread.State: BLOCKED (on object monitor) at org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo ol.java:103) - waiting to lock <0xdae67640> (a org.xmlBlaster.util.dispatch.DispatchWorkerPool) at org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM anager.java:722) - locked <0xdff773b8> (a org.xmlBlaster.util.dispatch.DispatchManager) at org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp atchManager.java:674) at org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D ispatchManager.java:888) at org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Thread.java:619) "XmlBlaster.DispatchWorkerPool.xmlBlaster_10_1_13_13_3412-5" daemon prio=10 tid=0x007a7000 nid=0x171 waiting for monitor entry [0xd3c5f000..0xd3c5f870] java.lang.Thread.State: BLOCKED (on object monitor) at org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo ol.java:103) - waiting to lock <0xdae67640> (a org.xmlBlaster.util.dispatch.DispatchWorkerPool) at org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM anager.java:722) - locked <0xdd854e80> (a org.xmlBlaster.util.dispatch.DispatchManager) at org.xmlBlaster.util.dispatch.DispatchManager.activateDispatchWorker(Disp atchManager.java:674) at org.xmlBlaster.util.dispatch.DispatchManager.setDispatchWorkerIsActive(D ispatchManager.java:888) at org.xmlBlaster.util.dispatch.DispatchWorker.run(DispatchWorker.java:150) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Thread.java:619) "XmlBlaster.SubjectEntryShuffler" daemon prio=10 tid=0x007a9000 nid=0x170 in Object.wait() [0xd323f000..0xd323f9f0] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at EDU.oswego.cs.dl.util.concurrent.SynchronousChannel.put(Unknown Source) - locked <0xe1a81d48> (a EDU.oswego.cs.dl.util.concurrent.LinkedNode) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$WaitWhenBlocked.blockedA ction(Unknown Source) at EDU.oswego.cs.dl.util.concurrent.PooledExecutor.execute(Unknown Source) at org.xmlBlaster.util.dispatch.DispatchWorkerPool.execute(DispatchWorkerPo ol.java:107) - locked <0xdae67640> (a org.xmlBlaster.util.dispatch.DispatchWorkerPool) at org.xmlBlaster.util.dispatch.DispatchManager.startWorkerThread(DispatchM anager.java:722) - locked <0xe16a3f48> (a org.xmlBlaster.util.dispatch.DispatchManager) at org.xmlBlaster.util.dispatch.DispatchManager.notifyAboutNewEntry(Dispatc hManager.java:652) at org.xmlBlaster.util.dispatch.DispatchManager.putPost(DispatchManager.jav a:570) at org.xmlBlaster.util.queue.ram.RamQueuePlugin.put(RamQueuePlugin.java:764 ) at org.xmlBlaster.authentication.SessionInfo.queueMessage(SessionInfo.java: 497) at org.xmlBlaster.authentication.SubjectInfo.forwardToSession
Re: [xmlblaster] dead lock
Xavier Roques wrote: In my configuration, I want that the clients always try to connect back to the Server so the clients retries is set to -1. But I want to be informed as soon as a client is disconnected. That's why my configuration is dispatch/callback/retries=0 Otherwise I'm not informed when a client is abnormally disconnected. So, I have a question, is there a way to know that a client has been normally disconnected or not ? Xavier. If you set dispatch/callback/retries=-1 the server will never cleanup your session (it will only if you do an explicit disconnect() or kill it using the jconsole). If the client disappears the callback framework goes to state "POLLING", and all settings like subscribes or callback queue entries are preserved (even on server restart if marked ). You can detect polling clients using an EventPlugin, see http://www.xmlblaster.org/xmlBlaster/doc/requirements/admin.events.html and use the event client/[subjectId]/session/[publicSessionId]/event/callbackState Please use the newest xmlBlaster as we have made some coding changes for this in the last history, best regards Marcel -Original Message- From: [EMAIL PROTECTED] [mailto:owner- [EMAIL PROTECTED] On Behalf Of Marcel Ruff Sent: mercredi 26 septembre 2007 19:02 To: xmlblaster@server.xmlBlaster.org Subject: Re: [xmlblaster] dead lock Xavier Roques wrote: Hi David, When the dead lock occurred, I have one server with 144 connected clients. All my clients are configured in fail safe mode and all the messages are stored in RAM, the configuration of all my agents is: #server configuration dispatch/callback/retries=0 --> 0 no retry: XmlBlaster will clean up the session if the client does not respond to the callback ping if you need fail safe behaviour you should set dispatch/callback/retries=-1 regards Marcel dispatch/callback/delay=2000 dispatch/callback/pingInterval=2000 dispatch/callback/plugin/socket_ssl/multiThreaded=false #client configuration pingInterval=2000 retries=-1 delay=2000 The problem occurred after having restarted 30 of the agents running on the same box. I hope it may help, Xavier. *From:* [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] *On Behalf Of *David Robison *Sent:* mercredi 26 septembre 2007 14:46 *To:* xmlblaster@server.xmlBlaster.org *Subject:* Re: [xmlblaster] dead lock Can you tell me a little more about the conditions that caused the dead lock? We are experiencing a problem that may be related (although it may be just my wacky code). Any additional information would be helpful. David Robison *From:* Marcel Ruff [mailto:[EMAIL PROTECTED] *To:* xmlblaster@server.xmlBlaster.org *Sent:* Wed, 26 Sep 2007 07:03:15 -0400 *Subject:* Re: [xmlblaster] dead lock Hi Xavier, the dead lock is now fixed, the code is available with svn. thanks for reporting, Marcel Xavier Roques wrote: Hi, I'm using Xmlblaster 1.6. I restarted, some connected clients and unfortunately one of my client never connects back :( Even if I restart several times this client, it never successes to connect. That's why I dumped the threads' stack on the server side and I found the following dead lock (see below) Is there a way to avoid it ? Thanks, Xavier. "XmlBlaster.ssl_socket.SSL.tcpListener-alouettebench17-agent": INFO | jvm 1 | 2007/09/25 11:31:29 | at org.xmlBlaster.util.dispatch.DispatchManager.getConnectionStatusListener s(DispatchManager.java:206) INFO | jvm 1 | 2007/09/25 11:31:29 | - waiting to lock <0xdb74d9c0> (a org.xmlBlaster.util.dispatch.DispatchManager) INFO | jvm 1 | 2007/09/25 11:31:29 | at org.xmlBlaster.util.dispatch.DispatchManager.toAlive(DispatchManager.jav a:293) INFO | jvm 1 | 2007/09/25 11:31:29 | - locked <0xdb7a0930> (a java.lang.Object) INFO | jvm 1 | 2007/09/25 11:31:29 | at org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.updateState(Disp atchConnectionsHandler.java:332) INFO | jvm 1 | 2007/09/25 11:31:29 | - locked <0xdb7a0f28> (a java.util.ArrayList) INFO | jvm 1 | 2007/09/25 11:31:29 | at org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.toAlive(Dispatch ConnectionsHandler.java:302) INFO | jvm 1 | 2007/09/25 11:31:29 | at org.xmlBlaster.util.dispatch.DispatchConnection.handleTransition(Dispatc hConnection.java:575) INFO | jvm 1 | 2007/09/25 11:31:29 | - locked <0xdb7b2838> (a org.xmlBlaster.engine.dispatch.CbDispatchConnection) INFO | jvm 1 | 2007/09/25 11:31:29 | at org.xmlBlaster.util.dispatch.DispatchConnection.initialize(DispatchConne ction.java:132) INFO | jvm 1 | 2007/09/25 11:31:29 | at org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.initialize(Dispa tchConnectionsHandler.java:179) INFO | jvm 1 | 2007/09/25 11:31:29 |
RE: [xmlblaster] dead lock
In my configuration, I want that the clients always try to connect back to the Server so the clients retries is set to -1. But I want to be informed as soon as a client is disconnected. That's why my configuration is dispatch/callback/retries=0 Otherwise I'm not informed when a client is abnormally disconnected. So, I have a question, is there a way to know that a client has been normally disconnected or not ? Xavier. > -Original Message- > From: [EMAIL PROTECTED] [mailto:owner- > [EMAIL PROTECTED] On Behalf Of Marcel Ruff > Sent: mercredi 26 septembre 2007 19:02 > To: xmlblaster@server.xmlBlaster.org > Subject: Re: [xmlblaster] dead lock > > Xavier Roques wrote: > > > > Hi David, > > > > > > > > When the dead lock occurred, I have one server with 144 connected clients. > > > > > > > > All my clients are configured in fail safe mode and all the messages > > are stored in RAM, the configuration of all my agents is: > > > > #server configuration > > > > dispatch/callback/retries=0 > > > --> 0 no retry: XmlBlaster will clean up the session if the client does > not respond to the callback ping > if you need fail safe behaviour you should set dispatch/callback/retries=-1 > > regards > Marcel > > > > dispatch/callback/delay=2000 > > > > dispatch/callback/pingInterval=2000 > > > > dispatch/callback/plugin/socket_ssl/multiThreaded=false > > > > #client configuration > > > > pingInterval=2000 > > > > retries=-1 > > > > delay=2000 > > > > > > > > The problem occurred after having restarted 30 of the agents running > > on the same box. > > > > > > > > I hope it may help, > > > > > > > > Xavier. > > > > > > > > > > > > *From:* [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] *On Behalf Of *David > > Robison > > *Sent:* mercredi 26 septembre 2007 14:46 > > *To:* xmlblaster@server.xmlBlaster.org > > *Subject:* Re: [xmlblaster] dead lock > > > > > > > > Can you tell me a little more about the conditions that caused the > > dead lock? We are experiencing a problem that may be related (although > > it may be just my wacky code). Any additional information would be > > helpful. > > > > David Robison > > > > > > > > *From:* Marcel Ruff [mailto:[EMAIL PROTECTED] > > *To:* xmlblaster@server.xmlBlaster.org > > *Sent:* Wed, 26 Sep 2007 07:03:15 -0400 > > *Subject:* Re: [xmlblaster] dead lock > > > > Hi Xavier, > > > > the dead lock is now fixed, the code is available with svn. > > > > thanks for reporting, > > Marcel > > > > > > Xavier Roques wrote: > > > Hi, > > > > > > I'm using Xmlblaster 1.6. > > > > > > I restarted, some connected clients and unfortunately one of my client > > > never connects back :( > > > > > > Even if I restart several times this client, it never successes to > > > connect. > > > > > > That's why I dumped the threads' stack on the server side and I found > > > the following dead lock (see below) > > > > > > Is there a way to avoid it ? > > > > > > Thanks, > > > > > > Xavier. > > > > > > > > > "XmlBlaster.ssl_socket.SSL.tcpListener-alouettebench17-agent": > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.util.dispatch.DispatchManager.getConnectionStatusListener > > > s(DispatchManager.java:206) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | - waiting to lock > > > <0xdb74d9c0> (a org.xmlBlaster.util.dispatch.DispatchManager) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.util.dispatch.DispatchManager.toAlive(DispatchManager.jav > > > a:293) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | - locked <0xdb7a0930> (a > > > java.lang.Object) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.updateState(Disp > > > atchConnectionsHandler.java:332) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | - locked <0xdb7a0f28> (a > > > java.util.ArrayList) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.toAlive(Dispatch > > > ConnectionsHandler.java:302) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.util.dispatch.DispatchConnection.handleTransition(Dispatc > > > hConnection.java:575) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | - locked <0xdb7b2838> (a > > > org.xmlBlaster.engine.dispatch.CbDispatchConnection) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.util.dispatch.DispatchConnection.initialize(DispatchConne > > > ction.java:132) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.util.dispatch.DispatchConnectionsHandler.initialize(Dispa > > > tchConnectionsHandler.java:179) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | - locked <0xdb7a0f28> (a > > > java.util.ArrayList) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.util.dispatch.DispatchManager.updateProperty(DispatchMana > > > ger.java:160) > > > INFO | jvm 1 | 2007/09/25 11:31:29 | at > > > org.xmlBlaster.authentication.SessionInfo.updateConnectQos(SessionInfo.j > > > ava
