I think the blocked threads were all producers. This happened in a production system after about 6 days uptime, so I'm not sure yet if it is reproducible. I guess I'll find out in a few days :-)
If my store usage limit is set to 1gb, then why would there be a problem when my data folder is less than 700mb? On Thu, Aug 13, 2009 at 5:17 AM, Gary Tully<gary.tu...@gmail.com> wrote: > Using jconsole you should be able to examine the store usage and size. > > Store disk space is released after a store checkpoint and cleanup and in > unreferenced maxDataFileSize chunks. It is not released on a per message ack > basis. > > Do you have active consumers for the destinations that are waiting for > space? > > Can you reproduce this scenario? > > 2009/8/12 Chetan Sarva <che...@milabra.com> > >> Hi, >> >> We appear to be facing a deadlock during queue message delivery. See >> the following stack dump: >> >> "ActiveMQ Transport: tcp:///10.10.10.30:40919" daemon prio=10 >> tid=0x00002aab002a3800 nid=0x281e in Object.wait() >> [0x0000000057b77000] >> java.lang.Thread.State: TIMED_WAITING (on object monitor) >> at java.lang.Object.wait(Native Method) >> at org.apache.activemq.usage.Usage.waitForSpace(Usage.java:99) >> - locked <0x00002aaab37c9e68> (a java.lang.Object) >> at >> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:496) >> - locked <0x00002aaab3a8cb48> (a java.lang.Object) >> at org.apache.activemq.broker.region.Queue.send(Queue.java:480) >> >> [...snip...] >> >> and several (15) of these: >> >> "ActiveMQ Transport: tcp:///10.10.10.31:55890" daemon prio=10 >> tid=0x00002aab00fbd800 nid=0x27f8 waiting for monitor entry >> [0x000000004f26a000] >> java.lang.Thread.State: BLOCKED (on object monitor) >> >> >> at >> org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:489) >> >> >> - waiting to >> lock <0x00002aaab3a8cb48> (a java.lang.Object) >> at org.apache.activemq.broker.region.Queue.send(Queue.java:480) >> >> >> We're using 5.3 (via fusesource) with the Kaha persistence adapter. >> Full broker config can be seen here - >> >> http://pastie.org/581968 >> >> It would seem from the stack trace above and from looking at the >> source, that it thinks the store usage is greater than 100% (just a >> guess) and is waiting for it to fall back to normal levels, however >> the entire data folder was only 630mb at the time of the deadlock. On >> restarting the server (had to kill -9 due to unresponsive threads) the >> data folder was no longer readable so I had to remove it (have a >> backup) in order to get the server started again. >> >> Any ideas as to what's going on here? >> >> regards, >> >> chetan >> > > > > -- > http://blog.garytully.com > > Open Source Integration > http://fusesource.com >
