Gary, Thanks for the reply and advice. I'll definitely try disabling concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an IO exception handler.
Unfortunately the issue can't reliably be reproduced. We have a cluster of servers all running their own local embedded queue, and it happens on one here, one there. That said, somebody else already opened a JIRA ticket for what looks like a similar/nearly identical issue: https://issues.apache.org/jira/browse/AMQ-3051 Dan On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] < ml-node+3177500-224584974-140...@n4.nabble.com<ml-node%2b3177500-224584974-140...@n4.nabble.com> > wrote: > One thought, there may be an issue with the default setting of true > for concurrentStoreAndDispatchQueues that is causing the index to be > out of sync with the store. > > It is worth trying to disable that, use: > <persistenceAdapter> > <kahaDB directory="/usr/local/embedded/activemq-data/kahadb" > concurrentStoreAndDispatchQueues="false"/> > </persistenceAdapter> > > How often is this occurring, to really get to the bottom of this a > test case will be needed. > Does that data file eventually get written, or are there any disk space > issues? > It may help to configure an IO exception handler just in case. > http://activemq.apache.org/configurable-ioexception-handling.html > > Would be a good idea to create a jira issue to track this and attach > what ever log and application information you have to it. > > On 6 January 2011 12:39, dcheckoway <[hidden > email]<http://user/SendEmail.jtp?type=node&node=3177500&i=0>> > wrote: > > > > > I'm using an embedded broker (version 5.4.2) with persistence enabled. > We're > > pumping hundreds of millions of messages per day through this thing. > Every > > once in a while, all of a sudden the KahaDB directory starts growing > > uncontrollably, and these errors start spewing out in the log over and > > over... > > > > -------------------------- > > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; > > AbstractStoreCursor - Failed to fill batch > > java.lang.RuntimeException: java.io.IOException: Could not locate data > file > > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > > at > > > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) > > > at > > > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) > > > at > > > org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) > > > at > org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) > > at > > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) > > at > org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) > > at > > > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > > > at > > > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:619) > > Caused by: java.io.IOException: Could not locate data file > > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) > > > at org.apache.kahadb.journal.Journal.read(Journal.java:592) > > at > > > org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) > > > at > > > org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) > > > at > > > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) > > > at > org.apache.kahadb.page.Transaction.execute(Transaction.java:728) > > at > > > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) > > > at > > > org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) > > > at > > > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > > > at > > > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) > > > ... 10 more > > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue > - > > Failed to page in more queue messages > > java.lang.RuntimeException: java.lang.RuntimeException: > java.io.IOException: > > Could not locate data file > > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > > at > > > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151) > > > at > > > org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) > > > at > org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) > > at > > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) > > at > org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) > > at > > > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > > > at > > > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:619) > > Caused by: java.lang.RuntimeException: java.io.IOException: Could not > locate > > data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > > > at > > > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) > > > at > > > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) > > > ... 9 more > > Caused by: java.io.IOException: Could not locate data file > > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) > > > at org.apache.kahadb.journal.Journal.read(Journal.java:592) > > at > > > org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) > > > at > > > org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) > > > at > > > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) > > > at > org.apache.kahadb.page.Transaction.execute(Transaction.java:728) > > at > > > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) > > > at > > > org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) > > > at > > > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > > > at > > > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) > > > ... 10 more > > -------------------------- > > > > It doesn't happen consistently, but it has happened numerous times. The > > only way to "fix" it is to kill the app, blow away the KahaDB data > > directory, and restart (ugh). > > > > Has anybody else seen this issue? Any idea why it happens? > > > > Here's the embedded broker config, fwiw: > > -------------------------- > > <broker xmlns="http://activemq.apache.org/schema/core"; > > id="embeddedActivemqBroker" > > useJmx="false" > > persistent="true" > > dataDirectory="/usr/local/embedded/activemq-data" > > schedulerSupport="false" > > advisorySupport="false" > > enableStatistics="true"> > > <destinationPolicy> > > <policyMap> > > <policyEntries> > > <policyEntry queue=">" producerFlowControl="false" > > memoryLimit="20mb"/> > > </policyEntries> > > </policyMap> > > </destinationPolicy> > > </broker> > > -------------------------- > > > > Help?! > > > > ----- > > Dan Checkoway > > dcheckoway gmail com > > -- > > View this message in context: > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > > > > > > -- > http://blog.garytully.com > http://fusesource.com > > > ------------------------------ > View message @ > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html > To unsubscribe from KahaDB error: "Could not locate data file", click > here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>. > > ----- Dan Checkoway dcheckoway gmail com -- View this message in context: http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
