One thought, there may be an issue with the default setting of true
for concurrentStoreAndDispatchQueues that is causing the index to be
out of sync with the store.
It is worth trying to disable that, use:
<persistenceAdapter>
<kahaDB directory="/usr/local/embedded/activemq-data/kahadb"
concurrentStoreAndDispatchQueues="false"/>
</persistenceAdapter>
How often is this occurring, to really get to the bottom of this a
test case will be needed.
Does that data file eventually get written, or are there any disk space issues?
It may help to configure an IO exception handler just in case.
http://activemq.apache.org/configurable-ioexception-handling.html
Would be a good idea to create a jira issue to track this and attach
what ever log and application information you have to it.
On 6 January 2011 12:39, dcheckoway <dchecko...@gmail.com> wrote:
>
> I'm using an embedded broker (version 5.4.2) with persistence enabled. We're
> pumping hundreds of millions of messages per day through this thing. Every
> once in a while, all of a sudden the KahaDB directory starts growing
> uncontrollably, and these errors start spewing out in the log over and
> over...
>
> --------------------------
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task;
> AbstractStoreCursor - Failed to fill batch
> java.lang.RuntimeException: java.io.IOException: Could not locate data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Could not locate data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
> at
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
> ... 10 more
> ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue -
> Failed to page in more queue messages
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
> Could not locate data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151)
> at
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate
> data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148)
> ... 9 more
> Caused by: java.io.IOException: Could not locate data file
> /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log
> at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
> at org.apache.kahadb.journal.Journal.read(Journal.java:592)
> at
> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
> at
> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
> at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
> at
> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
> at
> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
> at
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
> at
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
> ... 10 more
> --------------------------
>
> It doesn't happen consistently, but it has happened numerous times. The
> only way to "fix" it is to kill the app, blow away the KahaDB data
> directory, and restart (ugh).
>
> Has anybody else seen this issue? Any idea why it happens?
>
> Here's the embedded broker config, fwiw:
> --------------------------
> <broker xmlns="http://activemq.apache.org/schema/core";
> id="embeddedActivemqBroker"
> useJmx="false"
> persistent="true"
> dataDirectory="/usr/local/embedded/activemq-data"
> schedulerSupport="false"
> advisorySupport="false"
> enableStatistics="true">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" producerFlowControl="false"
> memoryLimit="20mb"/>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> </broker>
> --------------------------
>
> Help?!
>
> -----
> Dan Checkoway
> dcheckoway gmail com
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
http://blog.garytully.com
http://fusesource.com
