AMQ-3051 is using the older AMQPersistenceAdapter and NFS, so it seems to be different from your use case?
On 6 January 2011 15:14, dcheckoway <dchecko...@gmail.com> wrote: > > Gary, > > Thanks for the reply and advice. I'll definitely try disabling > concurrentStoreAndDispatchQueues to see if that helps, and I'll set up an IO > exception handler. > > Unfortunately the issue can't reliably be reproduced. We have a cluster of > servers all running their own local embedded queue, and it happens on one > here, one there. > > That said, somebody else already opened a JIRA ticket for what looks like a > similar/nearly identical issue: > https://issues.apache.org/jira/browse/AMQ-3051 > > Dan > > > On Thu, Jan 6, 2011 at 9:29 AM, Gary Tully [via ActiveMQ] < > ml-node+3177500-224584974-140...@n4.nabble.com<ml-node%2b3177500-224584974-140...@n4.nabble.com> >> wrote: > >> One thought, there may be an issue with the default setting of true >> for concurrentStoreAndDispatchQueues that is causing the index to be >> out of sync with the store. >> >> It is worth trying to disable that, use: >> <persistenceAdapter> >> <kahaDB directory="/usr/local/embedded/activemq-data/kahadb" >> concurrentStoreAndDispatchQueues="false"/> >> </persistenceAdapter> >> >> How often is this occurring, to really get to the bottom of this a >> test case will be needed. >> Does that data file eventually get written, or are there any disk space >> issues? >> It may help to configure an IO exception handler just in case. >> http://activemq.apache.org/configurable-ioexception-handling.html >> >> Would be a good idea to create a jira issue to track this and attach >> what ever log and application information you have to it. >> >> On 6 January 2011 12:39, dcheckoway <[hidden >> email]<http://user/SendEmail.jtp?type=node&node=3177500&i=0>> >> wrote: >> >> > >> > I'm using an embedded broker (version 5.4.2) with persistence enabled. >> We're >> > pumping hundreds of millions of messages per day through this thing. >> Every >> > once in a while, all of a sudden the KahaDB directory starts growing >> > uncontrollably, and these errors start spewing out in the log over and >> > over... >> > >> > -------------------------- >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; >> > AbstractStoreCursor - Failed to fill batch >> > java.lang.RuntimeException: java.io.IOException: Could not locate data >> file >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> > at >> > >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) >> >> > at >> > >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) >> >> > at >> > >> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) >> >> > at >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) >> > at >> > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) >> > at >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) >> > at >> > >> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) >> >> > at >> > >> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >> > at java.lang.Thread.run(Thread.java:619) >> > Caused by: java.io.IOException: Could not locate data file >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) >> >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592) >> > at >> > >> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) >> >> > at >> > >> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) >> >> > at >> > >> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) >> >> > at >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728) >> > at >> > >> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) >> >> > at >> > >> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) >> >> > at >> > >> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) >> >> > at >> > >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) >> >> > ... 10 more >> > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue >> - >> > Failed to page in more queue messages >> > java.lang.RuntimeException: java.lang.RuntimeException: >> java.io.IOException: >> > Could not locate data file >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> > at >> > >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151) >> >> > at >> > >> org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) >> >> > at >> org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) >> > at >> > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) >> > at >> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) >> > at >> > >> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) >> >> > at >> > >> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> >> > at >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> >> > at java.lang.Thread.run(Thread.java:619) >> > Caused by: java.lang.RuntimeException: java.io.IOException: Could not >> locate >> > data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> >> > at >> > >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) >> >> > at >> > >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) >> >> > ... 9 more >> > Caused by: java.io.IOException: Could not locate data file >> > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log >> > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) >> >> > at org.apache.kahadb.journal.Journal.read(Journal.java:592) >> > at >> > >> org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) >> >> > at >> > >> org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) >> >> > at >> > >> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) >> >> > at >> org.apache.kahadb.page.Transaction.execute(Transaction.java:728) >> > at >> > >> org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) >> >> > at >> > >> org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) >> >> > at >> > >> org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) >> >> > at >> > >> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) >> >> > ... 10 more >> > -------------------------- >> > >> > It doesn't happen consistently, but it has happened numerous times. The >> > only way to "fix" it is to kill the app, blow away the KahaDB data >> > directory, and restart (ugh). >> > >> > Has anybody else seen this issue? Any idea why it happens? >> > >> > Here's the embedded broker config, fwiw: >> > -------------------------- >> > <broker xmlns="http://activemq.apache.org/schema/core"; >> > id="embeddedActivemqBroker" >> > useJmx="false" >> > persistent="true" >> > dataDirectory="/usr/local/embedded/activemq-data" >> > schedulerSupport="false" >> > advisorySupport="false" >> > enableStatistics="true"> >> > <destinationPolicy> >> > <policyMap> >> > <policyEntries> >> > <policyEntry queue=">" producerFlowControl="false" >> > memoryLimit="20mb"/> >> > </policyEntries> >> > </policyMap> >> > </destinationPolicy> >> > </broker> >> > -------------------------- >> > >> > Help?! >> > >> > ----- >> > Dan Checkoway >> > dcheckoway gmail com >> > -- >> > View this message in context: >> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html<http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177338.html?by-user=t> >> > Sent from the ActiveMQ - User mailing list archive at Nabble.com. >> > >> >> >> >> -- >> http://blog.garytully.com >> http://fusesource.com >> >> >> ------------------------------ >> View message @ >> http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177500.html >> To unsubscribe from KahaDB error: "Could not locate data file", click >> here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3177338&code=ZGNoZWNrb3dheUBnbWFpbC5jb218MzE3NzMzOHw3MDc4NzEwMTU=>. >> >> > > > ----- > Dan Checkoway > dcheckoway gmail com > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/KahaDB-error-Could-not-locate-data-file-tp3177338p3177577.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > -- http://blog.garytully.com http://fusesource.com
