[ https://issues.apache.org/jira/browse/AMQ-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully resolved AMQ-3120. ----------------------------- Resolution: Fixed fix committed in http://svn.apache.org/viewvc?rev=1070484&view=rev interleaved checkpoint with journal add could cause inadvertent increment on the last update location, leaving current journal file as a gc candidate. resulting in the exception and hung destination. Fix keeps lastUpdate independent of the current checkpoint. > KahaDB error: "Could not locate data file" > ------------------------------------------ > > Key: AMQ-3120 > URL: https://issues.apache.org/jira/browse/AMQ-3120 > Project: ActiveMQ > Issue Type: Bug > Components: Broker > Affects Versions: 5.4.2 > Environment: linux: CentOS 5.5 > happens on both dual- and quad-cpu boxes, 4gb ram > java version "1.6.0_21" > Java(TM) SE Runtime Environment (build 1.6.0_21-b06) > Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode) > Reporter: Dan Checkoway > Assignee: Gary Tully > Fix For: 5.5.0 > > > I'm using an embedded broker (version 5.4.2) with persistence enabled. We're > pumping hundreds of millions of messages per day through this thing. Every > once in a while, all of a sudden the KahaDB directory starts growing > uncontrollably, and these errors start spewing out in the log over and over... > -------------------------- > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; > AbstractStoreCursor - Failed to fill batch > java.lang.RuntimeException: java.io.IOException: Could not locate data file > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) > at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) > at > org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) > at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) > at > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) > at > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.io.IOException: Could not locate data file > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) > at org.apache.kahadb.journal.Journal.read(Journal.java:592) > at > org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) > at > org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) > at > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) > at org.apache.kahadb.page.Transaction.execute(Transaction.java:728) > at > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) > at > org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) > at > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) > ... 10 more > ERROR; Jan 5, 2011 16:37:57 PM; tid:BrokerService[localhost] Task; Queue - > Failed to page in more queue messages > java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: > Could not locate data file > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:151) > at > org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131) > at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1679) > at > org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898) > at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425) > at > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) > at > org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate > data file /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265) > at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:148) > ... 9 more > Caused by: java.io.IOException: Could not locate data file > /usr/local/embedded/activemq-data/localhost/KahaDB/db-28098.log > at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345) > at org.apache.kahadb.journal.Journal.read(Journal.java:592) > at > org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786) > at > org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956) > at > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494) > at org.apache.kahadb.page.Transaction.execute(Transaction.java:728) > at > org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485) > at > org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88) > at > org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97) > at > org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262) > ... 10 more > -------------------------- > It doesn't happen consistently, but it has happened numerous times. The only > way to "fix" it is to kill the app, blow away the KahaDB data directory, and > restart (ugh). > Unfortunately this issue is not reproduceable. > Here's the embedded broker config, fwiw: > -------------------------- > <broker xmlns="http://activemq.apache.org/schema/core" > id="embeddedActivemqBroker" > useJmx="false" > persistent="true" > dataDirectory="/usr/local/embedded/activemq-data" > schedulerSupport="false" > advisorySupport="false" > enableStatistics="true"> > <destinationPolicy> > <policyMap> > <policyEntries> > <policyEntry queue=">" producerFlowControl="false" > memoryLimit="20mb"/> > </policyEntries> > </policyMap> > </destinationPolicy> > </broker> > -------------------------- -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira