[
https://issues.apache.org/jira/browse/AMQ-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Botros Kfoury updated AMQ-5670:
-------------------------------
Description:
Dear Support,
We are having KHADB connected to a two nodes as master slave AMQ approach. The
system was running for about two weeks and suddenly it stopped as shown below
in the logs, please give me your recommendations.
NODE1:
INFO | jvm 1 | 2015/03/04 12:54:11 | INFO | Database
\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock is locked... waiting 10 seconds for
the database to be unlocked. Reason: java.io.IOException: File
'\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked.
INFO | jvm 1 | 2015/03/04 12:54:36 | WARN | Index corrupted. Recovering
the index through journal replay. Cause:java.io.IOException: The requested
resource is in use
INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Failed to start Apache
ActiveMQ ([CenBroker, null], java.io.IOException: Could not delete:
\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data)
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Apache ActiveMQ 5.10.0
(CenBroker, null) is shutting down
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Connector openwire stopped
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO |
PListStore:[\\HQ-QTCM1\data\CenBroker\tmp_storage] stopped
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async queue tasks
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async topic tasks
INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopped KahaDB
INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Could not stop service:
KahaDB:[\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e].
Reason: java.lang.IllegalStateException: Cannot unload the page file when it
is not loaded
INFO | jvm 1 | 2015/03/04 12:54:36 | java.lang.IllegalStateException:
Cannot unload the page file when it is not loaded
INFO | jvm 1 | 2015/03/04 12:54:36 |
at
org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:36 |
at
org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0]
Node2:
INFO | jvm 1 | 2015/03/04 12:54:07 | ERROR | Checkpoint failed
INFO | jvm 1 | 2015/03/04 12:54:07 | java.io.IOException: The requested
resource is in use
INFO | jvm 1 | 2015/03/04 12:54:07 |
at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 |
at
org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0]
INFO | jvm 1 | 2015/03/04 12:54:07 | INFO | Stopping
BrokerService[CenBroker] due to exception, java.io.IOException: The requested
resource is in use
Best Regadrs,
Botros Kfoury
> AMQ issue
> ---------
>
> Key: AMQ-5670
> URL: https://issues.apache.org/jira/browse/AMQ-5670
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
> Affects Versions: 5.10.0
> Reporter: Botros Kfoury
> Fix For: 5.x
>
>
> Dear Support,
> We are having KHADB connected to a two nodes as master slave AMQ approach.
> The system was running for about two weeks and suddenly it stopped as shown
> below in the logs, please give me your recommendations.
> NODE1:
> INFO | jvm 1 | 2015/03/04 12:54:11 | INFO | Database
> \\qatch-fs\AMQ_Data_PRO\data\kahadb\lock is locked... waiting 10 seconds for
> the database to be unlocked. Reason: java.io.IOException: File
> '\\qatch-fs\AMQ_Data_PRO\data\kahadb\lock' could not be locked.
> INFO | jvm 1 | 2015/03/04 12:54:36 | WARN | Index corrupted. Recovering
> the index through journal replay. Cause:java.io.IOException: The requested
> resource is in use
> INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Failed to start Apache
> ActiveMQ ([CenBroker, null], java.io.IOException: Could not delete:
> \\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e\db.data)
> INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Apache ActiveMQ 5.10.0
> (CenBroker, null) is shutting down
> INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Connector openwire stopped
> INFO | jvm 1 | 2015/03/04 12:54:36 | INFO |
> PListStore:[\\HQ-QTCM1\data\CenBroker\tmp_storage] stopped
> INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async queue tasks
> INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopping async topic tasks
> INFO | jvm 1 | 2015/03/04 12:54:36 | INFO | Stopped KahaDB
> INFO | jvm 1 | 2015/03/04 12:54:36 | ERROR | Could not stop service:
> KahaDB:[\\qatch-fs\AMQ_Data_PRO\data\kahadb\CEN\kahadb\DLQ\queue#3a#2f#2fDLQ.#3e].
> Reason: java.lang.IllegalStateException: Cannot unload the page file when it
> is not loaded
> INFO | jvm 1 | 2015/03/04 12:54:36 | java.lang.IllegalStateException:
> Cannot unload the page file when it is not loaded
> INFO | jvm 1 | 2015/03/04 12:54:36 |
> at
> org.apache.activemq.store.kahadb.disk.page.PageFile.unload(PageFile.java:484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:36 |
> at
> org.apache.activemq.store.kahadb.MessageDatabase.close(MessageDatabase.java:432)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> Node2:
> INFO | jvm 1 | 2015/03/04 12:54:07 | ERROR | Checkpoint failed
> INFO | jvm 1 | 2015/03/04 12:54:07 | java.io.IOException: The requested
> resource is in use
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at sun.nio.ch.FileDispatcherImpl.force0(Native Method)[:1.7.0_71]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at sun.nio.ch.FileDispatcherImpl.force(Unknown Source)[:1.7.0_71]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at sun.nio.ch.FileChannelImpl.force(Unknown Source)[:1.7.0_71]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.util.RecoverableRandomAccessFile.sync(RecoverableRandomAccessFile.java:402)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.store.kahadb.disk.page.PageFile.writeBatch(PageFile.java:1094)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.store.kahadb.disk.page.PageFile.flush(PageFile.java:516)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1512)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.store.kahadb.MessageDatabase$17.execute(MessageDatabase.java:1484)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1481)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:929)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 |
> at
> org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:357)[activemq-kahadb-store-5.10.0.jar:5.10.0]
> INFO | jvm 1 | 2015/03/04 12:54:07 | INFO | Stopping
> BrokerService[CenBroker] due to exception, java.io.IOException: The requested
> resource is in use
> Best Regadrs,
> Botros Kfoury
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
