[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14020977#comment-14020977
]
Arthur Naseef commented on AMQ-3725:
Git commits:
https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commit;h=582af3e74adca7efbf7d88d092764325341b719d
https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commit;h=5dacae368b125cd6371bc5d74cb1820ba0a5ca5a
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
>Assignee: Dejan Bosanac
> Fix For: 5.9.1, 5.10.0
>
> Attachments: AMQ-3725-10112013.txt
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.j
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13956663#comment-13956663
]
Claus Ibsen commented on AMQ-3725:
--
[~dejanb] should we assume this is fixed and close the ticket?
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
>Assignee: Dejan Bosanac
> Fix For: 5.9.1, 5.10.0
>
> Attachments: AMQ-3725-10112013.txt
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(Da
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13956662#comment-13956662
]
Claus Ibsen commented on AMQ-3725:
--
Did anyone test the SNAPSHOT with a fix
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.9.1, 5.10.0
>
> Attachments: AMQ-3725-10112013.txt
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.jou
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13811124#comment-13811124
]
Dejan Bosanac commented on AMQ-3725:
Here's a snapshot containing a fix
https://repository.apache.org/content/repositories/snapshots/org/apache/activemq/apache-activemq/5.10-SNAPSHOT/apache-activemq-5.10-20131101.033855-13-bin.tar.gz
to be tested
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.10.0
>
> Attachments: AMQ-3725-10112013.txt
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.op
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13810442#comment-13810442
]
Dejan Bosanac commented on AMQ-3725:
Hi,
I just pushed a change that should help with this scenario. I also tested ti
with the USB drive and it seems that now KahaDB recovers properly. I also used
a bit longer resume period (30 sec)
so I can have a bit more time to unplug and plug the drive back in before
broker tries to recreate the folder.
I started a new snapshot build, but I'm not sure if it's gonna be built soon.
You can build it locally anyways. Test it out and let me know if the problem is
still there.
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.10.0
>
> Attachments: AMQ-3725-10112013.txt
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR |
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13792876#comment-13792876
]
Rob Waite commented on AMQ-3725:
So I reattempted with the latest snapshot
(apache-activemq-5.9-20131011.032953-115-bin) and it did seem a lot better. My
test is really just to make a producer that sends messages in a tight loop and
to use a USB thumbdrive for the data directory. Then, while the producer is
writing as fast as it can, I yank out the thumbdrive, wait a bit, and put it
back in and wait for AMQ to handle it.
We have seen issues with our SAN becoming unavailable and seeing very similar
stack traces.
With the new snapshot... it seems like it recovers most of the time. Twice (in
maybe 15 minutes of testing) it would not start back up. The first time it
seemed like the USB drive "data" folder got corrupted... I couldnt read from
the data folder from windows (it would say it was corrupted). I reformatted it
and tried again and the most recent error seemed like it would recover but then
a very vague null pointer error happened.
Attached is the output (AMQ-3725-10112013.txt)... at the end you can see it
tried to restart and then a nullpointerexception occurred and it halted.
I know a thumbdrive would not be used in prod... but the exceptions seemed very
similar to what we have seen with our NFS store.
Overall, it is much more resilient than 5.7.0. Of the maybe 20 times I tried...
18 times it would restart automatically.
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.10.0
>
> Attachments: AMQ-3725-10112013.txt
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13791987#comment-13791987
]
Gary Tully commented on AMQ-3725:
-
@Rob Waite - do you have a scenario that is known to fail, a little test case.
It is possible to configure the IOExceptionHandler.{code}
{code}. There may be an easy fix if there is a simple
test case. The shutdown logic is better on trunk (5.9 snap)and we auto
recreate a corrupt index on restart so things could well be better. The
original test case for the IOException handler revolved around disk-Full and
disk-not-full recovery. Your scenario seems a little more random. But if we can
narrow it down or break it down there may be some simple tweaks we can do to
improve.
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.10.0
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppende
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784172#comment-13784172
]
Rob Waite commented on AMQ-3725:
The most recent stack traces occurred in 5.7.0. I have been watching commits
and have not seen anything that seemed like it would have addressed this.
I am pretty sure that losing the SAN or ability to write to disk is not handled
correctly by the KahaDB-related code. I have seen LevelDB commits. Is there an
opinion among the AMQ devs on whether KahaDB code will be deprecated in favor
of another store?
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.9.0
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAcc
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784107#comment-13784107
]
Claus Ibsen commented on AMQ-3725:
--
Any update on this ticket? There has been many fixes et all since 5.5.1 release
and we are closing in on 5.9.
So we need to know if this is resolved already or should be moved to 5.10.
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.9.0
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.D
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578649#comment-13578649
]
Rob Waite commented on AMQ-3725:
Also... today which is about 18 hours later... we still do see some errors
popping in our AMQ instance that seems related to this issue. Perhaps our data
store is still corrupted/odd and we only see it for certain queues. Things in
general do seem to still be working but I don't trust the state that it is in:
{noformat}
2013-02-14 11:02:39,710 | ERROR | Failed to page in more queue messages |
org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[habroker]
Task-789
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
Invalid location: 129:8716740, : java.lang.NegativeArraySizeException
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:150)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
at
org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1766)
at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1993)
at
org.apache.activemq.broker.region.Queue.iterate(Queue.java:1486)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.RuntimeException: java.io.IOException: Invalid location:
129:8716740, : java.lang.NegativeArraySizeException
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:147)
... 9 more
Caused by: java.io.IOException: Invalid location: 129:8716740, :
java.lang.NegativeArraySizeException
at
org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:92)
at org.apache.kahadb.journal.Journal.read(Journal.java:604)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:879)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1030)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:558)
at
org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:547)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:274)
... 10 more
{noformat}
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.9.0
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578646#comment-13578646
]
Rob Waite commented on AMQ-3725:
We made a build of 5.7.0 with the stopStartConnectors flag set to true in
DefaultIOExceptionHandler. Locally, I used a thumb drive as the datadirectory
and would get producers and consumers working and then yank it. We reproduced a
stack trace a bit more similar to above but still a bit different.
{noformat}
2013-02-14 11:16:16,032 | ERROR | KahaDB failed to store to Journal |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport:
tcp:///0:0:0:0:0:0:0:1:54836@61616
java.io.SyncFailedException: sync failed
at java.io.FileDescriptor.sync(Native Method)
at
org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:367)
at
org.apache.kahadb.journal.DataFileAppender$1.run(DataFileAppender.java:188)
2013-02-14 11:16:16,035 | INFO | Initiating stop/restart of broker transport
due to IO exception, java.io.SyncFailedException: sync failed |
org.apache.activemq.util.DefaultIOExceptionHandler | ActiveMQ Transport:
tcp:///0:0:0:0:0:0:0:1:54836@61616
java.io.SyncFailedException: sync failed
at java.io.FileDescriptor.sync(Native Method)
at
org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:367)
at
org.apache.kahadb.journal.DataFileAppender$1.run(DataFileAppender.java:188)
2013-02-14 11:16:16,041 | INFO | waiting for broker persistence adapter
checkpoint to succeed before restarting transports |
org.apache.activemq.util.DefaultIOExceptionHandler | restart transport
connectors post IO exception
2013-02-14 11:16:16,062 | WARN | Transport Connection to:
tcp://0:0:0:0:0:0:0:1:54836 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ Transport:
tcp:///0:0:0:0:0:0:0:1:54836@61616
2013-02-14 11:16:16,268 | ERROR | KahaDB failed to store to Journal |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Transport:
tcp:///0:0:0:0:0:0:0:1:54848@61616
java.io.IOException: The volume for a file has been externally altered so that
the opened file is no longer valid
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:338)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.kahadb.page.PageFile.readPage(PageFile.java:876)
{noformat}
Then we got a loop trying to get a checkpoint before restarting transports:
{noformat}
2013-02-14 11:16:21,046 | INFO | KahaDB: Recovering checkpoint thread after
death | org.apache.activemq.store.kahadb.MessageDatabase | restart transport
connectors post IO exception
2013-02-14 11:16:21,049 | INFO | waiting for broker persistence adapter
checkpoint to succeed before restarting transports |
org.apache.activemq.util.DefaultIOExceptionHandler | restart transport
connectors post IO exception
2013-02-14 11:16:26,055 | ERROR | Checkpoint failed |
org.apache.activemq.store.kahadb.MessageDatabase | ActiveMQ Journal Checkpoint
Worker
java.io.IOException: The volume for a file has been externally altered so that
the opened file is no longer valid
at java.io.RandomAccessFile.write(Native Method)
at
java.io.RandomAccessFile.writeLong(RandomAccessFile.java:1001)
at
org.apache.kahadb.page.PageFile.writeBatch(PageFile.java:1077)
at org.apache.kahadb.page.PageFile.flush(PageFile.java:515)
at
org.apache.activemq.store.kahadb.MessageDatabase.checkpointUpdate(MessageDatabase.java:1313)
at
org.apache.activemq.store.kahadb.MessageDatabase$10.execute(MessageDatabase.java:769)
at
org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
at
org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:767)
at
org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:323)
2013-02-14 11:16:26,064 | INFO | waiting for broker persistence adapter
checkpoint to succeed before restarting transports |
org.apache.activemq.util.DefaultIOExceptionHandler | restart transport
connectors post IO exception
2013-02-14 11:16:31,069 | INFO | KahaDB: Recovering checkpoint thread after
death | org.apache.activemq.store.kahadb.MessageDatabase | restart transport
connectors post IO exception
{noformat}
So it would not automatically come back... I then restarted AMQ and got the
following:
{noformat}
2013-02-14 11:33:46,398 | INFO | Using Persistence Adapter:
KahaDBPersistenceAdapter[E:\kahadb] | org.apache.activemq.broker.BrokerService
|
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578539#comment-13578539
]
Rob Waite commented on AMQ-3725:
I was looking at the source for JDBCIOExceptionHandler and it calls
setStopStartConnectors(true) from the base class DefaultIOExceptionHandler.
It seems like the DefaultIOExceptionHandler already has code that would attempt
to restart all the connectors. Don't know if this would resolve the issue but
seems like we might want to either set stopStartConnectors to true by default
or allow it to get set in activemq.xml.
Haven't pulled the repo yet... hopefully the history of this file doesn't
indicate that setting this to true is buggy..
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.9.0
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB fail
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578235#comment-13578235
]
Rob Waite commented on AMQ-3725:
We ran into a similar issue but we a missing some of the exceptions shown in
the description. We are running 5.7.0
{noformat}
KahaDB failed to store to Journal |
org.apache.activemq.store.kahadb.MessageDatabase |
ConcurrentQueueStoreAndDispatch
java.io.SyncFailedException: sync failed
at java.io.FileDescriptor.sync(Native Method)
at
org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:367)
at
org.apache.kahadb.journal.DataFileAppender$1.run(DataFileAppender.java:188)
2013-02-13 17:07:57,427 | INFO | Stopping the broker due to exception,
java.io.SyncFailedException: sync failed |
org.apache.activemq.util.DefaultIOExceptionHandler |
ConcurrentQueueStoreAndDispatch
java.io.SyncFailedException: sync failed
at java.io.FileDescriptor.sync(Native Method)
at
org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:367)
at
org.apache.kahadb.journal.DataFileAppender$1.run(DataFileAppender.java:188)
2013-02-13 17:07:57,517 | INFO | Apache ActiveMQ 5.7.0 (habroker,
ID:HOSTNAME-35138-1360038011880-0:1) is shutting down |
org.apache.activemq.broker.BrokerService | Stopping the broker due to IO
exception
{noformat}
After this, AMQ immediately went into shutdown mode. We did not see the
FileNotFoundException, messages about checkpoint recovery with the IOException
or the transport failures from EOFException. We did see the transport failures
on the clients. The Java client also had an EOFException. Our C# client which
is using NMS and gave the message "Unable to read beyond the end of the stream".
When we restarted AMQ, it would immediately shut down due to the journal files
missing. We set the parameter on KahaDB in activemq.xml
ignoreMissingJournalfiles=true. This time AMQ came up and said that it lost 33
messages.
We then saw a number of errors (they actually show up a number of times with
maybe three variations almost like multiple loggers are kicking off):
{noformat}
2013-02-13 17:46:14,892 | ERROR | Problem retrieving message for browse |
org.apache.activemq.broker.region.Queue | ActiveMQ Broker[habroker] Scheduler
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException:
Invalid location: 131:3886117, : java.lang.NegativeArraySizeException
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1066)
at
org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:832)
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:98)
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:138)
at
org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
at java.util.TimerThread.mainLoop(Timer.java:534)
at java.util.TimerThread.run(Timer.java:484)
Caused by: java.lang.RuntimeException: java.io.IOException: Invalid location:
131:3886117, : java.lang.NegativeArraySizeException
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:277)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
... 8 more
Caused by: java.io.IOException: Invalid location: 131:3886117, :
java.lang.NegativeArraySizeException
at
org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:92)
at org.apache.kahadb.journal.Journal.read(Journal.java:604)
at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:879)
at
org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:1030)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:558)
at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
at
org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:547)
at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:106)
at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:274)
... 9 more
{noformat}
This all happens during an 8 second period. Then... about 13 minutes later we
see some more:
{noformat}
2013-02-13 17:59:55,562 | ERROR | Failed to page in more queue messages |
org.apache.activemq.broker.region.Queue | ActiveMQ B
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443030#comment-13443030
]
Sree Panchajanyam D commented on AMQ-3725:
--
We have implemented a GFS based solution with i/o fencing (also called disk
fencing). I/O fencing is used to avoid this "split brain" scenario where both
active and passive servers write to the disk.
"One server cannot see changes made by the other if not remounting manually." .
Could you please make this sentence more clear?
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.7.0
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java
[jira] [Commented] (AMQ-3725) Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a similar manner as the JDBC store using the IOExceptionHandler
[
https://issues.apache.org/jira/browse/AMQ-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210843#comment-13210843
]
SuoNayi commented on AMQ-3725:
--
Jason,we are going to set up master-slave brokers with SAN but get into the
trouble of synchronization.
We mount SAN to the same path directory on two servers while operations such as
creating or deleting files in the shared directory seems not synchronous.
One server cannot see changes made by the other if not remounting manually.
I know that even NFS can work well but it's not recommended to be the shared
storage with AMQ.
Can you share some experience about setting up the shared storage with SAN?
Thanks.
> Kahadb error during SAN failover delayed write - Allow kahaDB to recover in a
> similar manner as the JDBC store using the IOExceptionHandler
> ---
>
> Key: AMQ-3725
> URL: https://issues.apache.org/jira/browse/AMQ-3725
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
>Affects Versions: 5.5.1
>Reporter: Jason Sherman
> Fix For: 5.6.0, 5.x
>
>
> An issue can arise that causes the broker to terminate when using kahaDB with
> a SAN, when the SAN fails over. In this case the failover process is
> seamless however, on fail back there is a 2-3 sec delay where writes are
> blocked and the broker terminates. With the JDBC datastore a similar
> situation can be handled by using the IOExceptionHandler. However with
> kahaDB, when this same IOExceptionHandler is added it prevents the broker
> from terminating but kahaDB retains an invalid index.
> {code}
> INFO | ActiveMQ JMS Message Broker (Broker1,
> ID:macbookpro-251a.home-56915-1328715089252-0:1) started
> INFO | jetty-7.1.6.v20100715
> INFO | ActiveMQ WebConsole initialized.
> INFO | Initializing Spring FrameworkServlet 'dispatcher'
> INFO | ActiveMQ Console at http://0.0.0.0:8161/admin
> INFO | ActiveMQ Web Demos at http://0.0.0.0:8161/demo
> INFO | RESTful file access application at http://0.0.0.0:8161/fileserver
> INFO | FUSE Web Console at http://0.0.0.0:8161/console
> INFO | Started SelectChannelConnector@0.0.0.0:8161
> ERROR | KahaDB failed to store to Journal
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | Checkpoint failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.SyncFailedException: sync failed
> java.io.SyncFailedException: sync failed
> at java.io.FileDescriptor.sync(Native Method)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:382)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> INFO | Ignoring IO exception, java.io.FileNotFoundException:
> /Volumes/NAS-01/data/kahadb/db-1.log (No such file or directory)
> java.io.FileNotFoundException: /Volumes/NAS-01/data/kahadb/db-1.log (No such
> file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:216)
> at
> org.apache.kahadb.journal.DataFile.openRandomAccessFile(DataFile.java:70)
> at
> org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:324)
> at
> org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:203)
> ERROR | KahaDB failed to store to Journal
> java
