[jira] [Commented] (AMQ-5659) Add safety measure against infinite loop when store exception prevents message removal
[ https://issues.apache.org/jira/browse/AMQ-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14360133#comment-14360133 ] metatech commented on AMQ-5659: --- Gary, breaking out of the "for" loop is not enough. We really want to break the 2 nested loops + skip the reset of the queue count performed at the end of the method. > Add safety measure against infinite loop when store exception prevents > message removal > -- > > Key: AMQ-5659 > URL: https://issues.apache.org/jira/browse/AMQ-5659 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.7.0 > Environment: ServiceMix 4.5.3 >Reporter: metatech > Attachments: purge_queue_abort_loop.patch, > purge_queue_abort_loop_v2.patch > > > When the broker is configured with a database store, the "purge" operation > enters an infinite loop when the message removal operation fails, for > instance when the broker datasource is being restarted (see example stack > trace below). > Here is a patch which adds a safety measure, in case the "dequeue" count of > the queue does not increase between 2 messages removal operations. The check > is not garanteed to detect the problem on the next iteration, because a > business consumer might also be dequeuing messages from the queue. But the > "purge" is probably much faster than the business consumer, so if it fails to > remove 2 messages in a row, it is enough to detect the problem and abort the > infinite loop. > {code} > 2015-03-05 15:38:30,353 | WARN | 14571659-2202099 | | > JDBCPersistenceAdapter | Could not get JDBC connection: Data source > is closed > java.sql.SQLException: Data source is closed > at > org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1362) > at > org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) > at > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) > at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:285) > at > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:787) > at > org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194) > at > org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:358) > at > org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:166) > at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:846) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1602) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1594) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1579) > at org.apache.activemq.broker.region.Queue.purge(Queue.java:1158) > at org.apache.activemq.broker.jmx.QueueView.purge(QueueView.java:54) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-5659) Add safety measure against infinite loop when store exception prevents message removal
[ https://issues.apache.org/jira/browse/AMQ-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358859#comment-14358859 ] ASF GitHub Bot commented on AMQ-5659: - GitHub user metatechbe opened a pull request: https://github.com/apache/activemq/pull/72 AMQ-5659 Safety measure against infinite loop in "purge" (V2) You can merge this pull request into a Git repository by running: $ git pull https://github.com/metatechbe/activemq patch-3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq/pull/72.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #72 commit 825b7bb72f635138d6578f9c19db88ac7ca77fcf Author: metatechbe Date: 2015-03-12T15:53:30Z AMQ-5659 Safety measure against infinite loop in "purge" (V2) > Add safety measure against infinite loop when store exception prevents > message removal > -- > > Key: AMQ-5659 > URL: https://issues.apache.org/jira/browse/AMQ-5659 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.7.0 > Environment: ServiceMix 4.5.3 >Reporter: metatech > Attachments: purge_queue_abort_loop.patch, > purge_queue_abort_loop_v2.patch > > > When the broker is configured with a database store, the "purge" operation > enters an infinite loop when the message removal operation fails, for > instance when the broker datasource is being restarted (see example stack > trace below). > Here is a patch which adds a safety measure, in case the "dequeue" count of > the queue does not increase between 2 messages removal operations. The check > is not garanteed to detect the problem on the next iteration, because a > business consumer might also be dequeuing messages from the queue. But the > "purge" is probably much faster than the business consumer, so if it fails to > remove 2 messages in a row, it is enough to detect the problem and abort the > infinite loop. > {code} > 2015-03-05 15:38:30,353 | WARN | 14571659-2202099 | | > JDBCPersistenceAdapter | Could not get JDBC connection: Data source > is closed > java.sql.SQLException: Data source is closed > at > org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1362) > at > org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) > at > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) > at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:285) > at > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:787) > at > org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194) > at > org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:358) > at > org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:166) > at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:846) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1602) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1594) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1579) > at org.apache.activemq.broker.region.Queue.purge(Queue.java:1158) > at org.apache.activemq.broker.jmx.QueueView.purge(QueueView.java:54) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-5659) Add safety measure against infinite loop when store exception prevents message removal
[ https://issues.apache.org/jira/browse/AMQ-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358860#comment-14358860 ] ASF GitHub Bot commented on AMQ-5659: - Github user metatechbe closed the pull request at: https://github.com/apache/activemq/pull/71 > Add safety measure against infinite loop when store exception prevents > message removal > -- > > Key: AMQ-5659 > URL: https://issues.apache.org/jira/browse/AMQ-5659 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.7.0 > Environment: ServiceMix 4.5.3 >Reporter: metatech > Attachments: purge_queue_abort_loop.patch, > purge_queue_abort_loop_v2.patch > > > When the broker is configured with a database store, the "purge" operation > enters an infinite loop when the message removal operation fails, for > instance when the broker datasource is being restarted (see example stack > trace below). > Here is a patch which adds a safety measure, in case the "dequeue" count of > the queue does not increase between 2 messages removal operations. The check > is not garanteed to detect the problem on the next iteration, because a > business consumer might also be dequeuing messages from the queue. But the > "purge" is probably much faster than the business consumer, so if it fails to > remove 2 messages in a row, it is enough to detect the problem and abort the > infinite loop. > {code} > 2015-03-05 15:38:30,353 | WARN | 14571659-2202099 | | > JDBCPersistenceAdapter | Could not get JDBC connection: Data source > is closed > java.sql.SQLException: Data source is closed > at > org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1362) > at > org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) > at > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) > at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:285) > at > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:787) > at > org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194) > at > org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:358) > at > org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:166) > at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:846) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1602) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1594) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1579) > at org.apache.activemq.broker.region.Queue.purge(Queue.java:1158) > at org.apache.activemq.broker.jmx.QueueView.purge(QueueView.java:54) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-5659) Add safety measure against infinite loop when store exception prevents message removal
[ https://issues.apache.org/jira/browse/AMQ-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358852#comment-14358852 ] metatech commented on AMQ-5659: --- Hi Gary. The code tries to abort only on fatal/unrecoverable errors. If the error is transient (only for one message), the purge goes on. Here is another version of the patch which detects 3 failures in a row before aborting, so that transient errors are ignored. > Add safety measure against infinite loop when store exception prevents > message removal > -- > > Key: AMQ-5659 > URL: https://issues.apache.org/jira/browse/AMQ-5659 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.7.0 > Environment: ServiceMix 4.5.3 >Reporter: metatech > Attachments: purge_queue_abort_loop.patch, > purge_queue_abort_loop_v2.patch > > > When the broker is configured with a database store, the "purge" operation > enters an infinite loop when the message removal operation fails, for > instance when the broker datasource is being restarted (see example stack > trace below). > Here is a patch which adds a safety measure, in case the "dequeue" count of > the queue does not increase between 2 messages removal operations. The check > is not garanteed to detect the problem on the next iteration, because a > business consumer might also be dequeuing messages from the queue. But the > "purge" is probably much faster than the business consumer, so if it fails to > remove 2 messages in a row, it is enough to detect the problem and abort the > infinite loop. > {code} > 2015-03-05 15:38:30,353 | WARN | 14571659-2202099 | | > JDBCPersistenceAdapter | Could not get JDBC connection: Data source > is closed > java.sql.SQLException: Data source is closed > at > org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1362) > at > org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) > at > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) > at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:285) > at > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:787) > at > org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194) > at > org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:358) > at > org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:166) > at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:846) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1602) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1594) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1579) > at org.apache.activemq.broker.region.Queue.purge(Queue.java:1158) > at org.apache.activemq.broker.jmx.QueueView.purge(QueueView.java:54) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-5659) Add safety measure against infinite loop when store exception prevents message removal
[ https://issues.apache.org/jira/browse/AMQ-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358771#comment-14358771 ] Gary Tully commented on AMQ-5659: - thanks for the patch. One thought. Would it not be simpler to just break out of the loop on an exception? {code} } catch(IOException e) { break; // ensure we don't loop }{code} > Add safety measure against infinite loop when store exception prevents > message removal > -- > > Key: AMQ-5659 > URL: https://issues.apache.org/jira/browse/AMQ-5659 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.7.0 > Environment: ServiceMix 4.5.3 >Reporter: metatech > Attachments: purge_queue_abort_loop.patch > > > When the broker is configured with a database store, the "purge" operation > enters an infinite loop when the message removal operation fails, for > instance when the broker datasource is being restarted (see example stack > trace below). > Here is a patch which adds a safety measure, in case the "dequeue" count of > the queue does not increase between 2 messages removal operations. The check > is not garanteed to detect the problem on the next iteration, because a > business consumer might also be dequeuing messages from the queue. But the > "purge" is probably much faster than the business consumer, so if it fails to > remove 2 messages in a row, it is enough to detect the problem and abort the > infinite loop. > {code} > 2015-03-05 15:38:30,353 | WARN | 14571659-2202099 | | > JDBCPersistenceAdapter | Could not get JDBC connection: Data source > is closed > java.sql.SQLException: Data source is closed > at > org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1362) > at > org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) > at > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) > at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:285) > at > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:787) > at > org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194) > at > org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:358) > at > org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:166) > at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:846) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1602) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1594) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1579) > at org.apache.activemq.broker.region.Queue.purge(Queue.java:1158) > at org.apache.activemq.broker.jmx.QueueView.purge(QueueView.java:54) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AMQ-5659) Add safety measure against infinite loop when store exception prevents message removal
[ https://issues.apache.org/jira/browse/AMQ-5659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358604#comment-14358604 ] ASF GitHub Bot commented on AMQ-5659: - GitHub user metatechbe opened a pull request: https://github.com/apache/activemq/pull/71 AMQ-5659 Safety measure against infinite loop in "purge" You can merge this pull request into a Git repository by running: $ git pull https://github.com/metatechbe/activemq patch-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq/pull/71.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #71 commit df703832a308d44d5a166e96aaf89876d4f19552 Author: metatechbe Date: 2015-03-12T12:46:56Z AMQ-5659 Safety measure against infinite loop in "purge" > Add safety measure against infinite loop when store exception prevents > message removal > -- > > Key: AMQ-5659 > URL: https://issues.apache.org/jira/browse/AMQ-5659 > Project: ActiveMQ > Issue Type: Improvement > Components: Broker >Affects Versions: 5.7.0 > Environment: ServiceMix 4.5.3 >Reporter: metatech > Attachments: purge_queue_abort_loop.patch > > > When the broker is configured with a database store, the "purge" operation > enters an infinite loop when the message removal operation fails, for > instance when the broker datasource is being restarted (see example stack > trace below). > Here is a patch which adds a safety measure, in case the "dequeue" count of > the queue does not increase between 2 messages removal operations. The check > is not garanteed to detect the problem on the next iteration, because a > business consumer might also be dequeuing messages from the queue. But the > "purge" is probably much faster than the business consumer, so if it fails to > remove 2 messages in a row, it is enough to detect the problem and abort the > infinite loop. > {code} > 2015-03-05 15:38:30,353 | WARN | 14571659-2202099 | | > JDBCPersistenceAdapter | Could not get JDBC connection: Data source > is closed > java.sql.SQLException: Data source is closed > at > org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1362) > at > org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) > at > org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:58) > at > org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.getStoreSequenceId(DefaultJDBCAdapter.java:285) > at > org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getStoreSequenceIdForMessageId(JDBCPersistenceAdapter.java:787) > at > org.apache.activemq.store.jdbc.JDBCMessageStore.removeMessage(JDBCMessageStore.java:194) > at > org.apache.activemq.store.memory.MemoryTransactionStore.removeMessage(MemoryTransactionStore.java:358) > at > org.apache.activemq.store.memory.MemoryTransactionStore$1.removeAsyncMessage(MemoryTransactionStore.java:166) > at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:846) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1602) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1594) > at > org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1579) > at org.apache.activemq.broker.region.Queue.purge(Queue.java:1158) > at org.apache.activemq.broker.jmx.QueueView.purge(QueueView.java:54) > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)