[jira] [Created] (AMQ-5141) Message expiry that is done as part of a removeSubscription command should not use the clients credentials.
Torsten Mielke created AMQ-5141:
---
Summary: Message expiry that is done as part of a
removeSubscription command should not use the clients credentials.
Key: AMQ-5141
URL: https://issues.apache.org/jira/browse/AMQ-5141
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.9.0
Reporter: Torsten Mielke
If the broker handles a RemoveInfo command it may also kick off a message
expiry check for (I presume) any prefetched messages. If messages are to be
expired they get sent to ActiveMQ.DLQ by default. See stack trace in next
comment.
If the broker is security enabled with authorization turned on and messages get
sent to DLQ as a result of the expiry check then the broker uses the client's
security context when sending the messages to DLQ.
This implies the client user needs to have write access to ActiveMQ.DLQ.
As this may happen with any other client, all client users will require write
access to ActiveMQ.DLQ, which may not be appropriate from a security point of
view.
The broker regularly runs an expiry check and uses a broker internal security
context for this task. In my opinion this same broker internal security context
should be used when expiring messages as part of the RemoveInfo command. The
broker should not use the client's security context.
[1]
The current behavior can raise the following SecurityException if the client
user does not have write access to ActiveMQ.DLQ
{code}
2014-04-11 08:11:22,229 | WARN | 2.38:61201@61616 | RegionBroker |
ivemq.broker.region.RegionBroker 703 |
105 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Caught an
exception sending to DLQ: Message
ID:S930A3085-50865-635327964441522304-1:1:363:2:1 dropped=false acked=false
locked=true
java.lang.SecurityException: User Test is not authorized to write to:
queue://ActiveMQ.DLQ
at
org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:197)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.util.BrokerSupport.doResend(BrokerSupport.java:68)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.util.BrokerSupport.resendNoCopy(BrokerSupport.java:38)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(RegionBroker.java:691)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.advisory.AdvisoryBroker.sendToDeadLetterQueue(AdvisoryBroker.java:413)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:274)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.util.RedeliveryPlugin.sendToDeadLetterQueue(RedeliveryPlugin.java:132)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:274)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.region.RegionBroker.messageExpired(RegionBroker.java:659)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.messageExpired(BrokerFilter.java:257)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.messageExpired(BrokerFilter.java:257)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.advisory.AdvisoryBroker.messageExpired(AdvisoryBroker.java:283)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
[jira] [Commented] (AMQ-5141) Message expiry that is done as part of a removeSubscription command should not use the clients credentials.
[
https://issues.apache.org/jira/browse/AMQ-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966427#comment-13966427
]
Torsten Mielke commented on AMQ-5141:
-
Don't have a Unit test at hand but could build one if time permits.
Message expiry that is done as part of a removeSubscription command should
not use the clients credentials.
---
Key: AMQ-5141
URL: https://issues.apache.org/jira/browse/AMQ-5141
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.9.0
Reporter: Torsten Mielke
Labels: DLQ, authorization, expiry, security
If the broker handles a RemoveInfo command it may also kick off a message
expiry check for (I presume) any prefetched messages. If messages are to be
expired they get sent to ActiveMQ.DLQ by default. See stack trace in next
comment.
If the broker is security enabled with authorization turned on and messages
get sent to DLQ as a result of the expiry check then the broker uses the
client's security context when sending the messages to DLQ.
This implies the client user needs to have write access to ActiveMQ.DLQ.
As this may happen with any other client, all client users will require write
access to ActiveMQ.DLQ, which may not be appropriate from a security point of
view.
The broker regularly runs an expiry check and uses a broker internal security
context for this task. In my opinion this same broker internal security
context should be used when expiring messages as part of the RemoveInfo
command. The broker should not use the client's security context.
[1]
The current behavior can raise the following SecurityException if the client
user does not have write access to ActiveMQ.DLQ
{code}
2014-04-11 08:11:22,229 | WARN | 2.38:61201@61616 | RegionBroker |
ivemq.broker.region.RegionBroker 703 |
105 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Caught an
exception sending to DLQ: Message
ID:S930A3085-50865-635327964441522304-1:1:363:2:1 dropped=false acked=false
locked=true
java.lang.SecurityException: User Test is not authorized to write to:
queue://ActiveMQ.DLQ
at
org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:197)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.util.BrokerSupport.doResend(BrokerSupport.java:68)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.util.BrokerSupport.resendNoCopy(BrokerSupport.java:38)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(RegionBroker.java:691)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.advisory.AdvisoryBroker.sendToDeadLetterQueue(AdvisoryBroker.java:413)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:274)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.util.RedeliveryPlugin.sendToDeadLetterQueue(RedeliveryPlugin.java:132)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:274)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.region.RegionBroker.messageExpired(RegionBroker.java:659)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
[jira] [Commented] (AMQ-5141) Message expiry that is done as part of a removeSubscription command should not use the clients credentials.
[
https://issues.apache.org/jira/browse/AMQ-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966450#comment-13966450
]
Torsten Mielke commented on AMQ-5141:
-
Here are two possible workarounds:
1)
The quick solution is to grant write permissions to user 'Test' but potentially
other users could be affected as well and may also need write permissions for
ActiveMQ.DLQ. In the worst case all users may require write permissions to
ActiveMQ.DLQ.
2)
In case you don't care about these expired messages you can also configure the
broker to simply discard expired messages using this configuration within the
policyEntry config
deadLetterStrategy
sharedDeadLetterStrategy processExpired=false /
/deadLetterStrategy
as per http://activemq.apache.org/message-redelivery-and-dlq-handling.html.
Then the permission problem won't arise.
Message expiry that is done as part of a removeSubscription command should
not use the clients credentials.
---
Key: AMQ-5141
URL: https://issues.apache.org/jira/browse/AMQ-5141
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.9.0
Reporter: Torsten Mielke
Labels: DLQ, authorization, expiry, security
If the broker handles a RemoveInfo command it may also kick off a message
expiry check for (I presume) any prefetched messages. If messages are to be
expired they get sent to ActiveMQ.DLQ by default. See stack trace in next
comment.
If the broker is security enabled with authorization turned on and messages
get sent to DLQ as a result of the expiry check then the broker uses the
client's security context when sending the messages to DLQ.
This implies the client user needs to have write access to ActiveMQ.DLQ.
As this may happen with any other client, all client users will require write
access to ActiveMQ.DLQ, which may not be appropriate from a security point of
view.
The broker regularly runs an expiry check and uses a broker internal security
context for this task. In my opinion this same broker internal security
context should be used when expiring messages as part of the RemoveInfo
command. The broker should not use the client's security context.
[1]
The current behavior can raise the following SecurityException if the client
user does not have write access to ActiveMQ.DLQ
{code}
2014-04-11 08:11:22,229 | WARN | 2.38:61201@61616 | RegionBroker |
ivemq.broker.region.RegionBroker 703 |
105 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Caught an
exception sending to DLQ: Message
ID:S930A3085-50865-635327964441522304-1:1:363:2:1 dropped=false acked=false
locked=true
java.lang.SecurityException: User Test is not authorized to write to:
queue://ActiveMQ.DLQ
at
org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:197)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.util.BrokerSupport.doResend(BrokerSupport.java:68)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.util.BrokerSupport.resendNoCopy(BrokerSupport.java:38)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(RegionBroker.java:691)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.advisory.AdvisoryBroker.sendToDeadLetterQueue(AdvisoryBroker.java:413)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.sendToDeadLetterQueue(MutableBrokerFilter.java:274)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.util.RedeliveryPlugin.sendToDeadLetterQueue(RedeliveryPlugin.java:132)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
[jira] [Comment Edited] (AMQ-5141) Message expiry that is done as part of a removeSubscription command should not use the clients credentials.
[
https://issues.apache.org/jira/browse/AMQ-5141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966450#comment-13966450
]
Torsten Mielke edited comment on AMQ-5141 at 4/11/14 12:13 PM:
---
Here are two possible workarounds:
1)
The quick solution is to grant write permissions to user 'Test' but potentially
other users could be affected as well and may also need write permissions for
ActiveMQ.DLQ. In the worst case all users may require write permissions to
ActiveMQ.DLQ.
2)
In case you don't care about these expired messages you can also configure the
broker to simply discard expired messages using this configuration within the
policyEntry config
{code:xml}
deadLetterStrategy
sharedDeadLetterStrategy processExpired=false /
/deadLetterStrategy
{code}
as per http://activemq.apache.org/message-redelivery-and-dlq-handling.html.
Then the permission problem won't arise.
was (Author: tmielke):
Here are two possible workarounds:
1)
The quick solution is to grant write permissions to user 'Test' but potentially
other users could be affected as well and may also need write permissions for
ActiveMQ.DLQ. In the worst case all users may require write permissions to
ActiveMQ.DLQ.
2)
In case you don't care about these expired messages you can also configure the
broker to simply discard expired messages using this configuration within the
policyEntry config
deadLetterStrategy
sharedDeadLetterStrategy processExpired=false /
/deadLetterStrategy
as per http://activemq.apache.org/message-redelivery-and-dlq-handling.html.
Then the permission problem won't arise.
Message expiry that is done as part of a removeSubscription command should
not use the clients credentials.
---
Key: AMQ-5141
URL: https://issues.apache.org/jira/browse/AMQ-5141
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.9.0
Reporter: Torsten Mielke
Labels: DLQ, authorization, expiry, security
If the broker handles a RemoveInfo command it may also kick off a message
expiry check for (I presume) any prefetched messages. If messages are to be
expired they get sent to ActiveMQ.DLQ by default. See stack trace in next
comment.
If the broker is security enabled with authorization turned on and messages
get sent to DLQ as a result of the expiry check then the broker uses the
client's security context when sending the messages to DLQ.
This implies the client user needs to have write access to ActiveMQ.DLQ.
As this may happen with any other client, all client users will require write
access to ActiveMQ.DLQ, which may not be appropriate from a security point of
view.
The broker regularly runs an expiry check and uses a broker internal security
context for this task. In my opinion this same broker internal security
context should be used when expiring messages as part of the RemoveInfo
command. The broker should not use the client's security context.
[1]
The current behavior can raise the following SecurityException if the client
user does not have write access to ActiveMQ.DLQ
{code}
2014-04-11 08:11:22,229 | WARN | 2.38:61201@61616 | RegionBroker |
ivemq.broker.region.RegionBroker 703 |
105 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Caught an
exception sending to DLQ: Message
ID:S930A3085-50865-635327964441522304-1:1:363:2:1 dropped=false acked=false
locked=true
java.lang.SecurityException: User Test is not authorized to write to:
queue://ActiveMQ.DLQ
at
org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:197)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.util.BrokerSupport.doResend(BrokerSupport.java:68)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.util.BrokerSupport.resendNoCopy(BrokerSupport.java:38)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.region.RegionBroker.sendToDeadLetterQueue(RegionBroker.java:691)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
org.apache.activemq.broker.BrokerFilter.sendToDeadLetterQueue(BrokerFilter.java:262)[105:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]
at
[jira] [Commented] (AMQ-5082) ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening
[
https://issues.apache.org/jira/browse/AMQ-5082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967034#comment-13967034
]
Kevin McLaughlin commented on AMQ-5082:
---
I believe I am seeing the same issue. We have a three node dev setup that
seems to consistently get into this state. Attached are thread dumps from each
of the nodes.
ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening
---
Key: AMQ-5082
URL: https://issues.apache.org/jira/browse/AMQ-5082
Project: ActiveMQ
Issue Type: Bug
Components: activemq-leveldb-store
Affects Versions: 5.9.0, 5.10.0
Reporter: Scott Feldstein
Priority: Critical
Attachments: 03-07.tgz, mq-node1-cluster.failure,
mq-node2-cluster.failure, mq-node3-cluster.failure,
zookeeper.out-cluster.failure
I have a 3 node amq cluster and one zookeeper node using a replicatedLevelDB
persistence adapter.
{code}
persistenceAdapter
replicatedLevelDB
directory=${activemq.data}/leveldb
replicas=3
bind=tcp://0.0.0.0:0
zkAddress=zookeep0:2181
zkPath=/activemq/leveldb-stores/
/persistenceAdapter
{code}
After about a day or so of sitting idle there are cascading failures and the
cluster completely stops listening all together.
I can reproduce this consistently on 5.9 and the latest 5.10 (commit
2360fb859694bacac1e48092e53a56b388e1d2f0). I am going to attach logs from
the three mq nodes and the zookeeper logs that reflect the time where the
cluster starts having issues.
The cluster stops listening Mar 4, 2014 4:56:50 AM (within 5 seconds).
The OSs are all centos 5.9 on one esx server, so I doubt networking is an
issue.
If you need more data it should be pretty easy to get whatever is needed
since it is consistently reproducible.
This bug may be related to AMQ-5026, but looks different enough to file a
separate issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (AMQ-5082) ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening
[
https://issues.apache.org/jira/browse/AMQ-5082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin McLaughlin updated AMQ-5082:
--
Attachment: amq_5082_threads.tar.gz
Thread dump of non-listening cluster.
ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening
---
Key: AMQ-5082
URL: https://issues.apache.org/jira/browse/AMQ-5082
Project: ActiveMQ
Issue Type: Bug
Components: activemq-leveldb-store
Affects Versions: 5.9.0, 5.10.0
Reporter: Scott Feldstein
Priority: Critical
Attachments: 03-07.tgz, amq_5082_threads.tar.gz,
mq-node1-cluster.failure, mq-node2-cluster.failure, mq-node3-cluster.failure,
zookeeper.out-cluster.failure
I have a 3 node amq cluster and one zookeeper node using a replicatedLevelDB
persistence adapter.
{code}
persistenceAdapter
replicatedLevelDB
directory=${activemq.data}/leveldb
replicas=3
bind=tcp://0.0.0.0:0
zkAddress=zookeep0:2181
zkPath=/activemq/leveldb-stores/
/persistenceAdapter
{code}
After about a day or so of sitting idle there are cascading failures and the
cluster completely stops listening all together.
I can reproduce this consistently on 5.9 and the latest 5.10 (commit
2360fb859694bacac1e48092e53a56b388e1d2f0). I am going to attach logs from
the three mq nodes and the zookeeper logs that reflect the time where the
cluster starts having issues.
The cluster stops listening Mar 4, 2014 4:56:50 AM (within 5 seconds).
The OSs are all centos 5.9 on one esx server, so I doubt networking is an
issue.
If you need more data it should be pretty easy to get whatever is needed
since it is consistently reproducible.
This bug may be related to AMQ-5026, but looks different enough to file a
separate issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (AMQ-5082) ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening
[
https://issues.apache.org/jira/browse/AMQ-5082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967040#comment-13967040
]
Kevin McLaughlin commented on AMQ-5082:
---
Shutting down node1 from the attached thread dumps properly promoted node3 to
be the master. So, it appears that node1 is somehow maintaining its leadership
(but not listening) when this happens.
ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening
---
Key: AMQ-5082
URL: https://issues.apache.org/jira/browse/AMQ-5082
Project: ActiveMQ
Issue Type: Bug
Components: activemq-leveldb-store
Affects Versions: 5.9.0, 5.10.0
Reporter: Scott Feldstein
Priority: Critical
Attachments: 03-07.tgz, amq_5082_threads.tar.gz,
mq-node1-cluster.failure, mq-node2-cluster.failure, mq-node3-cluster.failure,
zookeeper.out-cluster.failure
I have a 3 node amq cluster and one zookeeper node using a replicatedLevelDB
persistence adapter.
{code}
persistenceAdapter
replicatedLevelDB
directory=${activemq.data}/leveldb
replicas=3
bind=tcp://0.0.0.0:0
zkAddress=zookeep0:2181
zkPath=/activemq/leveldb-stores/
/persistenceAdapter
{code}
After about a day or so of sitting idle there are cascading failures and the
cluster completely stops listening all together.
I can reproduce this consistently on 5.9 and the latest 5.10 (commit
2360fb859694bacac1e48092e53a56b388e1d2f0). I am going to attach logs from
the three mq nodes and the zookeeper logs that reflect the time where the
cluster starts having issues.
The cluster stops listening Mar 4, 2014 4:56:50 AM (within 5 seconds).
The OSs are all centos 5.9 on one esx server, so I doubt networking is an
issue.
If you need more data it should be pretty easy to get whatever is needed
since it is consistently reproducible.
This bug may be related to AMQ-5026, but looks different enough to file a
separate issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (AMQ-5082) ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening
[
https://issues.apache.org/jira/browse/AMQ-5082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967073#comment-13967073
]
Scott Feldstein commented on AMQ-5082:
--
This can be reproduced by causing a network disruption where all the apache
nodes can't access any of the zookeeper cluster. The disruption needs to be
longer than the zk timeout. The quickest way I've been able to accomplish this
is by using my local firewall, although the state of the objects isn't always
the exact same as the logs that i posted the overall symptoms are.
In code I think the problem is in the
org.apache.activemq.leveldb.replicated.MasterElector.scala - change_listener.
As Kevin said, the node thinks that it is the master but it is not correctly
initializing the listener when it gets into this state.
ActiveMQ replicatedLevelDB cluster breaks, all nodes stop listening
---
Key: AMQ-5082
URL: https://issues.apache.org/jira/browse/AMQ-5082
Project: ActiveMQ
Issue Type: Bug
Components: activemq-leveldb-store
Affects Versions: 5.9.0, 5.10.0
Reporter: Scott Feldstein
Priority: Critical
Attachments: 03-07.tgz, amq_5082_threads.tar.gz,
mq-node1-cluster.failure, mq-node2-cluster.failure, mq-node3-cluster.failure,
zookeeper.out-cluster.failure
I have a 3 node amq cluster and one zookeeper node using a replicatedLevelDB
persistence adapter.
{code}
persistenceAdapter
replicatedLevelDB
directory=${activemq.data}/leveldb
replicas=3
bind=tcp://0.0.0.0:0
zkAddress=zookeep0:2181
zkPath=/activemq/leveldb-stores/
/persistenceAdapter
{code}
After about a day or so of sitting idle there are cascading failures and the
cluster completely stops listening all together.
I can reproduce this consistently on 5.9 and the latest 5.10 (commit
2360fb859694bacac1e48092e53a56b388e1d2f0). I am going to attach logs from
the three mq nodes and the zookeeper logs that reflect the time where the
cluster starts having issues.
The cluster stops listening Mar 4, 2014 4:56:50 AM (within 5 seconds).
The OSs are all centos 5.9 on one esx server, so I doubt networking is an
issue.
If you need more data it should be pretty easy to get whatever is needed
since it is consistently reproducible.
This bug may be related to AMQ-5026, but looks different enough to file a
separate issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (AMQ-2009) Problem with message dispatch after a while
[ https://issues.apache.org/jira/browse/AMQ-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967085#comment-13967085 ] Eduardo Teodoro Silva Júnior commented on AMQ-2009: --- I´m facing the same problem with version 5.9.0. Problem with message dispatch after a while --- Key: AMQ-2009 URL: https://issues.apache.org/jira/browse/AMQ-2009 Project: ActiveMQ Issue Type: Bug Components: Broker Affects Versions: 5.1.0, 5.2.0 Reporter: Rajani Chennamaneni Assignee: Rob Davies Fix For: NEEDS_REVIEW Attachments: AMQ-2009Testcase2.zip, DispatchMultipleConsumersTest.java, JConsole-screenshot.jpg, consumertest.zip, testcase.zip Messages are not getting dispatched after a while (although it accepts new incoming messages) until restart of the broker. This problem is described in several posts. http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html http://www.nabble.com/Stuck-messages---Dispatch-issues-td20467949.html There was also an issue opened in Spring project for this thinking it was Spring problem. http://jira.springframework.org/browse/SPR-5110 I am not able to reproduce with Junit test case having BrokerService started with in the test case. I guess I am not hitting the right stress conditions this way. But when I run the test case against an externally running ActiveMQ instance backed with oracle database persistence, it is reproducible most of the times. This is not a every time failure situation, it takes more time once than the other. I was able to hit this situation of stuck messages on queue using following scenario most of the times: 1) Start 2 concurrent consumers for the queue using Spring's DefaultMessageListenerContainer using cacheLevelName as CACHE_CONSUMER 2) Send messages using JMETER 2.3.2 to the queue on ActiveMQ stand alone broker instance with 50 threads looping 20 times. 3) After a while, you will notice that Spring logs that no messages are being received but the messages are shown jconsole of ActiveMQ and the database backing it for persistence. But in 5.2 RC3, the problem is that it dispatches duplicate messages and does not remove them from broker's database after acknowledge properly. Attached test case might help to reproduce when run against externally running stand alone ActiveMQ broker. Another way to see the problem is that try to load test using JMETER by sending messages to a queue with a camel route that moves messages from this queue to another and you will notice that it stops moving after while or copied duplicates in case of 5.2 RC3. Sorry about such a huge description but it is a real problem! A different team at our company are having this issue in production with 5.1. They are using it as an embedded broker with derby for persistence. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (AMQ-4120) SqlServer - SQL Syntax error in initial table creation
[ https://issues.apache.org/jira/browse/AMQ-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967250#comment-13967250 ] Gurinder commented on AMQ-4120: --- I see this also in 5.9.0 I am using jdbcPersistenceAdapter and my datasource is using the bean class: org.apache.commons.dbcp.BasicDataSource SqlServer - SQL Syntax error in initial table creation -- Key: AMQ-4120 URL: https://issues.apache.org/jira/browse/AMQ-4120 Project: ActiveMQ Issue Type: Improvement Affects Versions: 5.7.0 Environment: Windows Server 2008 R2, 64bit, MS SQL Server 2008 persistence adapter. Reporter: Rasitha Wijesinghe Priority: Minor Fix For: NEEDS_REVIEW When using SQL Server as the backing data store and creating the AMQ tables for the very first time, the following error is logged: INFO | jvm 1| 2012/10/19 11:12:03 | WARN | Could not create JDBC tables; they could already exist. Failure was: ALTER TABLE ACTIVEMQ_ACKS DROP PRIMARY KEY Message: Incorrect syntax near the keyword 'PRIMARY'. SQLState: S1000 Vendor code: 156 INFO | jvm 1| 2012/10/19 11:12:03 | WARN | Failure details: Incorrect syntax near the keyword 'PRIMARY'. INFO | jvm 1| 2012/10/19 11:12:03 | java.sql.SQLException: Incorrect syntax near the keyword 'PRIMARY'. Since there is no specific adapter for SQL Server it uses the DefaultJDBCAdapter which has getDropAckPKAlterStatementEnd() returning DROP PRIMARY KEY. Not sure if this is an issue with other databases as well. (Note: If you already have the tables, this error is still present but only logs a DEBUG entry). Does it make sense to add a MSSqlJDBCAdapter that overrides getDropAckPKAlterStatementEnd? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (AMQ-5142) Behavior when message expires while it's being processed
Mike Chao created AMQ-5142: -- Summary: Behavior when message expires while it's being processed Key: AMQ-5142 URL: https://issues.apache.org/jira/browse/AMQ-5142 Project: ActiveMQ Issue Type: Bug Affects Versions: 5.7.0 Environment: OS: Linux 3.5.0 (Mint 14) Java: 1.6 Camel: 2.8.2 Reporter: Mike Chao I have 2 jms queues, queue.start and queue.end. Using camel I setup a route from queue.start - sleepProcessor - queue.end. I also have individual dead letter queues setup for queue.start and queue.end. The sleepProcessor takes the message and sleeps for 5 seconds. I send 1 message with a JMSExpirationTime of 1 second from System.currentTimeMillis(). The behavior I see is that 1 message ends up in DLQ.queue.start and another message ends up in DLQ.queue.end. Transactions are enabled. The test program can be found here http://s000.tinyupload.com/?file_id=04225732819763428273 And can be run with mvn camel:run Is there a way to configure activeMQ so that only 1 message ends up in the dead letter queue? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Closed] (AMQ-5142) Behavior when message expires while it's being processed
[ https://issues.apache.org/jira/browse/AMQ-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Bish closed AMQ-5142. - Resolution: Invalid This is a user list question not a bug report. Behavior when message expires while it's being processed Key: AMQ-5142 URL: https://issues.apache.org/jira/browse/AMQ-5142 Project: ActiveMQ Issue Type: Bug Affects Versions: 5.7.0 Environment: OS: Linux 3.5.0 (Mint 14) Java: 1.6 Camel: 2.8.2 Reporter: Mike Chao I have 2 jms queues, queue.start and queue.end. Using camel I setup a route from queue.start - sleepProcessor - queue.end. I also have individual dead letter queues setup for queue.start and queue.end. The sleepProcessor takes the message and sleeps for 5 seconds. I send 1 message with a JMSExpirationTime of 1 second from System.currentTimeMillis(). The behavior I see is that 1 message ends up in DLQ.queue.start and another message ends up in DLQ.queue.end. Transactions are enabled. The test program can be found here http://s000.tinyupload.com/?file_id=04225732819763428273 And can be run with mvn camel:run Is there a way to configure activeMQ so that only 1 message ends up in the dead letter queue? -- This message was sent by Atlassian JIRA (v6.2#6252)
