Alex Rudyy created QPID-6143:
--------------------------------
Summary: [Java Broker] Broker can crash on publishing/consuming
messages during master transfer in BDB HA group
Key: QPID-6143
URL: https://issues.apache.org/jira/browse/QPID-6143
Project: Qpid
Issue Type: Bug
Components: Java Broker
Affects Versions: 0.30
Reporter: Alex Rudyy
I built 3 node cluster and started a sample java app publishing and consuming
messages into/from a Master node.
During publishing/consuming I was flipping mastership between 1st and 2d nodes
from 3d node using the following script
{noformat}
while true; do (curl --user admin:admin -X PUT -d '{"role":"MASTER"}'
http://localhost:8082/api/latest/replicationnode/node3/node2; sleep 1; curl -s
--user admin:admin -X GET
http://localhost:8081/api/latest/virtualhostnode/node2 | grep "name\|role" ;
curl --user admin:admin -X PUT -d '{"role":"MASTER"}'
http://localhost:8082/api/latest/replicationnode/node3/node1 ; sleep 1; curl -s
--user admin:admin -X GET
http://localhost:8080/api/latest/virtualhostnode/node1 | grep "name\|role") done
{noformat}
In some cases, on message delivery when node was in a process of state change
from MASTER to REPLICA I've seen the following broker crush
{noformat}
########################################################################
#
# Unhandled Exception java.lang.IllegalStateException: Environment is not valid
in Thread pool-187-thread-21
#
# Exiting
#
########################################################################
java.lang.IllegalStateException: Environment is not valid
at
org.apache.qpid.server.store.berkeleydb.replication.ReplicatedEnvironmentFacade.openDatabase(ReplicatedEnvironmentFacade.java:460)
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.getMessageContentDb(AbstractBDBMessageStore.java:1057)
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore.getAllContent(AbstractBDBMessageStore.java:579)
at
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage.getContent(AbstractBDBMessageStore.java:1319)
at
org.apache.qpid.server.message.AbstractServerMessageImpl.getContent(AbstractServerMessageImpl.java:171)
at
org.apache.qpid.server.protocol.v0_8.output.ProtocolOutputConverterImpl$MessageContentSourceBody.writePayload(ProtocolOutputConverterImpl.java:251)
at org.apache.qpid.framing.AMQFrame.writeFrames(AMQFrame.java:123)
at
org.apache.qpid.server.protocol.v0_8.output.ProtocolOutputConverterImpl$CompositeAMQBodyBlock.writePayload(ProtocolOutputConverterImpl.java:468)
at
org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine.asByteBuffer(AMQProtocolEngine.java:595)
at
org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine.writeFrame(AMQProtocolEngine.java:709)
at
org.apache.qpid.server.protocol.v0_8.output.ProtocolOutputConverterImpl.writeFrame(ProtocolOutputConverterImpl.java:430)
at
org.apache.qpid.server.protocol.v0_8.output.ProtocolOutputConverterImpl.writeMessageDeliveryUnchanged(ProtocolOutputConverterImpl.java:207)
at
org.apache.qpid.server.protocol.v0_8.output.ProtocolOutputConverterImpl.writeMessageDelivery(ProtocolOutputConverterImpl.java:141)
at
org.apache.qpid.server.protocol.v0_8.output.ProtocolOutputConverterImpl.writeMessageDelivery(ProtocolOutputConverterImpl.java:101)
at
org.apache.qpid.server.protocol.v0_8.output.ProtocolOutputConverterImpl.writeDeliver(ProtocolOutputConverterImpl.java:78)
at
org.apache.qpid.server.protocol.v0_8.AMQProtocolEngine$WriteDeliverMethod.deliverToClient(AMQProtocolEngine.java:1626)
at
org.apache.qpid.server.protocol.v0_8.ConsumerTarget_0_8.sendToClient(ConsumerTarget_0_8.java:513)
at
org.apache.qpid.server.protocol.v0_8.ConsumerTarget_0_8$AckConsumer.send(ConsumerTarget_0_8.java:317)
at
org.apache.qpid.server.queue.QueueConsumerImpl.send(QueueConsumerImpl.java:475)
at
org.apache.qpid.server.queue.AbstractQueue.deliverMessage(AbstractQueue.java:1235)
at
org.apache.qpid.server.queue.AbstractQueue.attemptDelivery(AbstractQueue.java:1989)
at
org.apache.qpid.server.queue.AbstractQueue.processQueue(AbstractQueue.java:2160)
at org.apache.qpid.server.queue.QueueRunner$1.run(QueueRunner.java:77)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at org.apache.qpid.server.queue.QueueRunner.run(QueueRunner.java:68)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
