Testing failover on dispatcher/java-broker cluster
Hello Qpid community, I’m testing the resilience of a dispatcher/broker infrastructure and I noticed the following behavior: I run a test with one JMS client connected to a dispatcher, which is connected to a broker. 1) Using JMS I establish a connection to the dispatcher and create a message producer 2) I’m able to send a message to the broker through the dispatcher 3) I stop and restart the broker 4) I cannot send any messages using the message producer I created before. 5) If a recreate a MessageProducer (new AMQP link), the message arrives to the broker In the failing scenario 4, I noticed using Wireshark that the dispatcher does not send any messages to the broker. So I deduced that the broker is not responsible for this behavior. *Is it an expected behavior? What can I change in the dispatcher/JMS configuration to avoid the failure?* You can find attached the Wireshark logs I produced from this experiment: - JMS – dispatcher – reuse sender: logs between JMS and the dispatcher when I reuse the message producer after the restart - JMS – dispatcher – new sender: logs between JMS and the dispatcher when I create a new message producer after the restart - dispatcher – broker – reuse sender: logs between the dispatcher and the broker, I reuse the message producer - dispatcher – broker – reuse sender: logs between the dispatcher and the broker, I create a new message producer I’m using qpid-dispatch 0.6.0, JMS 0.9.0 and qpid-java-broker 6.0.1. Thanks, Best regards, Antoine src.ip = JMS ip dst.ip = dispatcher ip # Client connection SourceDestinationProtocol Length Info src.ip dst.ipTCP 66 53505 â 5672 [SYN] Seq=0 Win=65535 Len=0 MSS=1460 WS=2 SACK_PERM=1 dst.ipsrc.ip TCP 66 5672 â 53505 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 SACK_PERM=1 WS=128 src.ip dst.ipTCP 54 53505 â 5672 [ACK] Seq=1 Ack=1 Win=65536 Len=0 src.ip dst.ipAMQP 62 Protocol-Header 1-0-0 dst.ipsrc.ip TCP 60 5672 â 53505 [ACK] Seq=1 Ack=9 Win=29312 Len=0 dst.ipsrc.ip AMQP 105Protocol-Header 1-0-0 sasl.mechanisms src.ip dst.ipAMQP 91 sasl.init dst.ipsrc.ip TCP 60 5672 â 53505 [ACK] Seq=52 Ack=46 Win=29312 Len=0 dst.ipsrc.ip AMQP 76 sasl.outcome src.ip dst.ipAMQP 300Protocol-Header 1-0-0 open dst.ipsrc.ip AMQP 185Protocol-Header 1-0-0 open # Creating the Session src.ip dst.ipAMQP 86 begin dst.ipsrc.ip AMQP 86 begin src.ip dst.ipAMQP 86 begin dst.ipsrc.ip AMQP 86 begin # Creating MessageProducer src.ip dst.ipAMQP 313attach dst.ipsrc.ip AMQP 374attach flow # Sending a message (success) src.ip dst.ipAMQP 405transfer dst.ipsrc.ip AMQP 131flow disposition src.ip dst.ipTCP 54 53505 â 5672 [ACK] Seq=966 Ack=666 Win=64870 Len=0 src.ip dst.ipAMQP 62 (empty) dst.ipsrc.ip TCP 60 5672 â 53505 [ACK] Seq=666 Ack=974 Win=32512 Len=0 src.ip dst.ipAMQP 62 (empty) dst.ipsrc.ip TCP 60 5672 â 53505 [ACK] Seq=666 Ack=982 Win=32512 Len=0 dst.ipsrc.ip AMQP 62 (empty) # Stopping broker src.ip dst.ipTCP 54 53505 â 5672 [ACK] Seq=982 Ack=674 Win=64862 Len=0 src.ip dst.ipAMQP 62 (empty) dst.ipsrc.ip TCP 60 5672 â 53505 [ACK] Seq=674 Ack=990 Win=32512 Len=0 # Trying to send a message (timeout) src.ip dst.ipAMQP 406transfer dst.ipsrc.ip TCP 60 5672 â 53505 [ACK] Seq=674 Ack=1342 Win=33536 Len=0 src.ip dst.ipAMQP 62 (empty) # Restarting broker dst.ipsrc.ip TCP 60 5672 â 53505 [ACK] Seq=674 Ack=1350 Win=33536 Len=0 # Trying to send a message (timeout) src.ip dst.ipAMQP 405transfer dst.ipsrc.ip TCP 60 5672 â 53505 [ACK] Seq=674 Ack=1701 Win=34560 Len=0 dst.ipsrc.ip AMQP 62 (empty) src.ip dst.ipTCP 54 53505 â 5672 [ACK] Seq=1701 Ack=682 Win=64854 Len=0 src.ip dst.ipTCP 54 53505 â 5672 [RST, ACK] Seq=1701 Ack=682 Win=0 Len=0 dst.ip = dispatcher ip src.ip = broker ip # Connecting t
Re: Testing failover on dispatcher/java-broker cluster
Two details I did not mention in my previous mail:
- I use the same JMS connection/session to recreate a MessageProducer
- I noticed that the dispatcher reconnects correctly to the broker and the
autolinks are 'active'.
This is the dispatcher config that I used for this test:
router {
id: router.5672
mode: interior
worker-threads: 4
}
listener {
host: 0.0.0.0
port: 5672
role: normal
saslMechanisms: ANONYMOUS
requireSsl: no
authenticatePeer: no
}
address {
name: perf.topic.addr
prefix: perf.topic
waypoint: true
}
connector {
role: route-container
addr: localhost
port: 10101
name: localhost.broker.10101.connector
}
autoLink {
addr: perf.topic
dir: out
connection: localhost.broker.10101.connector
name: localhost.broker.10101.perf.topic.out
}
Thanks,
Antoine
Re: Testing failover on dispatcher/java-broker cluster
Hi Antoine, In the broker traces, I see connection shutdown after the transfer but before you shut down the broker. Do you know what is happening there? What was the disposition of the delivery? -Ted On 09/14/2016 09:12 AM, Antoine Chevin wrote: Hello Qpid community, I’m testing the resilience of a dispatcher/broker infrastructure and I noticed the following behavior: I run a test with one JMS client connected to a dispatcher, which is connected to a broker. 1) Using JMS I establish a connection to the dispatcher and create a message producer 2) I’m able to send a message to the broker through the dispatcher 3) I stop and restart the broker 4) I cannot send any messages using the message producer I created before. 5) If a recreate a MessageProducer (new AMQP link), the message arrives to the broker In the failing scenario 4, I noticed using Wireshark that the dispatcher does not send any messages to the broker. So I deduced that the broker is not responsible for this behavior. *Is it an expected behavior? What can I change in the dispatcher/JMS configuration to avoid the failure?* You can find attached the Wireshark logs I produced from this experiment: - JMS – dispatcher – reuse sender: logs between JMS and the dispatcher when I reuse the message producer after the restart - JMS – dispatcher – new sender: logs between JMS and the dispatcher when I create a new message producer after the restart - dispatcher – broker – reuse sender: logs between the dispatcher and the broker, I reuse the message producer - dispatcher – broker – reuse sender: logs between the dispatcher and the broker, I create a new message producer I’m using qpid-dispatch 0.6.0, JMS 0.9.0 and qpid-java-broker 6.0.1. Thanks, Best regards, Antoine - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: Connecting to the router
On Tue, 2016-09-13 at 16:33 -0700, Justin Ross wrote:
> Some potentially interesting things.
>
> ## Components
>
> python-qpid-proton-0.13.1-1.fc24.x86_64
> python-qpid-messaging-0.34-9.fc24.x86_64 (qpid*_*messaging)
> qpid-dispatch-router-0.6.1-2.fc24.x86_64
>
> ## Listener config
>
> listener {
> host: 0.0.0.0
> port: amqp
> authenticatePeer: on
> saslMechanisms: ANONYMOUS
> }
>
> ## host: 0.0.0.0 allows connection via localhost from qpid_messaging
> but
> not from proton
>
> My qpid_messaging client can connect, using localhost in its
> connection
> info, but my proton client cannot (connection refused). Using '::'
> instead
> of '0.0.0.0' allows either to connect. Perhaps the way proton
> resolves
> localhost is different.
0.0.0.0 is an IPv4 address :: is the IPv6 equivalent.
Usually localhost will map to both of these addresses, and the usual
system defaults will try the IPv6 address first.
So if the client code doesn't correctly try all the resolved IP
addresses in order then you'll get this behaviour.
I know that qpid.messaging does this correctly.
> ## "authenticatePeer: on" strips ANONYMOUS
What do you mean "strips" ANONYMOUS? Do you mean the mechanism isn't
offered? The mechanism isn't in the allowable client list?
At the proton API level authenticatePeer means that we require
connections to be authenticated before using them. This can either by
via SASL or via SSL.
So I wouldn't expect that setting to take ANONYMOUS out of the offered
mechanisms, but I'm not 100% about the behaviour of Cyrus under all
circumstances.
>
> This was a surprise to me. I had explicitly enabled ANONYMOUS. And
> I
> confess I was mentally reading 'authenticatePeer' to mean enable
> SASL.
> Which component is stripping it, proton or dispatch or cyrus? Should
> it be
> stripped when I positively express a desire for it?
I'm not sure that forcing the ANONYMOUS mechanism together with
authentication isn't contradictory - ANONYMOUS is explicitly
unauthenticated (it's anonymous after all!).
> If we keep this
> behavior, we should add a note about it to the qdrouterd.conf man
> page.
Undoubtedly!
Andrew
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
