Re: Testing failover on dispatcher/java-broker cluster

[Antoine Chevin]

Hi Ted,

You’re right, the connection close looked strange before stopping of the
broker. I manually added the annotation (# stopping the broker) and was
wrong about the position of this one. I replayed the test and the
connection close happens *after* the broker stop. I assume it is the broker
that initiates it.

I found something interesting. In my test, I always sent a message when the
broker is down, expecting to get a JmsSendTimedOutException (waiting for
the disposition frame). I assumed this was harmless. But it turns out this
is not. When I don’t do that, I can send a message after the broker
restart. So to sum up the experiment I did:

* I use Wireshark between the JMS client and the dispatcher. *

1)  Using JMS I establish a connection to the dispatcher and create a
message producer (Wireshark: connection open -> attach)
2)  I’m able to send a message to the broker through the dispatcher (
Wireshark: transfer -> disposition)
3)  I stop the broker
4)  With the same link, I send a message and I get a
JmsSendTimedOutException (waiting for the disposition frame) (Wireshark:
transfer)
5)  I restart the broker
6)  With the same link, I try to send a message and I get a
JmsSendTimedOutException for the same reason (waiting for the disposition
frame) (Wireshark: transfer)

If I skip step (4), I cannot reproduce step (6) and my messages arrive
(Wireshark: transfer -> disposition) to the restarted broker.

I hope it makes it clearer for you. Sorry for my rookie mistakes :-).

Note: My colleague and I ran a small experiment to identify if the problem
comes from JMS or the AMQP protocol. He changed the code of the java broker
to not send the disposition frame one time out of two.

We got these results:

* I use Wireshark between the JMS client and the patched broker. *

1) Using JMS I establish a connection to the patched broker and create a
message producer (Wireshark: connection open -> attach)
2)  I send a message to the broker and it replies with the disposition
frame (Wireshark: transfer -> disposition)
3) I send a message to the broker which drops the disposition frame. I get
a send timeout in JMS (Wireshark: transfer)
2)  I send a message to the broker and it replies with the disposition frame
(Wireshark: transfer -> disposition). It works fine.

We assume that there is something going on in the dispatcher.


Thanks,
Antoine

Qpid Proton-J client heartbeat (for ActiveMQ Artemis Broker)

[Andy Redhead]

Hi,

I'm (very) inexperienced with AMQP so this may be a silly question...

I'm using Proton-J from a standalone Java program to interact with queues 
hosted by an ActiveMQ Artemis broker.

So long as there is a regular message flow between the client and the broker, 
everything is fine.

If there is a pause in the message flow (for 60 seconds, give or take) then the 
broker starts to complain, writing messages in the log such as:

"2016-09-15T12:42:32,308 155833 [Thread-7 
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$2@2368231c-1759089888)]
 WARN  o.a.activemq.artemis.core.client - AMQ212037: Connection failure has 
been detected: AMQ119014: Did not receive data from /127.0.0.1:62229. It is 
likely the client has exited or crashed without closing its connection, or the 
network between the server and client has failed. You also might have 
configured connection-ttl and client-failure-check-period incorrectly. Please 
check user manual for more information. The connection will now be closed. 
[code=CONNECTION_TIMEDOUT]"

I think I can work around this by making my clients send "heartbeat" messages 
on a regular basis, however if there is some kind of AMQP heartbeat capability 
in the Proton-J client I'd rather use that.

Does Proton-J have the ability to send whatever an AMQP broker expects to 
receive as a heartbeat? If so, how do I set it up?

Thanks, Andy

[cid:image012.png@01D17AF7.D972DF70]
Andy Redhead
Principal Consultant > Solutions > Advanced


Advanced
230 City Road, London, EC1V 2TT
t: 020 7880  > m: 0781 392 5246

www.oneadvanced.com
[cid:image018.png@01D17AF7.D972DF70]  
[cid:image019.png@01D17AF7.D972DF70] 
> A Sunday Times Top Track 250 Company 2015
> Ranked in UK's 50 fastest growing technology companies 2014


* Email confidentiality *

This message is private and confidential. If you have received this message in 
error, please notify us and remove it from your system. The dissemination, 
copying or distribution of this message, or related files, by anyone other than 
the intended recipient is strictly prohibited.



Any views or opinions expressed are solely those of the author and do not 
necessarily represent those of Advanced 365 Limited.



* Email monitoring *

Advanced 365 Limited may monitor email traffic data and also the content of 
email for the purposes of security and staff training.



* Email security *

In keeping with good computing practice, the recipient of this email should 
ensure that it is virus-free. Advanced 365 Limited does not accept 
responsibility for any virus that may be transferred by way of this email.



Email may be susceptible to data corruption, interception and/or unauthorised 
amendment. Advanced 365 Limited does not accept liability for any such 
corruption, interception or amendment or any consequences thereof.



This email has been scanned for viruses by the Symantec Email Security.cloud 
service.



Advanced 365 Limited, part of the Advanced Computer Software Group

Registered office: Ditton Park, Riding Court Road, Datchet, Berkshire, SL3 9LL, 
UK

Registered in England under number 2124540



Please consider the environment: Think before you print!

Re: Qpid Proton-J client heartbeat (for ActiveMQ Artemis Broker)

[Robbie Gemmell]

I'm guesing when you say proton-j that you mean Messenger specifically?

The proton-j engine has support for AMQP heartbeating/idle-timeout (with
Artemis using that), and the Proton-J Reactor bits look to utilize that
support (so it should send them automatically if needed), but from grep of
the code it doesn't seem like Messenger does so it won't be able to send
heartbeats to satisfy the brokers idle-timeout, you would have to generate
traffic yourself as you described.

I'd suggest you look at either using the Reactor, or the Qpid JMS client,
or another use of the engine such as https://github.com/vert-x3/vertx-proton
.

Robbie

On 15 September 2016 at 12:59, Andy Redhead 
wrote:

> Hi,
>
>
>
> I’m (very) inexperienced with AMQP so this may be a silly question…
>
>
>
> I’m using Proton-J from a standalone Java program to interact with queues
> hosted by an ActiveMQ Artemis broker.
>
>
>
> So long as there is a regular message flow between the client and the
> broker, everything is fine.
>
>
>
> If there is a pause in the message flow (for 60 seconds, give or take)
> then the broker starts to complain, writing messages in the log such as:
>
>
>
> “2016-09-15T12:42:32,308 155833 [Thread-7 (ActiveMQ-server-org.apache.
> activemq.artemis.core.server.impl.ActiveMQServerImpl$2@2368231c-1759089888)]
> WARN  o.a.activemq.artemis.core.client - AMQ212037: Connection failure
> has been detected: AMQ119014: Did not receive data from /127.0.0.1:62229.
> It is likely the client has exited or crashed without closing its
> connection, or the network between the server and client has failed. You
> also might have configured connection-ttl and client-failure-check-period
> incorrectly. Please check user manual for more information. The connection
> will now be closed. [code=CONNECTION_TIMEDOUT]”
>
>
>
> I think I can work around this by making my clients send “heartbeat”
> messages on a regular basis, however if there is some kind of AMQP
> heartbeat capability in the Proton-J client I’d rather use that.
>
>
>
> Does Proton-J have the ability to send whatever an AMQP broker expects to
> receive as a heartbeat? If so, how do I set it up?
>
>
>
> Thanks, Andy
>
>
>
> [image: cid:image012.png@01D17AF7.D972DF70] 
>
> *Andy Redhead*
> Principal Consultant > Solutions > Advanced
> **
>
> *Advanced*
> 230 City Road, London, EC1V 2TT
> t: 020 7880  > m: 0781 392 5246
>
> www.oneadvanced.com
>
> [image: cid:image018.png@01D17AF7.D972DF70]
> [image:
> cid:image019.png@01D17AF7.D972DF70] 
>
> *>*
> * A Sunday Times Top Track 250 Company 2015 **>** Ranked in UK's 50
> fastest growing technology companies 2014*
>
>
>
> * Email confidentiality *
>
> This message is private and confidential. If you have received this
> message in error, please notify us and remove it from your system. The
> dissemination, copying or distribution of this message, or related files,
> by anyone other than the intended recipient is strictly prohibited.
>
>
>
> Any views or opinions expressed are solely those of the author and do not
> necessarily represent those of Advanced 365 Limited.
>
>
>
> * Email monitoring *
>
> Advanced 365 Limited may monitor email traffic data and also the content
> of email for the purposes of security and staff training.
>
>
>
> * Email security *
>
> In keeping with good computing practice, the recipient of this email
> should ensure that it is virus-free. Advanced 365 Limited does not accept
> responsibility for any virus that may be transferred by way of this email.
>
>
>
> Email may be susceptible to data corruption, interception and/or
> unauthorised amendment. Advanced 365 Limited does not accept liability for
> any such corruption, interception or amendment or any consequences thereof.
>
>
>
> This email has been scanned for viruses by the Symantec Email
> Security.cloud service.
>
>
>
> Advanced 365 Limited, part of the Advanced Computer Software Group
>
> Registered office: Ditton Park, Riding Court Road, Datchet, Berkshire, SL3
> 9LL, UK
>
> Registered in England under number 2124540
>
> --
>
> Please consider the environment: Think before you print!
>

RE: Qpid Proton-J client heartbeat (for ActiveMQ Artemis Broker)

[Andy Redhead]

Hi,

Yes, it is the Messenger (should have stated that in my original email).

I'll take a look at Reactor (though I have been meaning to have a look at 
vertx).

Cheers, Andy

-Original Message-
From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com]
Sent: 15 September 2016 15:18
To: users@qpid.apache.org
Subject: Re: Qpid Proton-J client heartbeat (for ActiveMQ Artemis Broker)

I'm guesing when you say proton-j that you mean Messenger specifically?

The proton-j engine has support for AMQP heartbeating/idle-timeout (with 
Artemis using that), and the Proton-J Reactor bits look to utilize that support 
(so it should send them automatically if needed), but from grep of the code it 
doesn't seem like Messenger does so it won't be able to send heartbeats to 
satisfy the brokers idle-timeout, you would have to generate traffic yourself 
as you described.

I'd suggest you look at either using the Reactor, or the Qpid JMS client, or 
another use of the engine such as https://github.com/vert-x3/vertx-proton
.

Robbie

On 15 September 2016 at 12:59, Andy Redhead 
wrote:

> Hi,
>
>
>
> I’m (very) inexperienced with AMQP so this may be a silly question…
>
>
>
> I’m using Proton-J from a standalone Java program to interact with
> queues hosted by an ActiveMQ Artemis broker.
>
>
>
> So long as there is a regular message flow between the client and the
> broker, everything is fine.
>
>
>
> If there is a pause in the message flow (for 60 seconds, give or take)
> then the broker starts to complain, writing messages in the log such as:
>
>
>
> “2016-09-15T12:42:32,308 155833 [Thread-7 (ActiveMQ-server-org.apache.
> activemq.artemis.core.server.impl.ActiveMQServerImpl$2@2368231c-175908
> 9888)] WARN  o.a.activemq.artemis.core.client - AMQ212037: Connection
> failure has been detected: AMQ119014: Did not receive data from
> /127.0.0.1:62229.
> It is likely the client has exited or crashed without closing its
> connection, or the network between the server and client has failed.
> You also might have configured connection-ttl and
> client-failure-check-period incorrectly. Please check user manual for
> more information. The connection will now be closed. 
> [code=CONNECTION_TIMEDOUT]”
>
>
>
> I think I can work around this by making my clients send “heartbeat”
> messages on a regular basis, however if there is some kind of AMQP
> heartbeat capability in the Proton-J client I’d rather use that.
>
>
>
> Does Proton-J have the ability to send whatever an AMQP broker expects
> to receive as a heartbeat? If so, how do I set it up?
>
>
>
> Thanks, Andy
>
>
>
> [image: cid:image012.png@01D17AF7.D972DF70]
> 
>
> *Andy Redhead*
> Principal Consultant > Solutions > Advanced
> **
>
> *Advanced*
> 230 City Road, London, EC1V 2TT
> t: 020 7880  > m: 0781 392 5246
>
> www.oneadvanced.com
>
> [image: cid:image018.png@01D17AF7.D972DF70]
> [image:
> cid:image019.png@01D17AF7.D972DF70]
> 
>
> *>*
> * A Sunday Times Top Track 250 Company 2015 **>** Ranked in UK's 50
> fastest growing technology companies 2014*
>
>
>
> * Email confidentiality *
>
> This message is private and confidential. If you have received this
> message in error, please notify us and remove it from your system. The
> dissemination, copying or distribution of this message, or related
> files, by anyone other than the intended recipient is strictly prohibited.
>
>
>
> Any views or opinions expressed are solely those of the author and do
> not necessarily represent those of Advanced 365 Limited.
>
>
>
> * Email monitoring *
>
> Advanced 365 Limited may monitor email traffic data and also the
> content of email for the purposes of security and staff training.
>
>
>
> * Email security *
>
> In keeping with good computing practice, the recipient of this email
> should ensure that it is virus-free. Advanced 365 Limited does not
> accept responsibility for any virus that may be transferred by way of this 
> email.
>
>
>
> Email may be susceptible to data corruption, interception and/or
> unauthorised amendment. Advanced 365 Limited does not accept liability
> for any such corruption, interception or amendment or any consequences 
> thereof.
>
>
>
> This email has been scanned for viruses by the Symantec Email
> Security.cloud service.
>
>
>
> Advanced 365 Limited, part of the Advanced Computer Software Group
>
> Registered office: Ditton Park, Riding Court Road, Datchet, Berkshire,
> SL3 9LL, UK
>
> Registered in England under number 2124540
>
> --
>
> Please consider the environment: Think before you print!
>
* Email confidentiality *

This message is private and confidential. If you have received this message in 
error, please notify us and remove it from your system. The dissemination, 
copying or distribution of this message, or related files, by anyone other than 
the intended