Re: Reactor Heartbeat [was: RE: Rapid generation of IOExceptions on Proton-J client recv() when HornetQ stopped/becomes unavailable]

[Robbie Gemmell]

On 8 April 2016 at 10:02, Matthew Karlsen  wrote:
> Hello Robbie (and All),
>
> Many thanks for the post.
>
> I am aware that HornetQ has issues w.r.t. ttl and so on. However, I thought I 
> could send it an empty frame on a regular basis and this would keep the 
> connection alive despite this -- perhaps I was mistaken?
>

I think the idea is fine but the execution has issues (more below).

> I have now tested the connection-ttl-override in Artemis and it works as 
> expected. Unfortunately Artemis is not an option at present... and 
> connection-ttl-override in HornetQ does not work.
>
>> It might be possible to tinker via reflection to trick it into thinking it 
>> should be.
>
> Thank you for the suggestion -- I will look in to this.
>
>> You also mention sending an empty frame, can you elaborate more on what you 
>> are doing exactly?
>
> Yes, I can. Incidentally, I modified handleFrame() to be public at present 
> (clearly not a long term solution). Then, on connection init
> I have the following code:
>
> transport = Proton.transport();
> transport.bind(connection);
> transport.setIdleTimeout(0);
>
> Open open = new Open();
> open.setIdleTimeOut(new UnsignedInteger(0));
> TransportFrame openFrame = new TransportFrame(0, open, null);
>  ((TransportImpl) transport).handleFrame(openFrame);
>
> [plus a Task t = event.getReactor().schedule(1, this); to schedule a 
> regular event ]
>
> Then in onTimerTask() I have the following:
>
> byte[] emptyFrame = (new AmqpFramer()).createEmptyFrame(0);
>  ((TransportImpl) transport).input(emptyFrame, 0, emptyFrame.length);
> Task t = e.getReactor().schedule(1, this);
>
> I have experimented with various timeout values but none have achieve the 
> desired result so far.
>

Ok, so you probably want to stop doing everything above :)

The handleFrame() method processings *incoming* frames, so you are
just generating empty frames locally and given them back to youurself,
thus HornetQ never saw them and they had no effect.

The reactor creates the Transport by itself, after onConnectionInit,
so none of that will be doing you any good.

I'd suggest looking at toying with reflection during
onConnectionRemoteOpen to tinker with the _remoteIdleTimeout value in
TransportImpl if it remains 0 (i.e none was advertised), it might be
possible to coax the trnaport into thinking one was advertised, and if
so the reactor should then begin sending heartbeats itself without any
task scheduling trickery. Disclaimer: I haven't tried this :)

>> If you look at the protocol trace (environment variable PN_TRACE_FRM=true) 
>> do you actually see it sending anything?
>
> Yes, at present I get a number of [[596512129:0] <- Empty Frame] messages -- 
> however, when I do the same thing with Artemis I get Empty Frames going in 
> both directions, so I suspect HornetQ is not responding correctly.
>

As per the above, that probably shows an incoming empty frame that you
generated locally yourself. The reason you see them going in both
directions with Artemis is likely that it has actually advertised an
idleTimeout (see the Open frame logging to confirm) and so the reactor
is taking care of sending heartbeats as needed to satisfy it. As
above, you probably want to look at tricking the transport into
thinking HornetQ actually advertised an idleTimeout too.

> Thank you & regards,
> Matthew
>
>
> -Original Message-
> From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com]
> Sent: 07 April 2016 13:14
> To: users@qpid.apache.org
> Subject: Re: Reactor Heartbeat [was: RE: Rapid generation of IOExceptions on 
> Proton-J client recv() when HornetQ stopped/becomes unavailable]
>
> Hi Matthew,
>
> I think your initial issue here is likely that HornetQ is neither requesting 
> clients send it heartbeats, or supporting sending heartbeats to them if it so 
> requested. It is however separately enforcing a timeout in its IO layer. I 
> say that based on the heartbeat handling only quite recently being fixed in 
> ActiveMQ Artemis, which is based on the previously donated HornetQ codebase.
>
> In AMQP 1.0 each side announces their own independent timeout requirements to 
> the other side, with each then doing what they need to satisfy the peers 
> requirements, either through normal messaging activity or by explicit 
> heartbeat/empty frames in its absense. The proton engine satisfies/enforces 
> these requirements via the transport tick method, as it seems you might be 
> aware, which checks that requested heartbeats have been received and required 
> heartbeats get sent, returning when it next needs to be called to keep on top 
> of that.
>
> The reactor takes care of calling tick itself as far as I am aware and that 
> being the case you also calling it should have no real effect on overall 
> behaviour, which seems to be what you observed. In any case, if the broker 
> has indeed not advertised an idleTimeout, the tick() will not produce 
> heartbeat/empty frames because it 

RE: Reactor Heartbeat [was: RE: Rapid generation of IOExceptions on Proton-J client recv() when HornetQ stopped/becomes unavailable]

[Matthew Karlsen]

Hello Robbie (and All),

Many thanks for the post.

I am aware that HornetQ has issues w.r.t. ttl and so on. However, I thought I 
could send it an empty frame on a regular basis and this would keep the 
connection alive despite this -- perhaps I was mistaken?

I have now tested the connection-ttl-override in Artemis and it works as 
expected. Unfortunately Artemis is not an option at present... and 
connection-ttl-override in HornetQ does not work.

> It might be possible to tinker via reflection to trick it into thinking it 
> should be.

Thank you for the suggestion -- I will look in to this.

> You also mention sending an empty frame, can you elaborate more on what you 
> are doing exactly?

Yes, I can. Incidentally, I modified handleFrame() to be public at present 
(clearly not a long term solution). Then, on connection init 
I have the following code:

transport = Proton.transport();
transport.bind(connection);
transport.setIdleTimeout(0);

Open open = new Open();
open.setIdleTimeOut(new UnsignedInteger(0));
TransportFrame openFrame = new TransportFrame(0, open, null);
 ((TransportImpl) transport).handleFrame(openFrame);

[plus a Task t = event.getReactor().schedule(1, this); to schedule a 
regular event ]

Then in onTimerTask() I have the following:

byte[] emptyFrame = (new AmqpFramer()).createEmptyFrame(0);
 ((TransportImpl) transport).input(emptyFrame, 0, emptyFrame.length);
Task t = e.getReactor().schedule(1, this);

I have experimented with various timeout values but none have achieve the 
desired result so far.

> If you look at the protocol trace (environment variable PN_TRACE_FRM=true) do 
> you actually see it sending anything?

Yes, at present I get a number of [[596512129:0] <- Empty Frame] messages -- 
however, when I do the same thing with Artemis I get Empty Frames going in both 
directions, so I suspect HornetQ is not responding correctly.

Thank you & regards,
Matthew


-Original Message-
From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] 
Sent: 07 April 2016 13:14
To: users@qpid.apache.org
Subject: Re: Reactor Heartbeat [was: RE: Rapid generation of IOExceptions on 
Proton-J client recv() when HornetQ stopped/becomes unavailable]

Hi Matthew,

I think your initial issue here is likely that HornetQ is neither requesting 
clients send it heartbeats, or supporting sending heartbeats to them if it so 
requested. It is however separately enforcing a timeout in its IO layer. I say 
that based on the heartbeat handling only quite recently being fixed in 
ActiveMQ Artemis, which is based on the previously donated HornetQ codebase.

In AMQP 1.0 each side announces their own independent timeout requirements to 
the other side, with each then doing what they need to satisfy the peers 
requirements, either through normal messaging activity or by explicit 
heartbeat/empty frames in its absense. The proton engine satisfies/enforces 
these requirements via the transport tick method, as it seems you might be 
aware, which checks that requested heartbeats have been received and required 
heartbeats get sent, returning when it next needs to be called to keep on top 
of that.

The reactor takes care of calling tick itself as far as I am aware and that 
being the case you also calling it should have no real effect on overall 
behaviour, which seems to be what you observed. In any case, if the broker has 
indeed not advertised an idleTimeout, the tick() will not produce 
heartbeat/empty frames because it has effectively been told it doesn't need to. 
It might be possible to tinker via reflection to trick it into thinking it 
should be.

You also mention sending an empty frame, can you elaborate more on what you are 
doing exactly? If you look at the protocol trace (envieonment variable 
PN_TRACE_FRM=true) do you actually see it sending anything? The heartbeating 
stuff is all hidden behind tick(), so are you achieving that via reflection?

Robbie

On 6 April 2016 at 10:20, Matthew Karlsen  wrote:
> Hello All,
>
> After being directed towards the Proton-J reactor (away from the messenger), 
> I have been experimenting with reactor. I modified the supplied reactor 
> examples to send/recv to/from a queue on HornetQ fairly quickly.
>
> However, I appear to be having less success implementing a 
> heartbeat/keep-alive (HornetQ 2.4.0 will disconnect after a one minute 
> default if it does not receive data from the client).
>
> I have scheduled a regular event using [ e.getReactor().schedule(1, 
> this); ] with a view to implementing a heartbeat. I then override 
> onTimerTask() and call tick() or send an empty frame within onTimerTask() 
> then re-schedule another call of onTimerTask() -- I have tried several 
> permutations.
>
> However, whether I use tick(), or send an empty frame, it does not seem to 
> maintain the connection.
>
> I was wondering:
> 1) Are there any better approaches?
> 2) What is the "correct" approach/location for using tick()?
>
> Thank you 

Re: Reactor Heartbeat [was: RE: Rapid generation of IOExceptions on Proton-J client recv() when HornetQ stopped/becomes unavailable]

[Robbie Gemmell]

Hi Matthew,

I think your initial issue here is likely that HornetQ is neither
requesting clients send it heartbeats, or supporting sending
heartbeats to them if it so requested. It is however separately
enforcing a timeout in its IO layer. I say that based on the heartbeat
handling only quite recently being fixed in ActiveMQ Artemis, which is
based on the previously donated HornetQ codebase.

In AMQP 1.0 each side announces their own independent timeout
requirements to the other side, with each then doing what they need to
satisfy the peers requirements, either through normal messaging
activity or by explicit heartbeat/empty frames in its absense. The
proton engine satisfies/enforces these requirements via the transport
tick method, as it seems you might be aware, which checks that
requested heartbeats have been received and required heartbeats get
sent, returning when it next needs to be called to keep on top of
that.

The reactor takes care of calling tick itself as far as I am aware and
that being the case you also calling it should have no real effect on
overall behaviour, which seems to be what you observed. In any case,
if the broker has indeed not advertised an idleTimeout, the tick()
will not produce heartbeat/empty frames because it has effectively
been told it doesn't need to. It might be possible to tinker via
reflection to trick it into thinking it should be.

You also mention sending an empty frame, can you elaborate more on
what you are doing exactly? If you look at the protocol trace
(envieonment variable PN_TRACE_FRM=true) do you actually see it
sending anything? The heartbeating stuff is all hidden behind tick(),
so are you achieving that via reflection?

Robbie

On 6 April 2016 at 10:20, Matthew Karlsen  wrote:
> Hello All,
>
> After being directed towards the Proton-J reactor (away from the messenger), 
> I have been experimenting with reactor. I modified the supplied reactor 
> examples to send/recv to/from a queue on HornetQ fairly quickly.
>
> However, I appear to be having less success implementing a 
> heartbeat/keep-alive (HornetQ 2.4.0 will disconnect after a one minute 
> default if it does not receive data from the client).
>
> I have scheduled a regular event using [ e.getReactor().schedule(1, 
> this); ] with a view to implementing a heartbeat. I then override 
> onTimerTask() and call tick() or send an empty frame within onTimerTask() 
> then re-schedule another call of onTimerTask() -- I have tried several 
> permutations.
>
> However, whether I use tick(), or send an empty frame, it does not seem to 
> maintain the connection.
>
> I was wondering:
> 1) Are there any better approaches?
> 2) What is the "correct" approach/location for using tick()?
>
> Thank you & regards,
> Matthew
>
>
>
> -Original Message-
> From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com]
> Sent: 18 March 2016 17:59
> To: users@qpid.apache.org
> Subject: Re: Rapid generation of IOExceptions on Proton-J client recv() when 
> HornetQ stopped/becomes unavailable
>
> On 18 March 2016 at 12:15, Matthew Karlsen  wrote:
>> Hello All,
>>
>> We have a queue running in HornetQ 2.4.0 (embedded within Wildfly), with a 
>> Proton-J 0.12.0 client periodically connecting to this queue.
>>
>> If HornetQ stops or becomes unavailable when the Proton-J client is running, 
>> the Proton-J client continually generates exceptions similar to that below 
>> (very rapidly).
>>
>> The issue is that the IOException generated exception is handled in
>> MessengerImpl's processActive() by generating an "Error processing 
>> connection" message in the logs, rather than recv() throwing an exception 
>> and hence the higher level program logic is unable to react to the problem.
>>
>> Is this a bug or am I missing something important?
>>
>> Thank you,
>> Matthew
>>
>> java.io.IOException: An existing connection was forcibly closed by the 
>> remote host
>> at sun.nio.ch.SocketDispatcher.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> at 
>> org.apache.qpid.proton.driver.impl.ConnectorImpl.read(ConnectorImpl.java:129)
>> at 
>> org.apache.qpid.proton.driver.impl.ConnectorImpl.process(ConnectorImpl.java:94)
>> at 
>> org.apache.qpid.proton.messenger.impl.MessengerImpl.processActive(MessengerImpl.java:738)
>> at 
>> org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:895)
>> at 
>> org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:844)
>> at 
>> org.apache.qpid.proton.messenger.impl.MessengerImpl.recv(MessengerImpl.java:446)
>> at 
>> org.apache.qpid.proton.messenger.impl.MessengerImpl.recv(MessengerImpl.java:451)
>>   

Reactor Heartbeat [was: RE: Rapid generation of IOExceptions on Proton-J client recv() when HornetQ stopped/becomes unavailable]

[Matthew Karlsen]

Hello All,

After being directed towards the Proton-J reactor (away from the messenger), I 
have been experimenting with reactor. I modified the supplied reactor examples 
to send/recv to/from a queue on HornetQ fairly quickly.

However, I appear to be having less success implementing a heartbeat/keep-alive 
(HornetQ 2.4.0 will disconnect after a one minute default if it does not 
receive data from the client). 

I have scheduled a regular event using [ e.getReactor().schedule(1, this); 
] with a view to implementing a heartbeat. I then override onTimerTask() and 
call tick() or send an empty frame within onTimerTask() then re-schedule 
another call of onTimerTask() -- I have tried several permutations. 

However, whether I use tick(), or send an empty frame, it does not seem to 
maintain the connection.

I was wondering:
1) Are there any better approaches?
2) What is the "correct" approach/location for using tick()?

Thank you & regards,
Matthew



-Original Message-
From: Robbie Gemmell [mailto:robbie.gemm...@gmail.com] 
Sent: 18 March 2016 17:59
To: users@qpid.apache.org
Subject: Re: Rapid generation of IOExceptions on Proton-J client recv() when 
HornetQ stopped/becomes unavailable

On 18 March 2016 at 12:15, Matthew Karlsen  wrote:
> Hello All,
>
> We have a queue running in HornetQ 2.4.0 (embedded within Wildfly), with a 
> Proton-J 0.12.0 client periodically connecting to this queue.
>
> If HornetQ stops or becomes unavailable when the Proton-J client is running, 
> the Proton-J client continually generates exceptions similar to that below 
> (very rapidly).
>
> The issue is that the IOException generated exception is handled in 
> MessengerImpl's processActive() by generating an "Error processing 
> connection" message in the logs, rather than recv() throwing an exception and 
> hence the higher level program logic is unable to react to the problem.
>
> Is this a bug or am I missing something important?
>
> Thank you,
> Matthew
>
> java.io.IOException: An existing connection was forcibly closed by the remote 
> host
> at sun.nio.ch.SocketDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> at 
> org.apache.qpid.proton.driver.impl.ConnectorImpl.read(ConnectorImpl.java:129)
> at 
> org.apache.qpid.proton.driver.impl.ConnectorImpl.process(ConnectorImpl.java:94)
> at 
> org.apache.qpid.proton.messenger.impl.MessengerImpl.processActive(MessengerImpl.java:738)
> at 
> org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:895)
> at 
> org.apache.qpid.proton.messenger.impl.MessengerImpl.waitUntil(MessengerImpl.java:844)
> at 
> org.apache.qpid.proton.messenger.impl.MessengerImpl.recv(MessengerImpl.java:446)
> at 
> org.apache.qpid.proton.messenger.impl.MessengerImpl.recv(MessengerImpl.java:451)
> ...
> at java.lang.Thread.run(Thread.java:745)
>
> -
> To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For 
> additional commands, e-mail: users-h...@qpid.apache.org
>

Hi Matthew,

It certainly doesnt seem desirable, I'd guess its a bug, though someone 
actually familiar with Messenger could certainly correct me on that.

Messenger isn't widely used on the proton-j side, and in general Messenger 
isn't getting much developer attention these days (arguably never did in 
proton-j, which is mostly still used as an pure protocol engine as intended at 
the outset) since folks are concentrated more on newer reactive APIs.

Proton-J is a bit behind the curve in terms of the higher level reactive work 
being/already done in some of the other lanaguages (something I'll be looking 
to get to), but there was a Java port done of the 'reactor' that exists in 
proton-c and some of its bindings. I'm not particularly familiar with it 
either, or how it would handle this situation, but it may be worth you looking 
at as alternative to Messenger. You can see some example of it in the 
examples/java/reactor dir, e.g. the Send class (the Recv class in this case is 
actually a server/listener that accepts incoming connections, such as those 
made by Send).

Robbie

-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional 
commands, e-mail: users-h...@qpid.apache.org