Hi Sree, Your attachment didnt make it, the mailing list strips almost all attachments, particularly of any non trivial size. Best to attach stuff to JIRAs or provide links to them elsewhere.
I had another look at it and still don't see a way for it to do what it did before, I'd guess it may be something slightly different. I think I'll need more analysis and/or a reproducer from you to make any further progress. 0.12.2 currently has the votes to pass. If it stays that way I probably lean towards proceeding with its release unless more concrete details becomes obvious, since it does seem to give a good improvement over the previous behaviour. Robbie On 15 April 2016 at 01:35, Garlapati Sreeram Kumar <sreer...@live.com> wrote: > Hello Robbie, > > > > After couple hours of Stress Run with the proton-j change – we still could > repro the Receive Stuck problem. Although – the below bug fix places us in a > much better state now. > > Attached a screenshot of Objects sizes on Heap – which corresponds to the > codepath that you fixed (SimpleSslTransportWrapper). > > Please see if this rings any bells – I am more than happy to share more > details (proton traces & dump – pls suggest if you will need more details). > > > > Thx! > > Sree > > > > From: Garlapati Sreeram Kumar > Sent: Monday, April 11, 2016 11:50 AM > To: Robbie Gemmell; proton@qpid.apache.org > Cc: SeongJoon Kwak (SJ); hm...@microsoft.com > Subject: RE: Proton-j Reactor - Receiver > > > > Awesome. > > To make it easy - added you as collaborator to my fork of Proton & here’s > the branch from which I submitted the PR: > https://github.com/SreeramGarlapati/qpid-proton/tree/sg.recvstuck > > Thx! > Sree > > From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> > Sent: Monday, April 11, 2016 9:52 AM > To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> > Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; > hm...@microsoft.com<mailto:hm...@microsoft.com> > Subject: Re: Proton-j Reactor - Receiver > > Ah, excellent. I had actually started on testing this myself a little > earlier, so I'll take a look and see whats what before continuing > tomorrow. On taking an initial better look at things I think the > change itself may need augmented to account for some other conditions > too, need to investigate further to be sure. > > Robbie > > On 11 April 2016 at 17:37, Garlapati Sreeram Kumar <sreer...@live.com> > wrote: >> Thanks a lot for the Response Robbie! >> Per your suggestion, added the CIT to the Pull Request (& yes, as you >> already said – this issue is being tracked via JIRA - PROTON-1171). >> >> Thanks a lot for the Wonderful Collaboration! >> Sree >> >> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> >> Sent: Thursday, April 7, 2016 3:52 AM >> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> >> Cc: SeongJoon Kwak (SJ)<mailto:sjk...@microsoft.com>; >> hm...@microsoft.com<mailto:hm...@microsoft.com> >> Subject: Re: Proton-j Reactor - Receiver >> >> Hi Sree, >> >> Thanks for the analysis and PR, I'll try to take a proper look soon. >> It's not an area of the code I'm familiar with so I'll need to have a >> bit of a dig myself to see if the change seems ok. I'd note that any >> not-insignificant bug fix such as this should probably have a test >> with it (and a JIRA, though I see you have since created one of those) >> :) >> >> Robbie >> >> On 6 April 2016 at 01:23, Garlapati Sreeram Kumar <sreer...@live.com> >> wrote: >>> Hello Robbie, >>> >>> We are using proton-j client with SSL and many of our customers are >>> hitting this issue. >>> Here are my findings after debugging through this issue: >>> >>> - When incoming bytes arrive on the SocketChannel – proton-j >>> client gets signaled by nio & as a result it unwinds the transport stack – >>> as a result all the TransportInput implementations performs its task on the >>> Read Bytes and hands off to the Next Layer in the stack (transport to ssl, >>> ssl to frameparser etc). >>> >>> - While unwinding that stack, >>> SimpleSSLTransportWrapper.unwrapInput reads(16k bytes) from _inputBuffer and >>> the result - decoded bytes are written to _decodedInputBuffer – as an >>> intermediate buffer. >>> >>> - It then flushes bytes from intermediate buffer to the next >>> layer & invokes an _underlyingInput.Process() – to signal it that it has >>> bytes in its input buffer. >>> >>> - If the underlyingInput (lets say FrameParser) buffer size is >>> small – lets say 4k – then decodedInputBuffer will be left with 12k bytes & >>> Over time this accrues. >>> >>> The fix here is to flush decodedInputBuffer to the Next transport in the >>> Network Stack & call _underlyingInput.Process() - until decodedInputBuffer >>> is empty. Here’s the pull request - >>> https://github.com/apache/qpid-proton/pull/73 >>> >>> Pl. let me know if we need to do more to fix this issue comprehensively. >>> >>> Thx! >>> Sree >>> >>> From: Robbie Gemmell<mailto:robbie.gemm...@gmail.com> >>> Sent: Thursday, March 31, 2016 9:19 AM >>> To: proton@qpid.apache.org<mailto:proton@qpid.apache.org> >>> Subject: Re: Proton-j Reactor - Receiver >>> >>> On 31 March 2016 at 04:32, Garlapati Sreeram Kumar <sreer...@live.com> >>> wrote: >>>> Hello All! >>>> >>>> I am using Proton-J reactor API (Version 0.12.0) for receiving AMQP >>>> Messages (from Microsoft Azure Event Hubs): >>>> https://github.com/Azure/azure-event-hubs/blob/master/java/azure-eventhubs/src/main/java/com/microsoft/azure/servicebus/amqp/ReceiveLinkHandler.java#L124 >>>> >>>> Am using the onDelivery(Event) callback to receive messages. I really >>>> appreciate your help with this issue/behavior: >>>> >>>> ISSUE: I noticed that the last few messages on the Queue are not being >>>> issued to onDelivery(Event) callback by the Reactor >>>> - Then, I went ahead and enabled proton Frame tracing (PN_TRACE_FRM=1) >>>> and discovered that the Transfer frames corresponding to those messages >>>> were >>>> not even delivered to Client. Then, I looked at our Service Proton Frames >>>> and can clearly see that they are being delivered by the Service. And other >>>> AMQP clients (for ex: .net client can see the Transfer frames) >>>> - Is this a known behavior? >>>> Does Reactor code path disable Nagle on underlying socket – could this >>>> be related? or is there any other Configuration that we should be setting >>>> to >>>> see all Transfer frames received on the Socket? >>>> >>>> Please advice. >>>> >>>> Thanks a lot in Advance! >>>> Sree >>>> >>>> Sent from Mail for Windows 10 >>>> >>> >>> I'm not aware of anyone else reporting anything like that. I don't see >>> anything in the code suggesting the reactor sets TCP_NODELAY trueon >>> the socket, but I wouldn't think that should matter here. >>> >>> The frame trace logging is done after the bytes are given to the >>> Transport and are processed into frames, so a lack of logging could >>> suggest various things such as they didnt actually get there, they >>> werent processed, something went wrong before they did/were, something >>> went wrong decoding them, etc. Its hard to say much more without more >>> info. >>> >>> Robbie
