That means from isEndOfMsgAdded what we can do is only stop further waiting
on the blocking queue. But still we have to read all remaining content from
the queue.
On Thu, Jun 2, 2016 at 9:52 PM, Samiyuru Senarathne <samiy...@wso2.com>
wrote:
> I assume isEndOfMsgAdded only tell that all the chunks are loaded to the
> blocking queue. So still we have to complete the total blocking queue.
>
> On Thu, Jun 2, 2016 at 9:50 PM, Afkham Azeez <az...@wso2.com> wrote:
>
>> while(!carbonMsg.isEndOfMsgAdded) {
>> ByteBuffer chunk = carbonMsg.getMessageBody();
>>
>> }
>>
>> In the above segment, carbonMsg is the request CarbonMessage and the
>> above code segment doesn't work. Code inside the while loop never executes.
>>
>>
>> On Thu, Jun 2, 2016 at 9:47 PM, Isuru Ranawaka <isu...@wso2.com> wrote:
>>
>>> Hi Azeez,
>>> yes that should work. Anyhow content needs to be added to carbonMsg
>>> from one thread and read to be from another thread.
>>>
>>> thanks
>>>
>>> On Thu, Jun 2, 2016 at 9:23 PM, Afkham Azeez <az...@wso2.com> wrote:
>>>
>>>> Is the following code segment the way to read all the chunks;
>>>>
>>>> while(!carbonMsg.isEndOfMsgAdded) {
>>>> ByteBuffer chunk = carbonMsg.getMessageBody();
>>>>
>>>> }
>>>>
>>>> On Thu, Jun 2, 2016 at 9:21 PM, Afkham Azeez <az...@wso2.com> wrote:
>>>>
>>>>> Looks like we require changes for input streaming as well.
>>>>>
>>>>> Does CarbonMessage.getMessageBody() block until the next chunk is
>>>>> received?
>>>>>
>>>>> On Thu, Jun 2, 2016 at 9:18 PM, Afkham Azeez <az...@wso2.com> wrote:
>>>>>
>>>>>> Samiyuru/Isuru,
>>>>>> Does this mean that the input streaming in MSF4J is currently working
>>>>>> without any issue and only output streaming requires some work?
>>>>>>
>>>>>> On Tue, May 3, 2016 at 2:29 PM, Isuru Ranawaka <isu...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>
>>>>>>> Following are the details on streaming support of carbon transport.
>>>>>>> Basically we can looking to streaming support in request path and
>>>>>>> response
>>>>>>> path separately.
>>>>>>>
>>>>>>> Request Path (Streaming is working)
>>>>>>>
>>>>>>> [image: requestpath.png]
>>>>>>>
>>>>>>> According to above diagram
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> We have blocking queue in the carbon message which keeps the
>>>>>>> content . when headers are received through Netty worker thread it
>>>>>>> will
>>>>>>> create CarbonMessage with a blocking queue and publish carbon
>>>>>>> message to
>>>>>>> engine level thread.
>>>>>>> -
>>>>>>>
>>>>>>> Reference for blocking queue is cached in the connection and
>>>>>>> when content is received it will be filled to that queue from Netty
>>>>>>> worker
>>>>>>> thread.
>>>>>>> -
>>>>>>>
>>>>>>> Meanwhile engine level threads can consume content through
>>>>>>> queue(While IO worker is filling ) and can directly send to file
>>>>>>> system or
>>>>>>> use sender for send messages to external service.
>>>>>>> -
>>>>>>>
>>>>>>> According to that streaming should work in Request path in
>>>>>>> Integration Server or MSF4J without any problem.
>>>>>>>
>>>>>>>
>>>>>>> Response Path (Streaming working scenario)
>>>>>>>
>>>>>>> [image: responsepathworking.png]
>>>>>>>
>>>>>>>
>>>>>>> In Response path basic difference we have is we are handling
>>>>>>> responses through callbacks.
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> Similar to Request path architecture Sender side IO thread
>>>>>>> creates a CM when response headers are received and publish to
>>>>>>> engine
>>>>>>> level thread .
>>>>>>> -
>>>>>>>
>>>>>>> Engine level thread calls carbonCallback.done() and waits on
>>>>>>> queue for content.
>>>>>>> -
>>>>>>>
>>>>>>> Meanwhile IO thread writes the content to the Queue .
>>>>>>> -
>>>>>>>
>>>>>>> So writing to Queue and reading from queue happens parallel and
>>>>>>> streaming should work properly.
>>>>>>> -
>>>>>>>
>>>>>>> So streaming is working fine with Integration Server for this
>>>>>>> kind of scenarios.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Response Path (Streaming not working scenario)
>>>>>>>
>>>>>>> [image: responsepathnotworking.png]
>>>>>>>
>>>>>>> This scenario is basically, if we have a Echo mediator or assume
>>>>>>> MSF4J has a service which is reading a file as chunks and writes back
>>>>>>> to
>>>>>>> client .
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> Basic difference with the previous one is in this approach
>>>>>>> reading a file chunk or stream and writing that file chunk or
>>>>>>> stream to
>>>>>>> Queue is happened within the same engine level thread as well as
>>>>>>> with
>>>>>>> reading from queue and writing to Listener side Netty worker
>>>>>>> threads.(Same
>>>>>>> thread produce and consume causes for deadlock)
>>>>>>> -
>>>>>>>
>>>>>>> In the previous example reading from stream and filling the
>>>>>>> queue was happened through Sender side IO thread and consuming the
>>>>>>> queue
>>>>>>> and writing to Listener side IO thread was happened through engine
>>>>>>> level
>>>>>>> thread. (Two different threads produce and consume)
>>>>>>> -
>>>>>>>
>>>>>>> Streaming is not working in this kind of scenarios because we
>>>>>>> cannot write to queue and keep polling the queue within single
>>>>>>> thread.
>>>>>>>
>>>>>>>
>>>>>>> We can figure out following solutions and what will be the most
>>>>>>> suitable one.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> Introduce another thread for run the callback logic instead of
>>>>>>> running through calling thread.
>>>>>>>
>>>>>>> This will solve the above streaming problem but when
>>>>>>> it comes to general message flow this will add another level of thread
>>>>>>> which will actually does not need.
>>>>>>>
>>>>>>> -
>>>>>>>
>>>>>>> Introduce another method in ResponseCallback which will support
>>>>>>> streaming which does not queuing contents. it will directly call IO
>>>>>>> threads and write contents to IO when chunks are received. This can
>>>>>>> be used
>>>>>>> only within single thread scenario(File reading and writing ).
>>>>>>> -
>>>>>>>
>>>>>>> Introduce another thread for read File stream or call Callback
>>>>>>> from another thread other than to reading thread from MSF4J
>>>>>>> level.Then it
>>>>>>> will be equivalent to how we used it in Integration Server with the
>>>>>>> use of
>>>>>>> Sender.
>>>>>>>
>>>>>>> ThankYou
>>>>>>>
>>>>>>> IsuruR
>>>>>>>
>>>>>>> On Tue, May 3, 2016 at 12:09 PM, Kasun Indrasiri <ka...@wso2.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Had a chat with Ranawaka on this.. seems like we do have couple of
>>>>>>>> ways to handle this. He will share the details.
>>>>>>>>
>>>>>>>> On Mon, May 2, 2016 at 6:13 AM, Afkham Azeez <az...@wso2.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The problem is, we can't do the next MSF4J release because we
>>>>>>>>> can't lose a feature in a release. This is a blocker for us. We have
>>>>>>>>> plans
>>>>>>>>> to release in the next 2 weeks.
>>>>>>>>>
>>>>>>>>> On Mon, May 2, 2016 at 4:56 PM, Isuru Ranawaka <isu...@wso2.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Azeez,
>>>>>>>>>>
>>>>>>>>>> Currently streaming works if we used both sender side and
>>>>>>>>>> listener side only . But since MSF4J is using only listener side if
>>>>>>>>>> we did
>>>>>>>>>> not spawn separate thread from engine level for writing response it
>>>>>>>>>> will
>>>>>>>>>> not work because request reading and writing happens through same
>>>>>>>>>> thread.
>>>>>>>>>> But with the next release we will fix that. currently we are
>>>>>>>>>> finalizing on
>>>>>>>>>> removing engine level thread model from transport.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> thanks
>>>>>>>>>> IsuruR
>>>>>>>>>>
>>>>>>>>>> On Mon, May 2, 2016 at 3:50 PM, Afkham Azeez <az...@wso2.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Is this working after moving to the new transport framework?
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> *Afkham Azeez*
>>>>>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>>>>>>> * <http://www.apache.org/>*
>>>>>>>>>>> *email: **az...@wso2.com* <az...@wso2.com>
>>>>>>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>>>>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>>>>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>>>>>>>> <http://twitter.com/afkham_azeez>
>>>>>>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>>>>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>>>>>>>
>>>>>>>>>>> *Lean . Enterprise . Middleware*
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Dev mailing list
>>>>>>>>>>> Dev@wso2.org
>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Best Regards
>>>>>>>>>> Isuru Ranawaka
>>>>>>>>>> M: +94714629880
>>>>>>>>>> Blog : http://isurur.blogspot.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Afkham Azeez*
>>>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>>>>> * <http://www.apache.org/>*
>>>>>>>>> *email: **az...@wso2.com* <az...@wso2.com>
>>>>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>>>>>> <http://twitter.com/afkham_azeez>
>>>>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>>>>>
>>>>>>>>> *Lean . Enterprise . Middleware*
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Dev mailing list
>>>>>>>>> Dev@wso2.org
>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Kasun Indrasiri
>>>>>>>> Software Architect
>>>>>>>> WSO2, Inc.; http://wso2.com
>>>>>>>> lean.enterprise.middleware
>>>>>>>>
>>>>>>>> cell: +94 77 556 5206
>>>>>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards
>>>>>>> Isuru Ranawaka
>>>>>>> M: +94714629880
>>>>>>> Blog : http://isurur.blogspot.com/
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Afkham Azeez*
>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>> * <http://www.apache.org/>*
>>>>>> *email: **az...@wso2.com* <az...@wso2.com>
>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>>> <http://twitter.com/afkham_azeez>
>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>>
>>>>>> *Lean . Enterprise . Middleware*
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Afkham Azeez*
>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>> * <http://www.apache.org/>*
>>>>> *email: **az...@wso2.com* <az...@wso2.com>
>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>> <http://twitter.com/afkham_azeez>
>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>
>>>>> *Lean . Enterprise . Middleware*
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *Afkham Azeez*
>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>> * <http://www.apache.org/>*
>>>> *email: **az...@wso2.com* <az...@wso2.com>
>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>> <http://twitter.com/afkham_azeez>
>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>
>>>> *Lean . Enterprise . Middleware*
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards
>>> Isuru Ranawaka
>>> M: +94714629880
>>> Blog : http://isurur.blogspot.com/
>>>
>>
>>
>>
>> --
>> *Afkham Azeez*
>> Director of Architecture; WSO2, Inc.; http://wso2.com
>> Member; Apache Software Foundation; http://www.apache.org/
>> * <http://www.apache.org/>*
>> *email: **az...@wso2.com* <az...@wso2.com>
>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>> *http://blog.afkham.org* <http://blog.afkham.org>
>> *twitter: **http://twitter.com/afkham_azeez*
>> <http://twitter.com/afkham_azeez>
>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>> <http://lk.linkedin.com/in/afkhamazeez>*
>>
>> *Lean . Enterprise . Middleware*
>>
>
>
>
> --
> Samiyuru Senarathne
> *Software Engineer*
> Mobile : +94 (0) 71 134 6087
> samiy...@wso2.com
>
--
Samiyuru Senarathne
*Software Engineer*
Mobile : +94 (0) 71 134 6087
samiy...@wso2.com
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
