Hi Azeez,
yes that should work. Anyhow content needs to be added to carbonMsg from
one thread and read to be from another thread.
thanks
On Thu, Jun 2, 2016 at 9:23 PM, Afkham Azeez <az...@wso2.com> wrote:
> Is the following code segment the way to read all the chunks;
>
> while(!carbonMsg.isEndOfMsgAdded) {
> ByteBuffer chunk = carbonMsg.getMessageBody();
>
> }
>
> On Thu, Jun 2, 2016 at 9:21 PM, Afkham Azeez <az...@wso2.com> wrote:
>
>> Looks like we require changes for input streaming as well.
>>
>> Does CarbonMessage.getMessageBody() block until the next chunk is
>> received?
>>
>> On Thu, Jun 2, 2016 at 9:18 PM, Afkham Azeez <az...@wso2.com> wrote:
>>
>>> Samiyuru/Isuru,
>>> Does this mean that the input streaming in MSF4J is currently working
>>> without any issue and only output streaming requires some work?
>>>
>>> On Tue, May 3, 2016 at 2:29 PM, Isuru Ranawaka <isu...@wso2.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> Following are the details on streaming support of carbon transport.
>>>> Basically we can looking to streaming support in request path and response
>>>> path separately.
>>>>
>>>> Request Path (Streaming is working)
>>>>
>>>> [image: requestpath.png]
>>>>
>>>> According to above diagram
>>>>
>>>> -
>>>>
>>>> We have blocking queue in the carbon message which keeps the
>>>> content . when headers are received through Netty worker thread it will
>>>> create CarbonMessage with a blocking queue and publish carbon message to
>>>> engine level thread.
>>>> -
>>>>
>>>> Reference for blocking queue is cached in the connection and when
>>>> content is received it will be filled to that queue from Netty worker
>>>> thread.
>>>> -
>>>>
>>>> Meanwhile engine level threads can consume content through
>>>> queue(While IO worker is filling ) and can directly send to file system
>>>> or
>>>> use sender for send messages to external service.
>>>> -
>>>>
>>>> According to that streaming should work in Request path in
>>>> Integration Server or MSF4J without any problem.
>>>>
>>>>
>>>> Response Path (Streaming working scenario)
>>>>
>>>> [image: responsepathworking.png]
>>>>
>>>>
>>>> In Response path basic difference we have is we are handling responses
>>>> through callbacks.
>>>>
>>>>
>>>> -
>>>>
>>>> Similar to Request path architecture Sender side IO thread
>>>> creates a CM when response headers are received and publish to engine
>>>> level thread .
>>>> -
>>>>
>>>> Engine level thread calls carbonCallback.done() and waits on queue
>>>> for content.
>>>> -
>>>>
>>>> Meanwhile IO thread writes the content to the Queue .
>>>> -
>>>>
>>>> So writing to Queue and reading from queue happens parallel and
>>>> streaming should work properly.
>>>> -
>>>>
>>>> So streaming is working fine with Integration Server for this kind
>>>> of scenarios.
>>>>
>>>>
>>>>
>>>> Response Path (Streaming not working scenario)
>>>>
>>>> [image: responsepathnotworking.png]
>>>>
>>>> This scenario is basically, if we have a Echo mediator or assume MSF4J
>>>> has a service which is reading a file as chunks and writes back to client
>>>> .
>>>>
>>>>
>>>> -
>>>>
>>>> Basic difference with the previous one is in this approach reading
>>>> a file chunk or stream and writing that file chunk or stream to Queue
>>>> is
>>>> happened within the same engine level thread as well as with reading
>>>> from
>>>> queue and writing to Listener side Netty worker threads.(Same thread
>>>> produce and consume causes for deadlock)
>>>> -
>>>>
>>>> In the previous example reading from stream and filling the queue
>>>> was happened through Sender side IO thread and consuming the queue and
>>>> writing to Listener side IO thread was happened through engine level
>>>> thread. (Two different threads produce and consume)
>>>> -
>>>>
>>>> Streaming is not working in this kind of scenarios because we
>>>> cannot write to queue and keep polling the queue within single thread.
>>>>
>>>>
>>>> We can figure out following solutions and what will be the most
>>>> suitable one.
>>>>
>>>>
>>>>
>>>> -
>>>>
>>>> Introduce another thread for run the callback logic instead of
>>>> running through calling thread.
>>>>
>>>> This will solve the above streaming problem but when it
>>>> comes to general message flow this will add another level of thread which
>>>> will actually does not need.
>>>>
>>>> -
>>>>
>>>> Introduce another method in ResponseCallback which will support
>>>> streaming which does not queuing contents. it will directly call IO
>>>> threads and write contents to IO when chunks are received. This can be
>>>> used
>>>> only within single thread scenario(File reading and writing ).
>>>> -
>>>>
>>>> Introduce another thread for read File stream or call Callback from
>>>> another thread other than to reading thread from MSF4J level.Then it
>>>> will
>>>> be equivalent to how we used it in Integration Server with the use of
>>>> Sender.
>>>>
>>>> ThankYou
>>>>
>>>> IsuruR
>>>>
>>>> On Tue, May 3, 2016 at 12:09 PM, Kasun Indrasiri <ka...@wso2.com>
>>>> wrote:
>>>>
>>>>> Had a chat with Ranawaka on this.. seems like we do have couple of
>>>>> ways to handle this. He will share the details.
>>>>>
>>>>> On Mon, May 2, 2016 at 6:13 AM, Afkham Azeez <az...@wso2.com> wrote:
>>>>>
>>>>>> The problem is, we can't do the next MSF4J release because we can't
>>>>>> lose a feature in a release. This is a blocker for us. We have plans to
>>>>>> release in the next 2 weeks.
>>>>>>
>>>>>> On Mon, May 2, 2016 at 4:56 PM, Isuru Ranawaka <isu...@wso2.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Azeez,
>>>>>>>
>>>>>>> Currently streaming works if we used both sender side and listener
>>>>>>> side only . But since MSF4J is using only listener side if we did not
>>>>>>> spawn
>>>>>>> separate thread from engine level for writing response it will not work
>>>>>>> because request reading and writing happens through same thread. But
>>>>>>> with
>>>>>>> the next release we will fix that. currently we are finalizing on
>>>>>>> removing
>>>>>>> engine level thread model from transport.
>>>>>>>
>>>>>>>
>>>>>>> thanks
>>>>>>> IsuruR
>>>>>>>
>>>>>>> On Mon, May 2, 2016 at 3:50 PM, Afkham Azeez <az...@wso2.com> wrote:
>>>>>>>
>>>>>>>> Is this working after moving to the new transport framework?
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Afkham Azeez*
>>>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>>>> * <http://www.apache.org/>*
>>>>>>>> *email: **az...@wso2.com* <az...@wso2.com>
>>>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>>>>> <http://twitter.com/afkham_azeez>
>>>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>>>>
>>>>>>>> *Lean . Enterprise . Middleware*
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Dev mailing list
>>>>>>>> Dev@wso2.org
>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Best Regards
>>>>>>> Isuru Ranawaka
>>>>>>> M: +94714629880
>>>>>>> Blog : http://isurur.blogspot.com/
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Afkham Azeez*
>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>>>>> Member; Apache Software Foundation; http://www.apache.org/
>>>>>> * <http://www.apache.org/>*
>>>>>> *email: **az...@wso2.com* <az...@wso2.com>
>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>>>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>>>>> *twitter: **http://twitter.com/afkham_azeez*
>>>>>> <http://twitter.com/afkham_azeez>
>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>>>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>>>>
>>>>>> *Lean . Enterprise . Middleware*
>>>>>>
>>>>>> _______________________________________________
>>>>>> Dev mailing list
>>>>>> Dev@wso2.org
>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Kasun Indrasiri
>>>>> Software Architect
>>>>> WSO2, Inc.; http://wso2.com
>>>>> lean.enterprise.middleware
>>>>>
>>>>> cell: +94 77 556 5206
>>>>> Blog : http://kasunpanorama.blogspot.com/
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best Regards
>>>> Isuru Ranawaka
>>>> M: +94714629880
>>>> Blog : http://isurur.blogspot.com/
>>>>
>>>
>>>
>>>
>>> --
>>> *Afkham Azeez*
>>> Director of Architecture; WSO2, Inc.; http://wso2.com
>>> Member; Apache Software Foundation; http://www.apache.org/
>>> * <http://www.apache.org/>*
>>> *email: **az...@wso2.com* <az...@wso2.com>
>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>>> *http://blog.afkham.org* <http://blog.afkham.org>
>>> *twitter: **http://twitter.com/afkham_azeez*
>>> <http://twitter.com/afkham_azeez>
>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>>> <http://lk.linkedin.com/in/afkhamazeez>*
>>>
>>> *Lean . Enterprise . Middleware*
>>>
>>
>>
>>
>> --
>> *Afkham Azeez*
>> Director of Architecture; WSO2, Inc.; http://wso2.com
>> Member; Apache Software Foundation; http://www.apache.org/
>> * <http://www.apache.org/>*
>> *email: **az...@wso2.com* <az...@wso2.com>
>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
>> *http://blog.afkham.org* <http://blog.afkham.org>
>> *twitter: **http://twitter.com/afkham_azeez*
>> <http://twitter.com/afkham_azeez>
>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
>> <http://lk.linkedin.com/in/afkhamazeez>*
>>
>> *Lean . Enterprise . Middleware*
>>
>
>
>
> --
> *Afkham Azeez*
> Director of Architecture; WSO2, Inc.; http://wso2.com
> Member; Apache Software Foundation; http://www.apache.org/
> * <http://www.apache.org/>*
> *email: **az...@wso2.com* <az...@wso2.com>
> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: *
> *http://blog.afkham.org* <http://blog.afkham.org>
> *twitter: **http://twitter.com/afkham_azeez*
> <http://twitter.com/afkham_azeez>
> *linked-in: **http://lk.linkedin.com/in/afkhamazeez
> <http://lk.linkedin.com/in/afkhamazeez>*
>
> *Lean . Enterprise . Middleware*
>
--
Best Regards
Isuru Ranawaka
M: +94714629880
Blog : http://isurur.blogspot.com/
_______________________________________________
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
