Is the following code segment the way to read all the chunks; while(!carbonMsg.isEndOfMsgAdded) { ByteBuffer chunk = carbonMsg.getMessageBody();
} On Thu, Jun 2, 2016 at 9:21 PM, Afkham Azeez <az...@wso2.com> wrote: > Looks like we require changes for input streaming as well. > > Does CarbonMessage.getMessageBody() block until the next chunk is > received? > > On Thu, Jun 2, 2016 at 9:18 PM, Afkham Azeez <az...@wso2.com> wrote: > >> Samiyuru/Isuru, >> Does this mean that the input streaming in MSF4J is currently working >> without any issue and only output streaming requires some work? >> >> On Tue, May 3, 2016 at 2:29 PM, Isuru Ranawaka <isu...@wso2.com> wrote: >> >>> Hi All, >>> >>> Following are the details on streaming support of carbon transport. >>> Basically we can looking to streaming support in request path and response >>> path separately. >>> >>> Request Path (Streaming is working) >>> >>> [image: requestpath.png] >>> >>> According to above diagram >>> >>> - >>> >>> We have blocking queue in the carbon message which keeps the content >>> . when headers are received through Netty worker thread it will create >>> CarbonMessage with a blocking queue and publish carbon message to engine >>> level thread. >>> - >>> >>> Reference for blocking queue is cached in the connection and when >>> content is received it will be filled to that queue from Netty worker >>> thread. >>> - >>> >>> Meanwhile engine level threads can consume content through >>> queue(While IO worker is filling ) and can directly send to file system >>> or >>> use sender for send messages to external service. >>> - >>> >>> According to that streaming should work in Request path in >>> Integration Server or MSF4J without any problem. >>> >>> >>> Response Path (Streaming working scenario) >>> >>> [image: responsepathworking.png] >>> >>> >>> In Response path basic difference we have is we are handling responses >>> through callbacks. >>> >>> >>> - >>> >>> Similar to Request path architecture Sender side IO thread >>> creates a CM when response headers are received and publish to engine >>> level thread . >>> - >>> >>> Engine level thread calls carbonCallback.done() and waits on queue >>> for content. >>> - >>> >>> Meanwhile IO thread writes the content to the Queue . >>> - >>> >>> So writing to Queue and reading from queue happens parallel and >>> streaming should work properly. >>> - >>> >>> So streaming is working fine with Integration Server for this kind >>> of scenarios. >>> >>> >>> >>> Response Path (Streaming not working scenario) >>> >>> [image: responsepathnotworking.png] >>> >>> This scenario is basically, if we have a Echo mediator or assume MSF4J >>> has a service which is reading a file as chunks and writes back to client . >>> >>> >>> - >>> >>> Basic difference with the previous one is in this approach reading >>> a file chunk or stream and writing that file chunk or stream to Queue is >>> happened within the same engine level thread as well as with reading >>> from >>> queue and writing to Listener side Netty worker threads.(Same thread >>> produce and consume causes for deadlock) >>> - >>> >>> In the previous example reading from stream and filling the queue >>> was happened through Sender side IO thread and consuming the queue and >>> writing to Listener side IO thread was happened through engine level >>> thread. (Two different threads produce and consume) >>> - >>> >>> Streaming is not working in this kind of scenarios because we cannot >>> write to queue and keep polling the queue within single thread. >>> >>> >>> We can figure out following solutions and what will be the most suitable >>> one. >>> >>> >>> >>> - >>> >>> Introduce another thread for run the callback logic instead of >>> running through calling thread. >>> >>> This will solve the above streaming problem but when it >>> comes to general message flow this will add another level of thread which >>> will actually does not need. >>> >>> - >>> >>> Introduce another method in ResponseCallback which will support >>> streaming which does not queuing contents. it will directly call IO >>> threads and write contents to IO when chunks are received. This can be >>> used >>> only within single thread scenario(File reading and writing ). >>> - >>> >>> Introduce another thread for read File stream or call Callback from >>> another thread other than to reading thread from MSF4J level.Then it >>> will >>> be equivalent to how we used it in Integration Server with the use of >>> Sender. >>> >>> ThankYou >>> >>> IsuruR >>> >>> On Tue, May 3, 2016 at 12:09 PM, Kasun Indrasiri <ka...@wso2.com> wrote: >>> >>>> Had a chat with Ranawaka on this.. seems like we do have couple of ways >>>> to handle this. He will share the details. >>>> >>>> On Mon, May 2, 2016 at 6:13 AM, Afkham Azeez <az...@wso2.com> wrote: >>>> >>>>> The problem is, we can't do the next MSF4J release because we can't >>>>> lose a feature in a release. This is a blocker for us. We have plans to >>>>> release in the next 2 weeks. >>>>> >>>>> On Mon, May 2, 2016 at 4:56 PM, Isuru Ranawaka <isu...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Azeez, >>>>>> >>>>>> Currently streaming works if we used both sender side and listener >>>>>> side only . But since MSF4J is using only listener side if we did not >>>>>> spawn >>>>>> separate thread from engine level for writing response it will not work >>>>>> because request reading and writing happens through same thread. But with >>>>>> the next release we will fix that. currently we are finalizing on >>>>>> removing >>>>>> engine level thread model from transport. >>>>>> >>>>>> >>>>>> thanks >>>>>> IsuruR >>>>>> >>>>>> On Mon, May 2, 2016 at 3:50 PM, Afkham Azeez <az...@wso2.com> wrote: >>>>>> >>>>>>> Is this working after moving to the new transport framework? >>>>>>> >>>>>>> -- >>>>>>> *Afkham Azeez* >>>>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>>>> * <http://www.apache.org/>* >>>>>>> *email: **az...@wso2.com* <az...@wso2.com> >>>>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >>>>>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>>>>> *twitter: **http://twitter.com/afkham_azeez* >>>>>>> <http://twitter.com/afkham_azeez> >>>>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>>>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>>>>> >>>>>>> *Lean . Enterprise . Middleware* >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> Dev@wso2.org >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Best Regards >>>>>> Isuru Ranawaka >>>>>> M: +94714629880 >>>>>> Blog : http://isurur.blogspot.com/ >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Afkham Azeez* >>>>> Director of Architecture; WSO2, Inc.; http://wso2.com >>>>> Member; Apache Software Foundation; http://www.apache.org/ >>>>> * <http://www.apache.org/>* >>>>> *email: **az...@wso2.com* <az...@wso2.com> >>>>> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >>>>> *http://blog.afkham.org* <http://blog.afkham.org> >>>>> *twitter: **http://twitter.com/afkham_azeez* >>>>> <http://twitter.com/afkham_azeez> >>>>> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >>>>> <http://lk.linkedin.com/in/afkhamazeez>* >>>>> >>>>> *Lean . Enterprise . Middleware* >>>>> >>>>> _______________________________________________ >>>>> Dev mailing list >>>>> Dev@wso2.org >>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>> >>>>> >>>> >>>> >>>> -- >>>> Kasun Indrasiri >>>> Software Architect >>>> WSO2, Inc.; http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> cell: +94 77 556 5206 >>>> Blog : http://kasunpanorama.blogspot.com/ >>>> >>> >>> >>> >>> -- >>> Best Regards >>> Isuru Ranawaka >>> M: +94714629880 >>> Blog : http://isurur.blogspot.com/ >>> >> >> >> >> -- >> *Afkham Azeez* >> Director of Architecture; WSO2, Inc.; http://wso2.com >> Member; Apache Software Foundation; http://www.apache.org/ >> * <http://www.apache.org/>* >> *email: **az...@wso2.com* <az...@wso2.com> >> * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * >> *http://blog.afkham.org* <http://blog.afkham.org> >> *twitter: **http://twitter.com/afkham_azeez* >> <http://twitter.com/afkham_azeez> >> *linked-in: **http://lk.linkedin.com/in/afkhamazeez >> <http://lk.linkedin.com/in/afkhamazeez>* >> >> *Lean . Enterprise . Middleware* >> > > > > -- > *Afkham Azeez* > Director of Architecture; WSO2, Inc.; http://wso2.com > Member; Apache Software Foundation; http://www.apache.org/ > * <http://www.apache.org/>* > *email: **az...@wso2.com* <az...@wso2.com> > * cell: +94 77 3320919 <%2B94%2077%203320919>blog: * > *http://blog.afkham.org* <http://blog.afkham.org> > *twitter: **http://twitter.com/afkham_azeez* > <http://twitter.com/afkham_azeez> > *linked-in: **http://lk.linkedin.com/in/afkhamazeez > <http://lk.linkedin.com/in/afkhamazeez>* > > *Lean . Enterprise . Middleware* > -- *Afkham Azeez* Director of Architecture; WSO2, Inc.; http://wso2.com Member; Apache Software Foundation; http://www.apache.org/ * <http://www.apache.org/>* *email: **az...@wso2.com* <az...@wso2.com> * cell: +94 77 3320919blog: **http://blog.afkham.org* <http://blog.afkham.org> *twitter: **http://twitter.com/afkham_azeez* <http://twitter.com/afkham_azeez> *linked-in: **http://lk.linkedin.com/in/afkhamazeez <http://lk.linkedin.com/in/afkhamazeez>* *Lean . Enterprise . Middleware*
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev