Thanks John..this is helpful and yes a fully reactive result collector from Geode will be awesome...I do think for large data sets this a must.
On Fri, Dec 1, 2017 at 2:05 AM, John Blum <[email protected]> wrote: > 1 last thing... > > If you run the PeerCacheFunctionExecutionResultStreamingIntegrationTests test > class, you will see the effects of Function Stream Result processing in you > IDE (or on the command-line). > > Enjoy! > -j > > > On Thu, Nov 30, 2017 at 12:31 PM, John Blum <[email protected]> wrote: > >> Hi Amit, all- >> >> 2 things... >> >> >> 1. Have a look at... DATAGEODE-36 - "*Add Async, Reactive and Streaming >> capabilities to the Function Annotation support.*" [1] >> >> I already started experimenting with Streaming Function results here >> [2]. Essentially the only thing preventing Apache Geode from Streaming >> results now is the default o.a.g.cache.execute.ResultCollector >> implementation (i.e. o.a.g.internal.cache.execute.DefaultResultCollector >> [3]) provided by Apache Geode OOTB. >> >> >> 2. *Dan is correct* about the batching functionality provided in SDG! >> >> You can see how it work by looking at the o.s.d.g.function.BatchingResul >> tSender class [4], which is used by the o.s.d.g.function.PojoFunctionW >> rapper class [5] (which is how the entire Function "implementation" >> annotated POJO methods work, BTW! Well, that class and some *Spring >> (Data Geode)* infrastructure to enable it, ;-). >> >> >> Hope this helps! >> >> -John >> >> [1] https://jira.spring.io/browse/DATAGEODE-36 >> [2] https://github.com/jxblum/spring-gemfire-tests/blob/mast >> er/src/test/java/org/spring/data/gemfire/cache/PeerCacheFu >> nctionExecutionResultStreamingIntegrationTests.java >> [3] https://github.com/apache/geode/blob/rel/v1.3.0/geode-core/ >> src/main/java/org/apache/geode/internal/cache/execute/Defaul >> tResultCollector.java >> [4] https://github.com/spring-projects/spring-data-gemfire/b >> lob/2.0.2.RELEASE/src/main/java/org/springframework/data/gem >> fire/function/BatchingResultSender.java >> [5] https://github.com/spring-projects/spring-data-gemfire/b >> lob/2.0.2.RELEASE/src/main/java/org/springframework/data/gem >> fire/function/PojoFunctionWrapper.java#L143-L148 >> >> >> On Thu, Nov 30, 2017 at 7:59 AM, Anthony Baker <[email protected]> wrote: >> >>> FWIW, I really like this suggestion. There is some interaction with the >>> `isHA()` behavior but I would love to have a reactive Function that returns >>> a mono / flux [1]. >>> >>> Anthony >>> >>> [1] https://spring.io/blog/2016/04/19/understanding-reactive-types >>> >>> >>> On Nov 28, 2017, at 10:23 PM, Amit Pandey <[email protected]> >>> wrote: >>> >>> Hey Guys, >>> >>> I filed this JIRA ticket :- https://issues.apache.org/j >>> ira/browse/GEODE-4027 >>> >>> Regards >>> >>> On Wed, Nov 29, 2017 at 2:01 AM, Amit Pandey <[email protected]> >>> wrote: >>> >>>> Sure. >>>> >>>> On Wed, Nov 29, 2017 at 1:43 AM, Swapnil Bawaskar <[email protected] >>>> > wrote: >>>> >>>>> Amit, >>>>> Can you please file a JIRA? >>>>> >>>>> Thanks! >>>>> >>>>> On Tue, Nov 28, 2017 at 12:02 PM Amit Pandey < >>>>> [email protected]> wrote: >>>>> >>>>>> Hey Udo, >>>>>> >>>>>> if I was confident I wouldn't have griped about the non availability. >>>>>> I think though that it will be a good addition. Because for large data >>>>>> collections it's not enough to batch it from the server side. >>>>>> >>>>>> Without a streaming result collector its futile >>>>>> >>>>>> On Wed, Nov 29, 2017 at 1:05 AM, Udo Kohlmeyer <[email protected] >>>>>> > wrote: >>>>>> >>>>>>> Hi there Amit. >>>>>>> >>>>>>> Geode does also work for PRs.... We actually look forward to the >>>>>>> community contributing to the product... :) >>>>>>> >>>>>>> But on a more serious note... a streaming results collector does >>>>>>> require more thought and potentially significant (positive) changes to >>>>>>> the >>>>>>> system behavior. >>>>>>> >>>>>>> --Udo >>>>>>> >>>>>>> On Tue, Nov 28, 2017 at 11:21 AM, Amit Pandey < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Thanks Dan. >>>>>>>> >>>>>>>> A Streaming Result Collector in Geode is needed now . This will be >>>>>>>> great addition. I know we can make it, but its pretty error prone to >>>>>>>> do it >>>>>>>> >>>>>>>> On Wed, Nov 29, 2017 at 12:05 AM, Dan Smith <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Amit, >>>>>>>>> >>>>>>>>> Looking at the code, it looks like if your annotated method >>>>>>>>> returns an array, or an Iteratable (like a List), and you have >>>>>>>>> batchSize >>>>>>>>> set, then spring will send your data in batches back to the caller of >>>>>>>>> the >>>>>>>>> function. Underneath the covers is it using geode's ResultSender and >>>>>>>>> passing the batches to the ResultSender sendResult method. >>>>>>>>> >>>>>>>>> So it looks like this would reduce the amount of data that gets >>>>>>>>> sent to the client in a single message. >>>>>>>>> >>>>>>>>> -Dan >>>>>>>>> >>>>>>>>> On Sun, Nov 26, 2017 at 9:11 PM, Amit Pandey < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi John and everyone, >>>>>>>>>> >>>>>>>>>> How does Spring Data Geode Function annotation batch size work? >>>>>>>>>> >>>>>>>>>> So if we set it up like @GemfireFunction(id="lotMapService",HA=true, >>>>>>>>>> batchSize =1000) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> what are the implications ? What happens if there is a million >>>>>>>>>> data to be streamed on the server as well as on the client side. >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Kindest Regards >>>>>>> ----------------------------- >>>>>>> *Udo Kohlmeyer* | *Pivotal* >>>>>>> [email protected] >>>>>>> <http://www.gopivotal.com/> >>>>>>> www.pivotal.io >>>>>>> >>>>>> >>>>>> >>>> >>> >>> >> >> >> -- >> -John >> john.blum10101 (skype) >> > > > > -- > -John > john.blum10101 (skype) >
