I have wanted this from the early days of my using gemfire. Keep pushing. Wayne Lund Advisory Platform Architect 916.296.1893 [email protected] <mailto:[email protected]>
<https://springoneplatform.io/?utm_source=employee-email-footer&utm_medium=email-footer-link&utm_campaign=employee-gmail> <https://springoneplatform.io/?utm_source=employee-email-footer&utm_medium=email-footer-link&utm_campaign=employee-gmail> > On Nov 30, 2017, at 12:51 PM, Amit Pandey <[email protected]> wrote: > > Thanks John..this is helpful and yes a fully reactive result collector from > Geode will be awesome...I do think for large data sets this a must. > > On Fri, Dec 1, 2017 at 2:05 AM, John Blum <[email protected] > <mailto:[email protected]>> wrote: > 1 last thing... > > If you run the PeerCacheFunctionExecutionResultStreamingIntegrationTests test > class, you will see the effects of Function Stream Result processing in you > IDE (or on the command-line). > > Enjoy! > -j > > > On Thu, Nov 30, 2017 at 12:31 PM, John Blum <[email protected] > <mailto:[email protected]>> wrote: > Hi Amit, all- > > 2 things... > > > 1. Have a look at... DATAGEODE-36 - "Add Async, Reactive and Streaming > capabilities to the Function Annotation support." [1] > > I already started experimenting with Streaming Function results here [2]. > Essentially the only thing preventing Apache Geode from Streaming results now > is the default o.a.g.cache.execute.ResultCollector implementation (i.e. > o.a.g.internal.cache.execute.DefaultResultCollector [3]) provided by Apache > Geode OOTB. > > > 2. Dan is correct about the batching functionality provided in SDG! > > You can see how it work by looking at the > o.s.d.g.function.BatchingResultSender class [4], which is used by the > o.s.d.g.function.PojoFunctionWrapper class [5] (which is how the entire > Function "implementation" annotated POJO methods work, BTW! Well, that class > and some Spring (Data Geode) infrastructure to enable it, ;-). > > > Hope this helps! > > -John > > [1] https://jira.spring.io/browse/DATAGEODE-36 > <https://jira.spring.io/browse/DATAGEODE-36> > [2] > https://github.com/jxblum/spring-gemfire-tests/blob/master/src/test/java/org/spring/data/gemfire/cache/PeerCacheFunctionExecutionResultStreamingIntegrationTests.java > > <https://github.com/jxblum/spring-gemfire-tests/blob/master/src/test/java/org/spring/data/gemfire/cache/PeerCacheFunctionExecutionResultStreamingIntegrationTests.java> > [3] > https://github.com/apache/geode/blob/rel/v1.3.0/geode-core/src/main/java/org/apache/geode/internal/cache/execute/DefaultResultCollector.java > > <https://github.com/apache/geode/blob/rel/v1.3.0/geode-core/src/main/java/org/apache/geode/internal/cache/execute/DefaultResultCollector.java> > [4] > https://github.com/spring-projects/spring-data-gemfire/blob/2.0.2.RELEASE/src/main/java/org/springframework/data/gemfire/function/BatchingResultSender.java > > <https://github.com/spring-projects/spring-data-gemfire/blob/2.0.2.RELEASE/src/main/java/org/springframework/data/gemfire/function/BatchingResultSender.java> > [5] > https://github.com/spring-projects/spring-data-gemfire/blob/2.0.2.RELEASE/src/main/java/org/springframework/data/gemfire/function/PojoFunctionWrapper.java#L143-L148 > > <https://github.com/spring-projects/spring-data-gemfire/blob/2.0.2.RELEASE/src/main/java/org/springframework/data/gemfire/function/PojoFunctionWrapper.java#L143-L148> > > > On Thu, Nov 30, 2017 at 7:59 AM, Anthony Baker <[email protected] > <mailto:[email protected]>> wrote: > FWIW, I really like this suggestion. There is some interaction with the > `isHA()` behavior but I would love to have a reactive Function that returns a > mono / flux [1]. > > Anthony > > [1] https://spring.io/blog/2016/04/19/understanding-reactive-types > <https://spring.io/blog/2016/04/19/understanding-reactive-types> > > >> On Nov 28, 2017, at 10:23 PM, Amit Pandey <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hey Guys, >> >> I filed this JIRA ticket :- https://issues.apache.org/jira/browse/GEODE-4027 >> <https://issues.apache.org/jira/browse/GEODE-4027> >> >> Regards >> >> On Wed, Nov 29, 2017 at 2:01 AM, Amit Pandey <[email protected] >> <mailto:[email protected]>> wrote: >> Sure. >> >> On Wed, Nov 29, 2017 at 1:43 AM, Swapnil Bawaskar <[email protected] >> <mailto:[email protected]>> wrote: >> Amit, >> Can you please file a JIRA? >> >> Thanks! >> >> On Tue, Nov 28, 2017 at 12:02 PM Amit Pandey <[email protected] >> <mailto:[email protected]>> wrote: >> Hey Udo, >> >> if I was confident I wouldn't have griped about the non availability. I >> think though that it will be a good addition. Because for large data >> collections it's not enough to batch it from the server side. >> >> Without a streaming result collector its futile >> >> On Wed, Nov 29, 2017 at 1:05 AM, Udo Kohlmeyer <[email protected] >> <mailto:[email protected]>> wrote: >> Hi there Amit. >> >> Geode does also work for PRs.... We actually look forward to the community >> contributing to the product... :) >> >> But on a more serious note... a streaming results collector does require >> more thought and potentially significant (positive) changes to the system >> behavior. >> >> --Udo >> >> On Tue, Nov 28, 2017 at 11:21 AM, Amit Pandey <[email protected] >> <mailto:[email protected]>> wrote: >> Thanks Dan. >> >> A Streaming Result Collector in Geode is needed now . This will be great >> addition. I know we can make it, but its pretty error prone to do it >> >> On Wed, Nov 29, 2017 at 12:05 AM, Dan Smith <[email protected] >> <mailto:[email protected]>> wrote: >> Hi Amit, >> >> Looking at the code, it looks like if your annotated method returns an >> array, or an Iteratable (like a List), and you have batchSize set, then >> spring will send your data in batches back to the caller of the function. >> Underneath the covers is it using geode's ResultSender and passing the >> batches to the ResultSender sendResult method. >> >> So it looks like this would reduce the amount of data that gets sent to the >> client in a single message. >> >> -Dan >> >> On Sun, Nov 26, 2017 at 9:11 PM, Amit Pandey <[email protected] >> <mailto:[email protected]>> wrote: >> Hi John and everyone, >> >> How does Spring Data Geode Function annotation batch size work? >> >> So if we set it up like @GemfireFunction(id="lotMapService",HA=true, >> batchSize =1000) >> >> >> what are the implications ? What happens if there is a million data to be >> streamed on the server as well as on the client side. >> >> Regards >> >> >> >> >> >> -- >> Kindest Regards >> ----------------------------- >> Udo Kohlmeyer | Pivotal >> [email protected] <mailto:[email protected]> >> <http://www.gopivotal.com/> >> www.pivotal.io <http://www.pivotal.io/> >> >> >> > > > > > -- > -John > john.blum10101 (skype) > > > > -- > -John > john.blum10101 (skype) >
signature.asc
Description: Message signed with OpenPGP
