Hi Amit- To which Geode demo application are you referring? Something off of geode.apache.org?
Thanks, John On Sun, Dec 11, 2016 at 12:47 AM, Amit Pandey <[email protected]> wrote: > Hey John many thanks. > > This is really helpful. Yes I was confused why there were no Write Behind > examples and the one which talked of doing write behind seemed to be using > Spring Async Executor mechanism. > > I am really looking forward to your new refactored demoes. > > If you get times kindly let some one rework the Geode demo application > with the latest code or atleats provide me a guide to update to the latest > Spring Data and Geode dist , because that one seemed really useful to me to > get started on something. > > On Sun, Dec 11, 2016 at 6:32 AM, John Blum <[email protected]> wrote: > >> Amit- >> >> The example >> <https://github.com/spring-projects/spring-gemfire-examples/tree/master/basic/write-through> >> [1] >> to which you are referring is a "*Write-Through*" example only. This >> evident from the configuration >> <https://github.com/spring-projects/spring-gemfire-examples/blob/master/basic/write-through/src/main/resources/cache-config.xml#L16-L20> >> [2] >> of the "*Product*" Region used to store the application Product >> <https://github.com/spring-projects/spring-gemfire-examples/blob/master/spring-gemfire-examples-common/src/main/java/org/springframework/data/gemfire/examples/domain/Product.java> >> [3] >> data. >> >> Whenever CacheLoaders >> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheLoader.html> >> [4] >> (to pull from an external DataSource) and CacheWriters >> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheWriter.html> >> [5] >> (to write to an external DataSource) are in play, then you know this is >> a *synchronous* operation between the cache and the external data store, >> as explained here >> <http://geode.apache.org/docs/guide/developing/outside_data_sources/chapter_overview.html> >> [6]. >> >> In order to achieve asynchronous "*Write-Behind*" behavior to an >> external data store, you need to use an AsyncEventQueue >> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventQueue.html> >> [7] >> (set on the Region) along with an AsyncEventListener >> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html> >> [8] >> registered on the AEQ. The Listener serves the same purpose as the >> CacheWriter to write to the external DataSource, however does so in an >> asynchronous, "write-behind" manner given the nature of an AEQ, as the AEQ >> Listener Javadoc alludes to... >> >> "*A callback for events passing through the AsyncEventQueue to which >> this listener is attached. Implementers of >> interface AsyncEventListener process batches of AsyncEvent delivered by the >> corresponding AsyncEventQueue.*" >> >> Of course, the AEQ Listener can do whatever it wants with the (async) >> events, but like the CacheWriter, you can also use it to write-behind to >> an external data store. >> >> The AEQ controls >> <http://geode.apache.org/docs/guide/reference/topics/cache_xml.html#async-event-queue> >> [9] >> many things: the order of events (order-policy), and how many events are >> in a batch (batch-size), the time interval (batch-time-interval) at >> which time the Listener is called with the batch of events regardless of >> whether the batch-size is reached even, whether the AEQ is durable ( >> disk-store-ref/disk-synchronous), and so on and so forth. >> >> When Region data access operations occur (e.g. *puts*), then it writes >> an AsyncEvent >> <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html> >> [10] >> to the AEQ. Eventually (again, either determined by batch-size or >> batch-time-interval), the AEQ will trigger the Listener. All of this is >> explained in more detail here >> <http://geode.apache.org/docs/guide/developing/events/implementing_write_behind_event_handler.html> >> [11]. >> >> Now, that we have laid down the foundation for understanding >> "Write-Behind" and how this works in Apache Geode (as well as Pivotal >> GemFire)... *how * >> *to do this with Spring Data Geode (/GemFire)? *Simple! >> >> Assuming you are configuring your application and Geode components using >> *Spring >> (Data Geode*) XML, first you need to define your AEQ Listener... >> >> <bean id="ExampleQueueListener" class="example.ExternalDataSto >> reAsyncEventListener"> >> ... >> </bean> >> >> Obviously, you could inject a JDBC DataSource and wrap it with *Spring's* >> JdbcTemplate, or use a SD JPA Repository, or whatever you want, to >> access your RDBMS of choice. >> >> You would code the AEQ Listener to perform a data access operation for >> each AsyncEvent passed to the Listener to write to the underlying data >> store. Or, if you weren't using the SD JPA Repository abstraction, rather >> the *Spring* JdbcTemplate, you could perform a "batch" JDBC data access >> operation to the underlying data store as well since the AEQ and associated >> Listener deal in "batches" of events. Hint, hint, ;-) >> >> Next, you need to define your AEQ and register your custom Listener(s). >> >> <gfe:async-event-queue id="ExampleQueue" >> batch-conflation-enabled="true" >> batch-size="10" >> batch-time-interval="3" >> dispatcher-threads="4" >> disk-store-ref="ExampleQueueDiskStore" >> disk-synchronous="true" >> maximum-queue-memory="50" >> parallel="false" >> persistent="true" >> order-policy="KEY"> >> <gfe:async-event-listener ref="ExampleQueueListener"/> >> </gfe:async-event-queue> >> >> NOTE: I copied this configuration from one of the tests in the SDG test >> suite. This test is just ensuring that "all" the configuration settings >> (i.e. XML <gfe:async-event-queue> element attributes) are properly >> parsed and configured on the resulting Geode/GemFire AEQ component created >> by SDG. Therefore, this is an exhaustive list of attributes available for >> you to use and control the behavior of the AEQ. It is NOT representative >> of settings that are necessarily appropriate for your application. >> >> Also know I can register more than 1 AEQ Listener. In fact, I can >> register as many AEQ Listeners as I like, all performing a different tasks >> as required by my application. >> >> You may have noticed that the "disk-store-ref" and "disk-synchronous" >> attributes were specified. This makes the AEQ "durable". Additionally, >> you should keep in mind that the "disk-synchronous" attribute as nothing >> to do with the asynchronous, write-behind nature of an AEQ/Listener >> associated with the Region implementing asynchronous, "write-behind" >> semantics. The disk-synchronous setting specifically controls how events >> on the "queue" are stored to disk when the AEQ is durable. >> >> Finally, you associate the AEQ with your Region, like so... >> >> <gfe:replicated-region id="ExampleRegionWithAEQ" > >> <gfe:async-event-queue-ref bean="ExampleQueue"/*>* >> * ...* >> </gfe:replicated-region> >> >> >> Note, this entire set of configuration could have easily been achieved >> using JavaConfig was well. >> >> Also note, XML configuration is very flexible in that both the Listener >> and Queue could have been defined as "anonymous" *Spring* beans on the >> Region. >> >> You are probably wondering why there is not an example of "Write-Behind" >> in the Spring GemFire Examples >> <https://github.com/spring-projects/spring-gemfire-examples> [12], and >> you would be correct, there should be. However, I am in the process of >> completely re-working these examples, all of them, to be *Spring Boot* >> based along with JavaConfig based using my new Annotation configuration >> model that will greatly simplify configuring both Pivotal GemFire and >> Apache Geode. >> >> Anyway, I hope this helps. If you have additional questions, let me know. >> >> Cheers! >> -John >> >> [1] https://github.com/spring-projects/spring-gemfire-exampl >> es/tree/master/basic/write-through >> [2] https://github.com/spring-projects/spring-gemfire-exampl >> es/blob/master/basic/write-through/src/main/resources/ >> cache-config.xml#L16-L20 >> [3] https://github.com/spring-projects/spring-gemfire-exampl >> es/blob/master/spring-gemfire-examples-common/src/main/java/ >> org/springframework/data/gemfire/examples/domain/Product.java >> [4] http://geode.apache.org/releases/latest/javadoc/org/apac >> he/geode/cache/CacheLoader.html >> [5] http://geode.apache.org/releases/latest/javadoc/org/apac >> he/geode/cache/CacheWriter.html >> [6] http://geode.apache.org/docs/guide/developing/outside_da >> ta_sources/chapter_overview.html >> [7] http://geode.apache.org/releases/latest/javadoc/org/apac >> he/geode/cache/asyncqueue/AsyncEventQueue.html >> [8] http://geode.apache.org/releases/latest/javadoc/org/apac >> he/geode/cache/asyncqueue/AsyncEventListener.html >> [9] http://geode.apache.org/docs/guide/reference/topics/cach >> e_xml.html#async-event-queue >> [10] http://geode.apache.org/releases/latest/javadoc/org/apa >> che/geode/cache/asyncqueue/AsyncEventListener.html >> [11] http://geode.apache.org/docs/guide/developing/events/im >> plementing_write_behind_event_handler.html >> >> >> On Sat, Dec 10, 2016 at 8:15 AM, Amit Pandey <[email protected]> >> wrote: >> >>> Sorry to be pedantic. Thats the example which confused me most. That >>> says a async profile is setup . >>> >>> " Spring Data JPA Repository is used to access the database. The >>> 'async' command line argument enables the 'async' Spring profile which >>> enables the asynchronous behavior in ProductDBWriter." >>> >>> What I am confused is , we need to set up event queues etc for backup, >>> we can make them persistable ensuring no data loss, where are those done? >>> >>> Regards >>> >>> On Sat, Dec 10, 2016 at 9:04 PM, Nilkanth Patel < >>> [email protected]> wrote: >>> >>>> Have a look at following if that helps! >>>> >>>> https://github.com/spring-projects/spring-gemfire-examples >>>> >>>> Nilkanth Patel. >>>> >>>> On Sat, Dec 10, 2016 at 7:24 PM, Amit Pandey <[email protected] >>>> > wrote: >>>> >>>>> Thanks I understood this. >>>>> >>>>> I want to know how to do this with Spring Data geode? >>>>> >>>>> On Sat, Dec 10, 2016 at 6:24 PM, Avinash Dongre <[email protected]> >>>>> wrote: >>>>> >>>>>> This should help. >>>>>> >>>>>> http://geode.apache.org/docs/guide/developing/events/impleme >>>>>> nting_write_behind_event_handler.html >>>>>> >>>>>> Thanks >>>>>> Avinash >>>>>> >>>>>> >>>>>> On Sat, Dec 10, 2016 at 4:22 PM, Amit Pandey < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hey Guys can anyone please provide a link? >>>>>>> >>>>>>> On Fri, Dec 9, 2016 at 8:24 PM, Amit Pandey < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Guys, >>>>>>>> >>>>>>>> Is there any geode example of write behind with Spring Gemfire or >>>>>>>> even without it using reliable event queue (which persists to disk ) ? >>>>>>>> >>>>>>>> Looking at this example what config changes will enable write >>>>>>>> behind keeping write order and not write orders with a reliable event >>>>>>>> queue. >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >> >> -- >> -John >> 503-504-8657 >> john.blum10101 (skype) >> > > -- -John 503-504-8657 john.blum10101 (skype)
