Amit- The example <https://github.com/spring-projects/spring-gemfire-examples/tree/master/basic/write-through> [1] to which you are referring is a "*Write-Through*" example only. This evident from the configuration <https://github.com/spring-projects/spring-gemfire-examples/blob/master/basic/write-through/src/main/resources/cache-config.xml#L16-L20> [2] of the "*Product*" Region used to store the application Product <https://github.com/spring-projects/spring-gemfire-examples/blob/master/spring-gemfire-examples-common/src/main/java/org/springframework/data/gemfire/examples/domain/Product.java> [3] data.
Whenever CacheLoaders <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheLoader.html> [4] (to pull from an external DataSource) and CacheWriters <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheWriter.html> [5] (to write to an external DataSource) are in play, then you know this is a *synchronous* operation between the cache and the external data store, as explained here <http://geode.apache.org/docs/guide/developing/outside_data_sources/chapter_overview.html> [6]. In order to achieve asynchronous "*Write-Behind*" behavior to an external data store, you need to use an AsyncEventQueue <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventQueue.html> [7] (set on the Region) along with an AsyncEventListener <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html> [8] registered on the AEQ. The Listener serves the same purpose as the CacheWriter to write to the external DataSource, however does so in an asynchronous, "write-behind" manner given the nature of an AEQ, as the AEQ Listener Javadoc alludes to... "*A callback for events passing through the AsyncEventQueue to which this listener is attached. Implementers of interface AsyncEventListener process batches of AsyncEvent delivered by the corresponding AsyncEventQueue.*" Of course, the AEQ Listener can do whatever it wants with the (async) events, but like the CacheWriter, you can also use it to write-behind to an external data store. The AEQ controls <http://geode.apache.org/docs/guide/reference/topics/cache_xml.html#async-event-queue> [9] many things: the order of events (order-policy), and how many events are in a batch (batch-size), the time interval (batch-time-interval) at which time the Listener is called with the batch of events regardless of whether the batch-size is reached even, whether the AEQ is durable ( disk-store-ref/disk-synchronous), and so on and so forth. When Region data access operations occur (e.g. *puts*), then it writes an AsyncEvent <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html> [10] to the AEQ. Eventually (again, either determined by batch-size or batch-time-interval), the AEQ will trigger the Listener. All of this is explained in more detail here <http://geode.apache.org/docs/guide/developing/events/implementing_write_behind_event_handler.html> [11]. Now, that we have laid down the foundation for understanding "Write-Behind" and how this works in Apache Geode (as well as Pivotal GemFire)... *how * *to do this with Spring Data Geode (/GemFire)? *Simple! Assuming you are configuring your application and Geode components using *Spring (Data Geode*) XML, first you need to define your AEQ Listener... <bean id="ExampleQueueListener" class= "example.ExternalDataStoreAsyncEventListener"> ... </bean> Obviously, you could inject a JDBC DataSource and wrap it with *Spring's* JdbcTemplate, or use a SD JPA Repository, or whatever you want, to access your RDBMS of choice. You would code the AEQ Listener to perform a data access operation for each AsyncEvent passed to the Listener to write to the underlying data store. Or, if you weren't using the SD JPA Repository abstraction, rather the *Spring* JdbcTemplate, you could perform a "batch" JDBC data access operation to the underlying data store as well since the AEQ and associated Listener deal in "batches" of events. Hint, hint, ;-) Next, you need to define your AEQ and register your custom Listener(s). <gfe:async-event-queue id="ExampleQueue" batch-conflation-enabled="true" batch-size="10" batch-time-interval="3" dispatcher-threads="4" disk-store-ref="ExampleQueueDiskStore" disk-synchronous="true" maximum-queue-memory="50" parallel="false" persistent="true" order-policy="KEY"> <gfe:async-event-listener ref="ExampleQueueListener"/> </gfe:async-event-queue> NOTE: I copied this configuration from one of the tests in the SDG test suite. This test is just ensuring that "all" the configuration settings (i.e. XML <gfe:async-event-queue> element attributes) are properly parsed and configured on the resulting Geode/GemFire AEQ component created by SDG. Therefore, this is an exhaustive list of attributes available for you to use and control the behavior of the AEQ. It is NOT representative of settings that are necessarily appropriate for your application. Also know I can register more than 1 AEQ Listener. In fact, I can register as many AEQ Listeners as I like, all performing a different tasks as required by my application. You may have noticed that the "disk-store-ref" and "disk-synchronous" attributes were specified. This makes the AEQ "durable". Additionally, you should keep in mind that the "disk-synchronous" attribute as nothing to do with the asynchronous, write-behind nature of an AEQ/Listener associated with the Region implementing asynchronous, "write-behind" semantics. The disk-synchronous setting specifically controls how events on the "queue" are stored to disk when the AEQ is durable. Finally, you associate the AEQ with your Region, like so... <gfe:replicated-region id="ExampleRegionWithAEQ" > <gfe:async-event-queue-ref bean="ExampleQueue"/*>* * ...* </gfe:replicated-region> Note, this entire set of configuration could have easily been achieved using JavaConfig was well. Also note, XML configuration is very flexible in that both the Listener and Queue could have been defined as "anonymous" *Spring* beans on the Region. You are probably wondering why there is not an example of "Write-Behind" in the Spring GemFire Examples <https://github.com/spring-projects/spring-gemfire-examples> [12], and you would be correct, there should be. However, I am in the process of completely re-working these examples, all of them, to be *Spring Boot* based along with JavaConfig based using my new Annotation configuration model that will greatly simplify configuring both Pivotal GemFire and Apache Geode. Anyway, I hope this helps. If you have additional questions, let me know. Cheers! -John [1] https://github.com/spring-projects/spring-gemfire-examples/tree/master/basic/write-through [2] https://github.com/spring-projects/spring-gemfire-examples/blob/master/basic/write-through/src/main/resources/cache-config.xml#L16-L20 [3] https://github.com/spring-projects/spring-gemfire-examples/blob/master/spring-gemfire-examples-common/src/main/java/org/springframework/data/gemfire/examples/domain/Product.java [4] http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheLoader.html [5] http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheWriter.html [6] http://geode.apache.org/docs/guide/developing/outside_data_sources/chapter_overview.html [7] http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventQueue.html [8] http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html [9] http://geode.apache.org/docs/guide/reference/topics/cache_xml.html#async-event-queue [10] http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html [11] http://geode.apache.org/docs/guide/developing/events/implementing_write_behind_event_handler.html On Sat, Dec 10, 2016 at 8:15 AM, Amit Pandey <[email protected]> wrote: > Sorry to be pedantic. Thats the example which confused me most. That says > a async profile is setup . > > " Spring Data JPA Repository is used to access the database. The 'async' > command line argument enables the 'async' Spring profile which enables the > asynchronous behavior in ProductDBWriter." > > What I am confused is , we need to set up event queues etc for backup, we > can make them persistable ensuring no data loss, where are those done? > > Regards > > On Sat, Dec 10, 2016 at 9:04 PM, Nilkanth Patel <[email protected] > > wrote: > >> Have a look at following if that helps! >> >> https://github.com/spring-projects/spring-gemfire-examples >> >> Nilkanth Patel. >> >> On Sat, Dec 10, 2016 at 7:24 PM, Amit Pandey <[email protected]> >> wrote: >> >>> Thanks I understood this. >>> >>> I want to know how to do this with Spring Data geode? >>> >>> On Sat, Dec 10, 2016 at 6:24 PM, Avinash Dongre <[email protected]> >>> wrote: >>> >>>> This should help. >>>> >>>> http://geode.apache.org/docs/guide/developing/events/impleme >>>> nting_write_behind_event_handler.html >>>> >>>> Thanks >>>> Avinash >>>> >>>> >>>> On Sat, Dec 10, 2016 at 4:22 PM, Amit Pandey <[email protected] >>>> > wrote: >>>> >>>>> Hey Guys can anyone please provide a link? >>>>> >>>>> On Fri, Dec 9, 2016 at 8:24 PM, Amit Pandey <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi Guys, >>>>>> >>>>>> Is there any geode example of write behind with Spring Gemfire or >>>>>> even without it using reliable event queue (which persists to disk ) ? >>>>>> >>>>>> Looking at this example what config changes will enable write behind >>>>>> keeping write order and not write orders with a reliable event queue. >>>>>> >>>>>> Regards >>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > -- -John 503-504-8657 john.blum10101 (skype)
