Hey John many thanks. This is really helpful. Yes I was confused why there were no Write Behind examples and the one which talked of doing write behind seemed to be using Spring Async Executor mechanism.
I am really looking forward to your new refactored demoes. If you get times kindly let some one rework the Geode demo application with the latest code or atleats provide me a guide to update to the latest Spring Data and Geode dist , because that one seemed really useful to me to get started on something. On Sun, Dec 11, 2016 at 6:32 AM, John Blum <[email protected]> wrote: > Amit- > > The example > <https://github.com/spring-projects/spring-gemfire-examples/tree/master/basic/write-through> > [1] > to which you are referring is a "*Write-Through*" example only. This > evident from the configuration > <https://github.com/spring-projects/spring-gemfire-examples/blob/master/basic/write-through/src/main/resources/cache-config.xml#L16-L20> > [2] > of the "*Product*" Region used to store the application Product > <https://github.com/spring-projects/spring-gemfire-examples/blob/master/spring-gemfire-examples-common/src/main/java/org/springframework/data/gemfire/examples/domain/Product.java> > [3] > data. > > Whenever CacheLoaders > <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheLoader.html> > [4] > (to pull from an external DataSource) and CacheWriters > <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/CacheWriter.html> > [5] > (to write to an external DataSource) are in play, then you know this is a > *synchronous* operation between the cache and the external data store, as > explained here > <http://geode.apache.org/docs/guide/developing/outside_data_sources/chapter_overview.html> > [6]. > > In order to achieve asynchronous "*Write-Behind*" behavior to an external > data store, you need to use an AsyncEventQueue > <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventQueue.html> > [7] > (set on the Region) along with an AsyncEventListener > <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html> > [8] > registered on the AEQ. The Listener serves the same purpose as the > CacheWriter to write to the external DataSource, however does so in an > asynchronous, "write-behind" manner given the nature of an AEQ, as the AEQ > Listener Javadoc alludes to... > > "*A callback for events passing through the AsyncEventQueue to which this > listener is attached. Implementers of interface AsyncEventListener process > batches of AsyncEvent delivered by the corresponding AsyncEventQueue.*" > > Of course, the AEQ Listener can do whatever it wants with the (async) > events, but like the CacheWriter, you can also use it to write-behind to > an external data store. > > The AEQ controls > <http://geode.apache.org/docs/guide/reference/topics/cache_xml.html#async-event-queue> > [9] > many things: the order of events (order-policy), and how many events are > in a batch (batch-size), the time interval (batch-time-interval) at which > time the Listener is called with the batch of events regardless of whether > the batch-size is reached even, whether the AEQ is durable ( > disk-store-ref/disk-synchronous), and so on and so forth. > > When Region data access operations occur (e.g. *puts*), then it writes an > AsyncEvent > <http://geode.apache.org/releases/latest/javadoc/org/apache/geode/cache/asyncqueue/AsyncEventListener.html> > [10] > to the AEQ. Eventually (again, either determined by batch-size or > batch-time-interval), the AEQ will trigger the Listener. All of this is > explained in more detail here > <http://geode.apache.org/docs/guide/developing/events/implementing_write_behind_event_handler.html> > [11]. > > Now, that we have laid down the foundation for understanding > "Write-Behind" and how this works in Apache Geode (as well as Pivotal > GemFire)... *how * > *to do this with Spring Data Geode (/GemFire)? *Simple! > > Assuming you are configuring your application and Geode components using > *Spring > (Data Geode*) XML, first you need to define your AEQ Listener... > > <bean id="ExampleQueueListener" class="example. > ExternalDataStoreAsyncEventListener"> > ... > </bean> > > Obviously, you could inject a JDBC DataSource and wrap it with *Spring's* > JdbcTemplate, or use a SD JPA Repository, or whatever you want, to access > your RDBMS of choice. > > You would code the AEQ Listener to perform a data access operation for > each AsyncEvent passed to the Listener to write to the underlying data > store. Or, if you weren't using the SD JPA Repository abstraction, rather > the *Spring* JdbcTemplate, you could perform a "batch" JDBC data access > operation to the underlying data store as well since the AEQ and associated > Listener deal in "batches" of events. Hint, hint, ;-) > > Next, you need to define your AEQ and register your custom Listener(s). > > <gfe:async-event-queue id="ExampleQueue" > batch-conflation-enabled="true" > batch-size="10" > batch-time-interval="3" > dispatcher-threads="4" > disk-store-ref="ExampleQueueDiskStore" > disk-synchronous="true" > maximum-queue-memory="50" > parallel="false" > persistent="true" > order-policy="KEY"> > <gfe:async-event-listener ref="ExampleQueueListener"/> > </gfe:async-event-queue> > > NOTE: I copied this configuration from one of the tests in the SDG test > suite. This test is just ensuring that "all" the configuration settings > (i.e. XML <gfe:async-event-queue> element attributes) are properly parsed > and configured on the resulting Geode/GemFire AEQ component created by > SDG. Therefore, this is an exhaustive list of attributes available for you > to use and control the behavior of the AEQ. It is NOT representative of > settings that are necessarily appropriate for your application. > > Also know I can register more than 1 AEQ Listener. In fact, I can > register as many AEQ Listeners as I like, all performing a different tasks > as required by my application. > > You may have noticed that the "disk-store-ref" and "disk-synchronous" > attributes were specified. This makes the AEQ "durable". Additionally, > you should keep in mind that the "disk-synchronous" attribute as nothing > to do with the asynchronous, write-behind nature of an AEQ/Listener > associated with the Region implementing asynchronous, "write-behind" > semantics. The disk-synchronous setting specifically controls how events > on the "queue" are stored to disk when the AEQ is durable. > > Finally, you associate the AEQ with your Region, like so... > > <gfe:replicated-region id="ExampleRegionWithAEQ" > > <gfe:async-event-queue-ref bean="ExampleQueue"/*>* > * ...* > </gfe:replicated-region> > > > Note, this entire set of configuration could have easily been achieved > using JavaConfig was well. > > Also note, XML configuration is very flexible in that both the Listener > and Queue could have been defined as "anonymous" *Spring* beans on the > Region. > > You are probably wondering why there is not an example of "Write-Behind" > in the Spring GemFire Examples > <https://github.com/spring-projects/spring-gemfire-examples> [12], and > you would be correct, there should be. However, I am in the process of > completely re-working these examples, all of them, to be *Spring Boot* > based along with JavaConfig based using my new Annotation configuration > model that will greatly simplify configuring both Pivotal GemFire and > Apache Geode. > > Anyway, I hope this helps. If you have additional questions, let me know. > > Cheers! > -John > > [1] https://github.com/spring-projects/spring-gemfire- > examples/tree/master/basic/write-through > [2] https://github.com/spring-projects/spring-gemfire- > examples/blob/master/basic/write-through/src/main/ > resources/cache-config.xml#L16-L20 > [3] https://github.com/spring-projects/spring-gemfire- > examples/blob/master/spring-gemfire-examples-common/src/ > main/java/org/springframework/data/gemfire/examples/domain/Product.java > [4] http://geode.apache.org/releases/latest/javadoc/org/ > apache/geode/cache/CacheLoader.html > [5] http://geode.apache.org/releases/latest/javadoc/org/ > apache/geode/cache/CacheWriter.html > [6] http://geode.apache.org/docs/guide/developing/outside_ > data_sources/chapter_overview.html > [7] http://geode.apache.org/releases/latest/javadoc/org/ > apache/geode/cache/asyncqueue/AsyncEventQueue.html > [8] http://geode.apache.org/releases/latest/javadoc/org/ > apache/geode/cache/asyncqueue/AsyncEventListener.html > [9] http://geode.apache.org/docs/guide/reference/topics/ > cache_xml.html#async-event-queue > [10] http://geode.apache.org/releases/latest/javadoc/org/ > apache/geode/cache/asyncqueue/AsyncEventListener.html > [11] http://geode.apache.org/docs/guide/developing/events/ > implementing_write_behind_event_handler.html > > > On Sat, Dec 10, 2016 at 8:15 AM, Amit Pandey <[email protected]> > wrote: > >> Sorry to be pedantic. Thats the example which confused me most. That says >> a async profile is setup . >> >> " Spring Data JPA Repository is used to access the database. The 'async' >> command line argument enables the 'async' Spring profile which enables the >> asynchronous behavior in ProductDBWriter." >> >> What I am confused is , we need to set up event queues etc for backup, we >> can make them persistable ensuring no data loss, where are those done? >> >> Regards >> >> On Sat, Dec 10, 2016 at 9:04 PM, Nilkanth Patel < >> [email protected]> wrote: >> >>> Have a look at following if that helps! >>> >>> https://github.com/spring-projects/spring-gemfire-examples >>> >>> Nilkanth Patel. >>> >>> On Sat, Dec 10, 2016 at 7:24 PM, Amit Pandey <[email protected]> >>> wrote: >>> >>>> Thanks I understood this. >>>> >>>> I want to know how to do this with Spring Data geode? >>>> >>>> On Sat, Dec 10, 2016 at 6:24 PM, Avinash Dongre <[email protected]> >>>> wrote: >>>> >>>>> This should help. >>>>> >>>>> http://geode.apache.org/docs/guide/developing/events/impleme >>>>> nting_write_behind_event_handler.html >>>>> >>>>> Thanks >>>>> Avinash >>>>> >>>>> >>>>> On Sat, Dec 10, 2016 at 4:22 PM, Amit Pandey < >>>>> [email protected]> wrote: >>>>> >>>>>> Hey Guys can anyone please provide a link? >>>>>> >>>>>> On Fri, Dec 9, 2016 at 8:24 PM, Amit Pandey < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Guys, >>>>>>> >>>>>>> Is there any geode example of write behind with Spring Gemfire or >>>>>>> even without it using reliable event queue (which persists to disk ) ? >>>>>>> >>>>>>> Looking at this example what config changes will enable write behind >>>>>>> keeping write order and not write orders with a reliable event queue. >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > > > -- > -John > 503-504-8657 > john.blum10101 (skype) >
