FTR, I am not opposed to *Naba's* idea either. +1 I kind of like the idea of having a global, cache-wide call that can coordinate the background initialization/processing of all other Geode objects that have lifecycle processes starting in the background (e.g. Gateways, AEQs, etc).
On Tue, Aug 20, 2019 at 2:38 PM Kirk Lund <kl...@apache.org> wrote: > If we need to add pause/resume processing to Cache, I suggest adding > setAutostart(boolean) to CacheFactory and start() to Cache to do something > like this: > > Cache cache = new CacheFactory() > *.setAutostart(false)* > .create(); > cache.createRegionFactory(...)... > cache.createAsyncEventQueueFactory(...)... > cache.addCacheServer()... > cache.*start()*; > > AEQs, CacheServers, GatewaySenders/Receivers and any other endpoints that > are created would not be started until invoking cache.start(). > > Cheers, > Kirk > > On Tue, Aug 20, 2019 at 2:01 PM Nabarun Nag <n...@apache.org> wrote: > > > Hi Anil, > > > > Will it be possible to explain to the community how the starting AEQ in a > > paused state is different from creating gateway senders with manual start > > set to true. It may be of concern as 'manual start' for gateways is a > > deprecated. > > > > Just thinking out loud, will it be more feasible if we can set the flag > at > > cache level. Any framework that is starting up Apache Geode (E.g: > Spring) , > > creates the cache -> cache.pauseProcessing(); -> create regions -> create > > AEQs -> cache.unpauseProcessing() > > > > We can gate the processing of all event listener at dispatchBatch(). > > > > The advantage I feel is that > > - we avoid introducing a new API to the AEQ creation factory. > > - if we created 100 AEQs in paused state then we avoid having to have > 100 > > AEQ.unpause calls. > > > > > > Regards > > Naba > > > > > > On Tue, Aug 20, 2019 at 9:07 AM Michael Stolz <mst...@pivotal.io> wrote: > > > > > Manual start has caused a lot of trouble over the years. We should > > > definitely circle back on those issues before traveling very far down > > this > > > road. > > > > > > -- > > > Mike Stolz > > > Principal Engineer, Pivotal Cloud Cache > > > Mobile: +1-631-835-4771 > > > > > > > > > > > > On Tue, Aug 20, 2019 at 11:56 AM Juan José Ramos <jra...@pivotal.io> > > > wrote: > > > > > > > Hello Anil, > > > > > > > > +1 for the proposed solution. > > > > I'd change the method name from *pauseEventDispatchToListener* to > > > something > > > > more meaningful and understandable for our users, maybe > *startPaused*?, > > > > *setManualStart* (as we currently have for the > > *GatewaySenderFactory*)?, > > > > *startWithEventDispatcherPaused*?. > > > > Best regards. > > > > > > > > > > > > > > > > On Sat, Aug 17, 2019 at 12:55 AM Anilkumar Gingade < > > aging...@pivotal.io> > > > > wrote: > > > > > > > > > I have updated the wiki based on Dan's comment. > > > > > Changes with api: > > > > > > > > > > *On "AsyncEventQueueFactory" interface - * > > > > > > > > > > *AsyncEventQueueFactory pauseEventDispatchToListener(); *// This > > > causes > > > > > AEQ to be created with paused state. > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 4:36 PM Anilkumar Gingade < > > aging...@pivotal.io > > > > > > > > > wrote: > > > > > > > > > > > Dan, > > > > > > > > > > > > If you look into the API; the AEQ will be created with the pause > > > state. > > > > > > The user (application) has to call resume to dispatch the events. > > > > > > > > > > > > It will be slightly different from GatewaySender behavior; where > > > > > > GatewaySender will be created with run mode and then application > > has > > > to > > > > > > call pause on it. Here in this case AEQ will be created with > paused > > > > > state. > > > > > > > > > > > > -Anil. > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2019 at 4:31 PM Dan Smith <dsm...@pivotal.io> > > wrote: > > > > > > > > > > > >> Hi Anil, > > > > > >> > > > > > >> While I like the idea of matching the API of GatewaySender, I'm > > not > > > > > sure I > > > > > >> see how this solves the problem. Is it required of the user to > > call > > > > > pause > > > > > >> on the AsyncEventQueue as soon as it is created? How would > someone > > > do > > > > > that > > > > > >> when creating AEQs with xml or cluster configuration? Maybe it > > would > > > > be > > > > > >> better to not dispatch any events until we are done creating all > > > > > regions? > > > > > >> > > > > > >> -Dan > > > > > >> > > > > > >> On Fri, Aug 16, 2019 at 2:31 PM Anilkumar Gingade < > > > > aging...@pivotal.io> > > > > > >> wrote: > > > > > >> > > > > > >> > Proposal to support controlling capability with event dispatch > > to > > > > > >> > AsyncEventQueue Listener. > > > > > >> > > > > > > >> > Wiki proposal page: > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/GEODE/%5BDraft%5D+Controlling+event+dispatch+to+AsyncEventListener > > > > > >> > > > > > > >> > Here is the details from the wiki page: > > > > > >> > *Problem* > > > > > >> > > > > > > >> > *The Geode system requires AEQs to be configured before > regions > > > are > > > > > >> > created. If an AEQ listener is operating on a secondary > region, > > > this > > > > > >> could > > > > > >> > cause listener to operate on a region which is not yet created > > or > > > > > fully > > > > > >> > initialized (for region with co-located regions) which could > > > result > > > > in > > > > > >> > missing events or dead-lock scenario between region > (co-located > > > > > region) > > > > > >> > creation threads. This scenario is likely to happen during > > > > persistence > > > > > >> > recovery; when AEQs are created in the start, the recovered > AEQ > > > > events > > > > > >> are > > > > > >> > dispatched immediately, thus invoking the AEQ listeners.* > > > > > >> > Anti-Goals > > > > > >> > > > > > > >> > None > > > > > >> > *Solution* > > > > > >> > > > > > > >> > *The proposed solution is to provide a way to control > > dispatching > > > > AEQ > > > > > >> > events to the AEQ Listeners, this could be done by adding > > "pause" > > > > and > > > > > >> > "resume" capability to the AEQ, which will allow application > to > > > > decide > > > > > >> when > > > > > >> > to dispatch events to the listeners. * > > > > > >> > > > > > > >> > > > > > > >> > *The proposal is similar to existing "pause" and "resume" > > behavior > > > > on > > > > > >> the > > > > > >> > GatewaySender, on which the AEQ is based on (AEQ > implementation > > > is a > > > > > >> > wrapper around GatewaySender). * > > > > > >> > Changes and Additions to Public Interfaces > > > > > >> > > > > > > >> > *The proposed APIs are:* > > > > > >> > > > > > > >> > *On "AsyncEventQueueFactory" interface - * > > > > > >> > > > > > > >> > *AsyncEventQueue pauseEventDispatchToListener();* > > > > > >> > > > > > > >> > *On "AsyncEventQueue" interface -* > > > > > >> > > > > > > >> > *boolean resumeEventDispatchToListener(); **returns true or > > false > > > if > > > > > the > > > > > >> > event dispatch is resumed successfully.* > > > > > >> > > > > > > >> > > > > > > >> > *The constraints on the pauseEventDispatchToListener() will > > remain > > > > > >> similar > > > > > >> > to as in "GatewaySender.pause()" :* > > > > > >> > > > > > > >> > "It should be kept in mind that the events will still be > getting > > > > > queued > > > > > >> > into the queue. The scope of this operation is the VM on which > > it > > > is > > > > > >> > invoked. In case the AEQ is parallel, the AEQ will be paused > on > > > > > >> individual > > > > > >> > node where this API is called and the AEQ on other VM's can > > still > > > > > >> dispatch > > > > > >> > events. In case the AEQ is not parallel, and the running AEQ > on > > > > which > > > > > >> this > > > > > >> > API is invoked is not primary then primary AEQ will still > > continue > > > > > >> > dispatching events." > > > > > >> > Performance Impact > > > > > >> > > > > > > >> > > > > > > >> > *This will have similar performance and resource implication > as > > > with > > > > > the > > > > > >> > "GatewaySender.pause()" functionality. If the AEQ is not > resumed > > > or > > > > > >> kept in > > > > > >> > "pause" state for long, it may start consuming the configured > > > memory > > > > > and > > > > > >> > overflow it into disk and may cause disk full scenario.* > > > > > >> > Backwards Compatibility and Upgrade Path > > > > > >> > > > > > > >> > *Impact with rolling upgrade: * > > > > > >> > > > > > > >> > *As the api is applicable at individual VM level, there is no > > > > message > > > > > >> > serialization changes involved. And only applicable to the > > events > > > > > >> getting > > > > > >> > dispatched to the listeners on that VM. And the AEQ which are > > > > > replicated > > > > > >> > (for redundancy) continues to work as before.* > > > > > >> > > > > > > >> > *Backward compatibility requirements: * > > > > > >> > > > > > > >> > *None. The AEQs are configured and managed at the server side. > > > There > > > > > is > > > > > >> no > > > > > >> > messaging involved between client/server.* > > > > > >> > > > > > > >> > *Disk formatting changes:* > > > > > >> > > > > > > >> > *None.* > > > > > >> > > > > > > >> > *Deprecation and Application Changes:* > > > > > >> > > > > > > >> > > > > > > >> > *None. If needed, the existing application can be modified to > > > > control > > > > > >> event > > > > > >> > dispatch with AEQ listener.* > > > > > >> > Prior Art > > > > > >> > > > > > > >> > *Without this, the AEQ listeners operating on other regions > > could > > > > > >> > experience missing events or dead lock, if there are > co-located > > > > > >> regions.* > > > > > >> > > > > > > >> > *This approach is simple and can take advantage of the > existing > > > > > >> > functionality that is already supported in GatewaySender on > > which > > > > AEQ > > > > > is > > > > > >> > based on.* > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Juan José Ramos Cassella > > > > Senior Software Engineer > > > > Email: jra...@pivotal.io > > > > > > > > > > -- -John john.blum10101 (skype)
