Hi there Paul.
We will assume that you are using partitioned regions. In partitioned
regions you have the notion of "primary" and "redundant" data copies.
Any CUD (create,update,destroy) operations will ALWAYS only happen on
the primary node. Which means that with an AsyncEventListener, it will
only ever "fire" on the primary data node.
So no, you will not have the AsyncEventListener fire 3 times.
With a replicate region, the AsyncEventListener will fire 3 times.
The concept of serial vs parallel just means the amount to
threads/executors that each AsyncEventListener will use. With serial,
there will only be 1. With parallel you could have many threads, but
once again an event will only ever be processed by one of the
AsyncEventListener threads. (if you are using partitioned regions).
You can try this out if you want.
--Udo
On 1/19/17 10:42, Paul Perez wrote:
Hello Michael
I did not see your answer before replying to Udo so may be in my last
email I made mistakes and wrote wrong things.
We also though about AsyncEventListeners but we found a difficulty in.
Geode documentation says:
/“You can configure an AsyncEventQueue to be either serial or
parallel. A serial queue is deployed to one Geode member, and it
delivers all of a region’s events, in order of occurrence, to a
configured AsyncEventListener implementation. A parallel queue is
deployed to multiple Geode members, and each instance of the queue
delivers region events, possibly simultaneously, to a local
AsyncEventListener implementation.”/
Let’s say that for scalability reason we have 3 members in our
aggregation system, and we implement the aggregation process in the
Listener. We understand that it will be invoked 3 times. And since an
aggregation is not a stateless process the aggregation will be wrong.
May be I’m wrong and I did not understand the documentation. I would
be very happy with that.
Please let me know
Best regards
Paul
*From:*Michael Stolz [mailto:[email protected]]
*Sent:* 19 January 2017 16:55
*To:* [email protected]
*Cc:* [email protected]; [email protected]
*Subject:* Re: Send an asynchronous event to one client among many
Instead of hopping out to a client, you could get horizontal scale and
asynchronous processing by using an AsyncEventListener in the servers.
That will take care of multi-threading and queuing and all the
plumbing, and you just go ahead and write your processing code and
deploy it as AsyncEventListeners. This gives you guaranteed ordering
semantics for each key as well.
I *think* it even gives you a notion of H/A so that if the primary
fails the queued messages will be processed by a secondary. (I know
the WAN Gateway does and it uses pretty much the same plumbing under
the covers).
--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: 631-835-4771
On Thu, Jan 19, 2017 at 11:21 AM, Udo Kohlmeyer <[email protected]
<mailto:[email protected]>> wrote:
Hi there Paul,
Firstly, your use case looks really interesting and hope to see a
few more posts on how you use Geode further. Keep us informed we
like to hear what you guys are doing with GEODE! :)
The subscription or CQ (continuous query) paradigm is, as stated,
a 1:1 relationship. When a client registers interest on a region
that client will be notified. This is more of a topic semantic
rather than a queue semantic.
Although this is not the first time I've heard the request for
this kind of functionality. To best explain why GEODE, currently,
implements the 1:1 relationship has got to do with guaranteed
delivery and in-order delivery. If we use a queue semantic, with
multiple clients being able to process data in a balanced manner,
we end up with potential out-of-order processing of messages. In
addition to that it now becomes significantly harder to track and
n deal with client failures and the potential replaying of messages.
But that said, I have seen other users resolve this problem and
could detail some approaches in a later correspondence if you'd like.
--Udo
On 1/19/17 03:39, Paul Perez wrote:
Hello All,
As explained in a previous email, we try to use Geode to
process and aggregate a stream of Traces. Our requirement is
to process billions of simple traces every day.
We imagine the aggregation process in many steps.
One: traces are generated by a tiers tools and stored in a
first geode region
Two: once a trace put in the first region we use the async
event feature to invoke a client that executes the first
aggregation steps. Then the result will be put in a second
region.
Three: the second aggregation step is in the same way, when
traces are put in the second region, then an asynchronous
event is sent to the client to execute the second part of
the aggregation etc.…
For scalability purposes, we plan to use many clients that
could receive the events and execute the aggregation and put
the results back to Geode.
Consequently, as far as we understand the documentation, when
an entry is put in a region, each client that registered an
interest receives an event and aggregate the trace. So, the
trace will be aggregated many times.
My question is: If many clients are registered, could we
configure the region to send randomly, the event to one client
only.
A subsidiary question: Do we have the same behaviour with the
function execution feature or it could be an alternative in
that case
Thank you for your help
Best regards
Paul
