Hello Udo,
Thank you for your prompt answer. Managing another open source forum, I know how it is difficult to reply promptly and how we are happy to know what the users do with the product we are working on. So further to your answer, let me reformulate our concern with an architect's vision and point out a kind of lack in Geode in the field it wants to play in. We see two designs for our aggregation. The first could rely on a pull design. This means that the aggregator invokes the persistence system to get "food to eat". The second could be a push design where the persistence system sends messages to the aggregator through events and send the "Food to eat" to the aggregator. Our background pushes us to the second solution and it was the way we used Gemfire (LONG TIME AGO) to ridicule pull designs based on databases in bank and financial projects. In these projects, we had a "1:1 relationship" and each event type was processed by one and only one application. Since, many things changed and a 1:1 relationship in a push architecture cannot keep up the current requirements. Today, large applications need to scale linearly. So Geode highlight its capability to scale and store more and more data in a better way that anyone else and I believe it is true. In the same way, Geode offers many nice features such as the events and the functions. Unfortunately, these last features cannot scale linearly because of the "1:1 relationship". I'm impatient to read the result of your search. In the meantime, maybe a pull solution could be enough and scalable too. Another way would be to use an additional tool that could scale the aggregation process such as Storm. But scaling Geode persistence and processing capability at the same time would be perfect. Thank you for your time and your patience with me Best regards Paul From: Udo Kohlmeyer [mailto:[email protected]] Sent: 19 January 2017 16:21 To: [email protected]; [email protected] Cc: [email protected] Subject: Re: Send an asynchronous event to one client among many Hi there Paul, Firstly, your use case looks really interesting and hope to see a few more posts on how you use Geode further. Keep us informed we like to hear what you guys are doing with GEODE! :) The subscription or CQ (continuous query) paradigm is, as stated, a 1:1 relationship. When a client registers interest on a region that client will be notified. This is more of a topic semantic rather than a queue semantic. Although this is not the first time I've heard the request for this kind of functionality. To best explain why GEODE, currently, implements the 1:1 relationship has got to do with guaranteed delivery and in-order delivery. If we use a queue semantic, with multiple clients being able to process data in a balanced manner, we end up with potential out-of-order processing of messages. In addition to that it now becomes significantly harder to track and n deal with client failures and the potential replaying of messages. But that said, I have seen other users resolve this problem and could detail some approaches in a later correspondence if you'd like. --Udo On 1/19/17 03:39, Paul Perez wrote: Hello All, As explained in a previous email, we try to use Geode to process and aggregate a stream of Traces. Our requirement is to process billions of simple traces every day. We imagine the aggregation process in many steps. One: traces are generated by a tiers tools and stored in a first geode region Two: once a trace put in the first region we use the async event feature to invoke a client that executes the first aggregation steps. Then the result will be put in a second region. Three: the second aggregation step is in the same way, when traces are put in the second region, then an asynchronous event is sent to the client to execute the second part of the aggregation etc.. For scalability purposes, we plan to use many clients that could receive the events and execute the aggregation and put the results back to Geode. Consequently, as far as we understand the documentation, when an entry is put in a region, each client that registered an interest receives an event and aggregate the trace. So, the trace will be aggregated many times. My question is: If many clients are registered, could we configure the region to send randomly, the event to one client only. A subsidiary question: Do we have the same behaviour with the function execution feature or it could be an alternative in that case Thank you for your help Best regards Paul
