Re: Continuous Query Questions
So, if I define the CQ as a service, and the node crashes, wouldn't ignite start a new service with CQ already registered, say if the CQ registration is in service init? I can do initial query at the as well. On Wed, Apr 1, 2020 at 7:33 PM Evgenii Zhuravlev wrote: > Well, with this use case, if one of the nodes goes down, there is always a > chance to lost notifications. I don't think that it's possible to recover > lost notifications with out of the box solution, but if you will be able to > track the last processed notification and store update time in entries, you > will be able to find not processed entries. Otherwise, you will need to > register CQ again and process all the entries using initialQuery. > > Evgenii > > ср, 1 апр. 2020 г. в 13:16, narges saleh : > >> Thanks Evgenii for the recommendation and the heads up. >> >> Is there a way to recover the lost notifications or even know if a >> notification is lost? >> >> On Wed, Apr 1, 2020 at 12:15 PM Evgenii Zhuravlev < >> e.zhuravlev...@gmail.com> wrote: >> >>> Hi, >>> >>> 1) I would recommend checking ContinouousQuery.setLocal: >>> https://www.gridgain.com/sdk/ce/latest/javadoc/org/apache/ignite/cache/query/Query.html#setLocal-boolean-. >>> Please check if it fits your requirements. >>> 2) You will need to do this in a separate thread, because cache >>> operations shouldn't be used inside CQ listeners, as they are executed >>> synchronously. >>> >>> In case of using local CQ, there is a chance to miss notifications in >>> case of node failure, it's described in javadoc. >>> >>> Evgenii >>> >>> >>> вт, 31 мар. 2020 г. в 03:00, narges saleh : >>> Hi All, I'd like to get your feedback regarding the following pattern. 1) CQ setup that listens to the changes to a cache on the local node only. 2) Upon receiving notification on a change, the listener makes additions to two other caches, one being on the local node (partitioned) and the other cache being replicated across all the nodes in the cluster. Is this setup performant and reliable in terms of the data staying in sync across the cluster? thanks.
Re: Continuous Query Questions
Well, with this use case, if one of the nodes goes down, there is always a chance to lost notifications. I don't think that it's possible to recover lost notifications with out of the box solution, but if you will be able to track the last processed notification and store update time in entries, you will be able to find not processed entries. Otherwise, you will need to register CQ again and process all the entries using initialQuery. Evgenii ср, 1 апр. 2020 г. в 13:16, narges saleh : > Thanks Evgenii for the recommendation and the heads up. > > Is there a way to recover the lost notifications or even know if a > notification is lost? > > On Wed, Apr 1, 2020 at 12:15 PM Evgenii Zhuravlev < > e.zhuravlev...@gmail.com> wrote: > >> Hi, >> >> 1) I would recommend checking ContinouousQuery.setLocal: >> https://www.gridgain.com/sdk/ce/latest/javadoc/org/apache/ignite/cache/query/Query.html#setLocal-boolean-. >> Please check if it fits your requirements. >> 2) You will need to do this in a separate thread, because cache >> operations shouldn't be used inside CQ listeners, as they are executed >> synchronously. >> >> In case of using local CQ, there is a chance to miss notifications in >> case of node failure, it's described in javadoc. >> >> Evgenii >> >> >> вт, 31 мар. 2020 г. в 03:00, narges saleh : >> >>> Hi All, >>> I'd like to get your feedback regarding the following pattern. >>> >>> 1) CQ setup that listens to the changes to a cache on the local node >>> only. >>> 2) Upon receiving notification on a change, the listener makes additions >>> to two other caches, one being on the local node (partitioned) and the >>> other cache being replicated across all the nodes in the cluster. >>> >>> Is this setup performant and reliable in terms of the data staying in >>> sync across the cluster? >>> >>> thanks. >>> >>> >>>
Re: Continuous Query Questions
Thanks Evgenii for the recommendation and the heads up. Is there a way to recover the lost notifications or even know if a notification is lost? On Wed, Apr 1, 2020 at 12:15 PM Evgenii Zhuravlev wrote: > Hi, > > 1) I would recommend checking ContinouousQuery.setLocal: > https://www.gridgain.com/sdk/ce/latest/javadoc/org/apache/ignite/cache/query/Query.html#setLocal-boolean-. > Please check if it fits your requirements. > 2) You will need to do this in a separate thread, because cache operations > shouldn't be used inside CQ listeners, as they are executed synchronously. > > In case of using local CQ, there is a chance to miss notifications in case > of node failure, it's described in javadoc. > > Evgenii > > > вт, 31 мар. 2020 г. в 03:00, narges saleh : > >> Hi All, >> I'd like to get your feedback regarding the following pattern. >> >> 1) CQ setup that listens to the changes to a cache on the local node only. >> 2) Upon receiving notification on a change, the listener makes additions >> to two other caches, one being on the local node (partitioned) and the >> other cache being replicated across all the nodes in the cluster. >> >> Is this setup performant and reliable in terms of the data staying in >> sync across the cluster? >> >> thanks. >> >> >>
Re: Continuous Query Questions
Hi, 1) I would recommend checking ContinouousQuery.setLocal: https://www.gridgain.com/sdk/ce/latest/javadoc/org/apache/ignite/cache/query/Query.html#setLocal-boolean-. Please check if it fits your requirements. 2) You will need to do this in a separate thread, because cache operations shouldn't be used inside CQ listeners, as they are executed synchronously. In case of using local CQ, there is a chance to miss notifications in case of node failure, it's described in javadoc. Evgenii вт, 31 мар. 2020 г. в 03:00, narges saleh : > Hi All, > I'd like to get your feedback regarding the following pattern. > > 1) CQ setup that listens to the changes to a cache on the local node only. > 2) Upon receiving notification on a change, the listener makes additions > to two other caches, one being on the local node (partitioned) and the > other cache being replicated across all the nodes in the cluster. > > Is this setup performant and reliable in terms of the data staying in sync > across the cluster? > > thanks. > > >
Continuous Query Questions
Hi All, I'd like to get your feedback regarding the following pattern. 1) CQ setup that listens to the changes to a cache on the local node only. 2) Upon receiving notification on a change, the listener makes additions to two other caches, one being on the local node (partitioned) and the other cache being replicated across all the nodes in the cluster. Is this setup performant and reliable in terms of the data staying in sync across the cluster? thanks.
Re: Continuous Query Questions
I'm not sure, because the final overhead depends on the object sizes. There is a buffer for CQ, which stores 1000 entries by default, but you can decrease it using property IGNITE_CONTINUOUS_QUERY_SERVER_BUFFER_SIZE Evgenii вт, 18 февр. 2020 г. в 18:09, narges saleh : > Hi Evgeni, > > There will be several thousands notifications/day if I have it send > notification only when certain patterns are visited, in about 100+ caches, > which brings up another question: wouldn't having 100+ CQs be creating too > much overhead? > > thanks. > > On Tue, Feb 18, 2020 at 2:17 PM Evgenii Zhuravlev < > e.zhuravlev...@gmail.com> wrote: > >> Hi, >> >> How many notifications do you want to get? If it's just a several >> notifications, then you can even register separate CQ for each of the entry >> with its own remote filters. At the same time, if you have a requirement to >> send these notifications for a lot of entries, then this approach will >> create a big overhead. >> >> Its possible to unregister a CQ after you get first notification - you >> just need to return FALSE from a remote filter. Also, you can send not >> exact entry, but only some fields using Transformer: >> https://www.gridgain.com/docs/latest/developers-guide/key-value-api/continuous-queries#remote-transformer. >> You can create some another object, which will contain only part of the >> fields. >> >> Best Regards, >> Evgeni >> >> >> пн, 17 февр. 2020 г. в 03:58, narges saleh : >> >>> Hi All, >>> I am getting the following streams of the following records: >>> name, org, year, month, day >>> 1- john, acc, 2004, 2, 1 >>> 2- pete, rd, 2004, 3,1 >>> 3- jim,hr,2004, 5,2 >>> 4- jerry,math,2005,2,1 >>> 5- betty,park,2005,3,2 >>> 6- carry,acc,2006,1,1 >>> >>> I want to get notification for the first occurrence of a particular >>> value. So, I want to get notifications when I get records 1, 4 and 6, and >>> in this case, I want to get the fields, org, and year back only. >>> >>> Questions: >>> 1) Is CQ overkill in this case? If yes, what's a better alternative? >>> 2) If not, how can I set up CQ to get only one record per occurrence? >>> 3) How would I return only org and year back with the CQ transformer, >>> considering that I am working with a flat object? Note that in reality this >>> record has 25-30 fields (I am showing only 5 of them). >>> >>> thanks. >>> >>
Re: Continuous Query Questions
Hi Evgeni, There will be several thousands notifications/day if I have it send notification only when certain patterns are visited, in about 100+ caches, which brings up another question: wouldn't having 100+ CQs be creating too much overhead? thanks. On Tue, Feb 18, 2020 at 2:17 PM Evgenii Zhuravlev wrote: > Hi, > > How many notifications do you want to get? If it's just a several > notifications, then you can even register separate CQ for each of the entry > with its own remote filters. At the same time, if you have a requirement to > send these notifications for a lot of entries, then this approach will > create a big overhead. > > Its possible to unregister a CQ after you get first notification - you > just need to return FALSE from a remote filter. Also, you can send not > exact entry, but only some fields using Transformer: > https://www.gridgain.com/docs/latest/developers-guide/key-value-api/continuous-queries#remote-transformer. > You can create some another object, which will contain only part of the > fields. > > Best Regards, > Evgeni > > > пн, 17 февр. 2020 г. в 03:58, narges saleh : > >> Hi All, >> I am getting the following streams of the following records: >> name, org, year, month, day >> 1- john, acc, 2004, 2, 1 >> 2- pete, rd, 2004, 3,1 >> 3- jim,hr,2004, 5,2 >> 4- jerry,math,2005,2,1 >> 5- betty,park,2005,3,2 >> 6- carry,acc,2006,1,1 >> >> I want to get notification for the first occurrence of a particular >> value. So, I want to get notifications when I get records 1, 4 and 6, and >> in this case, I want to get the fields, org, and year back only. >> >> Questions: >> 1) Is CQ overkill in this case? If yes, what's a better alternative? >> 2) If not, how can I set up CQ to get only one record per occurrence? >> 3) How would I return only org and year back with the CQ transformer, >> considering that I am working with a flat object? Note that in reality this >> record has 25-30 fields (I am showing only 5 of them). >> >> thanks. >> >
Re: Continuous Query Questions
Hi, How many notifications do you want to get? If it's just a several notifications, then you can even register separate CQ for each of the entry with its own remote filters. At the same time, if you have a requirement to send these notifications for a lot of entries, then this approach will create a big overhead. Its possible to unregister a CQ after you get first notification - you just need to return FALSE from a remote filter. Also, you can send not exact entry, but only some fields using Transformer: https://www.gridgain.com/docs/latest/developers-guide/key-value-api/continuous-queries#remote-transformer. You can create some another object, which will contain only part of the fields. Best Regards, Evgeni пн, 17 февр. 2020 г. в 03:58, narges saleh : > Hi All, > I am getting the following streams of the following records: > name, org, year, month, day > 1- john, acc, 2004, 2, 1 > 2- pete, rd, 2004, 3,1 > 3- jim,hr,2004, 5,2 > 4- jerry,math,2005,2,1 > 5- betty,park,2005,3,2 > 6- carry,acc,2006,1,1 > > I want to get notification for the first occurrence of a particular value. > So, I want to get notifications when I get records 1, 4 and 6, and in this > case, I want to get the fields, org, and year back only. > > Questions: > 1) Is CQ overkill in this case? If yes, what's a better alternative? > 2) If not, how can I set up CQ to get only one record per occurrence? > 3) How would I return only org and year back with the CQ transformer, > considering that I am working with a flat object? Note that in reality this > record has 25-30 fields (I am showing only 5 of them). > > thanks. >
Continuous Query Questions
Hi All, I am getting the following streams of the following records: name, org, year, month, day 1- john, acc, 2004, 2, 1 2- pete, rd, 2004, 3,1 3- jim,hr,2004, 5,2 4- jerry,math,2005,2,1 5- betty,park,2005,3,2 6- carry,acc,2006,1,1 I want to get notification for the first occurrence of a particular value. So, I want to get notifications when I get records 1, 4 and 6, and in this case, I want to get the fields, org, and year back only. Questions: 1) Is CQ overkill in this case? If yes, what's a better alternative? 2) If not, how can I set up CQ to get only one record per occurrence? 3) How would I return only org and year back with the CQ transformer, considering that I am working with a flat object? Note that in reality this record has 25-30 fields (I am showing only 5 of them). thanks.
