Ok, thanks. Moved the discussion here: http://apache-ignite-developers.2346864.n4.nabble.com/Continuous-queries-and-duplicates-td39444.html
wt., 11 gru 2018 o 14:05 Ilya Kasnacheev <ilya.kasnach...@gmail.com> napisał(a): > Hello! > > You could write this message to developers list in a separate thread, see > if there will be any discussion. > > Regards, > -- > Ilya Kasnacheev > > > ср, 5 дек. 2018 г. в 18:29, piotr.romanski <piotr.roman...@gmail.com>: > >> Hi all, I think that Krzysztof raised a valid concern. Actually, in my >> opinion the manual deduplication in some use cases may lead to possible >> memory problems on the client side. In order to remove duplicated >> notifications which we are receiving in the local listener, we need to >> keep >> all initial query results in memory (or at least their unique ids). >> Unfortunately, there is no way (is there?) to find a point in time when we >> can be sure that no dups will arrive anymore. That would mean that we need >> to keep that data indefinitely and use it every time a new notification >> arrives. In case of multiple continuous queries run from a single JVM, >> this >> might eventually become a memory or performance problem. I can see the >> following possible improvements to Ignite: >> >> 1. The deduplication between initial query and incoming notification could >> be done fully in Ignite. As far as I know there is already the >> updateCounter >> and partition id for all the objects so it could be used internally. >> >> 2. Add a guarantee that notifications arriving in the local listener after >> query() method returns are not duplicates. This kind of functionality >> would >> require a specific synchronization inside Ignite. It would also mean that >> the query() method cannot return before all potential duplicates are >> processed by a local listener what looks wrong. >> >> 3. Notify users that starting from a given notification they can be sure >> they will not receive any duplicates anymore. This could be an additional >> boolean flag in the CacheQueryEntryEvent. >> >> 4. CacheQueryEntryEvent already exposes the partitionUpdateCounter. >> Unfortunately we don't have this information for initial query results. If >> we had, a client could manually deduplicate notifications and get rid of >> initial query results for a given partition after newer notifications >> arrive. Also it would be very convenient to expose partition id as well >> but >> now we can figure it out using the affinity service. The assumption here >> is >> that notifications are ordered by partitionUpdateCounter (is it true?). >> >> Please correct me if I'm missing anything. >> >> What do you think? >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
