I'll create a wiki to compile the details/ideas of this discussion so that we can hash this out further.
On May 18, 2011, at 1:13 AM, Bela Ban wrote: > This is exactly what JGroups digests do > > On 05/17/2011 10:38 AM, Manik Surtani wrote: >> Interesting discussions. Another approach may be to version data using >> lamport clocks or vector clocks. Then at the start of a rehash, a digest of >> keys and versions can be pushed, and the receiver 'decides' which keys are >> out of date and needs to be pulled from across the network. >> >> On 17 May 2011, at 12:06, Sanne Grinovero wrote: >> >>> 2011/5/17 Galder Zamarreño<gal...@redhat.com>: >>>> >>>> On May 16, 2011, at 1:18 PM, Sanne Grinovero wrote: >>>> >>>>> 2011/5/16 Galder Zamarreño<gal...@redhat.com>: >>>>>> Not sure if the idea has come up but while at GeeCON last week I was >>>>>> discussing to one of the attendees about state transfer improvements in >>>>>> replicated environments: >>>>>> >>>>>> The idea is that in a replicated environment, if a cache manager shuts >>>>>> down, it would dump its memory contents to a cache store (i.e. a local >>>>>> filesystem) and when it starts up, instead of going over the network to >>>>>> do state transfer, it would load the state from the local filesystem >>>>>> which would be much quicker. Obviously, at times the cache manager would >>>>>> crash or have some failure dumping the memory contents, so in that case >>>>>> it would fallback on state transfer over the network. I think it's an >>>>>> interesting idea since it could reduce the amount of state transfer to >>>>>> be done. It's true though that there're other tricks if you're having >>>>>> issues with state transfer, such as the use of a cluster cache loader. >>>>>> >>>>>> WDYT? >>>>> >>>>> Well if it's a shared cachestore, then we're using network at some >>>>> level anyway. If we're talking about a not-shared cachestore, how do >>>>> you know which keys/values are still valid and where not updated? and >>>>> about the new keys? >>>> >>>> I see this only being useful with a local cache store cos if you need to >>>> go remote over the network, might as well just do state transfer. >>> >>> +1 >>> >>>> Not sure if the timestamp of creation/update is available per all entries >>>> (i'd need to check the code but maybe immortals do not store it...), but >>>> anyway assuming that a timestamp was stored in the local cache store, on >>>> startup the node could send this timestamp and the coordinator could send >>>> anything new created/updated after that timestamp. >>> >>> This means we'll need an API on the cache stores to return the >>> "highest timestamp"; some like the JDBC cacheloader could implement >>> that with a single query. >>> >>> Not sure how you would handle cases about deleted entries; the other >>> nodes would need to keep a list of deleted keys with timestamps; maybe >>> there could be an option to never delete keys from a cacheloader, only >>> values - and record the timestamp of the operation. >>> >>>> >>>> This would be particularly efficient in situations where you have to >>>> quickly restart a machine for whatever reason and so the deltas are very >>>> small, or when the caches are big and state transfer would cost a lot from >>>> a bandwidth perspective. >>> >>> super; this would be quite useful in the Lucene case, as I can >>> actually figure out which keys should be deleted inferring which ones >>> are "obsolete" from the common metadata (a known key contains this >>> information); and indeed startup time is a point I'd like to improve. >>> >>>> >>>>> >>>>> I like the concept though, let's explore more in this direction. >>>>> >>>>> Sanne >>>>> >>>>>> -- >>>>>> Galder Zamarreño >>>>>> Sr. Software Engineer >>>>>> Infinispan, JBoss Cache >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> infinispan-dev mailing list >>>>>> infinispan-dev@lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>>>> >>>>> >>>>> _______________________________________________ >>>>> infinispan-dev mailing list >>>>> infinispan-dev@lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>>> -- >>>> Galder Zamarreño >>>> Sr. Software Engineer >>>> Infinispan, JBoss Cache >>>> >>>> >>>> _______________________________________________ >>>> infinispan-dev mailing list >>>> infinispan-dev@lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >>>> >>> >>> _______________________________________________ >>> infinispan-dev mailing list >>> infinispan-dev@lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/infinispan-dev >> >> -- >> Manik Surtani >> ma...@jboss.org >> twitter.com/maniksurtani >> >> Lead, Infinispan >> http://www.infinispan.org >> >> >> >> >> _______________________________________________ >> infinispan-dev mailing list >> infinispan-dev@lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/infinispan-dev > > -- > Bela Ban > Lead JGroups / Clustering Team > JBoss > _______________________________________________ > infinispan-dev mailing list > infinispan-dev@lists.jboss.org > https://lists.jboss.org/mailman/listinfo/infinispan-dev -- Galder Zamarreño Sr. Software Engineer Infinispan, JBoss Cache _______________________________________________ infinispan-dev mailing list infinispan-dev@lists.jboss.org https://lists.jboss.org/mailman/listinfo/infinispan-dev
