If we don't serialize the data when we store it in the cache, doesn't that defeat the purpose of having an OpaqueValue in order to keep transactional consistency and the processed exactly once semantics?
On Thu, Jun 12, 2014 at 8:57 AM, Raphael Hsieh <raffihs...@gmail.com> wrote: > How come we only serialize data when storing it into the external > datastore and not the local cache then? > > > On Thu, Jun 12, 2014 at 7:56 AM, Romain Leroux <leroux....@gmail.com> > wrote: > >> >> https://github.com/nathanmarz/trident-memcached/blob/master/src/jvm/trident/memcached/MemcachedState.java#L62 >> localCacheSize sets the size of the LRU cache used locally, so no >> CachedMap data are not replicated among all host. >> >> The idea is that you use this on a grouped stream to store the result of >> your aggregation. >> So based on the hash (the groupBy fields) all related tuples always go to >> the same host. >> You want to avoid storing the same cached data in all hosts, otherwise >> scaling out is meaningless. >> >> >> >> 2014-06-12 6:58 GMT+09:00 Raphael Hsieh <raffihs...@gmail.com>: >> >> I am imitating the MemCachedState class found here: >>> https://github.com/nathanmarz/trident-memcached/blob/master/src/jvm/trident/memcached/MemcachedState.java >>> . >>> >>> I was wondering what the CachedMap is used for. I see that it creates a >>> cache layer between the final datastore in order to retrieve values much >>> quicker than accessing the datastore every time, however I am unsure about >>> a couple details. >>> >>> Is this cache replicated among all hosts? When I do a 'multiGet' I >>> expect to retrieve data I had previously stored. If the cache is specific >>> to each host, I wouldn't necessarily get the same data I had most recently >>> stored. >>> >>> Also, how does this work with Opaque Transactional consistency? It seems >>> that in the MemCachedState example we serialize the data once we store it >>> to the external datastore, however the data in the cache is not serialized. >>> why is this? >>> Shouldn't the local cache have the same data as the external datastore ? >>> >>> thanks >>> -- >>> Raphael Hsieh >>> >>> >>> >>> >> >> > > > -- > Raphael Hsieh > > -- Raphael Hsieh Amazon.com Software Development Engineer I (978) 764-9014
