Alexandr, Could you describe your proposal in more details? Especially in case with several nodes.
On Fri, Nov 11, 2016 at 6:34 PM, Alexandr Kuramshin <[email protected]> wrote: > Hi, > > You know CacheStore API that is commonly used for read/write-through > relationship of the in-memory data with the persistence storage. > > There is also IgniteCache.loadCache method for hot-loading the cache on > startup. Invocation of this method causes execution of CacheStore.loadCache > on the all nodes storing the cache partitions. Because of none keys are > passed to the CacheStore.loadCache methods, the underlying implementation > is forced to read all the data from the persistence storage, but only part > of the data will be stored on each node. > > So, the current implementation have two general drawbacks: > > 1. Persistence storage is forced to perform as many identical queries as > many nodes on the cluster. Each query may involve much additional > computation on the persistence storage server. > > 2. Network is forced to transfer much more data, so obviously the big > disadvantage on large systems. > > The partition-aware data loading approach, described in > https://apacheignite.readme.io/docs/data-loading#section- > partition-aware-data-loading > , is not a choice. It requires persistence of the volatile data depended on > affinity function implementation and settings. > > I propose using something like IgniteDataStreamer inside > IgniteCache.loadCache implementation. > > > -- > Thanks, > Alexandr Kuramshin > -- Alexey Kuznetsov
