***UNCHECKED*** Re: Re[4]: Cache scan efficiency

Alexei, > Summing up, I suggest adding new public method IgniteCache.preloadPartition(partId). If I understand correctly use case, the user wants to retrieve whole data from the cache (not only single partition) having slow HDD. So, my suggestion is to create methods of public API like these: `p

Re[4]: Cache scan efficiency

Great, i don`t think about that. >Среда, 19 сентября 2018, 9:40 +03:00 от Vladimir Ozerov : > >Pinning is even worse thing, because you loose control on how data is moved >within a single region. Instead, I would suggest to use partition warmup + >separate data region to achieve "pinning" semanti

Re: Re[2]: Cache scan efficiency

Pinning is even worse thing, because you loose control on how data is moved within a single region. Instead, I would suggest to use partition warmup + separate data region to achieve "pinning" semantics. On Wed, Sep 19, 2018 at 8:34 AM Zhenya Stanilovsky wrote: > hi, but how to deal with page re

Re[2]: Cache scan efficiency

hi, but how to deal with page replacements, which Dmitriy Pavlov mentioned? this approach would be efficient if all data fits into memory, may be better to have method to pin some critical caches? >Среда, 19 сентября 2018, 0:26 +03:00 от Dmitriy Pavlov : > >Even better, if RAM is exhausted page

Re: Cache scan efficiency

Even better, if RAM is exhausted page replacement process will be started. https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pagereplacement(rotationwithdisk) Effect of the preloading will be still markable, but not as excell

Re: Cache scan efficiency

Agree, it's just a matter of the documentation. If a user stores 100% in RAM and on disk, and just wants to warm RAM up after a restart then he knows everything will fit there. If during the preloading we detect that the RAM is exhausted we can halt it and print out a warning. -- Denis On Tue, Se

Re: Cache scan efficiency

Hi, I totally support the idea of cache preload. IMO it can be expanded. We can iterate over local partitions of the cache group and preload each. But it should be really clear documented methods so a user can be aware of the benefits of such method (e.g. if RAM region is big enough, etc). Sinc

Re: Cache scan efficiency

Folks, Since we're adding a method that would preload a certain partition, can we add the one which will preload the whole cache? Ignite persistence users I've been working with look puzzled once they realize there is no way to warm up RAM after the restart. There are use cases that require this.

Re: Cache scan efficiency

On Tue, Sep 18, 2018 at 1:58 PM Alexei Scherbakov < alexey.scherbak...@gmail.com> wrote: > Summing up, I suggest adding new public > method IgniteCache.preloadPartition(partId). > > I will start preparing PR for IGNITE-8873 > if no more objection

Re: Cache scan efficiency

Summing up, I suggest adding new public method IgniteCache.preloadPartition(partId). I will start preparing PR for IGNITE-8873 if no more objections follow. вт, 18 сент. 2018 г. в 10:50, Alexey Goncharuk : > Dmitriy, > > In my understanding,

Re: Cache scan efficiency

Dmitriy, In my understanding, the proper fix for the scan query looks like a big change and it is unlikely that we include it in Ignite 2.7. On the other hand, the method suggested by Alexei is quite simple and it definitely fits Ignite 2.7, which will provide a better user experience. Even havin

Re: Cache scan efficiency

As I understood it is not a hack, it is an advanced feature for warming up the partition. We can build warm-up of the overall cache by calling its partitions warm-up. Users often ask about this feature and are not confident with our lazy upload. Please correct me if I misunderstood the idea. пн,

Re: Cache scan efficiency

I would rather fix the scan than hack the scan. Is there any technical reason for hacking it now instead of fixing it properly? Can some of the experts in this thread provide an estimate of complexity and difference in work that would be required for each approach? D. On Mon, Sep 17, 2018 at 4:42

Re: Cache scan efficiency

I think it would be beneficial for some Ignite users if we added such a partition warmup method to the public API. The method should be well-documented and state that it may invalidate existing page cache. It will be a very effective instrument until we add the proper scan ability that Vladimir was

Re: Cache scan efficiency

Folks, Such warming up can be an effective technique for performing calculations which required large cache data reads, but I think it's the single narrow use case of all over Ignite store usages. Like all other powerfull techniques, we should use it wisely. In the general case, I think we should

Re: Cache scan efficiency

Hi Alexei, I did not find any PRs associated with the ticket for check code changes behind this idea. Are there any PRs? If we create some forwards scan of pages, it should be a very intellectual algorithm including a lot of parameters (how much RAM is free, how probably we will need next page, e

Re: Cache scan efficiency

HI Alex, This is good that you observed speedup. But I do not think this solution works for the product in general case. Amount of RAM is limited, and even a single partition may need more space than RAM available. Moving a lot of pages to page memory for scan means that you evict a lot of other p

Re: Cache scan efficiency

Alexey, this is a great feature. Can you explain what you meant by "warm-up" when iterating through pages? Do you have this feature already implemented? D. On Sun, Sep 16, 2018 at 6:44 PM Alexei Scherbakov < alexey.scherbak...@gmail.com> wrote: > Igniters, > > My use case involves scenario where

Cache scan efficiency

Igniters, My use case involves scenario where it's necessary to iterate over large(many TBs) persistent cache doing some calculation on read data. The basic solution is to iterate cache using ScanQuery. This turns out to be slow because iteration over cache involves a lot of random disk access f