Darrel, Thanks for the explanation and workaround. This helps in
understanding the issue.
Nilkanth.
On Tue, Jul 19, 2016 at 2:59 AM, Darrel Schneider <dschnei...@pivotal.io>
wrote:
> This is a known issue in GemFire. I filed a corresponding Geode
> ticket: GEODE-1672.
>
> The issue involves recovering large amounts of when using the heap LRU. The
> data get recovered asynchronously before it can be evicted by the heap LRU.
> So it is possible you will run out of heap during recovery. Eviction only
> moves an entry value to disk. The entry key and entry meta-data remain in
> memory. Geode will stop recovering values once the resource manager hits
> the eviction limit but, since it is not able to evict values already
> recovered, it can run out of memory recovering the keys and meta-data for
> subsequent entries.
>
> In your case the only known workaround is to set this system property:
> -Dgemfire.disk.recoverValues=false
> This will cause values to not be asynchronously recovered from disk. So any
> time you ask for a recovered value by doing a Region get it will need to be
> faulted in at that time. For some uses cases this might be optimal since if
> prevents recovery from faulting in values that may never be read.
>
>
> On Fri, Jul 15, 2016 at 2:01 PM, Nilkanth Patel <nilkanth.hpa...@gmail.com
> >
> wrote:
>
> > Hello,
> >
> > Facing issue in recovering data for persisted regions when large amount
> > (more than heap) of data is persisted.
> >
> > brief about scenario .
> >
> > Creating 10 regions, lets call it R1, R2, R3, ... R10 with following
> > config.
> > For R1, R2, Total # of buckets = 113.
> > For R3, R4, R10, #of buckets = 511.
> >
> > All above regions are configured with Disk persistance enabled (ASYNCH)
> and
> > eviction action overflow to disk. like,
> >
> > RegionFactory<> rf =
> > cache.createRegionFactory(RegionShortcut.PARTITION_PERSISTENT_OVERFLOW);
> > rf.setDiskSynchronous(false) //for asynch writes.
> > rf.setDiskStoreName("myDiskStore");PartitionAttributesFactory paf =
> > new
> >
> PartitionAttributesFactory().setRedundantCopies(3).paf.setTotalNumBuckets(511);
> >
> >
> > For each server, Setting both --initial-heap and --max-heap to same, i.e
> > 16gb with --eviction-heap-percentage=81 --critical-heap-percentage=90
> >
> > I keep the system running (puts, gets, delete) for hours to add data over
> > time until i have overflowed tons of data approaching the heap size or
> > more.
> > Now i shutdown my cluster and then attempt to restart but it does not
> come
> > up. It seems during this early phase of recovery (large amount of data),
> > geode surpasses the critical threshold which kills it before successful
> > startup.
> >
> > Is this observation correct and is this a known limitation...? If so any
> > work around for this..?
> >
> > Also, Considering the above case, recovery for (1)
> > ForceDisconnect--->Autoconnect case and (2) normal_shutdown-->restart
> case
> > is a same mechanism or is there any differences?
> >
> > Thanks in advance,.
> >
> > Nilkanth.
> >
>
