It feels like Java hasn't hit the threshold when the GC would kick in.
From a Geode perspective if the entry count is zero all of the
references to the objects stored in Geode should be removed and are
waiting to be reclaimed by Java.
If you would like to force a GC try the gfsh command "gc". GC will
attempt to run the Java Garbage Collector. Here is an example run:
gfsh>gc
GC Summary
Member ID/Name| HeapSize (MB) Before GC | HeapSize(MB) After GC |
Time Taken for GC in ms
------------------------------- | ----------------------- |
--------------------- | -----------------------
192.168.5.1(foo:23089)<v0>:9696 | 189 | 142 | 84
On Tue, Dec 13, 2016 at 2:40 PM <[email protected]
<mailto:[email protected]>> wrote:
Indeed the eviction/offheap features or persistency can
workaround this, I considered the offheap but the problem for my
scenario is that upon reaching the threshold (which is tested by
checking heap status) the offheap will take place in I will have
throughput hit although I have free space in RAM (the server
elements usage are less than threshold after some keys removal,
but since offheap is based on querying the heap new puts will go
offheap).
My system requirements can’t afford working with disk while
having free RAM, it will impact it dramatically.
Please share if you have more ideas to overcome this.
Thanks a lot.
*From:*Eric Shu [mailto:[email protected] <mailto:[email protected]>]
*Sent:* יום ג 13 דצמבר 2016 23:27
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: Geode memory handling issues
Others might chime in as well. I think most users set eviction if
they know there will be a memory issue. Eviction is kicked in
before the critical heap setting (Low Memory Exception) with
correct settings.
Also, if it is partitioned region, you can add capacity by adding
more nodes so that system will be able to proceed. Users may also
shut down nodes and restart them again -- with persistence
enabled. The data will be recovered from disc.
Regards,
Eric
On Tue, Dec 13, 2016 at 1:16 PM, <[email protected]
<mailto:[email protected]>> wrote:
Thanks Dan & Jared,
I carefully read the documentation for the resource management
and noticed that part that tells the interaction between it and
the GC.
I saw that my server is working with ConcMarkSweep and set the
InitiatingOccupancyFraction to lower percentage but with no help,
GC just won’t kick in. I also tried the G1 GC with now help.
(also, just to make sure that is the cause, I attached to the jvm
using yourkit and initiate GC manually and indeed the server was
able to get put requests again)
For my experience with JVM memory management, counting on GC
collection can be problematic as you can't really control the
collection timing, in addition invoking it directly (System.gc)
is stated as bad practice. I would expect such resource manager
to track its elements usage directly and not by querying the heap
status.
This mem usage issue seems to be very basic, I mean how come
GemFire users (which I know runs in production) are not facing
this problem?
I really appreciate your help guys.
-----Original Message-----
From: Dan Smith [mailto:[email protected] <mailto:[email protected]>]
Sent: יוםג 13 דצמבר 2016 21:44
To: [email protected] <mailto:[email protected]>
Subject: Re: Geode memory handling issues
Hi Assaf,
+1 for that link Jared send out. In order to the resource manager to
work, you need to be using ConcMarkSweep with an
InitiatingOccupancyFraction that's less than your critical and
eviction heap thresholds. That will cause GC to kick in if your
heap is above those thresholds.
-Dan
On Tue, Dec 13, 2016 at 11:36 AM, Jared Stewart
<[email protected] <mailto:[email protected]>> wrote:
> Hi Assaf,
>
> There is some information about tuning the JVM’s garbage collection
> parameters to work with eviction available here:
>
http://geode.apache.org/docs/guide/managing/heap_use/heap_management.h
> tml#resource_manager
>
> Best,
> Jared
>
> On Dec 13, 2016, at 11:26 AM, <[email protected]
<mailto:[email protected]>>
> <[email protected] <mailto:[email protected]>> wrote:
>
> Hi Eric,
>
> Thanks for the quick response!
> Shutting down the server is problematic for me. In a real
scenario I
> won’t remove all items but only some of them, thus the server still
> contains data but have free space – still I will get the exception.
> Shutting down the server will cause me to lose other entries – I’m
> afraid it’s not applicable to me.
>
> I also understand that I can’t really control GC collection, so how
> does the critical-threshold is expected to work? Once you reach
it you
> can’t add more items to server even if you removed some.
>
> Thanks.
>
> From: Eric Shu [mailto:[email protected] <mailto:[email protected]>]
> Sent: יום ג 13 דצמבר 2016 21:21
> To: [email protected] <mailto:[email protected]>
> Subject: Re: Geode memory handling issues
>
> I am not sure if GC collection can be controlled. One possible
way to
> work around this is to shut down the server and restart it.
>
> Also you may want to try offheap region to cope this issue?
>
> Regards,
> Eric
>
> On Tue, Dec 13, 2016 at 11:02 AM, <[email protected]
<mailto:[email protected]>> wrote:
> Hi,
>
> I am facing some issues with geode memory/resource management.
> I have simple environment with a single locator and a single
server,
> both are launched via gfsh.
> The server is started with initial-heap=max-heap=4GBand with
> critical-heap-percentage=70%.
> I also created a single region of type partition using gfsh.
>
> In addition, I created a simple client application (client-server
> topology, accessing the region via ClientCache PROXY). The client
> simply iteratively put data elements (of 100MB each) into the
region.
> Upon putting total size of ~70% of 4GB, I get exception on
client side
> which tells the server is working at low memory as expected –
so far so good.
> While observing the server metrics (using show metrics) I see
that the
> server holds data entries as expected and the heap usage and total
> heap size are fine too.
>
> Now, I’m removing all elements from the region – the server metrics
> after this operation shows that the elements count is 0 (i.e. the
> server is empty) but still heap usage is high (probably because GC
> didn’t collect freed items). That’s shouldn’t bother me, but the
> problem is if I’m trying to put additional elements into the
region now I still get the exception on client.
> Although the server is empty the client can’t put items into it and
> this is very problematic from user point of view.
>
> I tried to play with GC flags, change GC to G1 but with no
success. I
> can’t control GC collection – having an idle empty server with
no way
> to add elements to it.
>
> What am I missing here? Is there some other configuration I
should follow?
>
> Thanks.
>
> Assaf Waizman
> SW Architect | Process Diagnostics and Control | Applied Materials
> 9 Oppenheimer Street, Rehovot, 76705. Israel.
> Office +972.8.948.8661 <tel:%2B972.8.948.8661> | Mobile
+972.54.80.10.799 <tel:%2B972.54.80.10.799> | Fax
> +972.8.948.8848 <tel:%2B972.8.948.8848>
>
>
--
---
[email protected] <mailto:[email protected]> | +1.858.480.9722
