Re[4]: apache ignite 2.10.0 heap starvation

[Zhenya Stanilovsky]

Node is going down due to full gc triggering, you can avoid it by [1] but you 
obtain ALL not only life objects.
Additionally you can try to attach with async prof [2] or probably visual vm
 
[1]  
https://stackoverflow.com/questions/23393480/can-heap-dump-be-created-for-analyzing-memory-leak-without-garbage-collection
[2]  https://github.com/jvm-profiling-tools/async-profiler
 
>heap dump generation does not seems to be working.
>whenever I tried to generate the heap dump, node is going down, bit strange,
>what else we could analyze  
>On Tue, Oct 12, 2021 at 7:35 PM Zhenya Stanilovsky < arzamas...@mail.ru > 
>wrote:
>>hi, highly likely the problem in your code - cpu usage grow synchronously 
>>with heap increasing between 00.00 and 12.00.
>>You need to analyze heap dump, no additional settings will help here.
>>   
>>>On the same subject, we have made the changes as suggested 
>>> 
>>>nodes are running on 8 CORE and 128 GB MEM VMs, i've added the following jvm 
>>>parameters
>>>
>>>-XX:ParallelGCThreads=4
>>>-XX:ConcGCThreads=2
>>>-XX:MaxGCPauseMillis=200
>>>-XX:InitiatingHeapOccupancyPercent=40
>>> 
>>>Not used any of these below, using the default values for all these, which 
>>>is 8 (as the number of cores)
>>> 
>>>        
>>>        
>>>        
>>>        
>>> 
>>>I could still see our heap is increasing,  but atleast I could see a pattern 
>>>now (not like earlier which is almost exponential)
>>> 
>>>Attaching the screenshots of heap, CPU, GC and start script with all the jvm 
>>>arguments used. 
>>>what do you think I should be changing to run to use heap effectively 
>>> 
>>>   
>>>On Wed, Sep 29, 2021 at 2:35 PM Ibrahim Altun < ibrahim.al...@segmentify.com 
 wrote:
after many configuration changes and optimizations, i think i've solved the 
heap problem.

here are the changes that i applied to the system;
JVM changes ->  
https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
 helped a lot

nodes are running on 12CORE and 64GB MEM servers, i've added the following 
jvm parameters

-XX:ParallelGCThreads=6
-XX:ConcGCThreads=2
-XX:MaxGCPauseMillis=200
-XX:InitiatingHeapOccupancyPercent=40

on ignite configuration i've changed all thread pool sizes, which were much 
more than these;
        
        
        
        
        
        
        

Here is the 16 hours of GC report;
https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB



On 2021/09/27 17:11:21, Ilya Korol < llivezk...@gmail.com > wrote:
> Actually Query interface doesn't define close() method, but QueryCursor
> does.
> In your snippets you're using try-with-resource construction for SELECT
> queries which is good, but when you run MERGE INTO query you would also
> get an QueryCursor as a result of
>
> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
>
> so maybe this QueryCursor objects still hold some resources/memory.
> Javadoc for QueryCursor states that you should always close cursors.
>
> To simplify cursor closing there is a cursor.getAll() method that will
> do this for you under the hood.
>
>
> On 2021/09/13 06:17:21, Ibrahim Altun < i...@segmentify.com > wrote:
>  > Hi Ilya,>
>  >
>  > since this is production environment i could not risk to take heap
> dump for now, but i will try to convince my superiors to get one and
> analyze it.>
>  >
>  > Queries are heavily used in our system but aren't they autoclosable
> objects? do we have to close them anyway?>
>  >
>  > here are some usage examples on our system;>
>  > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> "label", "language") VALUES (?, ?, ?)>
>  > igniteCacheService.getCache(ID,
> IgniteCacheType.LABEL).query(insertQuery);>
>  >
>  > another usage example;>
>  > --sqlFieldsQuery is like this; >
>  > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";>
>  > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
>  > sqlFieldsQuery.setLazy(true);>
>  > sqlFieldsQuery.setArgs(emails.toArray());>
>  >
>  > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
>  >
>  >
>  >
>  > On 2021/09/12 20:28:09, Shishkov Ilya < sh...@gmail.com > wrote: >
>  > > Hi, Ibrahim!>
>  > > Have you analyzed the heap dump of the server node JVMs?>
>  > > In case your application executes queries are their cursors closed?>
>  > > >
>  > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun < ib...@segmentify.com 
>>:>
>  > > >
>  > > > Igniters any comment on this issue, we are facing huge GC
> problems on>
>  > > > production 

Re: Re[2]: apache ignite 2.10.0 heap starvation

[Naveen Kumar]

heap dump generation does not seems to be working.
whenever I tried to generate the heap dump, node is going down, bit strange,
what else we could analyze

On Tue, Oct 12, 2021 at 7:35 PM Zhenya Stanilovsky 
wrote:

> hi, highly likely the problem in your code - cpu usage grow synchronously
> with heap increasing between 00.00 and 12.00.
> You need to analyze heap dump, no additional settings will help here.
>
>
> On the same subject, we have made the changes as suggested
>
> nodes are running on 8 CORE and 128 GB MEM VMs, i've added the following
> jvm parameters
>
> -XX:ParallelGCThreads=4
> -XX:ConcGCThreads=2
> -XX:MaxGCPauseMillis=200
> -XX:InitiatingHeapOccupancyPercent=40
>
> Not used any of these below, using the default values for all these,
> which is 8 (as the number of cores)
>
> 
> 
> 
> 
>
> I could still see our heap is increasing,  but atleast I could see a
> pattern now (not like earlier which is almost exponential)
>
> Attaching the screenshots of heap, CPU, GC and start script with all the
> jvm arguments used.
> what do you think I should be changing to run to use heap effectively
>
>
>
> On Wed, Sep 29, 2021 at 2:35 PM Ibrahim Altun <
> ibrahim.al...@segmentify.com
> >
> wrote:
>
> after many configuration changes and optimizations, i think i've solved
> the heap problem.
>
> here are the changes that i applied to the system;
> JVM changes ->
> https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
> helped a lot
>
> nodes are running on 12CORE and 64GB MEM servers, i've added the following
> jvm parameters
>
> -XX:ParallelGCThreads=6
> -XX:ConcGCThreads=2
> -XX:MaxGCPauseMillis=200
> -XX:InitiatingHeapOccupancyPercent=40
>
> on ignite configuration i've changed all thread pool sizes, which were
> much more than these;
> 
> 
> 
> 
> 
> 
> 
>
> Here is the 16 hours of GC report;
>
> https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
>
>
>
> On 2021/09/27 17:11:21, Ilya Korol  > wrote:
> > Actually Query interface doesn't define close() method, but QueryCursor
> > does.
> > In your snippets you're using try-with-resource construction for SELECT
> > queries which is good, but when you run MERGE INTO query you would also
> > get an QueryCursor as a result of
> >
> > igniteCacheService.getCache(ID,
> IgniteCacheType.LABEL).query(insertQuery);
> >
> > so maybe this QueryCursor objects still hold some resources/memory.
> > Javadoc for QueryCursor states that you should always close cursors.
> >
> > To simplify cursor closing there is a cursor.getAll() method that will
> > do this for you under the hood.
> >
> >
> > On 2021/09/13 06:17:21, Ibrahim Altun  > wrote:
> >  > Hi Ilya,>
> >  >
> >  > since this is production environment i could not risk to take heap
> > dump for now, but i will try to convince my superiors to get one and
> > analyze it.>
> >  >
> >  > Queries are heavily used in our system but aren't they autoclosable
> > objects? do we have to close them anyway?>
> >  >
> >  > here are some usage examples on our system;>
> >  > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> > "label", "language") VALUES (?, ?, ?)>
> >  > igniteCacheService.getCache(ID,
> > IgniteCacheType.LABEL).query(insertQuery);>
> >  >
> >  > another usage example;>
> >  > --sqlFieldsQuery is like this; >
> >  > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN
> (?)";>
> >  > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
> >  > sqlFieldsQuery.setLazy(true);>
> >  > sqlFieldsQuery.setArgs(emails.toArray());>
> >  >
> >  > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> > IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
> >  >
> >  >
> >  >
> >  > On 2021/09/12 20:28:09, Shishkov Ilya  > wrote: >
> >  > > Hi, Ibrahim!>
> >  > > Have you analyzed the heap dump of the server node JVMs?>
> >  > > In case your application executes queries are their cursors closed?>
> >  > > >
> >  > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun  >:>
> >  > > >
> >  > > > Igniters any comment on this issue, we are facing huge GC
> > problems on>
> >  > > > production environment, please advise.>
> >  > > >>
> >  > > > On 2021/09/07 14:11:09, Ibrahim Altun  >>
> >  > > > wrote:>
> >  > > > > Hi,>
> >  > > > >>
> >  > > > > totally 400 - 600K reads/writes/updates>
> >  > > > > 12core>
> >  > > > > 64GB RAM>
> >  > > > > no iowait>
> >  > > > > 10 nodes>
> >  > > > >>
> >  > > > > On 2021/09/07 12:51:28, Piotr Jagielski  > wrote:>
> >  > > > > > Hi,>
> >  > > > > > Can you provide some information on how you use the cluster?
> > How many>
> >  > > > reads/writes/updates per second? Also CPU / RAM spec of cluster
> > nodes?>
> >  > > > > >>
> >  > > > > > We observed full GC / CPU load / OOM killer when loading big
> > amount of>
> >  > > > data (15 mln records, data streamer + 

Re[2]: apache ignite 2.10.0 heap starvation

[Zhenya Stanilovsky]

hi, highly likely the problem in your code - cpu usage grow synchronously with 
heap increasing between 00.00 and 12.00.
You need to analyze heap dump, no additional settings will help here.
 
>On the same subject, we have made the changes as suggested 
> 
>nodes are running on 8 CORE and 128 GB MEM VMs, i've added the following jvm 
>parameters
>
>-XX:ParallelGCThreads=4
>-XX:ConcGCThreads=2
>-XX:MaxGCPauseMillis=200
>-XX:InitiatingHeapOccupancyPercent=40
> 
>Not used any of these below, using the default values for all these, which is 
>8 (as the number of cores)
> 
>        
>        
>        
>        
> 
>I could still see our heap is increasing,  but atleast I could see a pattern 
>now (not like earlier which is almost exponential)
> 
>Attaching the screenshots of heap, CPU, GC and start script with all the jvm 
>arguments used. 
>what do you think I should be changing to run to use heap effectively 
> 
>   
>On Wed, Sep 29, 2021 at 2:35 PM Ibrahim Altun < ibrahim.al...@segmentify.com > 
>wrote:
>>after many configuration changes and optimizations, i think i've solved the 
>>heap problem.
>>
>>here are the changes that i applied to the system;
>>JVM changes ->  
>>https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
>> helped a lot
>>
>>nodes are running on 12CORE and 64GB MEM servers, i've added the following 
>>jvm parameters
>>
>>-XX:ParallelGCThreads=6
>>-XX:ConcGCThreads=2
>>-XX:MaxGCPauseMillis=200
>>-XX:InitiatingHeapOccupancyPercent=40
>>
>>on ignite configuration i've changed all thread pool sizes, which were much 
>>more than these;
>>        
>>        
>>        
>>        
>>        
>>        
>>        
>>
>>Here is the 16 hours of GC report;
>>https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
>>
>>
>>
>>On 2021/09/27 17:11:21, Ilya Korol < llivezk...@gmail.com > wrote:
>>> Actually Query interface doesn't define close() method, but QueryCursor
>>> does.
>>> In your snippets you're using try-with-resource construction for SELECT
>>> queries which is good, but when you run MERGE INTO query you would also
>>> get an QueryCursor as a result of
>>>
>>> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
>>>
>>> so maybe this QueryCursor objects still hold some resources/memory.
>>> Javadoc for QueryCursor states that you should always close cursors.
>>>
>>> To simplify cursor closing there is a cursor.getAll() method that will
>>> do this for you under the hood.
>>>
>>>
>>> On 2021/09/13 06:17:21, Ibrahim Altun < i...@segmentify.com > wrote:
>>>  > Hi Ilya,>
>>>  >
>>>  > since this is production environment i could not risk to take heap
>>> dump for now, but i will try to convince my superiors to get one and
>>> analyze it.>
>>>  >
>>>  > Queries are heavily used in our system but aren't they autoclosable
>>> objects? do we have to close them anyway?>
>>>  >
>>>  > here are some usage examples on our system;>
>>>  > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
>>> "label", "language") VALUES (?, ?, ?)>
>>>  > igniteCacheService.getCache(ID,
>>> IgniteCacheType.LABEL).query(insertQuery);>
>>>  >
>>>  > another usage example;>
>>>  > --sqlFieldsQuery is like this; >
>>>  > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";>
>>>  > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
>>>  > sqlFieldsQuery.setLazy(true);>
>>>  > sqlFieldsQuery.setArgs(emails.toArray());>
>>>  >
>>>  > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
>>> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
>>>  >
>>>  >
>>>  >
>>>  > On 2021/09/12 20:28:09, Shishkov Ilya < sh...@gmail.com > wrote: >
>>>  > > Hi, Ibrahim!>
>>>  > > Have you analyzed the heap dump of the server node JVMs?>
>>>  > > In case your application executes queries are their cursors closed?>
>>>  > > >
>>>  > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun < ib...@segmentify.com >:>
>>>  > > >
>>>  > > > Igniters any comment on this issue, we are facing huge GC
>>> problems on>
>>>  > > > production environment, please advise.>
>>>  > > >>
>>>  > > > On 2021/09/07 14:11:09, Ibrahim Altun < ib...@segmentify.com >>
>>>  > > > wrote:>
>>>  > > > > Hi,>
>>>  > > > >>
>>>  > > > > totally 400 - 600K reads/writes/updates>
>>>  > > > > 12core>
>>>  > > > > 64GB RAM>
>>>  > > > > no iowait>
>>>  > > > > 10 nodes>
>>>  > > > >>
>>>  > > > > On 2021/09/07 12:51:28, Piotr Jagielski < pj...@touk.pl > wrote:>
>>>  > > > > > Hi,>
>>>  > > > > > Can you provide some information on how you use the cluster?
>>> How many>
>>>  > > > reads/writes/updates per second? Also CPU / RAM spec of cluster
>>> nodes?>
>>>  > > > > >>
>>>  > > > > > We observed full GC / CPU load / OOM killer when loading big
>>> amount of>
>>>  > > > data (15 mln records, data streamer + allowOverwrite=true). We've
>>> seen>
>>>  > > > 200-400k updates per sec on JMX metrics, but load up to 10 on

Re: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

oh sorry about that, 128 is in our configuration file.


On 2021/09/29 15:47:27, Stephen Darlington  
wrote: 
> Correct me if I’m wrong, but I think they set the size of the threadpool to 
> 128 in their configuration file.
> 
> > On 29 Sep 2021, at 16:33, Zhenya Stanilovsky  wrote:
> > 
> > Ok, i still can`t understand whats the source of 128 value.
> > Can you check Runtime.getRuntime().availableProcessors() returning value on 
> > your side ?
> >  
> >  
> >  
> >  
> > Hi Naveen,
> > 
> > my first change was to change jvm parameters, at first it seemed to be 
> > resolved but changing jvm parameters only delayed the problem. Before that 
> > heap problems occured after 14-16 hours after the start, but with jvm 
> > changes it took up to 36 hours.
> > 
> > while keeping jvm changes i updated threadpool configurations and heap 
> > problem solved. we can see
> > saw pattern in heap usage.
> > 
> > before: https://ibb.co/mqx4kYy 
> > after: https://ibb.co/y8B0hzS 
> > 
> > 
> > On 2021/09/29 13:37:53, Naveen Kumar  > > wrote:
> > > Good to hear from you , I have had the same issue for quite a long time 
> > > and
> > > am still looking for a fix.
> > >
> > > What do you think has exactly resolved the heap starvation issue, is it 
> > > the
> > > GC related configuration or the threadpool configuration. ?
> > > Default thread pool is the number of the cores of the server, if this is
> > > true, we don't need to specify any config for all these thread pool
> > >
> > > Thanks
> > > Naveen
> > >
> > >
> > >
> > > On Wed, Sep 29, 2021 at 2:35 PM Ibrahim Altun 
> > >  > > >
> > > wrote:
> > >
> > > > after many configuration changes and optimizations, i think i've solved
> > > > the heap problem.
> > > >
> > > > here are the changes that i applied to the system;
> > > > JVM changes ->
> > > > https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
> > > >  
> > > > 
> > > > helped a lot
> > > >
> > > > nodes are running on 12CORE and 64GB MEM servers, i've added the 
> > > > following
> > > > jvm parameters
> > > >
> > > > -XX:ParallelGCThreads=6
> > > > -XX:ConcGCThreads=2
> > > > -XX:MaxGCPauseMillis=200
> > > > -XX:InitiatingHeapOccupancyPercent=40
> > > >
> > > > on ignite configuration i've changed all thread pool sizes, which were
> > > > much more than these;
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > >
> > > > Here is the 16 hours of GC report;
> > > >
> > > > https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
> > > >  
> > > > 
> > > >
> > > >
> > > >
> > > > On 2021/09/27 17:11:21, Ilya Korol  > > > > wrote:
> > > > > Actually Query interface doesn't define close() method, but 
> > > > > QueryCursor
> > > > > does.
> > > > > In your snippets you're using try-with-resource construction for 
> > > > > SELECT
> > > > > queries which is good, but when you run MERGE INTO query you would 
> > > > > also
> > > > > get an QueryCursor as a result of
> > > > >
> > > > > igniteCacheService.getCache(ID,
> > > > IgniteCacheType.LABEL).query(insertQuery);
> > > > >
> > > > > so maybe this QueryCursor objects still hold some resources/memory.
> > > > > Javadoc for QueryCursor states that you should always close cursors.
> > > > >
> > > > > To simplify cursor closing there is a cursor.getAll() method that will
> > > > > do this for you under the hood.
> > > > >
> > > > >
> > > > > On 2021/09/13 06:17:21, Ibrahim Altun  > > > > > wrote:
> > > > > > Hi Ilya,>
> > > > > >
> > > > > > since this is production environment i could not risk to take heap
> > > > > dump for now, but i will try to convince my superiors to get one and
> > > > > analyze it.>
> > > > > >
> > > > > > Queries are heavily used in our system but aren't they autoclosable
> > > > > objects? do we have to close them anyway?>
> > > > > >
> > > > > > here are some usage examples on our system;>
> > > > > > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> > > > > "label", "language") VALUES (?, ?, ?)>
> > > > > > igniteCacheService.getCache(ID,
> > > > > IgniteCacheType.LABEL).query(insertQuery);>
> > > > > >
> > > > > > another usage example;>
> > > > > > --sqlFieldsQuery is like this; >
> > > > > > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN
> > > > (?)";>
> > > > > > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
> > > > > > sqlFieldsQuery.setLazy(true);>
> > > > > > sqlFieldsQuery.setArgs(emails.toArray());>
> > > > > >
> > > > > > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> > > > > IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 2021/09/12 

Re: apache ignite 2.10.0 heap starvation

[Stephen Darlington]

Correct me if I’m wrong, but I think they set the size of the threadpool to 128 
in their configuration file.

> On 29 Sep 2021, at 16:33, Zhenya Stanilovsky  wrote:
> 
> Ok, i still can`t understand whats the source of 128 value.
> Can you check Runtime.getRuntime().availableProcessors() returning value on 
> your side ?
>  
>  
>  
>  
> Hi Naveen,
> 
> my first change was to change jvm parameters, at first it seemed to be 
> resolved but changing jvm parameters only delayed the problem. Before that 
> heap problems occured after 14-16 hours after the start, but with jvm changes 
> it took up to 36 hours.
> 
> while keeping jvm changes i updated threadpool configurations and heap 
> problem solved. we can see
> saw pattern in heap usage.
> 
> before: https://ibb.co/mqx4kYy 
> after: https://ibb.co/y8B0hzS 
> 
> 
> On 2021/09/29 13:37:53, Naveen Kumar  > wrote:
> > Good to hear from you , I have had the same issue for quite a long time and
> > am still looking for a fix.
> >
> > What do you think has exactly resolved the heap starvation issue, is it the
> > GC related configuration or the threadpool configuration. ?
> > Default thread pool is the number of the cores of the server, if this is
> > true, we don't need to specify any config for all these thread pool
> >
> > Thanks
> > Naveen
> >
> >
> >
> > On Wed, Sep 29, 2021 at 2:35 PM Ibrahim Altun  > >
> > wrote:
> >
> > > after many configuration changes and optimizations, i think i've solved
> > > the heap problem.
> > >
> > > here are the changes that i applied to the system;
> > > JVM changes ->
> > > https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
> > >  
> > > 
> > > helped a lot
> > >
> > > nodes are running on 12CORE and 64GB MEM servers, i've added the following
> > > jvm parameters
> > >
> > > -XX:ParallelGCThreads=6
> > > -XX:ConcGCThreads=2
> > > -XX:MaxGCPauseMillis=200
> > > -XX:InitiatingHeapOccupancyPercent=40
> > >
> > > on ignite configuration i've changed all thread pool sizes, which were
> > > much more than these;
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >
> > > Here is the 16 hours of GC report;
> > >
> > > https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
> > >  
> > > 
> > >
> > >
> > >
> > > On 2021/09/27 17:11:21, Ilya Korol  > > > wrote:
> > > > Actually Query interface doesn't define close() method, but QueryCursor
> > > > does.
> > > > In your snippets you're using try-with-resource construction for SELECT
> > > > queries which is good, but when you run MERGE INTO query you would also
> > > > get an QueryCursor as a result of
> > > >
> > > > igniteCacheService.getCache(ID,
> > > IgniteCacheType.LABEL).query(insertQuery);
> > > >
> > > > so maybe this QueryCursor objects still hold some resources/memory.
> > > > Javadoc for QueryCursor states that you should always close cursors.
> > > >
> > > > To simplify cursor closing there is a cursor.getAll() method that will
> > > > do this for you under the hood.
> > > >
> > > >
> > > > On 2021/09/13 06:17:21, Ibrahim Altun  > > > > wrote:
> > > > > Hi Ilya,>
> > > > >
> > > > > since this is production environment i could not risk to take heap
> > > > dump for now, but i will try to convince my superiors to get one and
> > > > analyze it.>
> > > > >
> > > > > Queries are heavily used in our system but aren't they autoclosable
> > > > objects? do we have to close them anyway?>
> > > > >
> > > > > here are some usage examples on our system;>
> > > > > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> > > > "label", "language") VALUES (?, ?, ?)>
> > > > > igniteCacheService.getCache(ID,
> > > > IgniteCacheType.LABEL).query(insertQuery);>
> > > > >
> > > > > another usage example;>
> > > > > --sqlFieldsQuery is like this; >
> > > > > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN
> > > (?)";>
> > > > > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
> > > > > sqlFieldsQuery.setLazy(true);>
> > > > > sqlFieldsQuery.setArgs(emails.toArray());>
> > > > >
> > > > > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> > > > IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
> > > > >
> > > > >
> > > > >
> > > > > On 2021/09/12 20:28:09, Shishkov Ilya  > > > > > wrote: >
> > > > > > Hi, Ibrahim!>
> > > > > > Have you analyzed the heap dump of the server node JVMs?>
> > > > > > In case your application executes queries are their cursors closed?>
> > > > > > >
> > > > > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun  > > > > > 
> > > >:>
> > > > > > >
> > > > > > > Igniters any comment on this issue, we are facing huge GC
> > > > problems on>
> > > > > > > 

Re[2]: apache ignite 2.10.0 heap starvation

[Zhenya Stanilovsky]

Ok, i still can`t understand whats the source of 128 value.
Can you check Runtime.getRuntime().availableProcessors() returning value on 
your side ?
 
 
> 
>> 
>>>Hi Naveen,
>>>
>>>my first change was to change jvm parameters, at first it seemed to be 
>>>resolved but changing jvm parameters only delayed the problem. Before that 
>>>heap problems occured after 14-16 hours after the start, but with jvm 
>>>changes it took up to 36 hours.
>>>
>>>while keeping jvm changes i updated threadpool configurations and heap 
>>>problem solved. we can see
>>>saw pattern in heap usage.
>>>
>>>before:  https://ibb.co/mqx4kYy
>>>after:  https://ibb.co/y8B0hzS
>>>
>>>
>>>On 2021/09/29 13:37:53, Naveen Kumar < naveen.band...@gmail.com > wrote:
 Good to hear from you , I have had the same issue for quite a long time and
 am still looking for a fix.

 What do you think has exactly resolved the heap starvation issue, is it the
 GC related configuration or the threadpool configuration. ?
 Default thread pool is the number of the cores of the server, if this is
 true, we don't need to specify any config for all these thread pool

 Thanks
 Naveen



 On Wed, Sep 29, 2021 at 2:35 PM Ibrahim Altun < 
 ibrahim.al...@segmentify.com >
 wrote:

 > after many configuration changes and optimizations, i think i've solved
 > the heap problem.
 >
 > here are the changes that i applied to the system;
 > JVM changes ->
 >  
 > https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
 > helped a lot
 >
 > nodes are running on 12CORE and 64GB MEM servers, i've added the 
 > following
 > jvm parameters
 >
 > -XX:ParallelGCThreads=6
 > -XX:ConcGCThreads=2
 > -XX:MaxGCPauseMillis=200
 > -XX:InitiatingHeapOccupancyPercent=40
 >
 > on ignite configuration i've changed all thread pool sizes, which were
 > much more than these;
 > 
 > 
 > 
 > 
 > 
 > 
 > 
 >
 > Here is the 16 hours of GC report;
 >
 >  
 > https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
 >
 >
 >
 > On 2021/09/27 17:11:21, Ilya Korol < llivezk...@gmail.com > wrote:
 > > Actually Query interface doesn't define close() method, but QueryCursor
 > > does.
 > > In your snippets you're using try-with-resource construction for SELECT
 > > queries which is good, but when you run MERGE INTO query you would also
 > > get an QueryCursor as a result of
 > >
 > > igniteCacheService.getCache(ID,
 > IgniteCacheType.LABEL).query(insertQuery);
 > >
 > > so maybe this QueryCursor objects still hold some resources/memory.
 > > Javadoc for QueryCursor states that you should always close cursors.
 > >
 > > To simplify cursor closing there is a cursor.getAll() method that will
 > > do this for you under the hood.
 > >
 > >
 > > On 2021/09/13 06:17:21, Ibrahim Altun < i...@segmentify.com > wrote:
 > > > Hi Ilya,>
 > > >
 > > > since this is production environment i could not risk to take heap
 > > dump for now, but i will try to convince my superiors to get one and
 > > analyze it.>
 > > >
 > > > Queries are heavily used in our system but aren't they autoclosable
 > > objects? do we have to close them anyway?>
 > > >
 > > > here are some usage examples on our system;>
 > > > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
 > > "label", "language") VALUES (?, ?, ?)>
 > > > igniteCacheService.getCache(ID,
 > > IgniteCacheType.LABEL).query(insertQuery);>
 > > >
 > > > another usage example;>
 > > > --sqlFieldsQuery is like this; >
 > > > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN
 > (?)";>
 > > > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
 > > > sqlFieldsQuery.setLazy(true);>
 > > > sqlFieldsQuery.setArgs(emails.toArray());>
 > > >
 > > > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
 > > IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
 > > >
 > > >
 > > >
 > > > On 2021/09/12 20:28:09, Shishkov Ilya < sh...@gmail.com > wrote: >
 > > > > Hi, Ibrahim!>
 > > > > Have you analyzed the heap dump of the server node JVMs?>
 > > > > In case your application executes queries are their cursors 
 > > > > closed?>
 > > > > >
 > > > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun < ib...@segmentify.com
 > >:>
 > > > > >
 > > > > > Igniters any comment on this issue, we are facing huge GC
 > > problems on>
 > > > > > production environment, please advise.>
 > > > > >>
 > > > > > On 2021/09/07 14:11:09, Ibrahim Altun < ib...@segmentify.com >>
 > > > > > wrote:>
 > > > > > > Hi,>
 > > > > > >>
 

Re: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

Hi Naveen,

my first change was to change jvm parameters, at first it seemed to be resolved 
but changing jvm parameters only delayed the problem. Before that heap problems 
occured after 14-16 hours after the start, but with jvm changes it took up to 
36 hours.

while keeping jvm changes i updated threadpool configurations and heap problem 
solved. we can see 
saw pattern in heap usage.

before: https://ibb.co/mqx4kYy
after: https://ibb.co/y8B0hzS


On 2021/09/29 13:37:53, Naveen Kumar  wrote: 
> Good to hear from you , I have had the same issue for quite a long time and
> am still looking for a fix.
> 
> What do you think has exactly resolved the heap starvation issue, is it the
> GC related configuration or the threadpool configuration. ?
>  Default thread pool is the number of the cores of the server, if this is
> true, we don't need to specify any config for all these thread pool
> 
> Thanks
> Naveen
> 
> 
> 
> On Wed, Sep 29, 2021 at 2:35 PM Ibrahim Altun 
> wrote:
> 
> > after many configuration changes and optimizations, i think i've solved
> > the heap problem.
> >
> > here are the changes that i applied to the system;
> > JVM changes ->
> > https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
> > helped a lot
> >
> > nodes are running on 12CORE and 64GB MEM servers, i've added the following
> > jvm parameters
> >
> > -XX:ParallelGCThreads=6
> > -XX:ConcGCThreads=2
> > -XX:MaxGCPauseMillis=200
> > -XX:InitiatingHeapOccupancyPercent=40
> >
> > on ignite configuration i've changed all thread pool sizes, which were
> > much more than these;
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >
> > Here is the 16 hours of GC report;
> >
> > https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
> >
> >
> >
> > On 2021/09/27 17:11:21, Ilya Korol  wrote:
> > > Actually Query interface doesn't define close() method, but QueryCursor
> > > does.
> > > In your snippets you're using try-with-resource construction for SELECT
> > > queries which is good, but when you run MERGE INTO query you would also
> > > get an QueryCursor as a result of
> > >
> > > igniteCacheService.getCache(ID,
> > IgniteCacheType.LABEL).query(insertQuery);
> > >
> > > so maybe this QueryCursor objects still hold some resources/memory.
> > > Javadoc for QueryCursor states that you should always close cursors.
> > >
> > > To simplify cursor closing there is a cursor.getAll() method that will
> > > do this for you under the hood.
> > >
> > >
> > > On 2021/09/13 06:17:21, Ibrahim Altun  wrote:
> > >  > Hi Ilya,>
> > >  >
> > >  > since this is production environment i could not risk to take heap
> > > dump for now, but i will try to convince my superiors to get one and
> > > analyze it.>
> > >  >
> > >  > Queries are heavily used in our system but aren't they autoclosable
> > > objects? do we have to close them anyway?>
> > >  >
> > >  > here are some usage examples on our system;>
> > >  > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> > > "label", "language") VALUES (?, ?, ?)>
> > >  > igniteCacheService.getCache(ID,
> > > IgniteCacheType.LABEL).query(insertQuery);>
> > >  >
> > >  > another usage example;>
> > >  > --sqlFieldsQuery is like this; >
> > >  > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN
> > (?)";>
> > >  > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
> > >  > sqlFieldsQuery.setLazy(true);>
> > >  > sqlFieldsQuery.setArgs(emails.toArray());>
> > >  >
> > >  > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> > > IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
> > >  >
> > >  >
> > >  >
> > >  > On 2021/09/12 20:28:09, Shishkov Ilya  wrote: >
> > >  > > Hi, Ibrahim!>
> > >  > > Have you analyzed the heap dump of the server node JVMs?>
> > >  > > In case your application executes queries are their cursors closed?>
> > >  > > >
> > >  > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun  > >:>
> > >  > > >
> > >  > > > Igniters any comment on this issue, we are facing huge GC
> > > problems on>
> > >  > > > production environment, please advise.>
> > >  > > >>
> > >  > > > On 2021/09/07 14:11:09, Ibrahim Altun >
> > >  > > > wrote:>
> > >  > > > > Hi,>
> > >  > > > >>
> > >  > > > > totally 400 - 600K reads/writes/updates>
> > >  > > > > 12core>
> > >  > > > > 64GB RAM>
> > >  > > > > no iowait>
> > >  > > > > 10 nodes>
> > >  > > > >>
> > >  > > > > On 2021/09/07 12:51:28, Piotr Jagielski  wrote:>
> > >  > > > > > Hi,>
> > >  > > > > > Can you provide some information on how you use the cluster?
> > > How many>
> > >  > > > reads/writes/updates per second? Also CPU / RAM spec of cluster
> > > nodes?>
> > >  > > > > >>
> > >  > > > > > We observed full GC / CPU load / OOM killer when loading big
> > > amount of>
> > >  > > > data (15 mln records, data streamer + allowOverwrite=true). We've
> > > seen>
> > >  > > > 

Re: apache ignite 2.10.0 heap starvation

[Naveen Kumar]

Good to hear from you , I have had the same issue for quite a long time and
am still looking for a fix.

What do you think has exactly resolved the heap starvation issue, is it the
GC related configuration or the threadpool configuration. ?
 Default thread pool is the number of the cores of the server, if this is
true, we don't need to specify any config for all these thread pool

Thanks
Naveen



On Wed, Sep 29, 2021 at 2:35 PM Ibrahim Altun 
wrote:

> after many configuration changes and optimizations, i think i've solved
> the heap problem.
>
> here are the changes that i applied to the system;
> JVM changes ->
> https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
> helped a lot
>
> nodes are running on 12CORE and 64GB MEM servers, i've added the following
> jvm parameters
>
> -XX:ParallelGCThreads=6
> -XX:ConcGCThreads=2
> -XX:MaxGCPauseMillis=200
> -XX:InitiatingHeapOccupancyPercent=40
>
> on ignite configuration i've changed all thread pool sizes, which were
> much more than these;
> 
> 
> 
> 
> 
> 
> 
>
> Here is the 16 hours of GC report;
>
> https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
>
>
>
> On 2021/09/27 17:11:21, Ilya Korol  wrote:
> > Actually Query interface doesn't define close() method, but QueryCursor
> > does.
> > In your snippets you're using try-with-resource construction for SELECT
> > queries which is good, but when you run MERGE INTO query you would also
> > get an QueryCursor as a result of
> >
> > igniteCacheService.getCache(ID,
> IgniteCacheType.LABEL).query(insertQuery);
> >
> > so maybe this QueryCursor objects still hold some resources/memory.
> > Javadoc for QueryCursor states that you should always close cursors.
> >
> > To simplify cursor closing there is a cursor.getAll() method that will
> > do this for you under the hood.
> >
> >
> > On 2021/09/13 06:17:21, Ibrahim Altun  wrote:
> >  > Hi Ilya,>
> >  >
> >  > since this is production environment i could not risk to take heap
> > dump for now, but i will try to convince my superiors to get one and
> > analyze it.>
> >  >
> >  > Queries are heavily used in our system but aren't they autoclosable
> > objects? do we have to close them anyway?>
> >  >
> >  > here are some usage examples on our system;>
> >  > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> > "label", "language") VALUES (?, ?, ?)>
> >  > igniteCacheService.getCache(ID,
> > IgniteCacheType.LABEL).query(insertQuery);>
> >  >
> >  > another usage example;>
> >  > --sqlFieldsQuery is like this; >
> >  > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN
> (?)";>
> >  > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
> >  > sqlFieldsQuery.setLazy(true);>
> >  > sqlFieldsQuery.setArgs(emails.toArray());>
> >  >
> >  > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> > IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
> >  >
> >  >
> >  >
> >  > On 2021/09/12 20:28:09, Shishkov Ilya  wrote: >
> >  > > Hi, Ibrahim!>
> >  > > Have you analyzed the heap dump of the server node JVMs?>
> >  > > In case your application executes queries are their cursors closed?>
> >  > > >
> >  > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun  >:>
> >  > > >
> >  > > > Igniters any comment on this issue, we are facing huge GC
> > problems on>
> >  > > > production environment, please advise.>
> >  > > >>
> >  > > > On 2021/09/07 14:11:09, Ibrahim Altun >
> >  > > > wrote:>
> >  > > > > Hi,>
> >  > > > >>
> >  > > > > totally 400 - 600K reads/writes/updates>
> >  > > > > 12core>
> >  > > > > 64GB RAM>
> >  > > > > no iowait>
> >  > > > > 10 nodes>
> >  > > > >>
> >  > > > > On 2021/09/07 12:51:28, Piotr Jagielski  wrote:>
> >  > > > > > Hi,>
> >  > > > > > Can you provide some information on how you use the cluster?
> > How many>
> >  > > > reads/writes/updates per second? Also CPU / RAM spec of cluster
> > nodes?>
> >  > > > > >>
> >  > > > > > We observed full GC / CPU load / OOM killer when loading big
> > amount of>
> >  > > > data (15 mln records, data streamer + allowOverwrite=true). We've
> > seen>
> >  > > > 200-400k updates per sec on JMX metrics, but load up to 10 on
> > nodes, iowait>
> >  > > > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto
> > 8CPU, 32GB>
> >  > > > RAM). Ignite 2.10>
> >  > > > > >>
> >  > > > > > Regards,>
> >  > > > > > Piotr>
> >  > > > > >>
> >  > > > > > On 2021/09/02 08:36:07, Ibrahim Altun >
> >  > > > wrote:>
> >  > > > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite
> > nodes>
> >  > > > facing>
> >  > > > > > > huge full GC operations after 24-36 hours after node start.>
> >  > > > > > >>
> >  > > > > > > We try to increase heap size but no luck, here is the start>
> >  > > > configuration>
> >  > > > > > > for nodes;>
> >  > > > > > >>
> >  > > > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g 

Re: Re[2]: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

yes i was able to get 128 with the following configuration;


here is a sample log;

[2021-09-27T00:00:19,359][INFO ][grid-timeout-worker-#198][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=f0025abe, uptime=2 days, 21:40:21.138]
^-- Cluster [hosts=39, CPUs=236, servers=10, clients=29, topVer=49, 
minorTopVer=0]
^-- Network [addrs=[10.240.242.1, 10.240.245.1, 127.0.0.1], 
discoPort=47500, commPort=47100]
^-- CPU [CPUs=12, curLoad=3.53%, avgLoad=7.1%, GC=0%]
^-- Heap [used=10076MB, free=17.99%, comm=12288MB]
^-- Off-heap memory [used=36152MB, free=2.98%, allocated=37063MB]
^-- Page memory [pages=9147903]
^--   sysMemPlc region [type=internal, persistence=true, lazyAlloc=false,
  ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, 
allocRam=99MB, allocTotal=0MB]
^--   default region [type=default, persistence=true, lazyAlloc=true,
  ...  initCfg=256MB, maxCfg=36864MB, usedRam=36152MB, freeRam=1.93%, 
allocRam=36864MB, allocTotal=81296MB]
^--   metastoreMemPlc region [type=internal, persistence=true, 
lazyAlloc=false,
  ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.57%, 
allocRam=0MB, allocTotal=0MB]
^--   TxLog region [type=internal, persistence=true, lazyAlloc=false,
  ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, 
allocRam=99MB, allocTotal=0MB]
^--   volatileDsMemPlc region [type=user, persistence=false, lazyAlloc=true,
  ...  initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, allocRam=0MB]
^-- Ignite persistence [used=81296MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=128, qSize=0]


On 2021/09/29 12:24:19, Zhenya Stanilovsky  wrote: 
> 
> 
> systemThreadPoolSize and other pools are defined by default from:
> Runtime.getRuntime().availableProcessors(), if you somehow obtain 128, plz 
> fill the ticket with all env info.
> thanks !
> 
> 
>  
> >after many configuration changes and optimizations, i think i've solved the 
> >heap problem.
> >
> >here are the changes that i applied to the system;
> >JVM changes ->  
> >https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
> > helped a lot
> >
> >nodes are running on 12CORE and 64GB MEM servers, i've added the following 
> >jvm parameters
> >
> >-XX:ParallelGCThreads=6
> >-XX:ConcGCThreads=2
> >-XX:MaxGCPauseMillis=200
> >-XX:InitiatingHeapOccupancyPercent=40
> >
> >on ignite configuration i've changed all thread pool sizes, which were much 
> >more than these;
> >
> >
> >
> >
> >
> >
> >
> >
> >Here is the 16 hours of GC report;
> >https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
> >
> >
> >
> >On 2021/09/27 17:11:21, Ilya Korol < llivezk...@gmail.com > wrote:
> >> Actually Query interface doesn't define close() method, but QueryCursor
> >> does.
> >> In your snippets you're using try-with-resource construction for SELECT
> >> queries which is good, but when you run MERGE INTO query you would also
> >> get an QueryCursor as a result of
> >>
> >> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
> >>
> >> so maybe this QueryCursor objects still hold some resources/memory.
> >> Javadoc for QueryCursor states that you should always close cursors.
> >>
> >> To simplify cursor closing there is a cursor.getAll() method that will
> >> do this for you under the hood.
> >>
> >>
> >> On 2021/09/13 06:17:21, Ibrahim Altun < i...@segmentify.com > wrote:
> >> > Hi Ilya,>
> >> >
> >> > since this is production environment i could not risk to take heap
> >> dump for now, but i will try to convince my superiors to get one and
> >> analyze it.>
> >> >
> >> > Queries are heavily used in our system but aren't they autoclosable
> >> objects? do we have to close them anyway?>
> >> >
> >> > here are some usage examples on our system;>
> >> > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> >> "label", "language") VALUES (?, ?, ?)>
> >> > igniteCacheService.getCache(ID,
> >> IgniteCacheType.LABEL).query(insertQuery);>
> >> >
> >> > another usage example;>
> >> > --sqlFieldsQuery is like this; >
> >> > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";>
> >> > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
> >> > sqlFieldsQuery.setLazy(true);>
> >> > sqlFieldsQuery.setArgs(emails.toArray());>
> >> >
> >> > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> >> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
> >> >
> >> >
> >> >
> >> > On 2021/09/12 20:28:09, Shishkov Ilya < sh...@gmail.com > wrote: >
> >> > > Hi, Ibrahim!>
> >> > > Have you analyzed the heap dump of the server node JVMs?>
> >> > > In case your application executes queries are their cursors closed?>
> >> > > >
> >> > 

Re[2]: apache ignite 2.10.0 heap starvation

[Zhenya Stanilovsky]

systemThreadPoolSize and other pools are defined by default from:
Runtime.getRuntime().availableProcessors(), if you somehow obtain 128, plz fill 
the ticket with all env info.
thanks !


 
>after many configuration changes and optimizations, i think i've solved the 
>heap problem.
>
>here are the changes that i applied to the system;
>JVM changes ->  
>https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
> helped a lot
>
>nodes are running on 12CORE and 64GB MEM servers, i've added the following jvm 
>parameters
>
>-XX:ParallelGCThreads=6
>-XX:ConcGCThreads=2
>-XX:MaxGCPauseMillis=200
>-XX:InitiatingHeapOccupancyPercent=40
>
>on ignite configuration i've changed all thread pool sizes, which were much 
>more than these;
>
>
>
>
>
>
>
>
>Here is the 16 hours of GC report;
>https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB
>
>
>
>On 2021/09/27 17:11:21, Ilya Korol < llivezk...@gmail.com > wrote:
>> Actually Query interface doesn't define close() method, but QueryCursor
>> does.
>> In your snippets you're using try-with-resource construction for SELECT
>> queries which is good, but when you run MERGE INTO query you would also
>> get an QueryCursor as a result of
>>
>> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
>>
>> so maybe this QueryCursor objects still hold some resources/memory.
>> Javadoc for QueryCursor states that you should always close cursors.
>>
>> To simplify cursor closing there is a cursor.getAll() method that will
>> do this for you under the hood.
>>
>>
>> On 2021/09/13 06:17:21, Ibrahim Altun < i...@segmentify.com > wrote:
>> > Hi Ilya,>
>> >
>> > since this is production environment i could not risk to take heap
>> dump for now, but i will try to convince my superiors to get one and
>> analyze it.>
>> >
>> > Queries are heavily used in our system but aren't they autoclosable
>> objects? do we have to close them anyway?>
>> >
>> > here are some usage examples on our system;>
>> > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
>> "label", "language") VALUES (?, ?, ?)>
>> > igniteCacheService.getCache(ID,
>> IgniteCacheType.LABEL).query(insertQuery);>
>> >
>> > another usage example;>
>> > --sqlFieldsQuery is like this; >
>> > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";>
>> > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
>> > sqlFieldsQuery.setLazy(true);>
>> > sqlFieldsQuery.setArgs(emails.toArray());>
>> >
>> > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
>> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
>> >
>> >
>> >
>> > On 2021/09/12 20:28:09, Shishkov Ilya < sh...@gmail.com > wrote: >
>> > > Hi, Ibrahim!>
>> > > Have you analyzed the heap dump of the server node JVMs?>
>> > > In case your application executes queries are their cursors closed?>
>> > > >
>> > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun < ib...@segmentify.com >:>
>> > > >
>> > > > Igniters any comment on this issue, we are facing huge GC
>> problems on>
>> > > > production environment, please advise.>
>> > > >>
>> > > > On 2021/09/07 14:11:09, Ibrahim Altun < ib...@segmentify.com >>
>> > > > wrote:>
>> > > > > Hi,>
>> > > > >>
>> > > > > totally 400 - 600K reads/writes/updates>
>> > > > > 12core>
>> > > > > 64GB RAM>
>> > > > > no iowait>
>> > > > > 10 nodes>
>> > > > >>
>> > > > > On 2021/09/07 12:51:28, Piotr Jagielski < pj...@touk.pl > wrote:>
>> > > > > > Hi,>
>> > > > > > Can you provide some information on how you use the cluster?
>> How many>
>> > > > reads/writes/updates per second? Also CPU / RAM spec of cluster
>> nodes?>
>> > > > > >>
>> > > > > > We observed full GC / CPU load / OOM killer when loading big
>> amount of>
>> > > > data (15 mln records, data streamer + allowOverwrite=true). We've
>> seen>
>> > > > 200-400k updates per sec on JMX metrics, but load up to 10 on
>> nodes, iowait>
>> > > > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto
>> 8CPU, 32GB>
>> > > > RAM). Ignite 2.10>
>> > > > > >>
>> > > > > > Regards,>
>> > > > > > Piotr>
>> > > > > >>
>> > > > > > On 2021/09/02 08:36:07, Ibrahim Altun < ib...@segmentify.com >>
>> > > > wrote:>
>> > > > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite
>> nodes>
>> > > > facing>
>> > > > > > > huge full GC operations after 24-36 hours after node start.>
>> > > > > > >>
>> > > > > > > We try to increase heap size but no luck, here is the start>
>> > > > configuration>
>> > > > > > > for nodes;>
>> > > > > > >>
>> > > > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server>
>> > > > > > >>
>> > > >
>> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml>
>>
>> > > > > > > -Dcom.sun.management.jmxremote>
>> > > > > > > -Dcom.sun.management.jmxremote.authenticate=false>
>> > > > > > > 

Re: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

after many configuration changes and optimizations, i think i've solved the 
heap problem.

here are the changes that i applied to the system;
JVM changes -> 
https://medium.com/@hoan.nguyen.it/how-did-g1gc-tuning-flags-affect-our-back-end-web-app-c121d38dfe56
 helped a lot

nodes are running on 12CORE and 64GB MEM servers, i've added the following jvm 
parameters

-XX:ParallelGCThreads=6 
-XX:ConcGCThreads=2 
-XX:MaxGCPauseMillis=200 
-XX:InitiatingHeapOccupancyPercent=40

on ignite configuration i've changed all thread pool sizes, which were much 
more than these;








Here is the 16 hours of GC report;
https://gceasy.io/diamondgc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjkvLS1nYy5sb2cuMC5jdXJyZW50LS04LTU4LTMx=WEB



On 2021/09/27 17:11:21, Ilya Korol  wrote: 
> Actually Query interface doesn't define close() method, but QueryCursor 
> does.
> In your snippets you're using try-with-resource construction for SELECT 
> queries which is good, but when you run MERGE INTO query you would also 
> get an QueryCursor as a result of
> 
> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
> 
> so maybe this QueryCursor objects still hold some resources/memory. 
> Javadoc for QueryCursor states that you should always close cursors.
> 
> To simplify cursor closing there is a cursor.getAll() method that will 
> do this for you under the hood.
> 
> 
> On 2021/09/13 06:17:21, Ibrahim Altun  wrote:
>  > Hi Ilya,>
>  >
>  > since this is production environment i could not risk to take heap 
> dump for now, but i will try to convince my superiors to get one and 
> analyze it.>
>  >
>  > Queries are heavily used in our system but aren't they autoclosable 
> objects? do we have to close them anyway?>
>  >
>  > here are some usage examples on our system;>
>  > --insert query is like this; MERGE INTO "ProductLabel" ("productId", 
> "label", "language") VALUES (?, ?, ?)>
>  > igniteCacheService.getCache(ID, 
> IgniteCacheType.LABEL).query(insertQuery);>
>  >
>  > another usage example;>
>  > --sqlFieldsQuery is like this; >
>  > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";>
>  > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
>  > sqlFieldsQuery.setLazy(true);>
>  > sqlFieldsQuery.setArgs(emails.toArray());>
>  >
>  > try (QueryCursor> ignored = igniteCacheService.getCache(ID, 
> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>
>  >
>  >
>  >
>  > On 2021/09/12 20:28:09, Shishkov Ilya  wrote: >
>  > > Hi, Ibrahim!>
>  > > Have you analyzed the heap dump of the server node JVMs?>
>  > > In case your application executes queries are their cursors closed?>
>  > > >
>  > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun :>
>  > > >
>  > > > Igniters any comment on this issue, we are facing huge GC 
> problems on>
>  > > > production environment, please advise.>
>  > > >>
>  > > > On 2021/09/07 14:11:09, Ibrahim Altun >
>  > > > wrote:>
>  > > > > Hi,>
>  > > > >>
>  > > > > totally 400 - 600K reads/writes/updates>
>  > > > > 12core>
>  > > > > 64GB RAM>
>  > > > > no iowait>
>  > > > > 10 nodes>
>  > > > >>
>  > > > > On 2021/09/07 12:51:28, Piotr Jagielski  wrote:>
>  > > > > > Hi,>
>  > > > > > Can you provide some information on how you use the cluster? 
> How many>
>  > > > reads/writes/updates per second? Also CPU / RAM spec of cluster 
> nodes?>
>  > > > > >>
>  > > > > > We observed full GC / CPU load / OOM killer when loading big 
> amount of>
>  > > > data (15 mln records, data streamer + allowOverwrite=true). We've 
> seen>
>  > > > 200-400k updates per sec on JMX metrics, but load up to 10 on 
> nodes, iowait>
>  > > > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 
> 8CPU, 32GB>
>  > > > RAM). Ignite 2.10>
>  > > > > >>
>  > > > > > Regards,>
>  > > > > > Piotr>
>  > > > > >>
>  > > > > > On 2021/09/02 08:36:07, Ibrahim Altun >
>  > > > wrote:>
>  > > > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite 
> nodes>
>  > > > facing>
>  > > > > > > huge full GC operations after 24-36 hours after node start.>
>  > > > > > >>
>  > > > > > > We try to increase heap size but no luck, here is the start>
>  > > > configuration>
>  > > > > > > for nodes;>
>  > > > > > >>
>  > > > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server>
>  > > > > > >>
>  > > > 
> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml>
>  
> 
>  > > > > > > -Dcom.sun.management.jmxremote>
>  > > > > > > -Dcom.sun.management.jmxremote.authenticate=false>
>  > > > > > > -Dcom.sun.management.jmxremote.port=49165>
>  > > > > > > -Dcom.sun.management.jmxremote.host=localhost>
>  > > > > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g>
>  > > > > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true>
>  > > > > > > -DIGNITE_WAL_MMAP=true 
> -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10>
>  > > > > > > -Djava.net.preferIPv4Stack=true">
>  > > > > > >>
>  > > > > > > 

Re: apache ignite 2.10.0 heap starvation

[Ilya Korol]

Actually Query interface doesn't define close() method, but QueryCursor 
does.
In your snippets you're using try-with-resource construction for SELECT 
queries which is good, but when you run MERGE INTO query you would also 
get an QueryCursor as a result of


igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);

so maybe this QueryCursor objects still hold some resources/memory. 
Javadoc for QueryCursor states that you should always close cursors.


To simplify cursor closing there is a cursor.getAll() method that will 
do this for you under the hood.



On 2021/09/13 06:17:21, Ibrahim Altun  wrote:
> Hi Ilya,>
>
> since this is production environment i could not risk to take heap 
dump for now, but i will try to convince my superiors to get one and 
analyze it.>

>
> Queries are heavily used in our system but aren't they autoclosable 
objects? do we have to close them anyway?>

>
> here are some usage examples on our system;>
> --insert query is like this; MERGE INTO "ProductLabel" ("productId", 
"label", "language") VALUES (?, ?, ?)>
> igniteCacheService.getCache(ID, 
IgniteCacheType.LABEL).query(insertQuery);>

>
> another usage example;>
> --sqlFieldsQuery is like this; >
> String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";>
> SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);>
> sqlFieldsQuery.setLazy(true);>
> sqlFieldsQuery.setArgs(emails.toArray());>
>
> try (QueryCursor> ignored = igniteCacheService.getCache(ID, 
IgniteCacheType.USER).query(sqlFieldsQuery)) {...}>

>
>
>
> On 2021/09/12 20:28:09, Shishkov Ilya  wrote: >
> > Hi, Ibrahim!>
> > Have you analyzed the heap dump of the server node JVMs?>
> > In case your application executes queries are their cursors closed?>
> > >
> > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun :>
> > >
> > > Igniters any comment on this issue, we are facing huge GC 
problems on>

> > > production environment, please advise.>
> > >>
> > > On 2021/09/07 14:11:09, Ibrahim Altun >
> > > wrote:>
> > > > Hi,>
> > > >>
> > > > totally 400 - 600K reads/writes/updates>
> > > > 12core>
> > > > 64GB RAM>
> > > > no iowait>
> > > > 10 nodes>
> > > >>
> > > > On 2021/09/07 12:51:28, Piotr Jagielski  wrote:>
> > > > > Hi,>
> > > > > Can you provide some information on how you use the cluster? 
How many>
> > > reads/writes/updates per second? Also CPU / RAM spec of cluster 
nodes?>

> > > > >>
> > > > > We observed full GC / CPU load / OOM killer when loading big 
amount of>
> > > data (15 mln records, data streamer + allowOverwrite=true). We've 
seen>
> > > 200-400k updates per sec on JMX metrics, but load up to 10 on 
nodes, iowait>
> > > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 
8CPU, 32GB>

> > > RAM). Ignite 2.10>
> > > > >>
> > > > > Regards,>
> > > > > Piotr>
> > > > >>
> > > > > On 2021/09/02 08:36:07, Ibrahim Altun >
> > > wrote:>
> > > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite 
nodes>

> > > facing>
> > > > > > huge full GC operations after 24-36 hours after node start.>
> > > > > >>
> > > > > > We try to increase heap size but no luck, here is the start>
> > > configuration>
> > > > > > for nodes;>
> > > > > >>
> > > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server>
> > > > > >>
> > > 
-javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml> 


> > > > > > -Dcom.sun.management.jmxremote>
> > > > > > -Dcom.sun.management.jmxremote.authenticate=false>
> > > > > > -Dcom.sun.management.jmxremote.port=49165>
> > > > > > -Dcom.sun.management.jmxremote.host=localhost>
> > > > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g>
> > > > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true>
> > > > > > -DIGNITE_WAL_MMAP=true 
-DIGNITE_BPLUS_TREE_LOCK_RETRIES=10>

> > > > > > -Djava.net.preferIPv4Stack=true">
> > > > > >>
> > > > > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC>
> > > > > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC>
> > > > > > -XX:+UseStringDeduplication 
-Xloggc:/var/log/apache-ignite/gc.log>

> > > > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps>
> > > > > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause>
> > > > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10>
> > > > > > -XX:GCLogFileSize=100M">
> > > > > >>
> > > > > > here is the 80 hours of GC analyize report:>
> > > > > >>
> > > 
https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB> 


> > > > > >>
> > > > > > do we need more heap size or is there a BUG that we need to 
be aware?>

> > > > > >>
> > > > > > here is the node configuration:>
> > > > > >>
> > > > > > >
> > > > > > http://www.springframework.org/schema/beans;>
> > > > > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;>
> > > > > > xsi:schemaLocation=">
> > > > > > http://www.springframework.org/schema/beans>
> > > > > > 
http://www.springframework.org/schema/beans/spring-beans.xsd;>>

> > > > > > 
> > > > > > 

Re: apache ignite 2.10.0 heap starvation

[Stephen Darlington]

I can’t say whether it’s the problem but I can say that’s really not going to 
help. The default for most of these thread pools is going to be 12 (the number 
of cores). Each thread is going to have its own working set, stack, etc. If all 
the threads are thrashing, it’s not going to be able to garbage collect the 
memory.

> On 27 Sep 2021, at 16:19, Ibrahim Altun  wrote:
> 
>
>
>
>
>
>
>
> 
> out of curiosity, how does these poolSizes affects heap size usages? we are 
> using physical machines with total of 12 cores. we did have these sizes 
> before upgrade from 2.7.6 and had no heap issues at all. we do have same data 
> size and no changes on any ignite usage in code-base.
> 
> 
> On 2021/09/20 09:55:34, Stephen Darlington  
> wrote: 
>> I’m not sure if it’s possible to diagnose this without knowing what’s on the 
>> heap.
>> 
>> I don’t think there are any known issues around heap usage. Certainly a 
>> MERGE is going to use a lot of heap space, but it should be recovered once 
>> the statement is closed. Similarly, your select could use a lot of heap 
>> depending on the number of records scanned, but that should also be released 
>> once the statement is closed. 
>> 
>>> On 20 Sep 2021, at 07:34, Ibrahim Altun  
>>> wrote:
>>> 
>>> Hi igniters,
>>> 
>>> the situation getting frustrated keeps going really huge GCs. Here is the 
>>> last GC report;
>>> 
>>> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjAvLS1nYy5sb2cuMC5jdXJyZW50IDIuemlwLS02LTMxLTg==WEB
>>> 
>>> This is production environment we cannot get heap dump since the prodecure 
>>> stops the whole world.
>>> 
>>> please advise.
>>> 
>>> On 2021/09/13 07:47:16, Naveen Kumar  wrote: 
 Just to add what Ibrahim mentioned, I also have a similar issue but I am
 using 2.8.1 and we do have a good number of Insert/merge statements getting
 executed.
 We do get warnings for some of the MERGE statements, like "*The search row
 by explicit key isn't supported. The primary key is always used to used to
 search a row". *Does is have any impact on heap utilization
 
 In our case, heap memory more than the off heap memory and doing a
 frequent rolling-over restarts to avoid OOM.
 
 
 
 On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun 
 
 wrote:
 
> Hi Ilya,
> 
> since this is production environment i could not risk to take heap dump
> for now, but i will try to convince my superiors to get one and analyze 
> it.
> 
> Queries are heavily used in our system but aren't they autoclosable
> objects? do we have to close them anyway?
> 
> here are some usage examples on our system;
> --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> "label", "language") VALUES (?, ?, ?)
> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
> 
> another usage example;
> --sqlFieldsQuery is like this;
> String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";
> SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);
> sqlFieldsQuery.setLazy(true);
> sqlFieldsQuery.setArgs(emails.toArray());
> 
> try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}
> 
> 
> 
> On 2021/09/12 20:28:09, Shishkov Ilya  wrote:
>> Hi, Ibrahim!
>> Have you analyzed the heap dump of the server node JVMs?
>> In case your application executes queries are their cursors closed?
>> 
>> пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun <
> ibrahim.al...@segmentify.com>:
>> 
>>> Igniters any comment on this issue, we are facing huge GC problems on
>>> production environment, please advise.
>>> 
>>> On 2021/09/07 14:11:09, Ibrahim Altun 
>>> wrote:
 Hi,
 
 totally 400 - 600K reads/writes/updates
 12core
 64GB RAM
 no iowait
 10 nodes
 
 On 2021/09/07 12:51:28, Piotr Jagielski  wrote:
> Hi,
> Can you provide some information on how you use the cluster? How
> many
>>> reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> 
> We observed full GC / CPU load / OOM killer when loading big
> amount of
>>> data (15 mln records, data streamer + allowOverwrite=true). We've seen
>>> 200-400k updates per sec on JMX metrics, but load up to 10 on nodes,
> iowait
>>> to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU,
> 32GB
>>> RAM). Ignite 2.10
> 
> Regards,
> Piotr
> 
> On 2021/09/02 08:36:07, Ibrahim Altun <
> ibrahim.al...@segmentify.com>
>>> wrote:
>> After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
>>> facing
>> huge full GC operations after 24-36 hours after node start.

Re: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

out of curiosity, how does these poolSizes affects heap size usages? we are 
using physical machines with total of 12 cores. we did have these sizes before 
upgrade from 2.7.6 and had no heap issues at all. we do have same data size and 
no changes on any ignite usage in code-base.


On 2021/09/20 09:55:34, Stephen Darlington  
wrote: 
> I’m not sure if it’s possible to diagnose this without knowing what’s on the 
> heap.
> 
> I don’t think there are any known issues around heap usage. Certainly a MERGE 
> is going to use a lot of heap space, but it should be recovered once the 
> statement is closed. Similarly, your select could use a lot of heap depending 
> on the number of records scanned, but that should also be released once the 
> statement is closed. 
> 
> > On 20 Sep 2021, at 07:34, Ibrahim Altun  
> > wrote:
> > 
> > Hi igniters,
> > 
> > the situation getting frustrated keeps going really huge GCs. Here is the 
> > last GC report;
> > 
> > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjAvLS1nYy5sb2cuMC5jdXJyZW50IDIuemlwLS02LTMxLTg==WEB
> > 
> > This is production environment we cannot get heap dump since the prodecure 
> > stops the whole world.
> > 
> > please advise.
> > 
> > On 2021/09/13 07:47:16, Naveen Kumar  wrote: 
> >> Just to add what Ibrahim mentioned, I also have a similar issue but I am
> >> using 2.8.1 and we do have a good number of Insert/merge statements getting
> >> executed.
> >> We do get warnings for some of the MERGE statements, like "*The search row
> >> by explicit key isn't supported. The primary key is always used to used to
> >> search a row". *Does is have any impact on heap utilization
> >> 
> >> In our case, heap memory more than the off heap memory and doing a
> >> frequent rolling-over restarts to avoid OOM.
> >> 
> >> 
> >> 
> >> On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun 
> >> 
> >> wrote:
> >> 
> >>> Hi Ilya,
> >>> 
> >>> since this is production environment i could not risk to take heap dump
> >>> for now, but i will try to convince my superiors to get one and analyze 
> >>> it.
> >>> 
> >>> Queries are heavily used in our system but aren't they autoclosable
> >>> objects? do we have to close them anyway?
> >>> 
> >>> here are some usage examples on our system;
> >>> --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> >>> "label", "language") VALUES (?, ?, ?)
> >>> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
> >>> 
> >>> another usage example;
> >>> --sqlFieldsQuery is like this;
> >>> String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";
> >>> SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);
> >>> sqlFieldsQuery.setLazy(true);
> >>> sqlFieldsQuery.setArgs(emails.toArray());
> >>> 
> >>> try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> >>> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}
> >>> 
> >>> 
> >>> 
> >>> On 2021/09/12 20:28:09, Shishkov Ilya  wrote:
>  Hi, Ibrahim!
>  Have you analyzed the heap dump of the server node JVMs?
>  In case your application executes queries are their cursors closed?
>  
>  пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun <
> >>> ibrahim.al...@segmentify.com>:
>  
> > Igniters any comment on this issue, we are facing huge GC problems on
> > production environment, please advise.
> > 
> > On 2021/09/07 14:11:09, Ibrahim Altun 
> > wrote:
> >> Hi,
> >> 
> >> totally 400 - 600K reads/writes/updates
> >> 12core
> >> 64GB RAM
> >> no iowait
> >> 10 nodes
> >> 
> >> On 2021/09/07 12:51:28, Piotr Jagielski  wrote:
> >>> Hi,
> >>> Can you provide some information on how you use the cluster? How
> >>> many
> > reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> >>> 
> >>> We observed full GC / CPU load / OOM killer when loading big
> >>> amount of
> > data (15 mln records, data streamer + allowOverwrite=true). We've seen
> > 200-400k updates per sec on JMX metrics, but load up to 10 on nodes,
> >>> iowait
> > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU,
> >>> 32GB
> > RAM). Ignite 2.10
> >>> 
> >>> Regards,
> >>> Piotr
> >>> 
> >>> On 2021/09/02 08:36:07, Ibrahim Altun <
> >>> ibrahim.al...@segmentify.com>
> > wrote:
>  After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
> > facing
>  huge full GC operations after 24-36 hours after node start.
>  
>  We try to increase heap size but no luck, here is the start
> > configuration
>  for nodes;
>  
>  JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
>  
> > 
> >>> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
>  -Dcom.sun.management.jmxremote
>  -Dcom.sun.management.jmxremote.authenticate=false
> 

Re: apache ignite 2.10.0 heap starvation

[Stephen Darlington]

I’m not sure if it’s possible to diagnose this without knowing what’s on the 
heap.

I don’t think there are any known issues around heap usage. Certainly a MERGE 
is going to use a lot of heap space, but it should be recovered once the 
statement is closed. Similarly, your select could use a lot of heap depending 
on the number of records scanned, but that should also be released once the 
statement is closed. 

> On 20 Sep 2021, at 07:34, Ibrahim Altun  wrote:
> 
> Hi igniters,
> 
> the situation getting frustrated keeps going really huge GCs. Here is the 
> last GC report;
> 
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjAvLS1nYy5sb2cuMC5jdXJyZW50IDIuemlwLS02LTMxLTg==WEB
> 
> This is production environment we cannot get heap dump since the prodecure 
> stops the whole world.
> 
> please advise.
> 
> On 2021/09/13 07:47:16, Naveen Kumar  wrote: 
>> Just to add what Ibrahim mentioned, I also have a similar issue but I am
>> using 2.8.1 and we do have a good number of Insert/merge statements getting
>> executed.
>> We do get warnings for some of the MERGE statements, like "*The search row
>> by explicit key isn't supported. The primary key is always used to used to
>> search a row". *Does is have any impact on heap utilization
>> 
>> In our case, heap memory more than the off heap memory and doing a
>> frequent rolling-over restarts to avoid OOM.
>> 
>> 
>> 
>> On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun 
>> wrote:
>> 
>>> Hi Ilya,
>>> 
>>> since this is production environment i could not risk to take heap dump
>>> for now, but i will try to convince my superiors to get one and analyze it.
>>> 
>>> Queries are heavily used in our system but aren't they autoclosable
>>> objects? do we have to close them anyway?
>>> 
>>> here are some usage examples on our system;
>>> --insert query is like this; MERGE INTO "ProductLabel" ("productId",
>>> "label", "language") VALUES (?, ?, ?)
>>> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
>>> 
>>> another usage example;
>>> --sqlFieldsQuery is like this;
>>> String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";
>>> SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);
>>> sqlFieldsQuery.setLazy(true);
>>> sqlFieldsQuery.setArgs(emails.toArray());
>>> 
>>> try (QueryCursor> ignored = igniteCacheService.getCache(ID,
>>> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}
>>> 
>>> 
>>> 
>>> On 2021/09/12 20:28:09, Shishkov Ilya  wrote:
 Hi, Ibrahim!
 Have you analyzed the heap dump of the server node JVMs?
 In case your application executes queries are their cursors closed?
 
 пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun <
>>> ibrahim.al...@segmentify.com>:
 
> Igniters any comment on this issue, we are facing huge GC problems on
> production environment, please advise.
> 
> On 2021/09/07 14:11:09, Ibrahim Altun 
> wrote:
>> Hi,
>> 
>> totally 400 - 600K reads/writes/updates
>> 12core
>> 64GB RAM
>> no iowait
>> 10 nodes
>> 
>> On 2021/09/07 12:51:28, Piotr Jagielski  wrote:
>>> Hi,
>>> Can you provide some information on how you use the cluster? How
>>> many
> reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
>>> 
>>> We observed full GC / CPU load / OOM killer when loading big
>>> amount of
> data (15 mln records, data streamer + allowOverwrite=true). We've seen
> 200-400k updates per sec on JMX metrics, but load up to 10 on nodes,
>>> iowait
> to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU,
>>> 32GB
> RAM). Ignite 2.10
>>> 
>>> Regards,
>>> Piotr
>>> 
>>> On 2021/09/02 08:36:07, Ibrahim Altun <
>>> ibrahim.al...@segmentify.com>
> wrote:
 After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
> facing
 huge full GC operations after 24-36 hours after node start.
 
 We try to increase heap size but no luck, here is the start
> configuration
 for nodes;
 
 JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
 
> 
>>> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
 -Dcom.sun.management.jmxremote
 -Dcom.sun.management.jmxremote.authenticate=false
 -Dcom.sun.management.jmxremote.port=49165
 -Dcom.sun.management.jmxremote.host=localhost
 -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
 -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
 -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
 -Djava.net.preferIPv4Stack=true"
 
 JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
 -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
 -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
 -XX:+PrintGCDetails -XX:+PrintGCDateStamps
 -XX:+PrintTenuringDistribution -XX:+PrintGCCause

Re: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

Hi igniters,

the situation getting frustrated keeps going really huge GCs. Here is the last 
GC report;

https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDkvMjAvLS1nYy5sb2cuMC5jdXJyZW50IDIuemlwLS02LTMxLTg==WEB

This is production environment we cannot get heap dump since the prodecure 
stops the whole world.

please advise.

On 2021/09/13 07:47:16, Naveen Kumar  wrote: 
> Just to add what Ibrahim mentioned, I also have a similar issue but I am
> using 2.8.1 and we do have a good number of Insert/merge statements getting
> executed.
> We do get warnings for some of the MERGE statements, like "*The search row
> by explicit key isn't supported. The primary key is always used to used to
> search a row". *Does is have any impact on heap utilization
> 
> In our case, heap memory more than the off heap memory and doing a
> frequent rolling-over restarts to avoid OOM.
> 
> 
> 
> On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun 
> wrote:
> 
> > Hi Ilya,
> >
> > since this is production environment i could not risk to take heap dump
> > for now, but i will try to convince my superiors to get one and analyze it.
> >
> > Queries are heavily used in our system but aren't they autoclosable
> > objects? do we have to close them anyway?
> >
> > here are some usage examples on our system;
> > --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> > "label", "language") VALUES (?, ?, ?)
> > igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
> >
> > another usage example;
> > --sqlFieldsQuery is like this;
> > String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";
> > SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);
> > sqlFieldsQuery.setLazy(true);
> > sqlFieldsQuery.setArgs(emails.toArray());
> >
> > try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> > IgniteCacheType.USER).query(sqlFieldsQuery)) {...}
> >
> >
> >
> > On 2021/09/12 20:28:09, Shishkov Ilya  wrote:
> > > Hi, Ibrahim!
> > > Have you analyzed the heap dump of the server node JVMs?
> > > In case your application executes queries are their cursors closed?
> > >
> > > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun <
> > ibrahim.al...@segmentify.com>:
> > >
> > > > Igniters any comment on this issue, we are facing huge GC problems on
> > > > production environment, please advise.
> > > >
> > > > On 2021/09/07 14:11:09, Ibrahim Altun 
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > totally 400 - 600K reads/writes/updates
> > > > > 12core
> > > > > 64GB RAM
> > > > > no iowait
> > > > > 10 nodes
> > > > >
> > > > > On 2021/09/07 12:51:28, Piotr Jagielski  wrote:
> > > > > > Hi,
> > > > > > Can you provide some information on how you use the cluster? How
> > many
> > > > reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> > > > > >
> > > > > > We observed full GC / CPU load / OOM killer when loading big
> > amount of
> > > > data (15 mln records, data streamer + allowOverwrite=true). We've seen
> > > > 200-400k updates per sec on JMX metrics, but load up to 10 on nodes,
> > iowait
> > > > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU,
> > 32GB
> > > > RAM). Ignite 2.10
> > > > > >
> > > > > > Regards,
> > > > > > Piotr
> > > > > >
> > > > > > On 2021/09/02 08:36:07, Ibrahim Altun <
> > ibrahim.al...@segmentify.com>
> > > > wrote:
> > > > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
> > > > facing
> > > > > > > huge full GC operations after 24-36 hours after node start.
> > > > > > >
> > > > > > > We try to increase heap size but no luck, here is the start
> > > > configuration
> > > > > > > for nodes;
> > > > > > >
> > > > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > > > > > >
> > > >
> > -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > > > > > > -Dcom.sun.management.jmxremote
> > > > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > > > -Dcom.sun.management.jmxremote.port=49165
> > > > > > > -Dcom.sun.management.jmxremote.host=localhost
> > > > > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > > > > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > > > > > > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> > > > > > > -Djava.net.preferIPv4Stack=true"
> > > > > > >
> > > > > > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > > > > > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > > > > > > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > > > > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > > > > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > > > > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > > > > > > -XX:GCLogFileSize=100M"
> > > > > > >
> > > > > > > here is the 80 hours of GC analyize report:
> > > > > > >
> > > >
> > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB
> > > > > > >
> > > 

Re: apache ignite 2.10.0 heap starvation

[Naveen Kumar]

Just to add what Ibrahim mentioned, I also have a similar issue but I am
using 2.8.1 and we do have a good number of Insert/merge statements getting
executed.
We do get warnings for some of the MERGE statements, like "*The search row
by explicit key isn't supported. The primary key is always used to used to
search a row". *Does is have any impact on heap utilization

In our case, heap memory more than the off heap memory and doing a
frequent rolling-over restarts to avoid OOM.



On Mon, Sep 13, 2021 at 11:47 AM Ibrahim Altun 
wrote:

> Hi Ilya,
>
> since this is production environment i could not risk to take heap dump
> for now, but i will try to convince my superiors to get one and analyze it.
>
> Queries are heavily used in our system but aren't they autoclosable
> objects? do we have to close them anyway?
>
> here are some usage examples on our system;
> --insert query is like this; MERGE INTO "ProductLabel" ("productId",
> "label", "language") VALUES (?, ?, ?)
> igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);
>
> another usage example;
> --sqlFieldsQuery is like this;
> String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";
> SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);
> sqlFieldsQuery.setLazy(true);
> sqlFieldsQuery.setArgs(emails.toArray());
>
> try (QueryCursor> ignored = igniteCacheService.getCache(ID,
> IgniteCacheType.USER).query(sqlFieldsQuery)) {...}
>
>
>
> On 2021/09/12 20:28:09, Shishkov Ilya  wrote:
> > Hi, Ibrahim!
> > Have you analyzed the heap dump of the server node JVMs?
> > In case your application executes queries are their cursors closed?
> >
> > пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun <
> ibrahim.al...@segmentify.com>:
> >
> > > Igniters any comment on this issue, we are facing huge GC problems on
> > > production environment, please advise.
> > >
> > > On 2021/09/07 14:11:09, Ibrahim Altun 
> > > wrote:
> > > > Hi,
> > > >
> > > > totally 400 - 600K reads/writes/updates
> > > > 12core
> > > > 64GB RAM
> > > > no iowait
> > > > 10 nodes
> > > >
> > > > On 2021/09/07 12:51:28, Piotr Jagielski  wrote:
> > > > > Hi,
> > > > > Can you provide some information on how you use the cluster? How
> many
> > > reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> > > > >
> > > > > We observed full GC / CPU load / OOM killer when loading big
> amount of
> > > data (15 mln records, data streamer + allowOverwrite=true). We've seen
> > > 200-400k updates per sec on JMX metrics, but load up to 10 on nodes,
> iowait
> > > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU,
> 32GB
> > > RAM). Ignite 2.10
> > > > >
> > > > > Regards,
> > > > > Piotr
> > > > >
> > > > > On 2021/09/02 08:36:07, Ibrahim Altun <
> ibrahim.al...@segmentify.com>
> > > wrote:
> > > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
> > > facing
> > > > > > huge full GC operations after 24-36 hours after node start.
> > > > > >
> > > > > > We try to increase heap size but no luck, here is the start
> > > configuration
> > > > > > for nodes;
> > > > > >
> > > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > > > > >
> > >
> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > > > > > -Dcom.sun.management.jmxremote
> > > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > > -Dcom.sun.management.jmxremote.port=49165
> > > > > > -Dcom.sun.management.jmxremote.host=localhost
> > > > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > > > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > > > > > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> > > > > > -Djava.net.preferIPv4Stack=true"
> > > > > >
> > > > > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > > > > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > > > > > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > > > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > > > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > > > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > > > > > -XX:GCLogFileSize=100M"
> > > > > >
> > > > > > here is the 80 hours of GC analyize report:
> > > > > >
> > >
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB
> > > > > >
> > > > > > do we need more heap size or is there a BUG that we need to be
> aware?
> > > > > >
> > > > > > here is the node configuration:
> > > > > >
> > > > > > 
> > > > > > http://www.springframework.org/schema/beans;
> > > > > >xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> > > > > >xsi:schemaLocation="
> > > > > > http://www.springframework.org/schema/beans
> > > > > > http://www.springframework.org/schema/beans/spring-beans.xsd
> ">
> > > > > >  > > > > > class="org.apache.ignite.configuration.IgniteConfiguration">
> > > > > > 
> > > > > >  

Re: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

Hi Ilya,

since this is production environment i could not risk to take heap dump for 
now, but i will try to convince my superiors to get one and analyze it.

Queries are heavily used in our system but aren't they autoclosable objects? do 
we have to close them anyway?

here are some usage examples on our system;
--insert query is like this; MERGE INTO "ProductLabel" ("productId", "label", 
"language") VALUES (?, ?, ?)
igniteCacheService.getCache(ID, IgniteCacheType.LABEL).query(insertQuery);

another usage example;
--sqlFieldsQuery is like this; 
String sql = "SELECT _val FROM \"UserRecord\" WHERE \"email\" IN (?)";
SqlFieldsQuery sqlFieldsQuery = new SqlFieldsQuery(sql);
sqlFieldsQuery.setLazy(true);
sqlFieldsQuery.setArgs(emails.toArray());

try (QueryCursor> ignored = igniteCacheService.getCache(ID, 
IgniteCacheType.USER).query(sqlFieldsQuery)) {...}



On 2021/09/12 20:28:09, Shishkov Ilya  wrote: 
> Hi, Ibrahim!
> Have you analyzed the heap dump of the server node JVMs?
> In case your application executes queries are their cursors closed?
> 
> пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun :
> 
> > Igniters any comment on this issue, we are facing huge GC problems on
> > production environment, please advise.
> >
> > On 2021/09/07 14:11:09, Ibrahim Altun 
> > wrote:
> > > Hi,
> > >
> > > totally 400 - 600K reads/writes/updates
> > > 12core
> > > 64GB RAM
> > > no iowait
> > > 10 nodes
> > >
> > > On 2021/09/07 12:51:28, Piotr Jagielski  wrote:
> > > > Hi,
> > > > Can you provide some information on how you use the cluster? How many
> > reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> > > >
> > > > We observed full GC / CPU load / OOM killer when loading big amount of
> > data (15 mln records, data streamer + allowOverwrite=true). We've seen
> > 200-400k updates per sec on JMX metrics, but load up to 10 on nodes, iowait
> > to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, 32GB
> > RAM). Ignite 2.10
> > > >
> > > > Regards,
> > > > Piotr
> > > >
> > > > On 2021/09/02 08:36:07, Ibrahim Altun 
> > wrote:
> > > > > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
> > facing
> > > > > huge full GC operations after 24-36 hours after node start.
> > > > >
> > > > > We try to increase heap size but no luck, here is the start
> > configuration
> > > > > for nodes;
> > > > >
> > > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > > > >
> > -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > > > > -Dcom.sun.management.jmxremote
> > > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > > -Dcom.sun.management.jmxremote.port=49165
> > > > > -Dcom.sun.management.jmxremote.host=localhost
> > > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > > > > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> > > > > -Djava.net.preferIPv4Stack=true"
> > > > >
> > > > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > > > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > > > > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > > > > -XX:GCLogFileSize=100M"
> > > > >
> > > > > here is the 80 hours of GC analyize report:
> > > > >
> > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB
> > > > >
> > > > > do we need more heap size or is there a BUG that we need to be aware?
> > > > >
> > > > > here is the node configuration:
> > > > >
> > > > > 
> > > > > http://www.springframework.org/schema/beans;
> > > > >xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> > > > >xsi:schemaLocation="
> > > > > http://www.springframework.org/schema/beans
> > > > > http://www.springframework.org/schema/beans/spring-beans.xsd;>
> > > > >  > > > > class="org.apache.ignite.configuration.IgniteConfiguration">
> > > > > 
> > > > > 
> > > > >  > > > > value="/etc/apache-ignite/ignite-log4j2.xml"/>
> > > > > 
> > > > > 
> > > > > 
> > > > >  > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > >
> > > > > 
> > > > > 
> > > > >
> > > > > 
> > > > > 
> > > > >  > class="org.apache.ignite.spi.deployment.uri.UriDeploymentSpi">
> > > > >  > > > > value="/tmp/temp_ignite_libs"/>
> > > > > 
> > > > > 
> > > > >
> > > > > file://freq=5000@localhost
> > /usr/share/apache-ignite/libs/segmentify/
> > > > > 
> > > > > 
> > > > > 
> > > > > 

Re: apache ignite 2.10.0 heap starvation

[Shishkov Ilya]

Hi, Ibrahim!
Have you analyzed the heap dump of the server node JVMs?
In case your application executes queries are their cursors closed?

пт, 10 сент. 2021 г. в 11:54, Ibrahim Altun :

> Igniters any comment on this issue, we are facing huge GC problems on
> production environment, please advise.
>
> On 2021/09/07 14:11:09, Ibrahim Altun 
> wrote:
> > Hi,
> >
> > totally 400 - 600K reads/writes/updates
> > 12core
> > 64GB RAM
> > no iowait
> > 10 nodes
> >
> > On 2021/09/07 12:51:28, Piotr Jagielski  wrote:
> > > Hi,
> > > Can you provide some information on how you use the cluster? How many
> reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> > >
> > > We observed full GC / CPU load / OOM killer when loading big amount of
> data (15 mln records, data streamer + allowOverwrite=true). We've seen
> 200-400k updates per sec on JMX metrics, but load up to 10 on nodes, iowait
> to 30%. Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, 32GB
> RAM). Ignite 2.10
> > >
> > > Regards,
> > > Piotr
> > >
> > > On 2021/09/02 08:36:07, Ibrahim Altun 
> wrote:
> > > > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes
> facing
> > > > huge full GC operations after 24-36 hours after node start.
> > > >
> > > > We try to increase heap size but no luck, here is the start
> configuration
> > > > for nodes;
> > > >
> > > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > > >
> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > > > -Dcom.sun.management.jmxremote
> > > > -Dcom.sun.management.jmxremote.authenticate=false
> > > > -Dcom.sun.management.jmxremote.port=49165
> > > > -Dcom.sun.management.jmxremote.host=localhost
> > > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > > > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> > > > -Djava.net.preferIPv4Stack=true"
> > > >
> > > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > > > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > > > -XX:GCLogFileSize=100M"
> > > >
> > > > here is the 80 hours of GC analyize report:
> > > >
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB
> > > >
> > > > do we need more heap size or is there a BUG that we need to be aware?
> > > >
> > > > here is the node configuration:
> > > >
> > > > 
> > > > http://www.springframework.org/schema/beans;
> > > >xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> > > >xsi:schemaLocation="
> > > > http://www.springframework.org/schema/beans
> > > > http://www.springframework.org/schema/beans/spring-beans.xsd;>
> > > >  > > > class="org.apache.ignite.configuration.IgniteConfiguration">
> > > > 
> > > > 
> > > >  > > > value="/etc/apache-ignite/ignite-log4j2.xml"/>
> > > > 
> > > > 
> > > > 
> > > >  class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > >
> > > > 
> > > > 
> > > >
> > > > 
> > > > 
> > > >  class="org.apache.ignite.spi.deployment.uri.UriDeploymentSpi">
> > > >  > > > value="/tmp/temp_ignite_libs"/>
> > > > 
> > > > 
> > > >
> > > > file://freq=5000@localhost
> /usr/share/apache-ignite/libs/segmentify/
> > > > 
> > > > 
> > > > 
> > > > 
> > > >
> > > > 
> > > > 
> > > > 
> > > >  class="org.apache.ignite.configuration.CacheConfiguration">
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > >
> > > > 
> > > > 
> > > >  class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> > > > 
> > > > 
> > > >  > > >
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > >
> > > > 
> > > > 
> > > >  class="org.apache.ignite.configuration.DataStorageConfiguration">
> > > > 
> > > >  > > > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > > >  value="true"/>
> > > >  > > > value="#{ 2L * 1024 * 1024 * 1024}"/>
> > > > 
> > > > 

Re: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

Igniters any comment on this issue, we are facing huge GC problems on 
production environment, please advise.

On 2021/09/07 14:11:09, Ibrahim Altun  wrote: 
> Hi,
> 
> totally 400 - 600K reads/writes/updates
> 12core
> 64GB RAM
> no iowait
> 10 nodes
> 
> On 2021/09/07 12:51:28, Piotr Jagielski  wrote: 
> > Hi,
> > Can you provide some information on how you use the cluster? How many 
> > reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> > 
> > We observed full GC / CPU load / OOM killer when loading big amount of data 
> > (15 mln records, data streamer + allowOverwrite=true). We've seen 200-400k 
> > updates per sec on JMX metrics, but load up to 10 on nodes, iowait to 30%. 
> > Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, 32GB RAM). 
> > Ignite 2.10
> > 
> > Regards,
> > Piotr
> > 
> > On 2021/09/02 08:36:07, Ibrahim Altun  wrote: 
> > > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes facing
> > > huge full GC operations after 24-36 hours after node start.
> > > 
> > > We try to increase heap size but no luck, here is the start configuration
> > > for nodes;
> > > 
> > > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > > -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > > -Dcom.sun.management.jmxremote
> > > -Dcom.sun.management.jmxremote.authenticate=false
> > > -Dcom.sun.management.jmxremote.port=49165
> > > -Dcom.sun.management.jmxremote.host=localhost
> > > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> > > -Djava.net.preferIPv4Stack=true"
> > > 
> > > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > > -XX:GCLogFileSize=100M"
> > > 
> > > here is the 80 hours of GC analyize report:
> > > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB
> > > 
> > > do we need more heap size or is there a BUG that we need to be aware?
> > > 
> > > here is the node configuration:
> > > 
> > > 
> > > http://www.springframework.org/schema/beans;
> > >xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> > >xsi:schemaLocation="
> > > http://www.springframework.org/schema/beans
> > > http://www.springframework.org/schema/beans/spring-beans.xsd;>
> > >  > > class="org.apache.ignite.configuration.IgniteConfiguration">
> > > 
> > > 
> > >  > > value="/etc/apache-ignite/ignite-log4j2.xml"/>
> > > 
> > > 
> > > 
> > >  > > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >  > > class="org.apache.ignite.spi.deployment.uri.UriDeploymentSpi">
> > >  > > value="/tmp/temp_ignite_libs"/>
> > > 
> > > 
> > > 
> > > file://freq=5000@localhost/usr/share/apache-ignite/libs/segmentify/
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >  > > class="org.apache.ignite.configuration.CacheConfiguration">
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >  > > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > >  > > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > > 
> > >  > > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > > 
> > >  > > value="#{ 2L * 1024 * 1024 * 1024}"/>
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > -- 
> > > İbrahim Halil AltunSenior Software 
> > > Engineer+90
> > > 536 3327510 • segmentify.com → UK • Germany •
> > > Turkey 

Re: apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

Hi,

totally 400 - 600K reads/writes/updates
12core
64GB RAM
no iowait
10 nodes

On 2021/09/07 12:51:28, Piotr Jagielski  wrote: 
> Hi,
> Can you provide some information on how you use the cluster? How many 
> reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?
> 
> We observed full GC / CPU load / OOM killer when loading big amount of data 
> (15 mln records, data streamer + allowOverwrite=true). We've seen 200-400k 
> updates per sec on JMX metrics, but load up to 10 on nodes, iowait to 30%. 
> Our cluster is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, 32GB RAM). 
> Ignite 2.10
> 
> Regards,
> Piotr
> 
> On 2021/09/02 08:36:07, Ibrahim Altun  wrote: 
> > After upgrading from 2.7.1 version to 2.10.0 version ignite nodes facing
> > huge full GC operations after 24-36 hours after node start.
> > 
> > We try to increase heap size but no luck, here is the start configuration
> > for nodes;
> > 
> > JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> > -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> > -Dcom.sun.management.jmxremote
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.port=49165
> > -Dcom.sun.management.jmxremote.host=localhost
> > -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> > -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> > -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> > -Djava.net.preferIPv4Stack=true"
> > 
> > JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> > -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> > -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> > -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> > -XX:GCLogFileSize=100M"
> > 
> > here is the 80 hours of GC analyize report:
> > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB
> > 
> > do we need more heap size or is there a BUG that we need to be aware?
> > 
> > here is the node configuration:
> > 
> > 
> > http://www.springframework.org/schema/beans;
> >xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> >xsi:schemaLocation="
> > http://www.springframework.org/schema/beans
> > http://www.springframework.org/schema/beans/spring-beans.xsd;>
> >  > class="org.apache.ignite.configuration.IgniteConfiguration">
> > 
> > 
> >  > value="/etc/apache-ignite/ignite-log4j2.xml"/>
> > 
> > 
> > 
> >  > class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  > value="/tmp/temp_ignite_libs"/>
> > 
> > 
> > 
> > file://freq=5000@localhost/usr/share/apache-ignite/libs/segmentify/
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  > class="org.apache.ignite.configuration.CacheConfiguration">
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  > class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> >  > class="org.apache.ignite.configuration.DataStorageConfiguration">
> > 
> >  > class="org.apache.ignite.configuration.DataRegionConfiguration">
> > 
> >  > value="#{ 2L * 1024 * 1024 * 1024}"/>
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > -- 
> > İbrahim Halil AltunSenior Software Engineer+90
> > 536 3327510 • segmentify.com → UK • Germany •
> > Turkey 
> > 
> > 
> 

Re: apache ignite 2.10.0 heap starvation

[Piotr Jagielski]

Hi,
Can you provide some information on how you use the cluster? How many 
reads/writes/updates per second? Also CPU / RAM spec of cluster nodes?

We observed full GC / CPU load / OOM killer when loading big amount of data (15 
mln records, data streamer + allowOverwrite=true). We've seen 200-400k updates 
per sec on JMX metrics, but load up to 10 on nodes, iowait to 30%. Our cluster 
is 3 x 4CPU, 16GB RAM (already upgradingto 8CPU, 32GB RAM). Ignite 2.10

Regards,
Piotr

On 2021/09/02 08:36:07, Ibrahim Altun  wrote: 
> After upgrading from 2.7.1 version to 2.10.0 version ignite nodes facing
> huge full GC operations after 24-36 hours after node start.
> 
> We try to increase heap size but no luck, here is the start configuration
> for nodes;
> 
> JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
> -javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.port=49165
> -Dcom.sun.management.jmxremote.host=localhost
> -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
> -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
> -DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
> -Djava.net.preferIPv4Stack=true"
> 
> JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
> -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
> -XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> -XX:+PrintTenuringDistribution -XX:+PrintGCCause
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
> -XX:GCLogFileSize=100M"
> 
> here is the 80 hours of GC analyize report:
> https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB
> 
> do we need more heap size or is there a BUG that we need to be aware?
> 
> here is the node configuration:
> 
> 
> http://www.springframework.org/schema/beans;
>xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd;>
>  class="org.apache.ignite.configuration.IgniteConfiguration">
> 
> 
>  value="/etc/apache-ignite/ignite-log4j2.xml"/>
> 
> 
> 
>  class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  value="/tmp/temp_ignite_libs"/>
> 
> 
> 
> file://freq=5000@localhost/usr/share/apache-ignite/libs/segmentify/
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  class="org.apache.ignite.configuration.DataStorageConfiguration">
> 
>  class="org.apache.ignite.configuration.DataRegionConfiguration">
> 
>  value="#{ 2L * 1024 * 1024 * 1024}"/>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> İbrahim Halil AltunSenior Software Engineer+90
> 536 3327510 • segmentify.com → UK • Germany •
> Turkey 
> 
> 

apache ignite 2.10.0 heap starvation

[Ibrahim Altun]

After upgrading from 2.7.1 version to 2.10.0 version ignite nodes facing
huge full GC operations after 24-36 hours after node start.

We try to increase heap size but no luck, here is the start configuration
for nodes;

JVM_OPTS="$JVM_OPTS -Xms12g -Xmx12g -server
-javaagent:/etc/prometheus/jmx_prometheus_javaagent-0.14.0.jar=8090:/etc/prometheus/jmx.yml
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=49165
-Dcom.sun.management.jmxremote.host=localhost
-XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=1g
-DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true
-DIGNITE_WAL_MMAP=true -DIGNITE_BPLUS_TREE_LOCK_RETRIES=10
-Djava.net.preferIPv4Stack=true"

JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC
-XX:+UseStringDeduplication -Xloggc:/var/log/apache-ignite/gc.log
-XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintTenuringDistribution -XX:+PrintGCCause
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10
-XX:GCLogFileSize=100M"

here is the 80 hours of GC analyize report:
https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjEvMDgvMzEvLS1nYy5sb2cuMC5jdXJyZW50LnppcC0tNS01MS0yOQ===WEB

do we need more heap size or is there a BUG that we need to be aware?

here is the node configuration:


http://www.springframework.org/schema/beans;
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
   xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd;>






























file://freq=5000@localhost/usr/share/apache-ignite/libs/segmentify/

























































-- 
İbrahim Halil AltunSenior Software Engineer+90
536 3327510 • segmentify.com → UK • Germany •
Turkey