Hi!
Ok, it does not sound like very extreme, do you really need 10GB heap ?
as you said it takes two weeks before the problems start, it does sound
like you have something growing there pretty slow and it does have to do
some intense GC pass finally to clean all that up, using a smaller heap
might help there if you don't need the big heap (less garbage to cleanup
when it get full, less GC time, so it will GC more often but will not
lock up the JVM for such long time).
~2 seconds GC pause is not a huge disaster so I think you should be able
to solve it with some tweaking and/or change collector.
The G1 or ZGC collectors work pretty well these days but I don't know
what JVM you use.
Linux/Windows ? setting a large heap (in relation to ram) on a Linux
machine with default swappiness can cause problems, it often start
paging if you just fill up half the memory or so.
I am no export on GC so I hope someone with better knowledge can give
some ideas, but I think the best is to log the GC for a few days and see
what is going on with the heap, each application is unique so it is very
difficult to give any generic " do this" and all will be good tips, also
make sure it actually is GC pauses that is causing the problem so that
it isn't something else.
regards
Mikael
On 2022-01-12 05:52, satyajit.man...@barclays.com wrote:
Hi Mikael,
We are using below settings and we have default off heap
memory enabled. Heap size is 10 GB per node and we are
running 4 nodes as part of cluster. Data size is ~25MB but
it continuously updates and inserts records in every 5 mins
and 15 mins. It’s a low latency application. We haven’t enabled
persistence.
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=55555
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Xms10g -Xmx10g
-XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC
-XX:+DisableExplicitGC -Djava.net.preferIPv4Stack=true
Regards
Satyajit
*From:*Mikael <mikael.arons...@gmail.com>
*Sent:* Tuesday, January 11, 2022 11:29 PM
*To:* user@ignite.apache.org
*Subject:* Re: long JVM pauses
CAUTION: This email originated from outside our organisation -
mikael.arons...@gmail.com Do not click on links, open attachments, or
respond unless you recognize the sender and can validate the content
is safe.
Hi!
There are no generic settings, each case is unique, it will require
some tuning to get it right, how much java heap do you use ? and what
garbage collector are you using ? how much data do you have ?
persistence enabled ? how do you use the caches, is data deleted often
or do you keep it around for longer time ?
Mikael
On 2022-01-11 18:23, satyajit.man...@barclays.com wrote:
Hi Team,
We do see long JVM pauses and after that nodes in our
cluster stops. This happens in every two weeks usually.
What are the possible solutions to avoid long JVM pause.
Can someone advise generic settings which is recommended in
such cases?
[23:20:37,824][WARNING][jvm-pause-detector-worker][IgniteKernal]
Possible too long JVM pause: 2262 milliseconds.
Regards
Satyajit
Restricted - Internal
This message is for information purposes only. It is not a
recommendation, advice, offer or solicitation to buy or sell a
product or service, nor an official confirmation of any
transaction. It is directed at persons who are professionals and
is intended for the recipient(s) only. It is not directed at
retail customers. This message is subject to the terms at:
https://www.cib.barclays/disclosures/web-and-email-disclaimer.html
<https://clicktime.symantec.com/36Drmrd2f6GnrX5HMTwSVT36H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fweb-and-email-disclaimer.html>.
For important disclosures, please see:
https://www.cib.barclays/disclosures/sales-and-trading-disclaimer.html
<https://clicktime.symantec.com/3B4s9V77DYoyX9dN8ua84my6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fsales-and-trading-disclaimer.html>
regarding marketing commentary from Barclays Sales and/or Trading
desks, who are active market participants;
https://www.cib.barclays/disclosures/barclays-global-markets-disclosures.html
<https://clicktime.symantec.com/3JegRP7ErAjYtcC7TGJFKtS6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fbarclays-global-markets-disclosures.html>
regarding our standard terms for Barclays Corporate and Investment
Bank where we trade with you in principal-to-principal wholesale
markets transactions; and in respect to Barclays Research,
including disclosures relating to specific issuers, see:
http://publicresearch.barclays.com
<https://clicktime.symantec.com/3HZtwy1wDet5tLGDuUowvHv6H4?u=http%3A%2F%2Fpublicresearch.barclays.com>.
__________________________________________________________________________________
If you are incorporated or operating in Australia, read these
important disclosures:
https://www.cib.barclays/disclosures/important-disclosures-asia-pacific.html
<https://clicktime.symantec.com/35pi9P5XPXZcYsC5PhKqM1K6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fimportant-disclosures-asia-pacific.html>.
__________________________________________________________________________________
For more details about how we use personal information, see our
privacy notice:
https://www.cib.barclays/disclosures/personal-information-use.html
<https://clicktime.symantec.com/3H8MJpzhKqWY15E4QwLkCNv6H4?u=https%3A%2F%2Fwww.cib.barclays%2Fdisclosures%2Fpersonal-information-use.html>.
__________________________________________________________________________________
This message is for information purposes only. It is not a
recommendation, advice, offer or solicitation to buy or sell a product
or service, nor an official confirmation of any transaction. It is
directed at persons who are professionals and is intended for the
recipient(s) only. It is not directed at retail customers. This
message is subject to the terms at:
https://www.cib.barclays/disclosures/web-and-email-disclaimer.html.
For important disclosures, please see:
https://www.cib.barclays/disclosures/sales-and-trading-disclaimer.html
regarding marketing commentary from Barclays Sales and/or Trading
desks, who are active market participants;
https://www.cib.barclays/disclosures/barclays-global-markets-disclosures.html
regarding our standard terms for Barclays Corporate and Investment
Bank where we trade with you in principal-to-principal wholesale
markets transactions; and in respect to Barclays Research, including
disclosures relating to specific issuers, see:
http://publicresearch.barclays.com.
__________________________________________________________________________________
If you are incorporated or operating in Australia, read these
important disclosures:
https://www.cib.barclays/disclosures/important-disclosures-asia-pacific.html.
__________________________________________________________________________________
For more details about how we use personal information, see our
privacy notice:
https://www.cib.barclays/disclosures/personal-information-use.html.
__________________________________________________________________________________
