Thanks Everyone! I could not regenerate issue by dropping memory to 30% or by
reducing tombstone-gc-timeout. It does work and that’s what I can understand
from below logs.
In fact it was used to work for this long running application and we had never
seen this in past.
[debug 2018/07/10 14:11:17.139 IST <P2P message reader for
host001(MyGeodeServer:18076)<ec><v10>:1026 shared ordered uid=7 port=54273>
tid=0x56] gcTombstones invoked for region
org.apache.geode.internal.cache.DistributedRegion[path='/Sample';scope=DISTRIBUTED_ACK';dataPolicy=REPLICATE;
concurrencyChecksEnabled] and version map { host001
(MyGeodeServer)<ec><v13>:1027=0, host001 (MyGeodeServer)<ec><v10>:1026=2448235}
Let me share my thoughts based on Charlie Black’s idea
1. Initially I thought, TombstoneSweeper might have been referenced
somewhere using reflection and due to private visibility, it could have caused
the issue. But that’s not the case.
2. So i thought what if class might have been unloaded by JVM? That’s
where Charlie’s point reminded me about classloader leaks in java.
3. Spring boot app was not restarted but internal geode distributed
system was reloaded/bootstrapped/redeployed after force disconnect attempt in
my case. And after that only issue was reported.
4. In fact, simply restarting the process solved the issue at that time
but I wanted to find root cause.
Note: I don’t have deep understanding in classloader leaks but that’s the
obvious one right now. Issue is there so problem must be there. I don’t have
anything user specific but it is trivial spring boot geode server,
Few more similar ones:
https://stackoverflow.com/questions/22918923/noclassdeffounderror-on-previously-operational-function?noredirect=1&lq=1
https://stackoverflow.com/questions/18706970/what-could-cause-a-sudden-classnotfoundexception-in-a-long-running-process
Thanks,
Dharam
From: John Blum [mailto:[email protected]]
Sent: Tuesday, July 10, 2018 1:47 AM
To: [email protected]
Subject: Re: Issue with TombstoneService
As a general rule of thumb, SDG fixes it's version of Apache Geode once it
reaches a Release Candidate (e.g. RC1) for any particular SD Release Train.
For instance, currently SDG Lovelace (i.e. 2.1) is on Apache Geode 1.6. If
there is not another release of Geode before SDG 2.1/Lovelace reaches RC1, then
SDG 2.1/Lovelace will be fixed on Geode 1.6. As a result, the version of Geode
on which SDG is based will not change until the master branch becomes SDG 2.2,
or SD Moore. Furthermore, master will not become SDG 2.2/Moore until SDG
2.1/Lovelace reaches GA. There can any number of Release Candidates before
final GA. For example, SDG 2.0/Kay had 3 Release Candidates (2.0.0.RC1,
2.0.0.RC2, 2.0.0.RC3). See the tags here [1]. I suspect there will be only 1
RC for SDG Lovelace, 2 at the most.
[1]
https://github.com/spring-projects/spring-data-geode<https://secureweb.jpmchase.net/readonly/https:/github.com/spring-projects/spring-data-geode>
On Mon, Jul 9, 2018 at 12:44 PM, John Blum
<[email protected]<mailto:[email protected]>> wrote:
Apache Geode 1.3 and higher is not going to work with SDG 2.0.x (SD Kay). SDG
2.0.x is fixed on Apache Geode 1.2.x [1] since there are significant API
changes in Geode after 1.2.
To use SDG with the latest version of Apache Geode (i.e. 1.6), you must use SD
Lovelace, or SDG 2.1.x, which is currently at M3 [2].
[1]
https://github.com/spring-projects/spring-data-geode/blob/2.0.8.RELEASE/pom.xml#L25<https://secureweb.jpmchase.net/readonly/https:/github.com/spring-projects/spring-data-geode/blob/2.0.8.RELEASE/pom.xml#L25>
[2]
https://github.com/spring-projects/spring-data-geode/blob/2.1.0.M3/pom.xml#L25<https://secureweb.jpmchase.net/readonly/https:/github.com/spring-projects/spring-data-geode/blob/2.1.0.M3/pom.xml#L25>
On Mon, Jul 9, 2018 at 11:56 AM, Udo Kohlmeyer
<[email protected]<mailto:[email protected]>> wrote:
Hi there Dharam,
I've been trying to reproduce your "NoClassDefFoundError" and I've been
unsuccessful.
Would you be willing to share the steps you've used to successfully reproduce
this error? In addition to that, to avoid the Geode community chasing its tail,
could you possibly test with a later version of Spring Data Geode (maybe
2.0.8.Release) using either the default geode-core 1.2.0 or updating to use the
1.6.0?
I'd be interested in understanding how this error can happen (other than a
class not being on the CP) but also to see if this is an issue in the current
release.
--Udo
On 7/8/18 23:21, Thacker, Dharam wrote:
Hi Anthony,
I have attached classpath portion of log in attachment.
It’s true that I am booting up geode server using spring boot and
spring-data-geode.
But I see all geode-* jars in classpath as expected.
In local, I could regenerate this issue as well.
Option1: I don’t have enough number of tombstones to trigger GC (100,000
default) but server memory dropped below 30% which should have triggered this.
Option2: Simulate force disconnect of member from distributed system and let
server member make an attempt to rejoin distributed system. Once it’s done,
load 100K dummy records and destroy them in sometime with REPLICATED mode, 2
peers.
Thanks,
Dharam
From: Charlie Black [[email protected]<mailto:[email protected]>]
Sent: Thursday, July 05, 2018 10:29 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Issue with TombstoneService
Are we redeploying the geode jar to the jvm? Or have some kind of interesting
classloader?
On Thu, Jul 5, 2018 at 8:36 AM Anthony Baker
<[email protected]<mailto:[email protected]>> wrote:
Thanks for the error report. Can you share the portion of the log that dumps
the classpath? It looks like you’re starting geode with springboot and perhaps
the classpath is incorrect. If geode-*.jar is on the classpath I don’t see how
you could get this error.
Anthony
On Jul 4, 2018, at 12:16 AM, Thacker, Dharam
<[email protected]<mailto:[email protected]>> wrote:
Hello Team,
Today we encountered below issue with Geode 1.1.1 (Should exists in Geode 1.6.0
as well as per codebase).
[severe 2018/07/03 11:03:53.914 EDT event-server-2 <Replicate/Partition Region
Garbage Collector> tid=0x4d] GemFire garbage collection service encountered an
unexpected exception
java.lang.NoClassDefFoundError:
org/apache/geode/internal/cache/TombstoneService$ReplicateTombstoneSweeper$1
at
org.apache.geode.internal.cache.TombstoneService$ReplicateTombstoneSweeper.expireBatch(TombstoneService.java:566)
at
org.apache.geode.internal.cache.TombstoneService$ReplicateTombstoneSweeper.checkExpiredTombstoneGC(TombstoneService.java:596)
at
org.apache.geode.internal.cache.TombstoneService$TombstoneSweeper.run(TombstoneService.java:882)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
org.apache.geode.internal.cache.TombstoneService$ReplicateTombstoneSweeper$1
at
java.net<https://secureweb.jpmchase.net/readonly/http:/java.net>.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at
org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
This makes sense to me as ReplicateTombstoneSweeper is a private static class
within TombstoneService.
Could you also verify at your end?
Another one as below which I think should not come with 1.6.0 as ThreadUtils
has been deprecated/removed from newer versions.
[severe 2018/07/03 11:03:53.916 EDT event-server-2 <Replicate/Partition Region
Garbage Collector> tid=0x4d] Uncaught exception in thread
Thread[Replicate/Partition Region Garbage Collector,5,Destroyed Entries
Processors]
java.lang.NoClassDefFoundError: org/apache/geode/internal/lang/ThreadUtils
at
org.apache.geode.internal.logging.log4j.AlertAppender.append(AlertAppender.java:141)
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:447)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:432)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:438)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:402)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at
org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217)
at
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
at
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1813)
at
org.apache.logging.log4j.spi.AbstractLogger.fatal(AbstractLogger.java:1005)
at
org.apache.geode.internal.cache.TombstoneService$TombstoneSweeper.run(TombstoneService.java:895)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
org.apache.geode.internal.lang.ThreadUtils
Thanks,
Dharam
This message is confidential and subject to terms at:
http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of
electronic messages. If you are not the intended recipient, please delete this
message and notify the sender immediately. Any unauthorized use is strictly
prohibited.
--
[email protected]<mailto:[email protected]> | +1.858.480.9722
Principal Realtime Data Engineer
This message is confidential and subject to terms at:
http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer>
including on confidentiality, legal privilege, viruses and monitoring of
electronic messages. If you are not the intended recipient, please delete this
message and notify the sender immediately. Any unauthorized use is strictly
prohibited.
--
-John
john.blum10101 (skype)
--
-John
john.blum10101 (skype)
This message is confidential and subject to terms at:
http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
privilege, viruses and monitoring of electronic messages. If you are not the
intended recipient, please delete this message and notify the sender
immediately. Any unauthorized use is strictly prohibited.
