Ok, I could regenerate the same today. Here are the server and client logs.
Could there be any chances where below classes might get unloaded?
Server2 logs:
[severe 2017/09/18 15:01:45.550 EDT Server2 <ServerConnection on port 40404
Thread 5> tid=0x536] Server connection from
[identity(HOST1XX(4062:loner):42960:efff897f,connection=2; port=57550] :
Unexpected Error on server
java.lang.NoClassDefFoundError:
org/apache/geode/internal/cache/tier/sockets/OldClientSupportService
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.getClientException(BaseCommand.java:574)
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.writeException(BaseCommand.java:584)
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.writeException(BaseCommand.java:567)
at
org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:424)
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:141)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1138)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
at java.lang.Thread.run(Thread.java:745)
[severe 2017/09/18 15:01:45.552 EDT Server2 <ServerConnection on port 40404
Thread 5> tid=0x536] Uncaught exception in thread Thread[ServerConnection on
port 40404 Thread 5,5,ServerConnection on port 40404]
java.lang.NoClassDefFoundError: org/apache/geode/internal/lang/ThreadUtils
at
org.apache.geode.internal.logging.log4j.AlertAppender.append(AlertAppender.java:141)
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:447)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:432)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:438)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:402)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at
org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217)
at
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
at
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1813)
at
org.apache.logging.log4j.spi.AbstractLogger.fatal(AbstractLogger.java:1005)
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.handleThrowable(BaseCommand.java:469)
at
org.apache.geode.internal.cache.tier.sockets.BaseCommand.execute(BaseCommand.java:162)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doNormalMsg(ServerConnection.java:783)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.doOneMessage(ServerConnection.java:914)
at
org.apache.geode.internal.cache.tier.sockets.ServerConnection.run(ServerConnection.java:1138)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.geode.internal.cache.tier.sockets.AcceptorImpl$1$1.run(AcceptorImpl.java:519)
at java.lang.Thread.run(Thread.java:745)
Server1 logs:
[severe 2017/09/18 14:00:57.044 EDT Server1 <Asynchronous disk writer for
region event_disk_store> tid=0x81] Fatal error from asynchronous flusher thread
java.lang.NoClassDefFoundError: org/apache/geode/internal/cache/Oplog$1
at org.apache.geode.internal.cache.Oplog.switchOpLog(Oplog.java:3810)
at org.apache.geode.internal.cache.Oplog.basicModify(Oplog.java:4617)
at org.apache.geode.internal.cache.Oplog.modify(Oplog.java:4442)
at
org.apache.geode.internal.cache.PersistentOplogSet.modify(PersistentOplogSet.java:190)
at
org.apache.geode.internal.cache.DiskStoreImpl.put(DiskStoreImpl.java:699)
at org.apache.geode.internal.cache.DiskRegion.put(DiskRegion.java:350)
at
org.apache.geode.internal.cache.DiskEntry$Helper.writeBytesToDisk(DiskEntry.java:825)
at
org.apache.geode.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:817)
at
org.apache.geode.internal.cache.DiskEntry$Helper.writeToDisk(DiskEntry.java:805)
at
org.apache.geode.internal.cache.DiskEntry$Helper.writeEntryToDisk(DiskEntry.java:1592)
at
org.apache.geode.internal.cache.DiskEntry$Helper.doAsyncFlush(DiskEntry.java:1546)
at
org.apache.geode.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1700)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
org.apache.geode.internal.cache.Oplog$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at
org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
[severe 2017/09/18 14:00:57.049 EDT Server1 <Asynchronous disk writer for
region event_disk_store> tid=0x81] Uncaught exception in thread
Thread[Asynchronous disk writer for region event_disk_store,5,Disk Writers]
java.lang.NoClassDefFoundError: org/apache/geode/internal/lang/ThreadUtils
at
org.apache.geode.internal.logging.log4j.AlertAppender.append(AlertAppender.java:141)
at
org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at
org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at
org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:447)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:432)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at
org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:438)
at
org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at
org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:402)
at
org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at
org.apache.logging.log4j.spi.ExtendedLoggerWrapper.logMessage(ExtendedLoggerWrapper.java:217)
at
org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
at
org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1813)
at
org.apache.logging.log4j.spi.AbstractLogger.fatal(AbstractLogger.java:1005)
at
org.apache.geode.internal.cache.DiskStoreImpl$FlusherThread.run(DiskStoreImpl.java:1742)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
org.apache.geode.internal.lang.ThreadUtils
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at
org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 20 more
[error 2017/09/18 14:01:05.604 EDT Server1 <P2P message reader for
HOST2XX(Server2:10459)<ec><v3>:1025 shared ordered uid=6 port=55866> tid=0x7e]
A DiskAccessException has occurred while writing to the disk for region /Event.
The cache will be closed.
org.apache.geode.cache.DiskAccessException: For DiskStore: event_disk_store:
Could not schedule asynchronous write because the flusher thread had been
terminated.
at
org.apache.geode.internal.cache.DiskStoreImpl.checkForFlusherThreadTermination(DiskStoreImpl.java:1207)
at
org.apache.geode.internal.cache.DiskStoreImpl.addAsyncItem(DiskStoreImpl.java:1254)
at
org.apache.geode.internal.cache.DiskStoreImpl.scheduleAsyncWrite(DiskStoreImpl.java:1318)
at
org.apache.geode.internal.cache.DiskRegion.scheduleAsyncWrite(DiskRegion.java:449)
at
org.apache.geode.internal.cache.DiskEntry$Helper.scheduleAsyncWrite(DiskEntry.java:1516)
at
org.apache.geode.internal.cache.DiskEntry$Helper.update(DiskEntry.java:867)
at
org.apache.geode.internal.cache.AbstractDiskRegionEntry.setValue(AbstractDiskRegionEntry.java:39)
at
org.apache.geode.internal.cache.AbstractRegionEntry.setValueWithTombstoneCheck(AbstractRegionEntry.java:269)
at
org.apache.geode.internal.cache.EntryEventImpl.setNewValueInRegion(EntryEventImpl.java:1630)
at
org.apache.geode.internal.cache.EntryEventImpl.putExistingEntry(EntryEventImpl.java:1504)
at
org.apache.geode.internal.cache.AbstractRegionMap.updateEntry(AbstractRegionMap.java:2954)
at
org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2777)
at
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5752)
at
org.apache.geode.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:337)
at
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:151)
at
org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5732)
at
org.apache.geode.internal.cache.AbstractUpdateOperation.doPutOrCreate(AbstractUpdateOperation.java:184)
at
org.apache.geode.internal.cache.AbstractUpdateOperation$AbstractUpdateMessage.basicOperateOnRegion(AbstractUpdateOperation.java:291)
at
org.apache.geode.internal.cache.DistributedPutAllOperation$PutAllMessage.doEntryPut(DistributedPutAllOperation.java:1100)
at
org.apache.geode.internal.cache.DistributedPutAllOperation$PutAllMessage$1.run(DistributedPutAllOperation.java:1185)
at
org.apache.geode.internal.cache.EventTracker.syncBulkOp(EventTracker.java:558)
at
org.apache.geode.internal.cache.LocalRegion.syncBulkOp(LocalRegion.java:6243)
at
org.apache.geode.internal.cache.DistributedPutAllOperation$PutAllMessage.basicOperateOnRegion(DistributedPutAllOperation.java:1176)
at
org.apache.geode.internal.cache.AbstractUpdateOperation$AbstractUpdateMessage.operateOnRegion(AbstractUpdateOperation.java:262)
at
org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.basicProcess(DistributedCacheOperation.java:1175)
at
org.apache.geode.internal.cache.DistributedCacheOperation$CacheOperationMessage.process(DistributedCacheOperation.java:1076)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
at
org.apache.geode.distributed.internal.DistributionMessage.schedule(DistributionMessage.java:434)
at
org.apache.geode.distributed.internal.DistributionManager.scheduleIncomingMessage(DistributionManager.java:3504)
at
org.apache.geode.distributed.internal.DistributionManager.handleIncomingDMsg(DistributionManager.java:3137)
at
org.apache.geode.distributed.internal.DistributionManager$MyListener.messageReceived(DistributionManager.java:4311)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.dispatchMessage(GMSMembershipManager.java:1115)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.handleOrDeferMessage(GMSMembershipManager.java:1039)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager$MyDCReceiver.messageReceived(GMSMembershipManager.java:407)
at
org.apache.geode.distributed.internal.direct.DirectChannel.receive(DirectChannel.java:714)
at
org.apache.geode.internal.tcp.TCPConduit.messageReceived(TCPConduit.java:877)
at
org.apache.geode.internal.tcp.Connection.dispatchMessage(Connection.java:4033)
at
org.apache.geode.internal.tcp.Connection.processNIOBuffer(Connection.java:3750)
at
org.apache.geode.internal.tcp.Connection.runNioReader(Connection.java:1865)
at org.apache.geode.internal.tcp.Connection.run(Connection.java:1726)
at java.lang.Thread.run(Thread.java:745)
Client logs:
[fine 2017/09/19 05:38:30.943 EDT <poolTimer-DEFAULT-2724> tid=0x9f08b]
Sending query to locator HOST1XX/169.X.Y.Z:10334: LocatorListRequest{group=null}
[fine 2017/09/19 05:38:30.944 EDT <poolTimer-DEFAULT-2724> tid=0x9f08b]
TcpClient sending LocatorListRequest{group=null} to HOST1XX/169.X.Y.Z:10334
[fine 2017/09/19 05:38:30.945 EDT <poolTimer-DEFAULT-2724> tid=0x9f08b]
received response: LocatorListResponse{locators=[HOST2XX:10334,
HOST1XX:10334],isBalanced=true}
[fine 2017/09/19 05:38:30.946 EDT <poolTimer-DEFAULT-2724> tid=0x9f08b]
Received query response from locator HOST1XX/169.X.Y.Z:10334:
LocatorListResponse{locators=[HOST2XX:10334, HOST1XX:10334],isBalanced=true}
[fine 2017/09/19 05:38:31.245 EDT <poolTimer-poc-geode-pool-3363> tid=0x9f089]
Sending query to locator HOST1XX/169.X.Y.Z:10334: LocatorListRequest{group=null}
[fine 2017/09/19 05:38:31.246 EDT <poolTimer-poc-geode-pool-3363> tid=0x9f089]
TcpClient sending LocatorListRequest{group=null} to HOST1XX/169.X.Y.Z:10334
[fine 2017/09/19 05:38:31.247 EDT <poolTimer-poc-geode-pool-3363> tid=0x9f089]
received response: LocatorListResponse{locators=[HOST2XX:10334,
HOST1XX:10334],isBalanced=true}
[fine 2017/09/19 05:38:31.247 EDT <poolTimer-poc-geode-pool-3363> tid=0x9f089]
Received query response from locator HOST1XX/169.X.Y.Z:10334:
LocatorListResponse{locators=[HOST2XX:10334, HOST1XX:10334],isBalanced=true}
[fine 2017/09/19 05:38:38.222 EDT <poolTimer-DEFAULT-2724> tid=0x9f08b]
OpExecutor.handleException on Connection to HOST1XX:40404
java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at
org.apache.geode.internal.cache.tier.sockets.Message.flushBuffer(Message.java:645)
at
org.apache.geode.internal.cache.tier.sockets.Message.sendBytes(Message.java:623)
at
org.apache.geode.internal.cache.tier.sockets.Message.send(Message.java:1096)
at
org.apache.geode.cache.client.internal.PingOp$PingOpImpl.sendMessage(PingOp.java:68)
at
org.apache.geode.cache.client.internal.AbstractOp.attemptSend(AbstractOp.java:94)
at
org.apache.geode.cache.client.internal.AbstractOp.attempt(AbstractOp.java:375)
at
org.apache.geode.cache.client.internal.ConnectionImpl.execute(ConnectionImpl.java:266)
at
org.apache.geode.cache.client.internal.pooling.PooledConnection.execute(PooledConnection.java:332)
at
org.apache.geode.cache.client.internal.OpExecutorImpl.executeWithPossibleReAuthentication(OpExecutorImpl.java:900)
at
org.apache.geode.cache.client.internal.OpExecutorImpl.executeOnServer(OpExecutorImpl.java:379)
at
org.apache.geode.cache.client.internal.OpExecutorImpl.executeOn(OpExecutorImpl.java:341)
at
org.apache.geode.cache.client.internal.PoolImpl.executeOn(PoolImpl.java:777)
at org.apache.geode.cache.client.internal.PingOp.execute(PingOp.java:36)
at
org.apache.geode.cache.client.internal.LiveServerPinger$PingTask.run2(LiveServerPinger.java:90)
at
org.apache.geode.cache.client.internal.PoolImpl$PoolTask.run(PoolImpl.java:1291)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
org.apache.geode.internal.ScheduledThreadPoolExecutorWithKeepAlive$DelegatingScheduledFuture.run(ScheduledThreadPoolExecutorWithKeepAlive.java:257)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[error 2017/09/19 05:38:38.222 EDT <poolTimer-DEFAULT-2724> tid=0x9f08b]
Unexpected error in pool task
<org.apache.geode.cache.client.internal.LiveServerPinger$PingTask@8559db4>
java.lang.NoClassDefFoundError:
org/apache/geode/internal/cache/tier/BatchException
at
org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:669)
at
org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:623)
at
org.apache.geode.cache.client.internal.OpExecutorImpl.executeOnServer(OpExecutorImpl.java:385)
at
org.apache.geode.cache.client.internal.OpExecutorImpl.executeOn(OpExecutorImpl.java:341)
at
org.apache.geode.cache.client.internal.PoolImpl.executeOn(PoolImpl.java:777)
at org.apache.geode.cache.client.internal.PingOp.execute(PingOp.java:36)
at
org.apache.geode.cache.client.internal.LiveServerPinger$PingTask.run2(LiveServerPinger.java:90)
at
org.apache.geode.cache.client.internal.PoolImpl$PoolTask.run(PoolImpl.java:1291)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
org.apache.geode.internal.ScheduledThreadPoolExecutorWithKeepAlive$DelegatingScheduledFuture.run(ScheduledThreadPoolExecutorWithKeepAlive.java:257)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Thanks,
Dharam
From: Xiawei Zhang [mailto:[email protected]]
Sent: Thursday, September 07, 2017 11:37 PM
To: [email protected]
Subject: Re: NoClassDefFoundError on client shutdown
The stacktrace says it is an exception coming back from server side. Client is
just logging it out here.
at
org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:669)
at
org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:623)
at
org.apache.geode.cache.client.internal.OpExecutorImpl.executeOnServer(OpExecutorImpl.java:385)
On Thu, 7 Sep 2017 at 11:52 PM, Dharam Thacker
<[email protected]<mailto:[email protected]>> wrote:
Yeah I have enabled debug logs in client and server both today. I am trying to
reproduce it.
For me it's coming at geode client application though.
Thanks,
Dharam
On Sep 7, 2017 9:18 PM, "Xiawei Zhang"
<[email protected]<mailto:[email protected]>> wrote:
By any chance you have debug log enabled on the server side? I remember I
encountered a similar problem before, and it turns out the real exception was
logged at debug level on the server side.
-Xiawei
On Thu, 7 Sep 2017 at 12:30 AM, Thacker, Dharam
<[email protected]<mailto:[email protected]>> wrote:
Hello Anthony,
Thank you for guidance on that!
I am sending you output of '-verbose:class' along with JVM GC args I have used
for spring boot geode client app.
-Xms4g -Xmx4g -XX:+UseG1GC -XX:+ParallelRefProcEnabled
-XX:+UseStringDeduplication -XX:+PrintFlagsFinal -XX:+PrintReferenceGC
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:+PrintAdaptiveSizePolicy -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/apps/logs/heap_dump
-Xloggc:/local/apps/ravenapps/logs/gc/event-publisher/jvm-gc-%t.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -client -XX:+AlwaysPreTouch
-XX:+UnlockDiagnosticVMOptions -XX:+G1SummarizeConcMark -Xss10M
-XX:InitiatingHeapOccupancyPercent=40 -XX:ParallelGCThreads=20
-XX:ConcGCThreads=5
Thanks,
Dharam
-----Original Message-----
From: Anthony Baker [mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, September 06, 2017 8:44 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: NoClassDefFoundError on client shutdown
You might trying adding '-verbose:class’ to your client startup options to help
troubleshoot this issue.
Anthony
> On Sep 6, 2017, at 7:41 AM, Thacker, Dharam
> <[email protected]<mailto:[email protected]>> wrote:
>
> Hi Anthony,
>
> Versions:
>
> Apache Geode: 1.1.1
> Spring Boot: 1.5.2.RELEASE
> Spring Data Geode: 1.0.0.INCUBATING-RELEASE [Within Maven POM Apache
> Geode jars have been overridden using version 1.1.1 – Attached
> pom.xml]
>
>
> [Using config level logging in geode client logs]
>
> Class path: /dev/lib/event-publisher.jar [It’s a fat UBER jar using
> spring boot maven plugin] Library Path:
> /usr/java/packages/lib/amd64
> /usr/lib64
> /lib64
> /lib
> /usr/lib
>
>
> Thanks & Regards,
> Dharam
> From: Anthony Baker [mailto:[email protected]<mailto:[email protected]>]
> Sent: Wednesday, September 06, 2017 7:51 PM
> To: [email protected]<mailto:[email protected]>
> Subject: Re: NoClassDefFoundError on client shutdown
>
> What is your classpath? What versions of Spring Boot and Geode are you using
> on the client and server?
>
> Anthony
>
> On Sep 5, 2017, at 10:35 PM, Thacker, Dharam
> <[email protected]<mailto:[email protected]>> wrote:
>
> Hi Team,
>
> I am observing few “NoClassDefFoundError” errors on unexpected termination of
> client under medium load.
>
> Could you help me to verify? Looks like a bug where it’s not able to find
> expected classes in available runtime.
>
> [error 2017/09/05 06:29:57.677 EDT <poolTimer-DEFAULT-17374>
> tid=0xa9830] Unexpected error in pool task
> <org.apache.geode.cache.client.internal.LiveServerPinger$PingTask@5ba4
<mailto:org.apache.geode.cache.client.internal.LiveServerPinger$PingTask@5ba4%0b>>
4553>
> java.lang.NoClassDefFoundError:
> org/apache/geode/internal/cache/tier/BatchException
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:669)
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:623)
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.executeOnServer(OpExecutorImpl.java:385)
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.executeOn(OpExecutorImpl.java:341)
> at
> org.apache.geode.cache.client.internal.PoolImpl.executeOn(PoolImpl.java:777)
> at
> org.apache.geode.cache.client.internal.PingOp.execute(PingOp.java:36)
> at
> org.apache.geode.cache.client.internal.LiveServerPinger$PingTask.run2(LiveServerPinger.java:90)
> at
> org.apache.geode.cache.client.internal.PoolImpl$PoolTask.run(PoolImpl.java:1291)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> org.apache.geode.internal.ScheduledThreadPoolExecutorWithKeepAlive$DelegatingScheduledFuture.run(ScheduledThreadPoolExecutorWithKeepAlive.java:257)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> [info 2017/09/05 06:30:06.327 EDT <Distributed system shutdown hook>
> tid=0x21] VM is exiting - shutting down distributed system
>
> [info 2017/09/05 06:30:06.330 EDT <Distributed system shutdown hook>
> tid=0x21] GemFireCache[id = 260620222; isClosing = true; isShutDownAll =
> false; created = Wed Aug 16 07:22:16 EDT 2017; server = false; copyOnRead =
> false; lockLease = 120; lockTimeout = 60]: Now closing.
>
> [warning 2017/09/05 06:30:06.333 EDT <Distributed system shutdown
> hook> tid=0x21] Exception trying to close cache
> java.lang.NoClassDefFoundError:
> org/apache/geode/internal/cache/partitioned/RegionAdvisor
> at
> org.apache.geode.internal.cache.GemFireCacheImpl.close(GemFireCacheImpl.java:2177)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1314)
> at
> org.apache.geode.distributed.internal.InternalDistributedSystem$6.run(InternalDistributedSystem.java:2283)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.geode.internal.cache.partitioned.RegionAdvisor
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at
> org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 4 more
>
> Thanks,
> Dharam
> This message is confidential and subject to terms at:
> http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not the
> intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> This message is confidential and subject to terms at:
> http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
> privilege, viruses and monitoring of electronic messages. If you are not the
> intended recipient, please delete this message and notify the sender
> immediately. Any unauthorized use is strictly prohibited.
>
> <pom.xml>
This message is confidential and subject to terms at:
http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
privilege, viruses and monitoring of electronic messages. If you are not the
intended recipient, please delete this message and notify the sender
immediately. Any unauthorized use is strictly prohibited.
This message is confidential and subject to terms at:
http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
privilege, viruses and monitoring of electronic messages. If you are not the
intended recipient, please delete this message and notify the sender
immediately. Any unauthorized use is strictly prohibited.
