[
https://issues.apache.org/jira/browse/GEODE-7763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039243#comment-17039243
]
Barrett Oglesby commented on GEODE-7763:
----------------------------------------
Some server thread dumps show ServerConnection threads waiting for one thread
to serialize the DeltaCapableGemFireSession for the ClientUpdateMessage.
Several ServerConnection threads waiting for the bucket lock in
BucketRegion.waitUntilLocked during a put operation:
{noformat}
"ServerConnection on port 49245 Thread 53" #111 daemon prio=5 os_prio=31
tid=0x00007fe691820800 nid=0x11903 in Object.wait() [0x000070000de16000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
org.apache.geode.internal.cache.BucketRegion.waitUntilLocked(BucketRegion.java:487)
- locked <0x000000079633bad8> (a
org.apache.geode.internal.cache.partitioned.LockObject)
at
org.apache.geode.internal.cache.BucketRegion.lockKeysAndPrimary(BucketRegion.java:753)
at
org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:521)
at
org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1212)
at
org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:2984)
at
org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2194)
at
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:162)
at
org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5560)
at
org.apache.geode.internal.cache.LocalRegion.basicBridgePut(LocalRegion.java:5211)
at
org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:411)
{noformat}
And 1 ServerConnection thread with the bucket lock serializing the
DeltaCapableGemFireSession for the ClientUpdateMessage (which is queued and
sent to the clients):
{noformat}
"ServerConnection on port 49245 Thread 103" #164 daemon prio=5 os_prio=31
tid=0x00007fe68d01f000 nid=0x16903 runnable [0x00007000113b4000]
java.lang.Thread.State: RUNNABLE
at
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$GemFireSessionAttributes.getAttribute(AbstractGemFireOperationsSessionRepository.java:1346)
- locked <0x00000007425d11b0> (a
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.lambda$null$2(DataSerializableSessionAttributesSerializer.java:98)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer$$Lambda$609/1945305806.doWrite(Unknown
Source)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.safeWrite(AbstractDataSerializableSessionSerializer.java:136)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.lambda$serialize$3(DataSerializableSessionAttributesSerializer.java:98)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer$$Lambda$604/243216168.accept(Unknown
Source)
at java.util.HashMap$KeySet.forEach(HashMap.java:932)
at
java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.serialize(DataSerializableSessionAttributesSerializer.java:96)
- locked <0x00000007425d11b0> (a
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionAttributesSerializer.serialize(DataSerializableSessionAttributesSerializer.java:50)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.lambda$toData$0(AbstractDataSerializableSessionSerializer.java:91)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer$$Lambda$596/1792537377.apply(Unknown
Source)
at java.util.Optional.map(Optional.java:215)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.toData(AbstractDataSerializableSessionSerializer.java:90)
at
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1564)
at
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1526)
at
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2105)
at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2839)
at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2808)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.serializeObject(AbstractDataSerializableSessionSerializer.java:102)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.serializeObject(AbstractDataSerializableSessionSerializer.java:98)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer.lambda$serialize$6(DataSerializableSessionSerializer.java:115)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer$$Lambda$602/594218789.doWrite(Unknown
Source)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.safeWrite(AbstractDataSerializableSessionSerializer.java:136)
at
org.springframework.session.data.gemfire.serialization.data.provider.DataSerializableSessionSerializer.serialize(DataSerializableSessionSerializer.java:115)
- locked <0x00000007425d11b0> (a
org.springframework.session.data.gemfire.AbstractGemFireOperationsSessionRepository$DeltaCapableGemFireSession)
at
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.serialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:468)
at
org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests$SpyingDataSerializableSessionSerializer.serialize(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:439)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.lambda$toData$0(AbstractDataSerializableSessionSerializer.java:91)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer$$Lambda$596/1792537377.apply(Unknown
Source)
at java.util.Optional.map(Optional.java:215)
at
org.springframework.session.data.gemfire.serialization.data.AbstractDataSerializableSessionSerializer.toData(AbstractDataSerializableSessionSerializer.java:90)
at
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1564)
at
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1526)
at
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2105)
at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2839)
at
org.apache.geode.internal.util.BlobHelper.serializeToBlob(BlobHelper.java:53)
at
org.apache.geode.internal.util.BlobHelper.serializeToBlob(BlobHelper.java:43)
at
org.apache.geode.internal.cache.tier.sockets.CacheServerHelper.serialize(CacheServerHelper.java:59)
at
org.apache.geode.internal.cache.tier.sockets.CacheServerHelper.serialize(CacheServerHelper.java:55)
at
org.apache.geode.internal.cache.tier.sockets.ClientUpdateMessageImpl.importNewObject(ClientUpdateMessageImpl.java:1608)
at
org.apache.geode.internal.cache.EntryEventImpl.exportNewValue(EntryEventImpl.java:1332)
at
org.apache.geode.internal.cache.tier.sockets.CacheClientNotifier.initializeMessage(CacheClientNotifier.java:1032)
at
org.apache.geode.internal.cache.tier.sockets.CacheClientNotifier.constructClientMessage(CacheClientNotifier.java:594)
at
org.apache.geode.internal.cache.tier.sockets.CacheClientNotifier.singletonNotifyClients(CacheClientNotifier.java:662)
at
org.apache.geode.internal.cache.tier.sockets.CacheClientNotifier.notifyClients(CacheClientNotifier.java:611)
at
org.apache.geode.internal.cache.LocalRegion.notifyBridgeClients(LocalRegion.java:6056)
at
org.apache.geode.internal.cache.LocalRegion.invokePutCallbacks(LocalRegion.java:5892)
at
org.apache.geode.internal.cache.BucketRegion.invokePutCallbacks(BucketRegion.java:1783)
at
org.apache.geode.internal.cache.EntryEventImpl.invokeCallbacks(EntryEventImpl.java:2437)
at
org.apache.geode.internal.cache.entries.AbstractRegionEntry.dispatchListenerEvents(AbstractRegionEntry.java:164)
at
org.apache.geode.internal.cache.LocalRegion.basicPutPart2(LocalRegion.java:5756)
at
org.apache.geode.internal.cache.BucketRegion.basicPutPart2(BucketRegion.java:703)
at
org.apache.geode.internal.cache.map.RegionMapPut.doBeforeCompletionActions(RegionMapPut.java:282)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutAndDeliverEvent(AbstractRegionMapPut.java:301)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut$$Lambda$479/1637000661.run(Unknown
Source)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWithIndexUpdatingInProgress(AbstractRegionMapPut.java:308)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutIfPreconditionsSatisified(AbstractRegionMapPut.java:296)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutOnSynchronizedRegionEntry(AbstractRegionMapPut.java:282)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutOnRegionEntryInMap(AbstractRegionMapPut.java:273)
- locked <0x00000007425d1168> (a
org.apache.geode.internal.cache.entries.VersionedStatsRegionEntryHeapObjectKey)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.addRegionEntryToMapAndDoPut(AbstractRegionMapPut.java:251)
- locked <0x00000007425d1168> (a
org.apache.geode.internal.cache.entries.VersionedStatsRegionEntryHeapObjectKey)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPutRetryingIfNeeded(AbstractRegionMapPut.java:216)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut$$Lambda$478/669501311.run(Unknown
Source)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doWithIndexInUpdateMode(AbstractRegionMapPut.java:193)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.doPut(AbstractRegionMapPut.java:180)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut$$Lambda$477/801996095.run(Unknown
Source)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.runWhileLockedForCacheModification(AbstractRegionMapPut.java:119)
at
org.apache.geode.internal.cache.map.RegionMapPut.runWhileLockedForCacheModification(RegionMapPut.java:161)
at
org.apache.geode.internal.cache.map.AbstractRegionMapPut.put(AbstractRegionMapPut.java:169)
at
org.apache.geode.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2044)
at
org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:529)
at
org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1212)
at
org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:2984)
at
org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2194)
at
org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:162)
at
org.apache.geode.internal.cache.LocalRegion.basicUpdate(LocalRegion.java:5560)
at
org.apache.geode.internal.cache.LocalRegion.basicBridgePut(LocalRegion.java:5211)
at
org.apache.geode.internal.cache.tier.sockets.command.Put65.cmdExecute(Put65.java:411)
{noformat}
In this case, this time spent serializing the DeltaCapableGemFireSession is for
no purpose. There are no clients interested in the event to since it originated
in the only one that exists.
I made a change to serialize the DeltaCapableGemFireSession only if there are
interested clients.
> Apache Geode 1.11 severely and negatively impacts performance and resource
> utilization
> --------------------------------------------------------------------------------------
>
> Key: GEODE-7763
> URL: https://issues.apache.org/jira/browse/GEODE-7763
> Project: Geode
> Issue Type: Bug
> Components: client/server
> Affects Versions: 1.10.0, 1.11.0
> Reporter: John Blum
> Priority: Critical
> Labels: performance
> Attachments: 1.11-client-stats.gfs, 1.11-server-stats.gfs,
> 1.11_thread_dumps.rtf, 1.9-client-stats.gfs, 1.9-server-stats.gfs, 1.9.log,
> apache-geode-1.10-client-server-interaction-output.txt,
> apache-geode-1.10-client-server-startup-output.txt,
> apache-geode-1.11-client-server-interaction-output.txt,
> apache-geode-1.11-client-server-startup-output.txt
>
>
> This problem was first observed in Apache Geode 1.11.0. The problem was not
> present in Apache Geode 1.9.2. This problem is an issue for Apache Geode
> 1.10 as well!
> After upgrading _Spring Session for Apache Geode_ (SSDG) 2.3 to _Spring Data
> for Apache Geode_ (SDG) Neumann/2.3, which is based on Apache Geode 1.11,
> this problem with SSDG's test suite started occurring.
> _Spring Session for Apache Geode_ (SSDG) 2.2, which is based on _Spring Data
> for Apache Geode_ (SDG) Moore/2.2, pulls in Apache Geode 1.9.2. This problem
> did not occur in SSDG 2.2. with Apache Geode 1.9.2.
> Out of curiosity, I wondered whether this problem affects (i.e. was actually
> introduced in) Apache Geode 1.10.0. So, I configured SSDG 2.3 to pull in SDG
> Moore/2.2 but run with Apache Geode 1.10. The problem occurred with Apache
> Geode 1.10 as well!
> The SSDG test class in question, affected by Geode's deficiencies, is the
> [MultiThreadedHighlyConcurrentClientServerSessionOperationsIntegrationTests|https://github.com/spring-projects/spring-session-data-geode/blob/2.2.2.RELEASE/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java].
> The test class was modeled after a customer UC, who were using Spring Session
> and Apache Geode/Pivotal GemFire as the HTTP Session state management
> provider, therefore it simulates their highly concurrent environment.
> The test class has 2 primary parameters: [Thread
> Count|https://github.com/spring-projects/spring-session-data-geode/blob/2.2.2.RELEASE/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java#L90]
> and the [Workload
> Size|https://github.com/spring-projects/spring-session-data-geode/blob/2.2.2.RELEASE/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java#L91].
> The "_Workload Size_" should not be confused with the "_Payload Size_" of the
> individual objects passed to the Geode data access operations (i.e. {{gets}},
> {{puts}}, {{removes}}). The "_Workload Size_" merely determines the number
> of {{get}}, {{put}} or {{remove}} operations performed on the (Session)
> Region over the duration of the test run. Certain operations are "favored"
> over others, therefore the number of {{gets}}, {{puts}} and {{removes}} is
> weighted.
> The "_Payload_" in this case is a (HTTP) {{Session}} object and the "size" is
> directly proportional to the number of Session attributes stored in the
> Session.
> As you can see from the [test class
> configuration|https://github.com/spring-projects/spring-session-data-geode/blob/2.2.2.RELEASE/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java#L90-L91]
> in *SSDG* {{2.2}}, the *Thread Count* was set to *180* and the *Workload
> Size* (or number of Region operations) was set to *10,000*.
> This had to be significantly adjusted in SSDG 2.3 using Apache Geode 1.11
> (and, as it turns out, Apache Geode 1.10 as well), as can be seen in the
> {{2.3.0.M1}} release bits source,
> [here|https://github.com/spring-projects/spring-session-data-geode/blob/2.3.0.M1/spring-session-data-geode/src/integration-test/java/org/springframework/session/data/gemfire/MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java#L94-L95].
> It turns out different combinations of the Thread Count (number of workers,
> or "concurrent Sessions") and Workload Size ultimately determine whether this
> test class passes or not.
> In other words, if I increase the Thread Count, then the Workload Size must
> decrease, otherwise the test fails! If I increase the Workload Size, then
> the Thread Count must decrease, otherwise again the test fails!
> I tried with different combinations of Thread Count and Workload Size until
> the test passed. More often than not 180 Threads with 3000 Regions
> operations worked, but was right on the cusp of failing, therefore, I settled
> on 180 Threads (which nearly matches the customers environment of 200
> concurrent client Sessions) and 2000 concurrent Region operations.
> The point of the test class is to assert the state of the Session is
> consistent at the end of the test run.
> However, before this test can even finish, the client, as in the
> {{ClientCache}} instance, starts failing with Exceptions, specifically:
> {code:java}
> java.lang.RuntimeException: Session Access Task Failed
> at
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.safeFutureGet(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:298)
> at
> java.util.stream.ReferencePipeline$4$1.accept(ReferencePipeline.java:210)
> at
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> at java.util.stream.IntPipeline.reduce(IntPipeline.java:456)
> at java.util.stream.IntPipeline.sum(IntPipeline.java:414)
> at
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.runSessionWorkload(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:313)
> at
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.concurrentSessionAccessIsCorrect(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:324)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:74)
> at
> org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:84)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
> at
> org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
> at
> org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
> at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
> Caused by: java.util.concurrent.ExecutionException:
> org.springframework.dao.DataAccessResourceFailureException: Pool unexpected
> socket timed out on client connection=Pooled Connection to localhost:60964:
> Connection[DESTROYED]). Server unreachable: could not connect after 1
> attempts; nested exception is
> org.apache.geode.cache.client.ServerConnectivityException: Pool unexpected
> socket timed out on client connection=Pooled Connection to localhost:60964:
> Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.safeFutureGet(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:295)
> ... 43 more
> Caused by: org.springframework.dao.DataAccessResourceFailureException: Pool
> unexpected socket timed out on client connection=Pooled Connection to
> localhost:60964: Connection[DESTROYED]). Server unreachable: could not
> connect after 1 attempts; nested exception is
> org.apache.geode.cache.client.ServerConnectivityException: Pool unexpected
> socket timed out on client connection=Pooled Connection to localhost:60964:
> Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts
> at
> org.springframework.data.gemfire.GemfireCacheUtils.convertGemfireAccessException(GemfireCacheUtils.java:235)
> at
> org.springframework.data.gemfire.GemfireAccessor.convertGemFireAccessException(GemfireAccessor.java:93)
> at
> org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:200)
> at
> org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.doSave(GemFireOperationsSessionRepository.java:226)
> at
> org.springframework.session.data.gemfire.GemFireOperationsSessionRepository.save(GemFireOperationsSessionRepository.java:186)
> at
> org.springframework.session.data.gemfire.AbstractGemFireIntegrationTests.save(AbstractGemFireIntegrationTests.java:409)
> at
> org.springframework.session.data.gemfire.MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.lambda$newAddSessionAttributeTask$2(MultiThreadedHighlyConcurrentClientServerHttpSessionAccessIntegrationTests.java:216)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.geode.cache.client.ServerConnectivityException: Pool
> unexpected socket timed out on client connection=Pooled Connection to
> localhost:60964: Connection[DESTROYED]). Server unreachable: could not
> connect after 1 attempts
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:659)
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.handleException(OpExecutorImpl.java:501)
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:153)
> at
> org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:108)
> at
> org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:772)
> at org.apache.geode.cache.client.internal.PutOp.execute(PutOp.java:89)
> at
> org.apache.geode.cache.client.internal.ServerRegionProxy.put(ServerRegionProxy.java:159)
> at
> org.apache.geode.internal.cache.LocalRegion.serverPut(LocalRegion.java:3035)
> at
> org.apache.geode.internal.cache.LocalRegion.cacheWriteBeforePut(LocalRegion.java:3152)
> at
> org.apache.geode.internal.cache.ProxyRegionMap.basicPut(ProxyRegionMap.java:238)
> at
> org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5580)
> at
> org.apache.geode.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:162)
> at
> org.apache.geode.internal.cache.LocalRegion.basicPut(LocalRegion.java:5036)
> at
> org.apache.geode.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1635)
> at
> org.apache.geode.internal.cache.LocalRegion.put(LocalRegion.java:1622)
> at
> org.apache.geode.internal.cache.AbstractRegion.put(AbstractRegion.java:442)
> at
> org.springframework.data.gemfire.GemfireTemplate.put(GemfireTemplate.java:197)
> ... 8 more
> {code}
> Attached to this issue are log output files from each of my runs using Apache
> Geode 1.10 and 1.11.
> The log files serve 2 purposes: 1) to show the version of Apache Geode used
> and 2) the errors occurs on the client and server during the run.
> Any lines in the log output prefixed with "{{[FORK]}}" originates from the
> cache server. The other lines come from the client. There is only a single
> client and server in this test case.
> It takes a bit of initial time during the run for the failures to start
> occurring, which is why this seems like a resource utilization problem.
> After first, I suspected issues with the client Pool configuration, or
> {{CacheServer}} configuration, adjusting timeouts and so on. I even
> suspected memory being an issue for the client and server processes, upping
> each to 2 GB+. However, it turns out none of the changes made a bit of
> difference. And the truth of the matter is, this (existing) configuration
> worked seamlessly until I upgraded to Apache Geode 1.10+ (specifically, 1.11).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
