[jira] [Updated] (GEODE-9311) It is possible that JedisCluster client may still retry if the Geode Redis server it is connected to shuts down

2022-02-08 Thread Anthony Baker (Jira)


 [ 
https://issues.apache.org/jira/browse/GEODE-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anthony Baker updated GEODE-9311:
-
Labels: unreleased  (was: )

> It is possible that JedisCluster client may still retry if the Geode Redis 
> server it is connected to shuts down
> ---
>
> Key: GEODE-9311
> URL: https://issues.apache.org/jira/browse/GEODE-9311
> Project: Geode
>  Issue Type: Bug
>  Components: redis
>Reporter: Eric Shu
>Priority: Major
>  Labels: unreleased
> Fix For: 1.15.0
>
>
> Even after the issue (GEODE-9310) is addressed, the retry issue may still 
> occur if the JedisCluster client is connected to the node is being shut down.
> Here is a test run result:
> vm2 gets the command:
> [vm2] [warn 2021/05/25 16:01:41.479 PDT server-2  
> tid=0x64] Executing Redis command: ZREM key member1-212 member1-211 
> member1-214 member1-213 member1-210 member1-209 member1-208 member1-205 
> member1-204 
> *** command is executed on the primary ***
> [vm2] [warn 2021/05/25 16:01:41.503 PDT server-2  Processor2> tid=0x51] bucket region is 
> BucketRegion[path='/__PR/_B__REDIS__DATA_123;serial=200;primary=true] key key
> [vm2] java.lang.Exception
> [vm2] at 
> org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:533)
> [vm2] at 
> org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
> [vm2] at 
> org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1213)
> [vm2] at 
> org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:3024)
> [vm2] at 
> org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2236)
> [vm2] at 
> org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
> And distributed to replica (vm1):
> vm1] [warn 2021/05/25 16:01:41.518 PDT server-1  192.168.0.14(server-2:83136):41002 shared ordered sender uid=7 local 
> port=58643 remote port=53459> tid=0x59] membersRemoveAll invoked
> [vm1] java.lang.Exception
> [vm1] at 
> org.apache.geode.redis.internal.data.RedisSortedSet.membersRemoveAll(RedisSortedSet.java:175)
> [vm1] at 
> org.apache.geode.redis.internal.data.RedisSortedSet.applyDelta(RedisSortedSet.java:89)
> [vm1] at 
> org.apache.geode.redis.internal.data.AbstractRedisData.fromDelta(AbstractRedisData.java:193)
> [vm1] at 
> org.apache.geode.internal.cache.EntryEventImpl.processDeltaBytes(EntryEventImpl.java:1841)
> [vm1] at 
> org.apache.geode.internal.cache.EntryEventImpl.setNewValueInRegion(EntryEventImpl.java:1696)
> [vm1] at 
> org.apache.geode.internal.cache.EntryEventImpl.putExistingEntry(EntryEventImpl.java:1643)
> [vm1] at 
> org.apache.geode.internal.cache.map.RegionMapPut.updateEntry(RegionMapPut.java:485)
> [vm1] at 
> org.apache.geode.internal.cache.map.RegionMapPut.createOrUpdateEntry(RegionMapPut.java:256)
> vm2 is bounced:
> [vm2] [info 2021/05/25 16:01:41.526 PDT server-2  Connection(1)-192.168.0.14> tid=0x14] Got result: 83136
> [vm2]  from org.apache.geode.test.dunit.VM$$Lambda$370/1141741369.call with 0 
> args on object: 
> org.apache.geode.test.dunit.VM$$Lambda$370/1141741369@68ecd55a (took 0 ms)
> [info 2021/05/25 16:01:41.527 PDT   tid=0x36] Bouncing 2 old 
> pid is 83136 and version is 10240.0.0
> The JedisCluster client did not get response back (possibly detects 
> connection is gone), and it does retry again, as the test failed with 
> following:
> java.util.concurrent.ExecutionException: 
> redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No more 
> cluster attempts left.
>   at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>   at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>   at 
> org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed(ZRemDUnitTest.java:177)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> 

[jira] [Updated] (GEODE-9311) It is possible that JedisCluster client may still retry if the Geode Redis server it is connected to shuts down

2021-05-25 Thread Eric Shu (Jira)


 [ 
https://issues.apache.org/jira/browse/GEODE-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Shu updated GEODE-9311:

Description: 
Even after the issue (GEODE-9310) is addressed, the retry issue may still occur 
if the JedisCluster client is connected to the node is being shut down.

Here is a test run result:
vm2 gets the command:
[vm2] [warn 2021/05/25 16:01:41.479 PDT server-2  
tid=0x64] Executing Redis command: ZREM key member1-212 member1-211 member1-214 
member1-213 member1-210 member1-209 member1-208 member1-205 member1-204 

*** command is executed on the primary ***
[vm2] [warn 2021/05/25 16:01:41.503 PDT server-2  tid=0x51] bucket region is 
BucketRegion[path='/__PR/_B__REDIS__DATA_123;serial=200;primary=true] key key
[vm2] java.lang.Exception
[vm2]   at 
org.apache.geode.internal.cache.BucketRegion.virtualPut(BucketRegion.java:533)
[vm2]   at 
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)
[vm2]   at 
org.apache.geode.internal.cache.PartitionedRegionDataStore.putLocally(PartitionedRegionDataStore.java:1213)
[vm2]   at 
org.apache.geode.internal.cache.PartitionedRegion.putInBucket(PartitionedRegion.java:3024)
[vm2]   at 
org.apache.geode.internal.cache.PartitionedRegion.virtualPut(PartitionedRegion.java:2236)
[vm2]   at 
org.apache.geode.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5586)

And distributed to replica (vm1):
vm1] [warn 2021/05/25 16:01:41.518 PDT server-1 :41002 shared ordered sender uid=7 local 
port=58643 remote port=53459> tid=0x59] membersRemoveAll invoked
[vm1] java.lang.Exception
[vm1]   at 
org.apache.geode.redis.internal.data.RedisSortedSet.membersRemoveAll(RedisSortedSet.java:175)
[vm1]   at 
org.apache.geode.redis.internal.data.RedisSortedSet.applyDelta(RedisSortedSet.java:89)
[vm1]   at 
org.apache.geode.redis.internal.data.AbstractRedisData.fromDelta(AbstractRedisData.java:193)
[vm1]   at 
org.apache.geode.internal.cache.EntryEventImpl.processDeltaBytes(EntryEventImpl.java:1841)
[vm1]   at 
org.apache.geode.internal.cache.EntryEventImpl.setNewValueInRegion(EntryEventImpl.java:1696)
[vm1]   at 
org.apache.geode.internal.cache.EntryEventImpl.putExistingEntry(EntryEventImpl.java:1643)
[vm1]   at 
org.apache.geode.internal.cache.map.RegionMapPut.updateEntry(RegionMapPut.java:485)
[vm1]   at 
org.apache.geode.internal.cache.map.RegionMapPut.createOrUpdateEntry(RegionMapPut.java:256)

vm2 is bounced:
[vm2] [info 2021/05/25 16:01:41.526 PDT server-2  tid=0x14] Got result: 83136
[vm2]  from org.apache.geode.test.dunit.VM$$Lambda$370/1141741369.call with 0 
args on object: org.apache.geode.test.dunit.VM$$Lambda$370/1141741369@68ecd55a 
(took 0 ms)

[info 2021/05/25 16:01:41.527 PDT   tid=0x36] Bouncing 2 old 
pid is 83136 and version is 10240.0.0

The JedisCluster client did not get response back (possibly detects connection 
is gone), and it does retry again, as the test failed with following:
java.util.concurrent.ExecutionException: 
redis.clients.jedis.exceptions.JedisClusterMaxAttemptsException: No more 
cluster attempts left.

at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at 
org.apache.geode.redis.internal.executor.sortedset.ZRemDUnitTest.zRemCanRemoveMembersFromSortedSetDuringPrimaryIsCrashed(ZRemDUnitTest.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.apache.geode.test.junit.rules.serializable.SerializableExternalResource$1.evaluate(SerializableExternalResource.java:38)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at