[jira] [Updated] (GEODE-9666) Client throws NoAvailableLocatorsException after locators change IP addresses
[ https://issues.apache.org/jira/browse/GEODE-9666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hale Bales updated GEODE-9666: -- Labels: pull-request-available (was: needsTriage pull-request-available) > Client throws NoAvailableLocatorsException after locators change IP addresses > - > > Key: GEODE-9666 > URL: https://issues.apache.org/jira/browse/GEODE-9666 > Project: Geode > Issue Type: Bug > Components: membership >Affects Versions: 1.15.0 >Reporter: Aaron Lindsey >Assignee: Aaron Lindsey >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > We have a test for Geode on Kubernetes which: > * Deploys a Geode cluster consisting of 2 locator Pods, 3 server Pods > * Deploys 5 Spring boot client Pods which continually do PUTs and GETs > * Triggers a rolling restart of the locator Pods > ** The rolling restart operation restarts one locator at a time, waiting for > each restarted locator to become fully online before restarting the next > locator > * Stops the client operations and validates there were no exceptions thrown > in the clients. > Occasionally, we see {{NoAvailableLocatorsException}} thrown on one of the > clients: > {code:none} > org.apache.geode.cache.client.NoAvailableLocatorsException: Unable to connect > to any locators in the list > [system-test-gemfire-locator-0.system-test-gemfire-locator.gemfire-system-test-3f1ecc74-b1ea-4288-b4d1-594bbb8364ab.svc.cluster.local:10334, > > system-test-gemfire-locator-1.system-test-gemfire-locator.gemfire-system-test-3f1ecc74-b1ea-4288-b4d1-594bbb8364ab.svc.cluster.local:10334] > at > org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.findServer(AutoConnectionSourceImpl.java:174) > at > org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:198) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.createPooledConnection(ConnectionManagerImpl.java:196) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.createPooledConnection(ConnectionManagerImpl.java:190) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:276) > at > org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:136) > at > org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:119) > at > org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:801) > at org.apache.geode.cache.client.internal.GetOp.execute(GetOp.java:92) > at > org.apache.geode.cache.client.internal.ServerRegionProxy.get(ServerRegionProxy.java:114) > at > org.apache.geode.internal.cache.LocalRegion.findObjectInSystem(LocalRegion.java:2802) > at > org.apache.geode.internal.cache.LocalRegion.getObject(LocalRegion.java:1469) > at > org.apache.geode.internal.cache.LocalRegion.nonTxnFindObject(LocalRegion.java:1442) > at > org.apache.geode.internal.cache.LocalRegionDataView.findObject(LocalRegionDataView.java:197) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1379) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1318) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1303) > at > org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:439) > at > org.apache.geode.kubernetes.client.service.AsyncOperationService.evaluate(AsyncOperationService.java:282) > at > org.apache.geode.kubernetes.client.api.Controller.evaluateRegion(Controller.java:88) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAda
[jira] [Updated] (GEODE-9666) Client throws NoAvailableLocatorsException after locators change IP addresses
[ https://issues.apache.org/jira/browse/GEODE-9666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated GEODE-9666: -- Labels: needsTriage pull-request-available (was: needsTriage) > Client throws NoAvailableLocatorsException after locators change IP addresses > - > > Key: GEODE-9666 > URL: https://issues.apache.org/jira/browse/GEODE-9666 > Project: Geode > Issue Type: Bug > Components: membership >Affects Versions: 1.15.0 >Reporter: Aaron Lindsey >Assignee: Aaron Lindsey >Priority: Major > Labels: needsTriage, pull-request-available > > We have a test for Geode on Kubernetes which: > * Deploys a Geode cluster consisting of 2 locator Pods, 3 server Pods > * Deploys 5 Spring boot client Pods which continually do PUTs and GETs > * Triggers a rolling restart of the locator Pods > ** The rolling restart operation restarts one locator at a time, waiting for > each restarted locator to become fully online before restarting the next > locator > * Stops the client operations and validates there were no exceptions thrown > in the clients. > Occasionally, we see {{NoAvailableLocatorsException}} thrown on one of the > clients: > {code:none} > org.apache.geode.cache.client.NoAvailableLocatorsException: Unable to connect > to any locators in the list > [system-test-gemfire-locator-0.system-test-gemfire-locator.gemfire-system-test-3f1ecc74-b1ea-4288-b4d1-594bbb8364ab.svc.cluster.local:10334, > > system-test-gemfire-locator-1.system-test-gemfire-locator.gemfire-system-test-3f1ecc74-b1ea-4288-b4d1-594bbb8364ab.svc.cluster.local:10334] > at > org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.findServer(AutoConnectionSourceImpl.java:174) > at > org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:198) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.createPooledConnection(ConnectionManagerImpl.java:196) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.createPooledConnection(ConnectionManagerImpl.java:190) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:276) > at > org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:136) > at > org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:119) > at > org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:801) > at org.apache.geode.cache.client.internal.GetOp.execute(GetOp.java:92) > at > org.apache.geode.cache.client.internal.ServerRegionProxy.get(ServerRegionProxy.java:114) > at > org.apache.geode.internal.cache.LocalRegion.findObjectInSystem(LocalRegion.java:2802) > at > org.apache.geode.internal.cache.LocalRegion.getObject(LocalRegion.java:1469) > at > org.apache.geode.internal.cache.LocalRegion.nonTxnFindObject(LocalRegion.java:1442) > at > org.apache.geode.internal.cache.LocalRegionDataView.findObject(LocalRegionDataView.java:197) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1379) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1318) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1303) > at > org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:439) > at > org.apache.geode.kubernetes.client.service.AsyncOperationService.evaluate(AsyncOperationService.java:282) > at > org.apache.geode.kubernetes.client.api.Controller.evaluateRegion(Controller.java:88) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(Re
[jira] [Updated] (GEODE-9666) Client throws NoAvailableLocatorsException after locators change IP addresses
[ https://issues.apache.org/jira/browse/GEODE-9666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Murmann updated GEODE-9666: - Labels: needsTriage (was: ) > Client throws NoAvailableLocatorsException after locators change IP addresses > - > > Key: GEODE-9666 > URL: https://issues.apache.org/jira/browse/GEODE-9666 > Project: Geode > Issue Type: Bug > Components: membership >Affects Versions: 1.15.0 >Reporter: Aaron Lindsey >Priority: Major > Labels: needsTriage > > We have a test for Geode on Kubernetes which: > * Deploys a Geode cluster consisting of 2 locator Pods, 3 server Pods > * Deploys 5 Spring boot client Pods which continually do PUTs and GETs > * Triggers a rolling restart of the locator Pods > ** The rolling restart operation restarts one locator at a time, waiting for > each restarted locator to become fully online before restarting the next > locator > * Stops the client operations and validates there were no exceptions thrown > in the clients. > Occasionally, we see {{NoAvailableLocatorsException}} thrown on one of the > clients: > {code:none} > org.apache.geode.cache.client.NoAvailableLocatorsException: Unable to connect > to any locators in the list > [system-test-gemfire-locator-0.system-test-gemfire-locator.gemfire-system-test-3f1ecc74-b1ea-4288-b4d1-594bbb8364ab.svc.cluster.local:10334, > > system-test-gemfire-locator-1.system-test-gemfire-locator.gemfire-system-test-3f1ecc74-b1ea-4288-b4d1-594bbb8364ab.svc.cluster.local:10334] > at > org.apache.geode.cache.client.internal.AutoConnectionSourceImpl.findServer(AutoConnectionSourceImpl.java:174) > at > org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:198) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.createPooledConnection(ConnectionManagerImpl.java:196) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.createPooledConnection(ConnectionManagerImpl.java:190) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:276) > at > org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:136) > at > org.apache.geode.cache.client.internal.OpExecutorImpl.execute(OpExecutorImpl.java:119) > at > org.apache.geode.cache.client.internal.PoolImpl.execute(PoolImpl.java:801) > at org.apache.geode.cache.client.internal.GetOp.execute(GetOp.java:92) > at > org.apache.geode.cache.client.internal.ServerRegionProxy.get(ServerRegionProxy.java:114) > at > org.apache.geode.internal.cache.LocalRegion.findObjectInSystem(LocalRegion.java:2802) > at > org.apache.geode.internal.cache.LocalRegion.getObject(LocalRegion.java:1469) > at > org.apache.geode.internal.cache.LocalRegion.nonTxnFindObject(LocalRegion.java:1442) > at > org.apache.geode.internal.cache.LocalRegionDataView.findObject(LocalRegionDataView.java:197) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1379) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1318) > at > org.apache.geode.internal.cache.LocalRegion.get(LocalRegion.java:1303) > at > org.apache.geode.internal.cache.AbstractRegion.get(AbstractRegion.java:439) > at > org.apache.geode.kubernetes.client.service.AsyncOperationService.evaluate(AsyncOperationService.java:282) > at > org.apache.geode.kubernetes.client.api.Controller.evaluateRegion(Controller.java:88) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) > at > org.springframework.web.servlet.mvc.me