[ https://issues.apache.org/jira/browse/GEODE-4451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anthony Baker closed GEODE-4451. -------------------------------- > When a gateway sender is created using gfsh and fails to connect due to > AuthenticationFailedException, it is left in an inconsistent state > ------------------------------------------------------------------------------------------------------------------------------------------ > > Key: GEODE-4451 > URL: https://issues.apache.org/jira/browse/GEODE-4451 > Project: Geode > Issue Type: Bug > Components: wan > Reporter: Barry Oglesby > Priority: Major > Labels: pull-request-available > Fix For: 1.6.0 > > Time Spent: 40m > Remaining Estimate: 0h > > If I attempt to create a gateway sender with invalid credentials using gfsh, > it fails like: > {noformat} > (2) Executing - create gateway-sender --remote-distributed-system-id=1 > --id=ny --dispatcher-threads=1 --parallel=true > Member | Status > ------ | > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > ln-1 | ERROR: Could not start a gateway sender ny because of exception Could > not start a gateway sender 84 because of exception > org.apache.geode.security.AuthenticationFailedException: Authentication > error. Please check your credentials. > {noformat} > The 84 should be the gateway sender id; instead its the thread id (from > \{{ConcurrentParallelGatewaySenderEventProcessor.waitForRunningStatus}}. > In the server log, I see 5 AuthenticationFailedException stacks like (one per > dispatcher-thread): > {noformat} > [severe 2018/01/31 16:13:12.638 PST ln-1 <Event Processor for > GatewaySender_ny_0> tid=0x55] Message dispatch failed due to unexpected > exception.. > org.apache.geode.internal.cache.wan.GatewaySenderException: > org.apache.geode.security.AuthenticationFailedException: Authentication > error. Please check your credentials., caused by > org.apache.geode.security.AuthenticationFailedException: Authentication > error. Please check your credentials. > at > org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:432) > at > org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.<init>(GatewaySenderEventRemoteDispatcher.java:75) > at > org.apache.geode.internal.cache.wan.parallel.RemoteParallelGatewaySenderEventProcessor.initializeEventDispatcher(RemoteParallelGatewaySenderEventProcessor.java:74) > at > org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.setRunningStatus(AbstractGatewaySenderEventProcessor.java:1078) > at > org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1050) > Caused by: org.apache.geode.security.AuthenticationFailedException: > Authentication error. Please check your credentials. > at > org.apache.geode.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1397) > at > org.apache.geode.internal.cache.tier.sockets.HandShake.handshakeWithServer(HandShake.java:1253) > at > org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:118) > at > org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:136) > at > org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:259) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:206) > at > org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:902) > at > org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:388) > ... 4 more > {noformat} > A thread dump shows the Event Processor threads are gone, but 5 > BatchRemovalThreads like this exist: > {noformat} > "BatchRemovalThread" #96 daemon prio=10 os_prio=31 tid=0x00007fc1dd009000 > nid=0xcb03 waiting on condition [0x000070000e245000] > java.lang.Thread.State: TIMED_WAITING (parking) > at sun.misc.Unsafe.park(Native Method) > - parking to wait for <0x00000007a04043c8> (a > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) > at > org.apache.geode.internal.util.concurrent.StoppableCondition.await(StoppableCondition.java:61) > at > org.apache.geode.internal.cache.wan.parallel.ParallelGatewaySenderQueue$BatchRemovalThread.run(ParallelGatewaySenderQueue.java:1613) > {noformat} > list gateways shows the sender in the \{{Not Running}} state: > {noformat} > GatewaySender Id | Member | Remote Cluster Id | Type | Status | Queued Events > | Receiver Location > ---------------- | -------------------------------- | ----------------- | > -------- | ----------- | ------------- | ----------------- > ny | 192.168.2.4(ln-1:13883)<v3>:1027 | 1 | Parallel | Not Running | 0 | null > {noformat} > The sender is left in an inconsistent state. It should either be completely > gone or retrying to connect. > {\{GatewaySenderEventRemoteDispatcher.<init>}} catches the > \{{GatewaySenderException}} attempting to \{{initializeConnection}} and > rethrows it if is caused by \{{GemFireSecurityException}}: > {noformat} > public GatewaySenderEventRemoteDispatcher(AbstractGatewaySenderEventProcessor > eventProcessor) { > this.processor = eventProcessor; > this.sender = eventProcessor.getSender(); > // this.ackReaderThread = new AckReaderThread(sender); > try { > initializeConnection(); > } catch (GatewaySenderException e) { > if (e.getCause() instanceof GemFireSecurityException) { > throw e; > } > } > } > {noformat} > {\{AbstractGatewaySenderEventProcessor.run}} catches that exception and logs > severe message and stack above. > If I change this area of code to log the warning instead of stopping the > Event Processor threads, I see this exception after puts are done: > {noformat} > [severe 2018/01/31 17:37:16.324 PST ln-1 <Event Processor for > GatewaySender_ny_0> tid=0x49] Stopping the processor because the following > exception occurred while processing a batch: > org.apache.geode.internal.cache.wan.GatewaySenderException: > org.apache.geode.security.AuthenticationFailedException: Authentication > error. Please check your credentials., caused by > org.apache.geode.security.AuthenticationFailedException: Authentication > error. Please check your credentials. > at > org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:431) > at > org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.getConnection(GatewaySenderEventRemoteDispatcher.java:320) > at > org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher._dispatchBatch(GatewaySenderEventRemoteDispatcher.java:198) > at > org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.dispatchBatch(GatewaySenderEventRemoteDispatcher.java:147) > at > org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.processQueue(AbstractGatewaySenderEventProcessor.java:609) > at > org.apache.geode.internal.cache.wan.AbstractGatewaySenderEventProcessor.run(AbstractGatewaySenderEventProcessor.java:1051) > Caused by: org.apache.geode.security.AuthenticationFailedException: > Authentication error. Please check your credentials. > at > org.apache.geode.internal.cache.tier.sockets.HandShake.readMessage(HandShake.java:1397) > at > org.apache.geode.internal.cache.tier.sockets.HandShake.handshakeWithServer(HandShake.java:1253) > at > org.apache.geode.cache.client.internal.ConnectionImpl.connect(ConnectionImpl.java:118) > at > org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:136) > at > org.apache.geode.cache.client.internal.ConnectionFactoryImpl.createClientToServerConnection(ConnectionFactoryImpl.java:259) > at > org.apache.geode.cache.client.internal.pooling.ConnectionManagerImpl.borrowConnection(ConnectionManagerImpl.java:206) > at > org.apache.geode.cache.client.internal.PoolImpl.acquireConnection(PoolImpl.java:902) > at > org.apache.geode.internal.cache.wan.GatewaySenderEventRemoteDispatcher.initializeConnection(GatewaySenderEventRemoteDispatcher.java:387) > ... 5 more > {noformat} > That is coming from \{{GatewaySenderEventRemoteDispatcher.dispatchBatch}}: > {noformat} > } catch (GatewaySenderException ge) { > if (...) { > ... > } else { > logger.fatal( > LocalizedMessage.create( > > LocalizedStrings.GatewayEventRemoteDispatcher_STOPPING_THE_PROCESSOR_BECAUSE_THE_FOLLOWING_EXCEPTION_OCCURRED_WHILE_PROCESSING_A_BATCH), > ge); > this.processor.setIsStopped(true); > } > {noformat} > So, \{{AuthenticationFailedExceptions}} are definitely meant to stop the > sender. -- This message was sent by Atlassian JIRA (v7.6.3#76005)