Re: Odg: Odg: Restart gateway-receiver

[Jacob Barrett]

It’s is likely that your firewall is closing idle connections. This is very 
common for a firewall to drop the state information for a connection that 
hasn’t seen any traffic in some set period of time. It’s a way for firewalls to 
reclaim resources for connections that were likely closed on one side or the 
other without any FIN sequence.

If you don’t have any control over your firewall then we should explore a keep 
alive method.

-Jake


> On Nov 26, 2019, at 2:06 AM, Mario Kevo  wrote:
> 
> Thanks a lot @Barry Oglesby!
> 
> It seems that this closing inactive connection is done by Kubernetes as we 
> run Geode on it.
> 
> BR,
> Mario
> 
> 
> Šalje: Barry Oglesby 
> Poslano: 22. studenog 2019. 22:35
> Prima: Mario Kevo 
> Kopija: dev@geode.apache.org 
> Predmet: Re: Odg: Restart gateway-receiver
> 
>> If we don't send any event, the connection will be closed after some time as 
>> connection is inactive.
> 
> Are you seeing this behavior? I don't think this is true by default.
> 
> AbstractRemoteGatewaySender.initProxy sets these fields on the PoolFactory:
> 
> pf.setReadTimeout(this.socketReadTimeout);
> pf.setIdleTimeout(connectionIdleTimeOut);
> 
> By default, socketReadTimeout is 0 (no timeout), and connectionIdleTimeOut is 
> -1 (disabled).
> 
> Each Event Processor thread will have its own connection to the remote site:
> 
> Event Processor for GatewaySender_ny.1: 
> GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
> Connection to 127.0.0.1:5452: 
> Connection[127.0.0.1:5452]@306907760
> Event Processor for GatewaySender_ny.2: 
> GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
> Connection to 127.0.0.1:5452: 
> Connection[127.0.0.1:5452]@608855137
> Event Processor for GatewaySender_ny.0: 
> GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
> Connection to 127.0.0.1:5452: 
> Connection[127.0.0.1:5452]@950613560
> Event Processor for GatewaySender_ny.4: 
> GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
> Connection to 127.0.0.1:5452: 
> Connection[127.0.0.1:5452]@1005378489
> Event Processor for GatewaySender_ny.3: 
> GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
> Connection to 127.0.0.1:5452: 
> Connection[127.0.0.1:5452]@629246640
> 
> There will be one Event Processor thread for each dispatcher thread (5 by 
> default).
> 
> There aren't any good public ways to monitor the connections other than JMX.
> 
> One way to monitor these connections is with ConnectionStats on the sender 
> side.
> 
> You can do that with vsd: ClientStats -> GatewaySenderStats -> connections
> 
> You can also do it with code like:
> 
> private int getConnectionCount(String gatewaySenderId) {
>  AbstractGatewaySender sender = (AbstractGatewaySender) 
> cache.getGatewaySender(gatewaySenderId);
>  int totalConnections = 0;
>  if (sender != null) {
>for (ConnectionStats connectionStats : 
> sender.getProxy().getEndpointManager().getAllStats().values()) {
>  totalConnections += connectionStats.getConnections();
>}
>System.out.println("Sender=" + gatewaySenderId + "; connectionCount=" + 
> totalConnections);
>  }
>  return totalConnections;
> }
> 
> You can also dump whether the dispatcher is connected like:
> 
> private void dumpConnected(String gatewaySenderId) {
>  AbstractGatewaySender sender = (AbstractGatewaySender) 
> cache.getGatewaySender(gatewaySenderId);
>  if (sender.isParallel()) {
>ConcurrentParallelGatewaySenderEventProcessor concurrentProcessor = 
> (ConcurrentParallelGatewaySenderEventProcessor) sender.getEventProcessor();
>for (ParallelGatewaySenderEventProcessor processor : 
> concurrentProcessor.getProcessors()) {
>  System.out.println("Processor=" + processor + "; isConnected=" + 
> processor.getDispatcher().isConnectedToRemote());
>}
>  } else {
>ConcurrentSerialGatewaySenderEventProcessor concurrentProcessor = 
> (ConcurrentSerialGatewaySenderEventProcessor) sender.getEventProcessor();
>List processors = 
> concurrentProcessor.getProcessors();
>for (SerialGatewaySenderEventProcessor processor : 
> concurrentProcessor.getProcessors()) {
>  System.out.println("Processor=" + processor + "; isConnected=" + 
> processor.getDispatcher().isConnectedToRemote());
>}
>  }
> }
> 
> The isConnectedToRemote method does:
> 
> return connection != null && !connection.isDestroyed();
> 
> Thanks,
> Barry Oglesby
> 
> 
> 
> On Thu, Nov 21, 2019 at 11:15 PM Mario Kevo  wrote:
> Hi,
> 
> @Barry Oglesby, thanks for the clarification.
> 
> If we don't send any event, the connection will be closed after some time as 
> connection is inactive.
> Is it possible to somehow monitor from the app if the replication is 
> established to get 

Odg: Odg: Restart gateway-receiver

[Mario Kevo]

Thanks a lot @Barry Oglesby!

It seems that this closing inactive connection is done by Kubernetes as we run 
Geode on it.

BR,
Mario


Šalje: Barry Oglesby 
Poslano: 22. studenog 2019. 22:35
Prima: Mario Kevo 
Kopija: dev@geode.apache.org 
Predmet: Re: Odg: Restart gateway-receiver

> If we don't send any event, the connection will be closed after some time as 
> connection is inactive.

Are you seeing this behavior? I don't think this is true by default.

AbstractRemoteGatewaySender.initProxy sets these fields on the PoolFactory:

pf.setReadTimeout(this.socketReadTimeout);
pf.setIdleTimeout(connectionIdleTimeOut);

By default, socketReadTimeout is 0 (no timeout), and connectionIdleTimeOut is 
-1 (disabled).

Each Event Processor thread will have its own connection to the remote site:

Event Processor for GatewaySender_ny.1: 
GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
Connection to 127.0.0.1:5452: 
Connection[127.0.0.1:5452]@306907760
Event Processor for GatewaySender_ny.2: 
GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
Connection to 127.0.0.1:5452: 
Connection[127.0.0.1:5452]@608855137
Event Processor for GatewaySender_ny.0: 
GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
Connection to 127.0.0.1:5452: 
Connection[127.0.0.1:5452]@950613560
Event Processor for GatewaySender_ny.4: 
GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
Connection to 127.0.0.1:5452: 
Connection[127.0.0.1:5452]@1005378489
Event Processor for GatewaySender_ny.3: 
GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled 
Connection to 127.0.0.1:5452: 
Connection[127.0.0.1:5452]@629246640

There will be one Event Processor thread for each dispatcher thread (5 by 
default).

There aren't any good public ways to monitor the connections other than JMX.

One way to monitor these connections is with ConnectionStats on the sender side.

You can do that with vsd: ClientStats -> GatewaySenderStats -> connections

You can also do it with code like:

private int getConnectionCount(String gatewaySenderId) {
  AbstractGatewaySender sender = (AbstractGatewaySender) 
cache.getGatewaySender(gatewaySenderId);
  int totalConnections = 0;
  if (sender != null) {
for (ConnectionStats connectionStats : 
sender.getProxy().getEndpointManager().getAllStats().values()) {
  totalConnections += connectionStats.getConnections();
}
System.out.println("Sender=" + gatewaySenderId + "; connectionCount=" + 
totalConnections);
  }
  return totalConnections;
}

You can also dump whether the dispatcher is connected like:

private void dumpConnected(String gatewaySenderId) {
  AbstractGatewaySender sender = (AbstractGatewaySender) 
cache.getGatewaySender(gatewaySenderId);
  if (sender.isParallel()) {
ConcurrentParallelGatewaySenderEventProcessor concurrentProcessor = 
(ConcurrentParallelGatewaySenderEventProcessor) sender.getEventProcessor();
for (ParallelGatewaySenderEventProcessor processor : 
concurrentProcessor.getProcessors()) {
  System.out.println("Processor=" + processor + "; isConnected=" + 
processor.getDispatcher().isConnectedToRemote());
}
  } else {
ConcurrentSerialGatewaySenderEventProcessor concurrentProcessor = 
(ConcurrentSerialGatewaySenderEventProcessor) sender.getEventProcessor();
List processors = 
concurrentProcessor.getProcessors();
for (SerialGatewaySenderEventProcessor processor : 
concurrentProcessor.getProcessors()) {
  System.out.println("Processor=" + processor + "; isConnected=" + 
processor.getDispatcher().isConnectedToRemote());
}
  }
}

The isConnectedToRemote method does:

return connection != null && !connection.isDestroyed();

Thanks,
Barry Oglesby



On Thu, Nov 21, 2019 at 11:15 PM Mario Kevo  wrote:
Hi,

@Barry Oglesby, thanks for the clarification.

If we don't send any event, the connection will be closed after some time as 
connection is inactive.
Is it possible to somehow monitor from the app if the replication is 
established to get information if there is a some problem with network or it is 
just closed due to inactivity?
Can we monitor the replication on some other way than looking "isConnected" 
state on JMX?

BR,
Mario

Šalje: Barry Oglesby mailto:bogle...@pivotal.io>>
Poslano: 14. studenog 2019. 18:29
Prima: dev@geode.apache.org 
mailto:dev@geode.apache.org>>
Predmet: Re: Restart gateway-receiver

Mario,

Thats the current behavior. When the sender is initially started, it
attempts to connect to the receiver. If it does not connect, it won't retry
until there is a batch of events to send. Look for callers of
GatewaySenderEventRemoteDispatcher.initializeConnection to see the