Re: Odg: Odg: Restart gateway-receiver
It’s is likely that your firewall is closing idle connections. This is very common for a firewall to drop the state information for a connection that hasn’t seen any traffic in some set period of time. It’s a way for firewalls to reclaim resources for connections that were likely closed on one side or the other without any FIN sequence. If you don’t have any control over your firewall then we should explore a keep alive method. -Jake > On Nov 26, 2019, at 2:06 AM, Mario Kevo wrote: > > Thanks a lot @Barry Oglesby<mailto:bogle...@pivotal.io>! > > It seems that this closing inactive connection is done by Kubernetes as we > run Geode on it. > > BR, > Mario > > > Šalje: Barry Oglesby > Poslano: 22. studenog 2019. 22:35 > Prima: Mario Kevo > Kopija: dev@geode.apache.org > Predmet: Re: Odg: Restart gateway-receiver > >> If we don't send any event, the connection will be closed after some time as >> connection is inactive. > > Are you seeing this behavior? I don't think this is true by default. > > AbstractRemoteGatewaySender.initProxy sets these fields on the PoolFactory: > > pf.setReadTimeout(this.socketReadTimeout); > pf.setIdleTimeout(connectionIdleTimeOut); > > By default, socketReadTimeout is 0 (no timeout), and connectionIdleTimeOut is > -1 (disabled). > > Each Event Processor thread will have its own connection to the remote site: > > Event Processor for GatewaySender_ny.1: > GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled > Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: > Connection[127.0.0.1:5452]@306907760 > Event Processor for GatewaySender_ny.2: > GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled > Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: > Connection[127.0.0.1:5452]@608855137 > Event Processor for GatewaySender_ny.0: > GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled > Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: > Connection[127.0.0.1:5452]@950613560 > Event Processor for GatewaySender_ny.4: > GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled > Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: > Connection[127.0.0.1:5452]@1005378489 > Event Processor for GatewaySender_ny.3: > GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled > Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: > Connection[127.0.0.1:5452]@629246640 > > There will be one Event Processor thread for each dispatcher thread (5 by > default). > > There aren't any good public ways to monitor the connections other than JMX. > > One way to monitor these connections is with ConnectionStats on the sender > side. > > You can do that with vsd: ClientStats -> GatewaySenderStats -> connections > > You can also do it with code like: > > private int getConnectionCount(String gatewaySenderId) { > AbstractGatewaySender sender = (AbstractGatewaySender) > cache.getGatewaySender(gatewaySenderId); > int totalConnections = 0; > if (sender != null) { >for (ConnectionStats connectionStats : > sender.getProxy().getEndpointManager().getAllStats().values()) { > totalConnections += connectionStats.getConnections(); >} >System.out.println("Sender=" + gatewaySenderId + "; connectionCount=" + > totalConnections); > } > return totalConnections; > } > > You can also dump whether the dispatcher is connected like: > > private void dumpConnected(String gatewaySenderId) { > AbstractGatewaySender sender = (AbstractGatewaySender) > cache.getGatewaySender(gatewaySenderId); > if (sender.isParallel()) { >ConcurrentParallelGatewaySenderEventProcessor concurrentProcessor = > (ConcurrentParallelGatewaySenderEventProcessor) sender.getEventProcessor(); >for (ParallelGatewaySenderEventProcessor processor : > concurrentProcessor.getProcessors()) { > System.out.println("Processor=" + processor + "; isConnected=" + > processor.getDispatcher().isConnectedToRemote()); >} > } else { >ConcurrentSerialGatewaySenderEventProcessor concurrentProcessor = > (ConcurrentSerialGatewaySenderEventProcessor) sender.getEventProcessor(); >List processors = > concurrentProcessor.getProcessors(); >for (SerialGatewaySenderEventProcessor processor : > concurrentProcessor.getProcessors()) { > System.out.println("Processor=" + processor + "; isConnected=" + > processor.getDispatcher().isConnectedToRemote()); >} > } > } > > The isConnectedToRemote method does: > >
Odg: Odg: Restart gateway-receiver
Thanks a lot @Barry Oglesby<mailto:bogle...@pivotal.io>! It seems that this closing inactive connection is done by Kubernetes as we run Geode on it. BR, Mario Šalje: Barry Oglesby Poslano: 22. studenog 2019. 22:35 Prima: Mario Kevo Kopija: dev@geode.apache.org Predmet: Re: Odg: Restart gateway-receiver > If we don't send any event, the connection will be closed after some time as > connection is inactive. Are you seeing this behavior? I don't think this is true by default. AbstractRemoteGatewaySender.initProxy sets these fields on the PoolFactory: pf.setReadTimeout(this.socketReadTimeout); pf.setIdleTimeout(connectionIdleTimeOut); By default, socketReadTimeout is 0 (no timeout), and connectionIdleTimeOut is -1 (disabled). Each Event Processor thread will have its own connection to the remote site: Event Processor for GatewaySender_ny.1: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@306907760 Event Processor for GatewaySender_ny.2: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@608855137 Event Processor for GatewaySender_ny.0: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@950613560 Event Processor for GatewaySender_ny.4: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@1005378489 Event Processor for GatewaySender_ny.3: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@629246640 There will be one Event Processor thread for each dispatcher thread (5 by default). There aren't any good public ways to monitor the connections other than JMX. One way to monitor these connections is with ConnectionStats on the sender side. You can do that with vsd: ClientStats -> GatewaySenderStats -> connections You can also do it with code like: private int getConnectionCount(String gatewaySenderId) { AbstractGatewaySender sender = (AbstractGatewaySender) cache.getGatewaySender(gatewaySenderId); int totalConnections = 0; if (sender != null) { for (ConnectionStats connectionStats : sender.getProxy().getEndpointManager().getAllStats().values()) { totalConnections += connectionStats.getConnections(); } System.out.println("Sender=" + gatewaySenderId + "; connectionCount=" + totalConnections); } return totalConnections; } You can also dump whether the dispatcher is connected like: private void dumpConnected(String gatewaySenderId) { AbstractGatewaySender sender = (AbstractGatewaySender) cache.getGatewaySender(gatewaySenderId); if (sender.isParallel()) { ConcurrentParallelGatewaySenderEventProcessor concurrentProcessor = (ConcurrentParallelGatewaySenderEventProcessor) sender.getEventProcessor(); for (ParallelGatewaySenderEventProcessor processor : concurrentProcessor.getProcessors()) { System.out.println("Processor=" + processor + "; isConnected=" + processor.getDispatcher().isConnectedToRemote()); } } else { ConcurrentSerialGatewaySenderEventProcessor concurrentProcessor = (ConcurrentSerialGatewaySenderEventProcessor) sender.getEventProcessor(); List processors = concurrentProcessor.getProcessors(); for (SerialGatewaySenderEventProcessor processor : concurrentProcessor.getProcessors()) { System.out.println("Processor=" + processor + "; isConnected=" + processor.getDispatcher().isConnectedToRemote()); } } } The isConnectedToRemote method does: return connection != null && !connection.isDestroyed(); Thanks, Barry Oglesby On Thu, Nov 21, 2019 at 11:15 PM Mario Kevo wrote: Hi, @Barry Oglesby<mailto:bogle...@pivotal.io>, thanks for the clarification. If we don't send any event, the connection will be closed after some time as connection is inactive. Is it possible to somehow monitor from the app if the replication is established to get information if there is a some problem with network or it is just closed due to inactivity? Can we monitor the replication on some other way than looking "isConnected" state on JMX? BR, Mario Šalje: Barry Oglesby mailto:bogle...@pivotal.io>> Poslano: 14. studenog 2019. 18:29 Prima: dev@geode.apache.org<mailto:dev@geode.apache.org> mailto:dev@geode.apache.org>> Predmet: Re: Restart gateway-receiver Mario, Thats the current behavior. When the sender is initially started, it attempts to connect to the receiver. If it does no
Re: Odg: Restart gateway-receiver
> If we don't send any event, the connection will be closed after some time as connection is inactive. Are you seeing this behavior? I don't think this is true by default. AbstractRemoteGatewaySender.initProxy sets these fields on the PoolFactory: pf.setReadTimeout(this.socketReadTimeout); pf.setIdleTimeout(connectionIdleTimeOut); By default, socketReadTimeout is 0 (no timeout), and connectionIdleTimeOut is -1 (disabled). Each Event Processor thread will have its own connection to the remote site: Event Processor for GatewaySender_ny.1: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452: Connection[127.0.0.1:5452]@306907760 Event Processor for GatewaySender_ny.2: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452: Connection[127.0.0.1:5452]@608855137 Event Processor for GatewaySender_ny.0: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452: Connection[127.0.0.1:5452]@950613560 Event Processor for GatewaySender_ny.4: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452: Connection[127.0.0.1:5452]@1005378489 Event Processor for GatewaySender_ny.3: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452: Connection[127.0.0.1:5452]@629246640 There will be one Event Processor thread for each dispatcher thread (5 by default). There aren't any good public ways to monitor the connections other than JMX. One way to monitor these connections is with ConnectionStats on the sender side. You can do that with vsd: ClientStats -> GatewaySenderStats -> connections You can also do it with code like: private int getConnectionCount(String gatewaySenderId) { AbstractGatewaySender sender = (AbstractGatewaySender) cache.getGatewaySender(gatewaySenderId); int totalConnections = 0; if (sender != null) { for (ConnectionStats connectionStats : sender.getProxy().getEndpointManager().getAllStats().values()) { totalConnections += connectionStats.getConnections(); } System.out.println("Sender=" + gatewaySenderId + "; connectionCount=" + totalConnections); } return totalConnections; } You can also dump whether the dispatcher is connected like: private void dumpConnected(String gatewaySenderId) { AbstractGatewaySender sender = (AbstractGatewaySender) cache.getGatewaySender(gatewaySenderId); if (sender.isParallel()) { ConcurrentParallelGatewaySenderEventProcessor concurrentProcessor = (ConcurrentParallelGatewaySenderEventProcessor) sender.getEventProcessor(); for (ParallelGatewaySenderEventProcessor processor : concurrentProcessor.getProcessors()) { System.out.println("Processor=" + processor + "; isConnected=" + processor.getDispatcher().isConnectedToRemote()); } } else { ConcurrentSerialGatewaySenderEventProcessor concurrentProcessor = (ConcurrentSerialGatewaySenderEventProcessor) sender.getEventProcessor(); List processors = concurrentProcessor.getProcessors(); for (SerialGatewaySenderEventProcessor processor : concurrentProcessor.getProcessors()) { System.out.println("Processor=" + processor + "; isConnected=" + processor.getDispatcher().isConnectedToRemote()); } } } The isConnectedToRemote method does: return connection != null && !connection.isDestroyed(); Thanks, Barry Oglesby On Thu, Nov 21, 2019 at 11:15 PM Mario Kevo wrote: > Hi, > > @Barry Oglesby , thanks for the clarification. > > If we don't send any event, the connection will be closed after some time > as connection is inactive. > Is it possible to somehow monitor from the app if the replication is > established to get information if there is a some problem with network or > it is just closed due to inactivity? > Can we monitor the replication on some other way than looking > "isConnected" state on JMX? > > BR, > Mario > ---------- > *Šalje:* Barry Oglesby > *Poslano:* 14. studenog 2019. 18:29 > *Prima:* dev@geode.apache.org > *Predmet:* Re: Restart gateway-receiver > > Mario, > > Thats the current behavior. When the sender is initially started, it > attempts to connect to the receiver. If it does not connect, it won't retry > until there is a batch of events to send. Look for callers of > GatewaySenderEventRemoteDispatcher.initializeConnection to see the > behavior. It could be changed to have a task to retry lost connections, but > generally there are events in the queue, so the connection is > re-established pretty quickly by the event processor thread. > > Thanks, > Barry Oglesby > > > > On Wed, Nov 13, 2019 at 4:55 AM Mario Kevo wrote: > > > Hi geode dev, > > > > Afte
Odg: Restart gateway-receiver
Hi, @Barry Oglesby<mailto:bogle...@pivotal.io>, thanks for the clarification. If we don't send any event, the connection will be closed after some time as connection is inactive. Is it possible to somehow monitor from the app if the replication is established to get information if there is a some problem with network or it is just closed due to inactivity? Can we monitor the replication on some other way than looking "isConnected" state on JMX? BR, Mario Šalje: Barry Oglesby Poslano: 14. studenog 2019. 18:29 Prima: dev@geode.apache.org Predmet: Re: Restart gateway-receiver Mario, Thats the current behavior. When the sender is initially started, it attempts to connect to the receiver. If it does not connect, it won't retry until there is a batch of events to send. Look for callers of GatewaySenderEventRemoteDispatcher.initializeConnection to see the behavior. It could be changed to have a task to retry lost connections, but generally there are events in the queue, so the connection is re-established pretty quickly by the event processor thread. Thanks, Barry Oglesby On Wed, Nov 13, 2019 at 4:55 AM Mario Kevo wrote: > Hi geode dev, > > After creating gateways senders and receivers between two geode clusters > replications is established. After restart gateway receiver, sender will > not connect to it until we send some entry from sender to receiver. > Is this a normal behavior or a bug? > Should geode have some mechanism for checking if connection is established > no matter if entry is sent or not? > > BR, > Mario >
Re: Restart gateway-receiver
Mario, Thats the current behavior. When the sender is initially started, it attempts to connect to the receiver. If it does not connect, it won't retry until there is a batch of events to send. Look for callers of GatewaySenderEventRemoteDispatcher.initializeConnection to see the behavior. It could be changed to have a task to retry lost connections, but generally there are events in the queue, so the connection is re-established pretty quickly by the event processor thread. Thanks, Barry Oglesby On Wed, Nov 13, 2019 at 4:55 AM Mario Kevo wrote: > Hi geode dev, > > After creating gateways senders and receivers between two geode clusters > replications is established. After restart gateway receiver, sender will > not connect to it until we send some entry from sender to receiver. > Is this a normal behavior or a bug? > Should geode have some mechanism for checking if connection is established > no matter if entry is sent or not? > > BR, > Mario >
Restart gateway-receiver
Hi geode dev, After creating gateways senders and receivers between two geode clusters replications is established. After restart gateway receiver, sender will not connect to it until we send some entry from sender to receiver. Is this a normal behavior or a bug? Should geode have some mechanism for checking if connection is established no matter if entry is sent or not? BR, Mario