Thanks a lot @Barry Oglesby<mailto:bogle...@pivotal.io>!
It seems that this closing inactive connection is done by Kubernetes as we run Geode on it. BR, Mario ________________________________ Šalje: Barry Oglesby <bogle...@pivotal.io> Poslano: 22. studenog 2019. 22:35 Prima: Mario Kevo <mario.k...@est.tech> Kopija: dev@geode.apache.org <dev@geode.apache.org> Predmet: Re: Odg: Restart gateway-receiver > If we don't send any event, the connection will be closed after some time as > connection is inactive. Are you seeing this behavior? I don't think this is true by default. AbstractRemoteGatewaySender.initProxy sets these fields on the PoolFactory: pf.setReadTimeout(this.socketReadTimeout); pf.setIdleTimeout(connectionIdleTimeOut); By default, socketReadTimeout is 0 (no timeout), and connectionIdleTimeOut is -1 (disabled). Each Event Processor thread will have its own connection to the remote site: Event Processor for GatewaySender_ny.1: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@306907760 Event Processor for GatewaySender_ny.2: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@608855137 Event Processor for GatewaySender_ny.0: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@950613560 Event Processor for GatewaySender_ny.4: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@1005378489 Event Processor for GatewaySender_ny.3: GatewaySenderEventRemoteDispatcher.initializeConnection connection=Pooled Connection to 127.0.0.1:5452<http://127.0.0.1:5452>: Connection[127.0.0.1:5452]@629246640 There will be one Event Processor thread for each dispatcher thread (5 by default). There aren't any good public ways to monitor the connections other than JMX. One way to monitor these connections is with ConnectionStats on the sender side. You can do that with vsd: ClientStats -> GatewaySenderStats -> connections You can also do it with code like: private int getConnectionCount(String gatewaySenderId) { AbstractGatewaySender sender = (AbstractGatewaySender) cache.getGatewaySender(gatewaySenderId); int totalConnections = 0; if (sender != null) { for (ConnectionStats connectionStats : sender.getProxy().getEndpointManager().getAllStats().values()) { totalConnections += connectionStats.getConnections(); } System.out.println("Sender=" + gatewaySenderId + "; connectionCount=" + totalConnections); } return totalConnections; } You can also dump whether the dispatcher is connected like: private void dumpConnected(String gatewaySenderId) { AbstractGatewaySender sender = (AbstractGatewaySender) cache.getGatewaySender(gatewaySenderId); if (sender.isParallel()) { ConcurrentParallelGatewaySenderEventProcessor concurrentProcessor = (ConcurrentParallelGatewaySenderEventProcessor) sender.getEventProcessor(); for (ParallelGatewaySenderEventProcessor processor : concurrentProcessor.getProcessors()) { System.out.println("Processor=" + processor + "; isConnected=" + processor.getDispatcher().isConnectedToRemote()); } } else { ConcurrentSerialGatewaySenderEventProcessor concurrentProcessor = (ConcurrentSerialGatewaySenderEventProcessor) sender.getEventProcessor(); List<SerialGatewaySenderEventProcessor> processors = concurrentProcessor.getProcessors(); for (SerialGatewaySenderEventProcessor processor : concurrentProcessor.getProcessors()) { System.out.println("Processor=" + processor + "; isConnected=" + processor.getDispatcher().isConnectedToRemote()); } } } The isConnectedToRemote method does: return connection != null && !connection.isDestroyed(); Thanks, Barry Oglesby On Thu, Nov 21, 2019 at 11:15 PM Mario Kevo <mario.k...@est.tech> wrote: Hi, @Barry Oglesby<mailto:bogle...@pivotal.io>, thanks for the clarification. If we don't send any event, the connection will be closed after some time as connection is inactive. Is it possible to somehow monitor from the app if the replication is established to get information if there is a some problem with network or it is just closed due to inactivity? Can we monitor the replication on some other way than looking "isConnected" state on JMX? BR, Mario ________________________________ Šalje: Barry Oglesby <bogle...@pivotal.io<mailto:bogle...@pivotal.io>> Poslano: 14. studenog 2019. 18:29 Prima: dev@geode.apache.org<mailto:dev@geode.apache.org> <dev@geode.apache.org<mailto:dev@geode.apache.org>> Predmet: Re: Restart gateway-receiver Mario, Thats the current behavior. When the sender is initially started, it attempts to connect to the receiver. If it does not connect, it won't retry until there is a batch of events to send. Look for callers of GatewaySenderEventRemoteDispatcher.initializeConnection to see the behavior. It could be changed to have a task to retry lost connections, but generally there are events in the queue, so the connection is re-established pretty quickly by the event processor thread. Thanks, Barry Oglesby On Wed, Nov 13, 2019 at 4:55 AM Mario Kevo <mario.k...@est.tech> wrote: > Hi geode dev, > > After creating gateways senders and receivers between two geode clusters > replications is established. After restart gateway receiver, sender will > not connect to it until we send some entry from sender to receiver. > Is this a normal behavior or a bug? > Should geode have some mechanism for checking if connection is established > no matter if entry is sent or not? > > BR, > Mario >
