Hello, Some links to give context,
(1) https://github.com/apache/zookeeper/blob/b79af153d0f98a4f3f3516910ed47234d7b3d74e/src/java/main/org/apache/zookeeper/server/FinalRequestProcessor.java#L178-188 (2) https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/ClientCnxnSocketNIO.java#L99-102 (3) https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/ClientCnxn.java#L826-836 (4) https://github.com/apache/zookeeper/blob/087abf83684167ae56864fe4c3be0079fa653266/src/java/main/org/apache/zookeeper/ClientCnxnSocketNIO.java#L111-112 Based on link #4, I think now that after a long GC pause, the SendThread would (or at least could) race to update the last send with any thread trying to gate writes using the value. This update of last send occurs before the socket IO, so wouldn't fail to happen even if the OS thought the socket was in an exceptional state. So, in light of this, my earlier idea is not going to work. But, I noticed that the server is returning an application level heartbeat based on link #1. And, based on link #2, this causes a "lastHeard" state to be registered. Also, based on link #3, the "latency" of the ping is logged, but not recorded. So, I've come up with a new proposal: * Create a ClientCnxnStat object. * Move all the current recorded state (packet stats, lastHeard, lastSend, etc) to ClientCnxnStat. * Also start recording the lastPingResponse in the location of link #3. * All the values could be volatile, the object only possessing getters. * Package private in org.apache.zookeeper to allow the connection objects to update the values. * alternatively, the method of exposure could deep copy the object, but some synchronization would be needed to get a "snapshot" effect. * Expose the ClientCnxnStat object via zookeeper using ZooKeeper.getClientCnxnStat(). This opens up many possibilities, like exposing some programmatic insight into the health of the network connection with the ensemble. For example, it would be possible to calculate the latency of pings. And, I, and others concerned about transactions in the presence of long GC pauses, could use a combination of lastHeard and lastPingResponse to gate transaction writes and increase confidence. I anticipate some concerns, like it might degrade performance by slowing down by needing to sync via volatile or a ReadWriteLock to get a snapshot effect. Thoughts? Austin -------------------------------------------------------------------------------- NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
