Ivan Andika created HDDS-15058:
----------------------------------

             Summary: Use gRPC stream for DN heartbeat mechanism
                 Key: HDDS-15058
                 URL: https://issues.apache.org/jira/browse/HDDS-15058
             Project: Apache Ozone
          Issue Type: New Feature
            Reporter: Ivan Andika
            Assignee: Ivan Andika


This is just an idea. Currently, we use normal gRPC blocking service as a 
heartbeat mechanism. The SCM runs a separate thread to mark a DN as stale or 
dead based on DN last heartbeat time. This is simple to implement, but seems to 
have drawbacks
* There is no way to differentiate between crashed datanodes and restarted 
datanodes
* Detection of crashed datanodes is pretty slow
* The stale and dead time might be too short or too long for a given cluster 
(e.g. a cluster with dense datanodes with a lot of containers per DN might 
require DN to have longer grace period).

An idea is to treat these datanodes similar to ZK Ephemeral node where a node 
exists as long as the session is still active and ZK is able to detect and 
generate an event if one node is down. A DN will create a persistent gRPC 
stream (or "session" in ZK term). If the DN is restarted normally, we can call 
onCompleted(). If the DN crashed, it will trigger onError(). Therefore we can 
handle DN lifecycle in higher granularity.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to