Ivan Andika created HDDS-15058:
----------------------------------
Summary: Use gRPC stream for DN heartbeat mechanism
Key: HDDS-15058
URL: https://issues.apache.org/jira/browse/HDDS-15058
Project: Apache Ozone
Issue Type: New Feature
Reporter: Ivan Andika
Assignee: Ivan Andika
This is just an idea. Currently, we use normal gRPC blocking service as a
heartbeat mechanism. The SCM runs a separate thread to mark a DN as stale or
dead based on DN last heartbeat time. This is simple to implement, but seems to
have drawbacks
* There is no way to differentiate between crashed datanodes and restarted
datanodes
* Detection of crashed datanodes is pretty slow
* The stale and dead time might be too short or too long for a given cluster
(e.g. a cluster with dense datanodes with a lot of containers per DN might
require DN to have longer grace period).
An idea is to treat these datanodes similar to ZK Ephemeral node where a node
exists as long as the session is still active and ZK is able to detect and
generate an event if one node is down. A DN will create a persistent gRPC
stream (or "session" in ZK term). If the DN is restarted normally, we can call
onCompleted(). If the DN crashed, it will trigger onError(). Therefore we can
handle DN lifecycle in higher granularity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]