[ 
https://issues.apache.org/jira/browse/HDDS-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Andika updated HDDS-15058:
-------------------------------
    Description: 
Currently, we use normal gRPC blocking service as a heartbeat mechanism. The 
SCM runs a separate thread to mark a DN as stale or dead based on DN last 
heartbeat time. This is simple to implement, but seems to have drawbacks
* There is no way to differentiate between crashed datanodes and restarted 
datanodes
* Detection of crashed datanodes is pretty slow
* The stale and dead time might be too short or too long for a given cluster 
(e.g. a cluster with dense datanodes with a lot of containers per DN might 
require DN to have longer grace period).

An idea is to treat these datanodes similar to ZK Ephemeral node where a node 
exists as long as the session is still active and ZK is able to detect and 
generate an event if one node is down. A DN will create a persistent gRPC 
stream (or "session" in ZK term). If the DN is restarted normally, we can call 
onCompleted() and SCM can mark datanodes to status other than DEAD. If the DN 
crashed, it will trigger onError(). Therefore we can handle DN lifecycle in 
higher granularity.

  was:
Currently, we use normal gRPC blocking service as a heartbeat mechanism. The 
SCM runs a separate thread to mark a DN as stale or dead based on DN last 
heartbeat time. This is simple to implement, but seems to have drawbacks
* There is no way to differentiate between crashed datanodes and restarted 
datanodes
* Detection of crashed datanodes is pretty slow
* The stale and dead time might be too short or too long for a given cluster 
(e.g. a cluster with dense datanodes with a lot of containers per DN might 
require DN to have longer grace period).

An idea is to treat these datanodes similar to ZK Ephemeral node where a node 
exists as long as the session is still active and ZK is able to detect and 
generate an event if one node is down. A DN will create a persistent gRPC 
stream (or "session" in ZK term). If the DN is restarted normally, we can call 
onCompleted(). If the DN crashed, it will trigger onError(). Therefore we can 
handle DN lifecycle in higher granularity.


> Use gRPC stream for DN heartbeat mechanism
> ------------------------------------------
>
>                 Key: HDDS-15058
>                 URL: https://issues.apache.org/jira/browse/HDDS-15058
>             Project: Apache Ozone
>          Issue Type: New Feature
>            Reporter: Ivan Andika
>            Assignee: Ivan Andika
>            Priority: Major
>
> Currently, we use normal gRPC blocking service as a heartbeat mechanism. The 
> SCM runs a separate thread to mark a DN as stale or dead based on DN last 
> heartbeat time. This is simple to implement, but seems to have drawbacks
> * There is no way to differentiate between crashed datanodes and restarted 
> datanodes
> * Detection of crashed datanodes is pretty slow
> * The stale and dead time might be too short or too long for a given cluster 
> (e.g. a cluster with dense datanodes with a lot of containers per DN might 
> require DN to have longer grace period).
> An idea is to treat these datanodes similar to ZK Ephemeral node where a node 
> exists as long as the session is still active and ZK is able to detect and 
> generate an event if one node is down. A DN will create a persistent gRPC 
> stream (or "session" in ZK term). If the DN is restarted normally, we can 
> call onCompleted() and SCM can mark datanodes to status other than DEAD. If 
> the DN crashed, it will trigger onError(). Therefore we can handle DN 
> lifecycle in higher granularity.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to