[
https://issues.apache.org/jira/browse/HDDS-15058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-15058:
-------------------------------
Description:
Currently, we use normal gRPC blocking service as a heartbeat mechanism. The
SCM runs a separate thread to mark a DN as stale or dead based on DN last
heartbeat time. This is simple to implement, but seems to have drawbacks
* There is no way to differentiate between crashed datanodes and restarted
datanodes
* Detection of crashed datanodes is pretty slow
* The stale and dead time might be too short or too long for a given cluster
(e.g. a cluster with dense datanodes with a lot of containers per DN might
require DN to have longer grace period).
An idea is to treat these datanodes similar to ZK Ephemeral node where a node
exists as long as the session is still active and ZK is able to detect and
generate an event if one node is down. A DN will create a persistent gRPC
stream (or "session" in ZK term). If the DN is restarted normally, we can call
onCompleted(). If the DN crashed, it will trigger onError(). Therefore we can
handle DN lifecycle in higher granularity.
was:
This is just an idea. Currently, we use normal gRPC blocking service as a
heartbeat mechanism. The SCM runs a separate thread to mark a DN as stale or
dead based on DN last heartbeat time. This is simple to implement, but seems to
have drawbacks
* There is no way to differentiate between crashed datanodes and restarted
datanodes
* Detection of crashed datanodes is pretty slow
* The stale and dead time might be too short or too long for a given cluster
(e.g. a cluster with dense datanodes with a lot of containers per DN might
require DN to have longer grace period).
An idea is to treat these datanodes similar to ZK Ephemeral node where a node
exists as long as the session is still active and ZK is able to detect and
generate an event if one node is down. A DN will create a persistent gRPC
stream (or "session" in ZK term). If the DN is restarted normally, we can call
onCompleted(). If the DN crashed, it will trigger onError(). Therefore we can
handle DN lifecycle in higher granularity.
> Use gRPC stream for DN heartbeat mechanism
> ------------------------------------------
>
> Key: HDDS-15058
> URL: https://issues.apache.org/jira/browse/HDDS-15058
> Project: Apache Ozone
> Issue Type: New Feature
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, we use normal gRPC blocking service as a heartbeat mechanism. The
> SCM runs a separate thread to mark a DN as stale or dead based on DN last
> heartbeat time. This is simple to implement, but seems to have drawbacks
> * There is no way to differentiate between crashed datanodes and restarted
> datanodes
> * Detection of crashed datanodes is pretty slow
> * The stale and dead time might be too short or too long for a given cluster
> (e.g. a cluster with dense datanodes with a lot of containers per DN might
> require DN to have longer grace period).
> An idea is to treat these datanodes similar to ZK Ephemeral node where a node
> exists as long as the session is still active and ZK is able to detect and
> generate an event if one node is down. A DN will create a persistent gRPC
> stream (or "session" in ZK term). If the DN is restarted normally, we can
> call onCompleted(). If the DN crashed, it will trigger onError(). Therefore
> we can handle DN lifecycle in higher granularity.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]