[jira] [Comment Edited] (HDFS-14651) DeadNodeDetector checks dead node periodically
[
https://issues.apache.org/jira/browse/HDFS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027943#comment-17027943
]
Ahmed Hussein edited comment on HDFS-14651 at 2/1/20 2:32 AM:
--
Hi [~leosun08] and [~linyiqun]!
I am looking at the timeouts caused by the changes in this patch HDFS-15149
I have couple of questions:
# what is the usage of {{deadNodeDetectInterval}} ? As far as I understand,
every call to {{checkDeadNodes()}} will change the state to {{IDLE}} forcing
the {{DeadNodeDetector}} to sleep for {{IDLE_SLEEP_MS}}. So, why do we need
{{deadNodeDetectInterval}} if the actual time gap between every check is
{{IDLE_SLEEP_MS}}?
# Correct me if I am wrong:
{{stopDeadNodeDetectorThread.stopDeadNodeDetectorThread()}} is supposed to stop
the deadNodeDetector thread; but it looks like the implementation of the
runnable never terminates. {{DeadNodeDetector}} surpresses all interrupts and
never checks for a termination flag. Therefore, the caller will just hang for 3
seconds waiting to join.
was (Author: ahussein):
Hi [~leosun08] and [~linyiqun]!
I am looking at the timeouts caused by the changes in this patch HDFS-15149
I have couple of questions:
# what is the usage of {{deadNodeDetectInterval}} ? As far as I understand,
every call to {{checkDeadNodes()}} will change the state to {{IDLE}} forcing
the {{DeadNodeDetector}} to sleep for {{IDLE_SLEEP_MS}}. So, why do we need
{{deadNodeDetectInterval}} if the actual time gap between every check is
{{IDLE_SLEEP_MS}}?
# Correct me if I am wrong:
{{stopDeadNodeDetectorThread.stopDeadNodeDetectorThread()}} is supposed to stop
the deadNodeDetector thread; but it looks like the implementation never of the
runnable never terminates. {{DeadNodeDetector}} surpresses all interrupts and
never checks for a termination flag. Therefore, the caller will just hang for 3
seconds waiting to join.
> DeadNodeDetector checks dead node periodically
> --
>
> Key: HDFS-14651
> URL: https://issues.apache.org/jira/browse/HDFS-14651
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14651.001.patch, HDFS-14651.002.patch,
> HDFS-14651.003.patch, HDFS-14651.004.patch, HDFS-14651.005.patch,
> HDFS-14651.006.patch, HDFS-14651.007.patch, HDFS-14651.008.patch
>
>
> DeadNodeDetector checks dead node periodically.
> DeadNodeDetector periodically detect the Node in DeadNodeDetector#deadnode,
> If the access is successful, the Node will be moved from
> DeadNodeDetector#deadnode. Continuous detection of the dead node is
> necessary. The DataNode need rejoin the cluster due to a service
> restart/machine repair. The DataNode may be permanently excluded if there is
> no added probe mechanism.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14651) DeadNodeDetector checks dead node periodically
[
https://issues.apache.org/jira/browse/HDFS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027943#comment-17027943
]
Ahmed Hussein edited comment on HDFS-14651 at 2/1/20 2:31 AM:
--
Hi [~leosun08] and [~linyiqun]!
I am looking at the timeouts caused by the changes in this patch HDFS-15149
I have couple of questions:
# what is the usage of {{deadNodeDetectInterval}} ? As far as I understand,
every call to {{checkDeadNodes()}} will change the state to {{IDLE}} forcing
the {{DeadNodeDetector}} to sleep for {{IDLE_SLEEP_MS}}. So, why do we need
{{deadNodeDetectInterval}} if the actual time gap between every check is
{{IDLE_SLEEP_MS}}?
# Correct me if I am wrong:
{{stopDeadNodeDetectorThread.stopDeadNodeDetectorThread()}} is supposed to stop
the deadNodeDetector thread; but it looks like the implementation never of the
runnable never terminates. {{DeadNodeDetector}} surpresses all interrupts and
never checks for a termination flag. Therefore, the caller will just hang for 3
seconds waiting to join.
was (Author: ahussein):
Hi [~leosun08] and [~linyiqun]!
I am looking at the timeouts caused by the changes in this patch HDFS-15147
I have couple of questions:
# what is the usage of {{deadNodeDetectInterval}} ? As far as I understand,
every call to {{checkDeadNodes()}} will change the state to {{IDLE}} forcing
the {{DeadNodeDetector}} to sleep for {{IDLE_SLEEP_MS}}. So, why do we need
{{deadNodeDetectInterval}} if the actual time gap between every check is
{{IDLE_SLEEP_MS}}?
# Correct me if I am wrong:
{{stopDeadNodeDetectorThread.stopDeadNodeDetectorThread()}} is supposed to stop
the deadNodeDetector thread; but it looks like the implementation never of the
runnable never terminates. {{DeadNodeDetector}} surpresses all interrupts and
never checks for a termination flag. Therefore, the caller will just hang for 3
seconds waiting to join.
> DeadNodeDetector checks dead node periodically
> --
>
> Key: HDFS-14651
> URL: https://issues.apache.org/jira/browse/HDFS-14651
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14651.001.patch, HDFS-14651.002.patch,
> HDFS-14651.003.patch, HDFS-14651.004.patch, HDFS-14651.005.patch,
> HDFS-14651.006.patch, HDFS-14651.007.patch, HDFS-14651.008.patch
>
>
> DeadNodeDetector checks dead node periodically.
> DeadNodeDetector periodically detect the Node in DeadNodeDetector#deadnode,
> If the access is successful, the Node will be moved from
> DeadNodeDetector#deadnode. Continuous detection of the dead node is
> necessary. The DataNode need rejoin the cluster due to a service
> restart/machine repair. The DataNode may be permanently excluded if there is
> no added probe mechanism.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14651) DeadNodeDetector checks dead node periodically
[ https://issues.apache.org/jira/browse/HDFS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978253#comment-16978253 ] Lisheng Sun edited comment on HDFS-14651 at 11/20/19 10:10 AM: --- [~linyiqun] when client read failed from Datanode, there are generally two reasons as follow: 1.There is a problem with the datanode itself. 2.There is a problem with the replica on datanode and datanode is good. The client can't distinguish between the two cases. For the second case, we should not add the datanode to dead list. it need to be confirmed by re-probing and requires a higher priority processing. so we add re-probing node to suspicious list. At the same time the datanode in suspicious list is accessed by other dfsinputstream. was (Author: leosun08): [~linyiqun] when client read failed from Datanode, there are generally two reasons as follow: 1.There is a problem with the datanode itself. 2.There is a problem with the replica on datanode and datanode is good. The client can't distinguish between the two cases. For the second case, we should not add the datanode to dead list. it need to be confirmed by re-probing and requires a higher priority processing. so we add re-probing node to suspicious list. At the same time the datanode in suspicious list is accessed from other dfsinputstream. > DeadNodeDetector checks dead node periodically > -- > > Key: HDFS-14651 > URL: https://issues.apache.org/jira/browse/HDFS-14651 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14651.001.patch, HDFS-14651.002.patch, > HDFS-14651.003.patch, HDFS-14651.004.patch, HDFS-14651.005.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14651) DeadNodeDetector checks dead node periodically
[ https://issues.apache.org/jira/browse/HDFS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978266#comment-16978266 ] Yiqun Lin edited comment on HDFS-14651 at 11/20/19 10:00 AM: - For the second case, if we add the node to the dead node by mistake but it can quickly be probed and removed from dfsinputstream in Detector. So this should not be a major problem, right? Okay, I got your idea. Let me take further review. was (Author: linyiqun): For the second case, if we add the node to the dead node by mistake but it can quickly be probed and removed from dfsinputstream in Detector. So this should not be a major problem, right? > DeadNodeDetector checks dead node periodically > -- > > Key: HDFS-14651 > URL: https://issues.apache.org/jira/browse/HDFS-14651 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14651.001.patch, HDFS-14651.002.patch, > HDFS-14651.003.patch, HDFS-14651.004.patch, HDFS-14651.005.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14651) DeadNodeDetector checks dead node periodically
[
https://issues.apache.org/jira/browse/HDFS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16978193#comment-16978193
]
Yiqun Lin edited comment on HDFS-14651 at 11/20/19 8:47 AM:
{quote}
HDFS-14649 will add CHECK_SUSPICIOUS type. it is executed in different probe
thread pool by different probe type. so we need define this type.
{quote}
I see now we meet the error, we all add the node to the dead list. Is there any
other place(case) we will put into suspicious list? From my personal opinion,
only one dead node type is enough. Currently, I prefer not to add suspicious
state firstly and keep deadnode detection more clear and simple. How do you
think of this?
was (Author: linyiqun):
{quote}
HDFS-14649 will add CHECK_SUSPICIOUS type. it is executed in different probe
thread pool by different probe type. so we need define this type.
{quote}
I see now we meet the error, we all add the node to the dead list. Is there any
other place(case) we will put into suspicious list? From my personal opinion,
only one dead node type is enough. How do you think of this?
> DeadNodeDetector checks dead node periodically
> --
>
> Key: HDFS-14651
> URL: https://issues.apache.org/jira/browse/HDFS-14651
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14651.001.patch, HDFS-14651.002.patch,
> HDFS-14651.003.patch, HDFS-14651.004.patch, HDFS-14651.005.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14651) DeadNodeDetector checks dead node periodically
[ https://issues.apache.org/jira/browse/HDFS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977534#comment-16977534 ] Yiqun Lin edited comment on HDFS-14651 at 11/19/19 2:40 PM: Some comments for the variable/setting naming, so that it will look more readable. * probeThreadPool --> rpcThreadPool * dfs.client.deadnode.detection.dead.queue.max --> dfs.client.deadnode.detection.probe.queue.max, description: The max queue size of probing dead node. * dfs.client.deadnode.detection.probe.threads --> dfs.client.deadnode.detection.rpc.threads, description: The maximum number of threads to use for calling RPC call to recheck the liveness of dead node. * dfs.client.deadnode.detection.dead.threads --> dfs.client.deadnode.detection.probe.threads, description: The maximum number of threads to use for probing dead node. * dfs.client.deadnode.detection.dead.interval --> dfs.client.deadnode.detection.probe.interval.ms. description: Interval time in milliseconds for probing dead node behavior. was (Author: linyiqun): Some comments for the variable/setting naming, so that it will look more readable. * probeThreadPool --> rpcThreadPool * dfs.client.deadnode.detection.dead.queue.max --> dfs.client.deadnode.detection.probe.queue.max, description: The max queue size of probing dead node. * dfs.client.deadnode.detection.probe.threads --> dfs.client.deadnode.detection.rpc.threads, description: The maximum number of threads to use for calling RPC call to recheck the liveness of dead node. * dfs.client.deadnode.detection.dead.threads --> dfs.client.deadnode.detection.probe.threads, description: The maximum number of threads to use for probing dead node. * dfs.client.deadnode.detection.dead.interval --> dfs.client.deadnode.detection.interval.ms. description: Interval time in milliseconds for probing dead node behavior. > DeadNodeDetector checks dead node periodically > -- > > Key: HDFS-14651 > URL: https://issues.apache.org/jira/browse/HDFS-14651 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14651.001.patch, HDFS-14651.002.patch, > HDFS-14651.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14651) DeadNodeDetector checks dead node periodically
[
https://issues.apache.org/jira/browse/HDFS-14651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977420#comment-16977420
]
Yiqun Lin edited comment on HDFS-14651 at 11/19/19 11:58 AM:
-
Thanks for addressing the comment, [~leosun08]. Some review comments:
{code}
private static long CONNECTION_TIMEOUT_MS = 20 * 1000; // 20ms
long DFS_CLIENT_DEAD_NODE_DETECTION_INTERVAL_DEFAULT = 60 * 1000; // 60ms
{code}
Here the comment should be 20s, 60s.
{code}
this.deadNodesProbeQueue = new ArrayBlockingQueue(maxDeadNodesProbeQueueLen);
{code}
Can we update this to new
ArrayBlockingQueue(maxDeadNodesProbeQueueLen);
I think we can simplify this
{code}
+ private void checkDeadNodes() {
+...
+ for (DatanodeInfo datanodeInfo : datanodeInfos) {
+LOG.debug("Add dead node to check: {}.", datanodeInfo);
+if (isDeadNodesProbeQueueFull()) {
+ break;
+}
+addToDeadNodesProbeQueue(datanodeInfo);
+ ...
+ }
{code}
to
{code}
for (DatanodeInfo datanodeInfo : datanodeInfos) {
LOG.debug("Add dead node to check: {}.", datanodeInfo);
if(!addToDeadNodesProbeQueue(datanodeInfo)) {
LOG.debug("Skip to add dead node {} to check since the proboe
queue is full.", datanodeInfo);
break;
}
}
...
private boolean addToDeadNodesProbeQueue(DatanodeInfo datanodeInfo) {
return deadNodesProbeQueue.offer(datanodeInfo);
}
{code}
And following three methods can be removed
* addToDeadNodesProbeQueue
* isDeadNodesProbeQueueFull
* pollFromDeadNodesProbeQueue
{code}
private void removeFromDead(DatanodeInfo datanodeInfo) {
deadNodes.remove(datanodeInfo.getDatanodeUuid());
}
{code}
When we remove one dead node, can we also remove the dead node from
dfsInputStreamNodes and DFSInputStream local dead nodes? DFSInputStream local
dead will hold stale dead node but actually we check that this is not dead node
now.
{code}
+ /**
+ * Start probe dead node thread.
+ */
+ private void startProbeScheduler() {
+probeDeadNodesSchedulerThr =
+new Thread(new ProbeScheduler(this, ProbeType.CHECK_DEAD));
+probeDeadNodesSchedulerThr.setDaemon(true);
+probeDeadNodesSchedulerThr.start();
+ }
{code}
We should have corresponding close operations of detector and stop these
threads.
I noticed that here we only have one type CHECK_DEAD not other probe type like
suspicious node type? Do we still need to define this type? Seems it's okay
that we don't need to add suspicious state.
was (Author: linyiqun):
Thanks for addressing the comment, [~leosun08]. Some review comments:
{code}
private static long CONNECTION_TIMEOUT_MS = 20 * 1000; // 20ms
long DFS_CLIENT_DEAD_NODE_DETECTION_INTERVAL_DEFAULT = 60 * 1000; // 60ms
{code}
Here the comment should be 20s, 60s.
{code}
this.deadNodesProbeQueue = new ArrayBlockingQueue(maxDeadNodesProbeQueueLen);
{code}
Can we update this to new
ArrayBlockingQueue(maxDeadNodesProbeQueueLen);
I think we can simplify this
{code}
+ private void checkDeadNodes() {
+...
+ for (DatanodeInfo datanodeInfo : datanodeInfos) {
+LOG.debug("Add dead node to check: {}.", datanodeInfo);
+if (isDeadNodesProbeQueueFull()) {
+ break;
+}
+addToDeadNodesProbeQueue(datanodeInfo);
+ ...
+ }
{code}
to
{code}
for (DatanodeInfo datanodeInfo : datanodeInfos) {
LOG.debug("Add dead node to check: {}.", datanodeInfo);
if(!addToDeadNodesProbeQueue(datanodeInfo)) {
LOG.debug("Skip to add dead node {} to check since the proboe
queue is full.", datanodeInfo);
break;
}
}
{code}
And following three methods can be removed
* addToDeadNodesProbeQueue
* isDeadNodesProbeQueueFull
* pollFromDeadNodesProbeQueue
{code}
private void removeFromDead(DatanodeInfo datanodeInfo) {
deadNodes.remove(datanodeInfo.getDatanodeUuid());
}
{code}
When we remove one dead node, can we also remove the dead node from
dfsInputStreamNodes and DFSInputStream local dead nodes? DFSInputStream local
dead will hold stale dead node but actually we check that this is not dead node
now.
{code}
+ /**
+ * Start probe dead node thread.
+ */
+ private void startProbeScheduler() {
+probeDeadNodesSchedulerThr =
+new Thread(new ProbeScheduler(this, ProbeType.CHECK_DEAD));
+probeDeadNodesSchedulerThr.setDaemon(true);
+probeDeadNodesSchedulerThr.start();
+ }
{code}
We should have corresponding close operations of detector and stop these
threads.
I noticed that here we only have one type CHECK_DEAD not other probe type like
suspicious node type? Do we still need to define this type? Seems it's okay
that we don't need to add suspicious state.
> DeadNodeDetector checks dead node periodically
> --
>
> Key: HDFS-14651
>
