Prabhu Joseph created HDFS-16676: ------------------------------------ Summary: DatanodeAdminManager$Monitor reports a node as invalid forever Key: HDFS-16676 URL: https://issues.apache.org/jira/browse/HDFS-16676 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.2.1 Reporter: Prabhu Joseph Assignee: Prabhu Joseph
DatanodeAdminManager$Monitor reports a node as invalid forever {code} 2022-07-21 06:54:38,562 WARN org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager (DatanodeAdminMonitor-0): DatanodeAdminMonitor caught exception when processing node 1.2.3.4:9866. java.lang.IllegalStateException: Node 1.2.3.4:9866 is in an invalid state! Invalid state: In Service 0 blocks are on this dn. at com.google.common.base.Preconditions.checkState(Preconditions.java:172) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager$Monitor.check(DatanodeAdminManager.java:601) at org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager$Monitor.run(DatanodeAdminManager.java:504) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) {code} A node goes into invalid state when stopDecommission sets the node to IN-Service and misses to remove from pendingNodes queues (HDFS-16675). This will be corrected only when user triggers startDecommission. Till then we need not keep the invalid state node in the queue as anyway startDecommission will add it back. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org