[ https://issues.apache.org/jira/browse/HDFS-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571193#comment-17571193 ]
Wei-Chiu Chuang commented on HDFS-16675: ---------------------------------------- [~sodonnell] Is this fixed by your HDFS-16583? They look similar but different. > DatanodeAdminManager.stopDecommission fails with > ConcurrentModificationException > -------------------------------------------------------------------------------- > > Key: HDFS-16675 > URL: https://issues.apache.org/jira/browse/HDFS-16675 > Project: Hadoop HDFS > Issue Type: Bug > Components: namanode > Affects Versions: 3.2.1 > Reporter: Prabhu Joseph > Assignee: Prabhu Joseph > Priority: Major > > DatanodeAdminManager.stopDecommission intermittently fails with > ConcurrentModificationException. > {code} > java.util.ConcurrentModificationException: Tree has been modified outside of > iterator > at > org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.checkForModification(FoldedTreeSet.java:311) > at > org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.hasNext(FoldedTreeSet.java:256) > at > org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.next(FoldedTreeSet.java:262) > at > java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processExtraRedundancyBlocksOnInService(BlockManager.java:4300) > at > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager.stopDecommission(DatanodeAdminManager.java:241) > {code} > This is intermittently seen on busy cluster with AutoScale Enabled. This > leads to a Node with state "IN Service" in pendingNodes queue > {code} > 2022-07-21 06:54:38,562 WARN > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager > (DatanodeAdminMonitor-0): DatanodeAdminMonitor caught exception when > processing node 1.2.3.4:9866. > java.lang.IllegalStateException: Node 1.2.3.4:9866 is in an invalid state! > Invalid state: In Service 0 blocks are on this dn. > at > com.google.common.base.Preconditions.checkState(Preconditions.java:172) > at > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager$Monitor.check(DatanodeAdminManager.java:601) > at > org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager$Monitor.run(DatanodeAdminManager.java:504) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:750) > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org