[
https://issues.apache.org/jira/browse/HDFS-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17571193#comment-17571193
]
Wei-Chiu Chuang commented on HDFS-16675:
----------------------------------------
[~sodonnell]
Is this fixed by your HDFS-16583? They look similar but different.
> DatanodeAdminManager.stopDecommission fails with
> ConcurrentModificationException
> --------------------------------------------------------------------------------
>
> Key: HDFS-16675
> URL: https://issues.apache.org/jira/browse/HDFS-16675
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
> Affects Versions: 3.2.1
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
>
> DatanodeAdminManager.stopDecommission intermittently fails with
> ConcurrentModificationException.
> {code}
> java.util.ConcurrentModificationException: Tree has been modified outside of
> iterator
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.checkForModification(FoldedTreeSet.java:311)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.hasNext(FoldedTreeSet.java:256)
> at
> org.apache.hadoop.hdfs.util.FoldedTreeSet$TreeSetIterator.next(FoldedTreeSet.java:262)
> at
> java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1044)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processExtraRedundancyBlocksOnInService(BlockManager.java:4300)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager.stopDecommission(DatanodeAdminManager.java:241)
> {code}
> This is intermittently seen on busy cluster with AutoScale Enabled. This
> leads to a Node with state "IN Service" in pendingNodes queue
> {code}
> 2022-07-21 06:54:38,562 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager
> (DatanodeAdminMonitor-0): DatanodeAdminMonitor caught exception when
> processing node 1.2.3.4:9866.
> java.lang.IllegalStateException: Node 1.2.3.4:9866 is in an invalid state!
> Invalid state: In Service 0 blocks are on this dn.
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:172)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager$Monitor.check(DatanodeAdminManager.java:601)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminManager$Monitor.run(DatanodeAdminManager.java:504)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
