[ https://issues.apache.org/jira/browse/HDFS-17564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17862973#comment-17862973 ]
ASF GitHub Bot commented on HDFS-17564: --------------------------------------- hadoop-yetus commented on PR #6911: URL: https://github.com/apache/hadoop/pull/6911#issuecomment-2208471652 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |:----:|----------:|--------:|:--------:|:-------:| | +0 :ok: | reexec | 14m 11s | | Docker mode activated. | |||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. | | +0 :ok: | codespell | 0m 0s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 0s | | The patch appears to include 1 new or modified test files. | |||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 45m 5s | | trunk passed | | +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | | +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | | +1 :green_heart: | checkstyle | 1m 11s | | trunk passed | | +1 :green_heart: | mvnsite | 1m 25s | | trunk passed | | +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | | +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | | +1 :green_heart: | spotbugs | 3m 20s | | trunk passed | | +1 :green_heart: | shadedclient | 35m 43s | | branch has no errors when building and testing our client artifacts. | |||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 1m 11s | | the patch passed | | +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | | +1 :green_heart: | javac | 1m 14s | | the patch passed | | +1 :green_heart: | compile | 1m 5s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | | +1 :green_heart: | javac | 1m 5s | | the patch passed | | +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 0m 58s | | the patch passed | | +1 :green_heart: | mvnsite | 1m 12s | | the patch passed | | +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 | | +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | | +1 :green_heart: | spotbugs | 3m 10s | | the patch passed | | +1 :green_heart: | shadedclient | 35m 56s | | patch has no errors when building and testing our client artifacts. | |||| _ Other Tests _ | | -1 :x: | unit | 227m 36s | [/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6911/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt) | hadoop-hdfs in the patch passed. | | +1 :green_heart: | asflicense | 0m 54s | | The patch does not generate ASF License warnings. | | | | 380m 17s | | | | Reason | Tests | |-------:|:------| | Failed junit tests | hadoop.hdfs.server.blockmanagement.TestDatanodeManager | | Subsystem | Report/Notes | |----------:|:-------------| | Docker | ClientAPI=1.46 ServerAPI=1.46 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6911/3/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/6911 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | Linux ea2255be361b 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/bin/hadoop.sh | | git revision | trunk / a4aaa06477dcd8603c70654dd5d63fd566507b40 | | Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6911/3/testReport/ | | Max. process+thread count | 4184 (vs. ulimit of 5500) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6911/3/console | | versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > Erasure Coding: Fix the issue of inaccurate metrics when decommission mark > busy DN > ---------------------------------------------------------------------------------- > > Key: HDFS-17564 > URL: https://issues.apache.org/jira/browse/HDFS-17564 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Haiyang Hu > Assignee: Haiyang Hu > Priority: Major > Labels: pull-request-available > > If DataNode is marked as busy and contains many EC blocks, when running > decommission DataNode, when execute ErasureCodingWork#addTaskToDatanode, here > will no replication work will be generated for ecBlocksToBeReplicated, but > related metrics (such as DatanodeDescriptor#currApproxBlocksScheduled, > pendingReconstruction and needReconstruction) will still updated. > *Specific codeļ¼* > BlockManager#scheduleReconstruction -> BlockManager#chooseSourceDatanodes > [2628~2650] > If DataNode is marked as busy and contains many EC blocks here will not add > to srcNodes. > . > {code:java} > @VisibleForTesting > DatanodeDescriptor[] chooseSourceDatanodes(BlockInfo block, > List<DatanodeDescriptor> containingNodes, > List<DatanodeStorageInfo> nodesContainingLiveReplicas, > NumberReplicas numReplicas, List<Byte> liveBlockIndices, > List<Byte> liveBusyBlockIndices, List<Byte> excludeReconstructed, int > priority) { > containingNodes.clear(); > nodesContainingLiveReplicas.clear(); > List<DatanodeDescriptor> srcNodes = new ArrayList<>(); > ... > for (DatanodeStorageInfo storage : blocksMap.getStorages(block)) { > final DatanodeDescriptor node = getDatanodeDescriptorFromStorage(storage); > final StoredReplicaState state = checkReplicaOnStorage(numReplicas, block, > storage, corruptReplicas.getNodes(block), false); > ... > // for EC here need to make sure the numReplicas replicates state correct > // because in the scheduleReconstruction it need the numReplicas to check > // whether need to reconstruct the ec internal block > byte blockIndex = -1; > if (isStriped) { > blockIndex = ((BlockInfoStriped) block) > .getStorageBlockIndex(storage); > countLiveAndDecommissioningReplicas(numReplicas, state, > liveBitSet, decommissioningBitSet, blockIndex); > } > if (priority != LowRedundancyBlocks.QUEUE_HIGHEST_PRIORITY > && (!node.isDecommissionInProgress() && !node.isEnteringMaintenance()) > && node.getNumberOfBlocksToBeReplicated() + > node.getNumberOfBlocksToBeErasureCoded() >= maxReplicationStreams) { > if (isStriped && (state == StoredReplicaState.LIVE > || state == StoredReplicaState.DECOMMISSIONING)) { > liveBusyBlockIndices.add(blockIndex); > //HDFS-16566 ExcludeReconstructed won't be reconstructed. > excludeReconstructed.add(blockIndex); > } > continue; // already reached replication limit > } > if (node.getNumberOfBlocksToBeReplicated() + > node.getNumberOfBlocksToBeErasureCoded() >= > replicationStreamsHardLimit) { > if (isStriped && (state == StoredReplicaState.LIVE > || state == StoredReplicaState.DECOMMISSIONING)) { > liveBusyBlockIndices.add(blockIndex); > //HDFS-16566 ExcludeReconstructed won't be reconstructed. > excludeReconstructed.add(blockIndex); > } > continue; > } > if(isStriped || srcNodes.isEmpty()) { > srcNodes.add(node); > if (isStriped) { > liveBlockIndices.add(blockIndex); > } > continue; > } > ... > {code} > ErasureCodingWork#addTaskToDatanode[149~157] > {code:java} > @Override > void addTaskToDatanode(NumberReplicas numberReplicas) { > final DatanodeStorageInfo[] targets = getTargets(); > assert targets.length > 0; > BlockInfoStriped stripedBlk = (BlockInfoStriped) getBlock(); > ... > } else if ((numberReplicas.decommissioning() > 0 || > numberReplicas.liveEnteringMaintenanceReplicas() > 0) && > hasAllInternalBlocks()) { > List<Integer> leavingServiceSources = findLeavingServiceSources(); > // decommissioningSources.size() should be >= targets.length > // if the leavingServiceSources size is 0, here will not to > createReplicationWork > final int num = Math.min(leavingServiceSources.size(), targets.length); > for (int i = 0; i < num; i++) { > createReplicationWork(leavingServiceSources.get(i), targets[i]); > } > ... > } > // Since there is no decommission busy datanode in srcNodes, here return the > set size of srcIndices as 0. > private List<Integer> findLeavingServiceSources() { > // Mark the block in normal node. > BlockInfoStriped block = (BlockInfoStriped)getBlock(); > BitSet bitSet = new BitSet(block.getRealTotalBlockNum()); > for (int i = 0; i < getSrcNodes().length; i++) { > if (getSrcNodes()[i].isInService()) { > bitSet.set(liveBlockIndices[i]); > } > } > // If the block is on the node which is decommissioning or > // entering_maintenance, and it doesn't exist on other normal nodes, > // we just add the node into source list. > List<Integer> srcIndices = new ArrayList<>(); > for (int i = 0; i < getSrcNodes().length; i++) { > if ((getSrcNodes()[i].isDecommissionInProgress() || > (getSrcNodes()[i].isEnteringMaintenance() && > getSrcNodes()[i].isAlive())) && > !bitSet.get(liveBlockIndices[i])) { > srcIndices.add(i); > } > } > return srcIndices; > } > {code} > so we need to fix this logic to avoid inaccurate metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org