[ https://issues.apache.org/jira/browse/HDFS-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17839929#comment-17839929 ]
ASF GitHub Bot commented on HDFS-17488: --------------------------------------- hadoop-yetus commented on PR #6759: URL: https://github.com/apache/hadoop/pull/6759#issuecomment-2071375244 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |:----:|----------:|--------:|:--------:|:-------:| |||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 02s | | No case conflicting files found. | | +0 :ok: | spotbugs | 0m 00s | | spotbugs executables are not available. | | +0 :ok: | codespell | 0m 00s | | codespell was not available. | | +0 :ok: | detsecrets | 0m 01s | | detect-secrets was not available. | | +1 :green_heart: | @author | 0m 00s | | The patch does not contain any @author tags. | | +1 :green_heart: | test4tests | 0m 00s | | The patch appears to include 1 new or modified test files. | |||| _ trunk Compile Tests _ | | +1 :green_heart: | mvninstall | 118m 49s | | trunk passed | | +1 :green_heart: | compile | 8m 15s | | trunk passed | | +1 :green_heart: | checkstyle | 6m 34s | | trunk passed | | +1 :green_heart: | mvnsite | 8m 28s | | trunk passed | | +1 :green_heart: | javadoc | 7m 31s | | trunk passed | | +1 :green_heart: | shadedclient | 190m 44s | | branch has no errors when building and testing our client artifacts. | |||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 6m 57s | | the patch passed | | +1 :green_heart: | compile | 4m 54s | | the patch passed | | +1 :green_heart: | javac | 4m 54s | | the patch passed | | +1 :green_heart: | blanks | 0m 00s | | The patch has no blanks issues. | | +1 :green_heart: | checkstyle | 3m 25s | | the patch passed | | +1 :green_heart: | mvnsite | 6m 07s | | the patch passed | | +1 :green_heart: | javadoc | 4m 56s | | the patch passed | | +1 :green_heart: | shadedclient | 207m 49s | | patch has no errors when building and testing our client artifacts. | |||| _ Other Tests _ | | +1 :green_heart: | asflicense | 9m 28s | | The patch does not generate ASF License warnings. | | | | 557m 52s | | | | Subsystem | Report/Notes | |----------:|:-------------| | GITHUB PR | https://github.com/apache/hadoop/pull/6759 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets | | uname | MINGW64_NT-10.0-17763 9dbcaf4dc965 3.4.10-87d57229.x86_64 2024-02-14 20:17 UTC x86_64 Msys | | Build tool | maven | | Personality | /c/hadoop/dev-support/bin/hadoop.sh | | git revision | trunk / d5a87e15db1c70fb0c3974832c75504ff184b0d1 | | Default Java | Azul Systems, Inc.-1.8.0_332-b09 | | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6759/1/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch-windows-10/job/PR-6759/1/console | | versions | git=2.44.0.windows.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > DN can fail IBRs with NPE when a volume is removed > -------------------------------------------------- > > Key: HDFS-17488 > URL: https://issues.apache.org/jira/browse/HDFS-17488 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs > Reporter: Felix N > Assignee: Felix N > Priority: Major > Labels: pull-request-available > > > Error logs > {code:java} > 2024-04-22 15:46:33,422 [BP-1842952724-10.22.68.249-1713771988830 > heartbeating to localhost/127.0.0.1:64977] ERROR datanode.DataNode > (BPServiceActor.java:run(922)) - Exception in BPOfferService for Block pool > BP-1842952724-10.22.68.249-1713771988830 (Datanode Uuid > 1659ffaf-1a80-4a8e-a542-643f6bd97ed4) service to localhost/127.0.0.1:64977 > java.lang.NullPointerException > at > org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReceivedAndDeleted(DatanodeProtocolClientSideTranslatorPB.java:246) > at > org.apache.hadoop.hdfs.server.datanode.IncrementalBlockReportManager.sendIBRs(IncrementalBlockReportManager.java:218) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:749) > at > org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:920) > at java.lang.Thread.run(Thread.java:748) {code} > The root cause is in BPOfferService#notifyNamenodeBlock, happens when it's > called on a block belonging to a volume already removed prior. Because the > volume was already removed > > {code:java} > private void notifyNamenodeBlock(ExtendedBlock block, BlockStatus status, > String delHint, String storageUuid, boolean isOnTransientStorage) { > checkBlock(block); > final ReceivedDeletedBlockInfo info = new ReceivedDeletedBlockInfo( > block.getLocalBlock(), status, delHint); > final DatanodeStorage storage = dn.getFSDataset().getStorage(storageUuid); > > // storage == null here because it's already removed earlier. > for (BPServiceActor actor : bpServices) { > actor.getIbrManager().notifyNamenodeBlock(info, storage, > isOnTransientStorage); > } > } {code} > so IBRs with a null storage are now pending. > The reason why notifyNamenodeBlock can trigger on such blocks is up in > DirectoryScanner#reconcile > {code:java} > public void reconcile() throws IOException { > LOG.debug("reconcile start DirectoryScanning"); > scan(); > // If a volume is removed here after scan() already finished running, > // diffs is stale and checkAndUpdate will run on a removed volume > // HDFS-14476: run checkAndUpdate with batch to avoid holding the lock too > // long > int loopCount = 0; > synchronized (diffs) { > for (final Map.Entry<String, ScanInfo> entry : diffs.getEntries()) { > dataset.checkAndUpdate(entry.getKey(), entry.getValue()); > ... > } {code} > Inside checkAndUpdate, memBlockInfo is null because all the block meta in > memory is removed during the volume removal, but diskFile still exists. Then > DataNode#notifyNamenodeDeletedBlock (and further down the line, > notifyNamenodeBlock) is called on this block. > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org