ConfX created HDFS-17860:
----------------------------
Summary: NPE in BlocksMap After NameNode Concat and Restart
Key: HDFS-17860
URL: https://issues.apache.org/jira/browse/HDFS-17860
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.3.5
Reporter: ConfX
Attachments: reproduce.sh, restart.patch
## Overview
A NullPointerException occurs in `BlocksMap.numNodes()` when calling
`getBlockLocations()` on a file that has been created via `concat()` operation,
after a NameNode restart. This is a critical production bug that could crash
the NameNode during normal file read operations.
---
## Reproduction
### Quick Start with Bash Script
**Use the provided `reproduce.sh` script to automatically reproduce this bug:**
This script automates the entire reproduction process by cloning Hadoop 3.3.5,
applying the test patch, building the project, and running the failing test
case.
```bash
cd bug1
./reproduce.sh
```
**What the script does:**
1. Clones the Hadoop repository (release 3.3.5 branch)
2. Applies the test patch (`restart.patch`) that adds the reproduction test
3. Builds the Hadoop HDFS module
4. Runs the test case `TestHDFSConcat#testConcatWithRestart` which demonstrates
the NullPointerException
The bug is confirmed if the test fails with a `NullPointerException` in
`BlocksMap.numNodes()`.
---
### Manual Reproduction Steps
If you prefer to run the test manually:
```bash
mvn surefire:test
-Dtest=TestHDFSConcat_RestartInjected#testConcat_AfterConcat_NN_Crash
```
### Test Scenario
1. Create a target file (`/trg`) with 3 blocks (512 bytes each)
2. Create 10 source files, each with 3 blocks
3. Call `dfs.concat(trgPath, files)` to concatenate all source files into target
4. Restart the NameNode
5. Call `nn.getBlockLocations(trg, 0, trgLen)` on the concatenated file
6. **NPE occurs** at `BlocksMap.numNodes()`
---
## Stack Trace
```
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.numNodes(BlocksMap.java:172)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1420)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1382)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:1353)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1503)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:179)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2124)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:769)
```
---
## Root Cause Analysis
### Code Location
**File:** `BlocksMap.java:172`
```java
int numNodes(Block b) {
BlockInfo info = blocks.get(b); // LINE 172 - NPE HERE
return info == null ? 0 : info.numNodes();
}
```
### Why the NPE Occurs
The NPE happens because a **null** `Block` parameter `b` is being passed to
`numNodes()`. When `b` is null, calling `blocks.get(null)` throws a
`NullPointerException` because:
1. The `blocks` map is likely a `LightWeightGSet` or similar hash map
implementation
2. The `get()` method calls `b.hashCode()` for lookup
3. Calling `hashCode()` on null throws NPE
### How Null Blocks Enter the System
#### Debug Investigation
I added debug logging to `BlockManager.createLocatedBlockList()` to inspect the
blocks array:
```java
LOG.info("RESTART_DEBUG: createLocatedBlockList called with blocks.length=" +
blocks.length);
for (int i = 0; i < blocks.length; i++) {
if (blocks[i] == null) {
LOG.error("RESTART_DEBUG: blocks[" + i + "] is NULL!");
}
}
```
**Finding:** The blocks array itself does **NOT** contain null elements after
restart. All 33 blocks in the concatenated file's blocks array are non-null
BlockInfo objects.
#### Hypothesis: Stale Block References
After concat, the file's INode contains blocks from the original target file
PLUS all blocks from the concatenated source files. The test logs show:
- Before restart: Each source file has 3 blocks (10 files × 3 = 30 blocks)
- After concat: Target file should have 3 + 30 = 33 blocks
- After restart: `blocks.length=33` - all blocks present
However, based on the stack trace and NPE location, the issue likely stems from:
1. **Concat operation** moves blocks from source files to target file
2. Source files are deleted after concat
3. Block metadata for source files may be marked for deletion/invalidation
4. **After restart**, when NameNode reloads FSImage/EditLog:
- The concatenated file's INode correctly references all 33 blocks
- BUT some blocks may have been removed from the `BlocksMap` during fsimage
load
- Or blocks from deleted source files were never added to BlocksMap
5. When `createLocatedBlock()` is called with one of these "ghost" block
references:
- The BlockInfo object exists in the file's blocks array
- But the actual Block/BlockInfo lookup in BlocksMap fails or returns
inconsistent state
- A null block reference propagates to `numNodes()`
---
## Evidence
### Test Log Analysis
**Before Restart (all blocks non-null):**
```
2025-12-01 12:44:06,136 [Time-limited test] INFO BlockManager -
RESTART_DEBUG: createLocatedBlockList called with blocks.length=3, offset=0,
length=1536
blocks[0] = blk_1073741825_1001, numBytes=512
blocks[1] = blk_1073741826_1002, numBytes=512
blocks[2] = blk_1073741827_1003, numBytes=512
```
**After Restart (file now has 33 blocks from concat, all non-null):**
```
2025-12-01 12:44:15,034 [IPC Server handler 0] INFO BlockManager -
RESTART_DEBUG: createLocatedBlockList called with blocks.length=33, offset=0,
length=5120
blocks[0] = blk_1073741825_1001, numBytes=512
blocks[1] = blk_1073741826_1002, numBytes=512
blocks[2] = blk_1073741827_1003, numBytes=512
blocks[3] = blk_1073741855_1031, numBytes=512
... (all 33 blocks non-null)
blocks[32] = blk_1073741830_1006, numBytes=512
```
**Then NPE occurs** - suggesting the null block comes from a code path not
instrumented by my debug logging, or there's a race condition during BlocksMap
access.
---
## Likely Bug Location
### Suspect Area 1: Concat Implementation
The concat operation (in `FSNamesystem` or `FSDirectory`) may not properly
handle block ownership transfer during the transaction that gets persisted to
edit log. On restart:
- FSImage loading might not correctly restore all blocks to BlocksMap
- Blocks from deleted source files might be in a transitional state
### Suspect Area 2: BlockInfo Reference vs BlocksMap Inconsistency
There may be a race or ordering issue where:
1. INodeFile's blocks array references BlockInfo objects
2. These BlockInfo objects are not yet added to (or have been removed from) the
global BlocksMap
3. When `createLocatedBlock()` tries to look up block locations, it accesses a
BlockInfo that's not in the map
### Code Path to NPE
```
getBlockLocations()
→ createLocatedBlocks()
→ createLocatedBlockList()
→ createLocatedBlock(blocks[curBlk], ...) // blocks[curBlk] might be
problematic
→ createLocatedBlock(blk, ...)
→ blocksMap.numNodes(blk) // NPE if blk is somehow null or invalid
```
---
## Impact
1. **NameNode Crash Risk:** NPE in RPC handler can crash the NameNode during
client `getBlockLocations()` calls
2. **Data Availability:** Files created via `concat()` become unreadable after
NameNode restart
3. **Silent Corruption:** The concat operation appears to succeed, but the file
is broken after restart
### Affected Operations
- Any `getBlockLocations()` call on concatenated files after restart
- File reads (since clients call getBlockLocations)
- MapReduce/Spark jobs reading concatenated files
- Backup/replication tools accessing these files
---
## Recommended Fix
### Immediate Mitigation
Add null-safety check in `BlocksMap.numNodes()`:
```java
int numNodes(Block b) {
if (b == null) {
LOG.error("Null block passed to numNodes()!", new Exception("Stack trace"));
return 0; // Or throw IOException
}
BlockInfo info = blocks.get(b);
return info == null ? 0 : info.numNodes();
}
```
And in `BlockManager.createLocatedBlock()`:
```java
private LocatedBlock createLocatedBlock(LocatedBlockBuilder locatedBlocks,
final BlockInfo blk, final long pos, final AccessMode mode)
throws IOException {
if (blk == null) {
LOG.error("Null block in createLocatedBlock at pos=" + pos, new
Exception());
throw new IOException("Null block reference in file's block list");
}
// ... rest of method
}
```
Please let me know if you think there could be a better fix and I'm happy to
discuss!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
