[
https://issues.apache.org/jira/browse/HDFS-17860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ConfX updated HDFS-17860:
-------------------------
Description:
h2. Overview
A NullPointerException occurs in `BlocksMap.numNodes()` when calling
`getBlockLocations()` on a file that has been created via `concat()` operation,
after a NameNode restart. This is a critical production bug that could crash
the NameNode during normal file read operations.
h2. Reproduction
h3. Quick Start with Bash Script
Use the provided `reproduce.sh` script to automatically reproduce this bug:
This script automates the entire reproduction process by cloning Hadoop 3.3.5,
applying the test patch, building the project, and running the failing test
case.
{code:java}
cd bug1
./reproduce.sh{code}
*What the script does:*
1. Clones the Hadoop repository (release 3.3.5 branch)
2. Applies the test patch (`restart.patch`) that adds the reproduction test
3. Builds the Hadoop HDFS module
4. Runs the test case `TestHDFSConcat#testConcatWithRestart` which demonstrates
the NullPointerException
The bug is confirmed if the test fails with a `NullPointerException` in
`BlocksMap.numNodes()`.
*Manual Reproduction Steps:*
If you prefer to run the test manually:
{code:java}
mvn surefire:test
-Dtest=TestHDFSConcat_RestartInjected#testConcat_AfterConcat_NN_Crash{code}
*Test Scenario*
1. Create a target file (`/trg`) with 3 blocks (512 bytes each)
2. Create 10 source files, each with 3 blocks
3. Call `dfs.concat(trgPath, files)` to concatenate all source files into target
4. Restart the NameNode
5. Call `nn.getBlockLocations(trg, 0, trgLen)` on the concatenated file
6. NPE occurs at `BlocksMap.numNodes()`
*Stack Trace*
{code:java}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.numNodes(BlocksMap.java:172)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1420)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1382)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:1353)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1503)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:179)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2124)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:769){code}
h2. Root Cause Analysis
Code Location
File: `BlocksMap.java:172`
{code:java}
int numNodes(Block b) {
BlockInfo info = blocks.get(b); // LINE 172 - NPE HERE
return info == null ? 0 : info.numNodes();
} {code}
*Why the NPE Occurs*
The NPE happens because a null `Block` parameter `b` is being passed to
`numNodes()`. When `b` is null, calling `blocks.get(null)` throws a
`NullPointerException` because:
1. The `blocks` map is likely a `LightWeightGSet` or similar hash map
implementation
2. The `get()` method calls `b.hashCode()` for lookup
3. Calling `hashCode()` on null throws NPE
*How Null Blocks Enter the System*
*Debug Investigation*
I added debug logging to `BlockManager.createLocatedBlockList()` to inspect the
blocks array:
{code:java}
LOG.info("RESTART_DEBUG: createLocatedBlockList called with blocks.length=" +
blocks.length);
for (int i = 0; i < blocks.length; i++) {
if (blocks[i] == null) {
LOG.error("RESTART_DEBUG: blocks[" + i + "] is NULL!");
}
}{code}
{*}Finding{*}: The blocks array itself does NOT contain null elements after
restart. All 33 blocks in the concatenated file's blocks array are non-null
BlockInfo objects.
Hypothesis: Stale Block References
After concat, the file's INode contains blocks from the original target file
PLUS all blocks from the concatenated source files. The test logs show:
- Before restart: Each source file has 3 blocks (10 files × 3 = 30 blocks)
- After concat: Target file should have 3 + 30 = 33 blocks
- After restart: `blocks.length=33` - all blocks present
However, based on the stack trace and NPE location, the issue likely stems from:
1. Concat operation moves blocks from source files to target file
2. Source files are deleted after concat
3. Block metadata for source files may be marked for deletion/invalidation
4. After restart, when NameNode reloads FSImage/EditLog:
- The concatenated file's INode correctly references all 33 blocks
- BUT some blocks may have been removed from the `BlocksMap` during fsimage
load
- Or blocks from deleted source files were never added to BlocksMap
5. When `createLocatedBlock()` is called with one of these "ghost" block
references:
- The BlockInfo object exists in the file's blocks array
- But the actual Block/BlockInfo lookup in BlocksMap fails or returns
inconsistent state
- A null block reference propagates to `numNodes()`
h3. Evidence
Test Log Analysis
*Before Restart (all blocks non-null):*
```
2025-12-01 12:44:06,136 [Time-limited test] INFO BlockManager -
RESTART_DEBUG: createLocatedBlockList called with blocks.length=3, offset=0,
length=1536
blocks[0] = blk_1073741825_1001, numBytes=512
blocks[1] = blk_1073741826_1002, numBytes=512
blocks[2] = blk_1073741827_1003, numBytes=512
```
*After Restart (file now has 33 blocks from concat, all non-null):*
```
2025-12-01 12:44:15,034 [IPC Server handler 0] INFO BlockManager -
RESTART_DEBUG: createLocatedBlockList called with blocks.length=33, offset=0,
length=5120
blocks[0] = blk_1073741825_1001, numBytes=512
blocks[1] = blk_1073741826_1002, numBytes=512
blocks[2] = blk_1073741827_1003, numBytes=512
blocks[3] = blk_1073741855_1031, numBytes=512
... (all 33 blocks non-null)
blocks[32] = blk_1073741830_1006, numBytes=512
```
*Then NPE occurs* - suggesting the null block comes from a code path not
instrumented by my debug logging, or there's a race condition during BlocksMap
access.
h3. *Likely Bug Location*
*Suspect Area 1: Concat Implementation*
The concat operation (in `FSNamesystem` or `FSDirectory`) may not properly
handle block ownership transfer during the transaction that gets persisted to
edit log. On restart:
- FSImage loading might not correctly restore all blocks to BlocksMap
- Blocks from deleted source files might be in a transitional state
*Suspect Area 2: BlockInfo Reference vs BlocksMap Inconsistency*
There may be a race or ordering issue where:
1. INodeFile's blocks array references BlockInfo objects
2. These BlockInfo objects are not yet added to (or have been removed from) the
global BlocksMap
3. When `createLocatedBlock()` tries to look up block locations, it accesses a
BlockInfo that's not in the map
*Code Path to NPE*
{code:java}
getBlockLocations()
→ createLocatedBlocks()
→ createLocatedBlockList()
→ createLocatedBlock(blocks[curBlk], ...) // blocks[curBlk] might be
problematic
→ createLocatedBlock(blk, ...)
→ blocksMap.numNodes(blk) // NPE if blk is somehow null or
invalid{code}
h2. Impact
Production Impact:
1. NameNode Crash Risk: NPE in RPC handler can crash the NameNode during client
`getBlockLocations()` calls
2. Data Availability: Files created via `concat()` become unreadable after
NameNode restart
3. Silent Corruption: The concat operation appears to succeed, but the file is
broken after restart
Affected Operations
- Any `getBlockLocations()` call on concatenated files after restart
- File reads (since clients call getBlockLocations)
- MapReduce/Spark jobs reading concatenated files
- Backup/replication tools accessing these files
h2.
Recommended Fix
*Immediate Mitigation*
Add null-safety check in `BlocksMap.numNodes()`:
{code:java}
int numNodes(Block b) {
if (b == null) {
LOG.error("Null block passed to numNodes()!", new Exception("Stack trace"));
return 0; // Or throw IOException
}
BlockInfo info = blocks.get(b);
return info == null ? 0 : info.numNodes();
} {code}
And in `BlockManager.createLocatedBlock()`:
{code:java}
private LocatedBlock createLocatedBlock(LocatedBlockBuilder locatedBlocks,
final BlockInfo blk, final long pos, final AccessMode mode)
throws IOException {
if (blk == null) {
LOG.error("Null block in createLocatedBlock at pos=" + pos, new
Exception());
throw new IOException("Null block reference in file's block list");
}
// ... rest of method
} {code}
*Root Cause Fix*
Requires deeper investigation:
1. Audit concat implementation to ensure all blocks are properly:
- Added to target file's INode
- Registered in BlocksMap
- Persisted correctly in edit log
- Loaded correctly from FSImage/EditLog on restart
2. Check FSImage/EditLog loading for concat transactions:
- Verify blocks from concatenated files are added to BlocksMap
- Ensure proper ordering of operations during replay
- Check for race conditions in block map population
3. Add consistency checks during NameNode startup:
- Verify all blocks referenced by INodes exist in BlocksMap
- Log warnings for orphaned block references
- Option to auto-repair or fail-safe mode
*Related Code Files to Investigate*
1. `BlocksMap.java` - Block storage map
2. `BlockManager.java` - Block management and location services
3. `FSNamesystem.java` / `FSDirConcatOp.java` - Concat operation implementation
4. `FSImageFormat.java` / `FSEditLog.java` - Persistence and loading
5. `INodeFile.java` - File inode and block array management
was:
h2. Overview
A NullPointerException occurs in `BlocksMap.numNodes()` when calling
`getBlockLocations()` on a file that has been created via `concat()` operation,
after a NameNode restart. This is a critical production bug that could crash
the NameNode during normal file read operations.
h2. Reproduction
h3. Quick Start with Bash Script
Use the provided `reproduce.sh` script to automatically reproduce this bug:
This script automates the entire reproduction process by cloning Hadoop 3.3.5,
applying the test patch, building the project, and running the failing test
case.
{code:java}
cd bug1
./reproduce.sh{code}
*What the script does:*
1. Clones the Hadoop repository (release 3.3.5 branch)
2. Applies the test patch (`restart.patch`) that adds the reproduction test
3. Builds the Hadoop HDFS module
4. Runs the test case `TestHDFSConcat#testConcatWithRestart` which demonstrates
the NullPointerException
The bug is confirmed if the test fails with a `NullPointerException` in
`BlocksMap.numNodes()`.
*Manual Reproduction Steps:*
If you prefer to run the test manually:
{code:java}
mvn surefire:test
-Dtest=TestHDFSConcat_RestartInjected#testConcat_AfterConcat_NN_Crash{code}
*Test Scenario*
1. Create a target file (`/trg`) with 3 blocks (512 bytes each)
2. Create 10 source files, each with 3 blocks
3. Call `dfs.concat(trgPath, files)` to concatenate all source files into target
4. Restart the NameNode
5. Call `nn.getBlockLocations(trg, 0, trgLen)` on the concatenated file
6. NPE occurs at `BlocksMap.numNodes()`
*Stack Trace*
{code:java}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.numNodes(BlocksMap.java:172)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1420)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1382)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:1353)
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1503)
at
org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:179)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2124)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:769){code}
h2. Root Cause Analysis
Code Location
File: `BlocksMap.java:172`
{code:java}
int numNodes(Block b) {
BlockInfo info = blocks.get(b); // LINE 172 - NPE HERE
return info == null ? 0 : info.numNodes();
} {code}
*Why the NPE Occurs*
The NPE happens because a null `Block` parameter `b` is being passed to
`numNodes()`. When `b` is null, calling `blocks.get(null)` throws a
`NullPointerException` because:
1. The `blocks` map is likely a `LightWeightGSet` or similar hash map
implementation
2. The `get()` method calls `b.hashCode()` for lookup
3. Calling `hashCode()` on null throws NPE
*How Null Blocks Enter the System*
*Debug Investigation*
I added debug logging to `BlockManager.createLocatedBlockList()` to inspect the
blocks array:
{code:java}
LOG.info("RESTART_DEBUG: createLocatedBlockList called with blocks.length=" +
blocks.length);
for (int i = 0; i < blocks.length; i++) {
if (blocks[i] == null) {
LOG.error("RESTART_DEBUG: blocks[" + i + "] is NULL!");
}
}{code}
{*}Finding{*}: The blocks array itself does NOT contain null elements after
restart. All 33 blocks in the concatenated file's blocks array are non-null
BlockInfo objects.
Hypothesis: Stale Block References
After concat, the file's INode contains blocks from the original target file
PLUS all blocks from the concatenated source files. The test logs show:
- Before restart: Each source file has 3 blocks (10 files × 3 = 30 blocks)
- After concat: Target file should have 3 + 30 = 33 blocks
- After restart: `blocks.length=33` - all blocks present
However, based on the stack trace and NPE location, the issue likely stems from:
1. Concat operation moves blocks from source files to target file
2. Source files are deleted after concat
3. Block metadata for source files may be marked for deletion/invalidation
4. After restart, when NameNode reloads FSImage/EditLog:
- The concatenated file's INode correctly references all 33 blocks
- BUT some blocks may have been removed from the `BlocksMap` during fsimage
load
- Or blocks from deleted source files were never added to BlocksMap
5. When `createLocatedBlock()` is called with one of these "ghost" block
references:
- The BlockInfo object exists in the file's blocks array
- But the actual Block/BlockInfo lookup in BlocksMap fails or returns
inconsistent state
- A null block reference propagates to `numNodes()`
h3. Evidence
Test Log Analysis
*Before Restart (all blocks non-null):*
```
2025-12-01 12:44:06,136 [Time-limited test] INFO BlockManager -
RESTART_DEBUG: createLocatedBlockList called with blocks.length=3, offset=0,
length=1536
blocks[0] = blk_1073741825_1001, numBytes=512
blocks[1] = blk_1073741826_1002, numBytes=512
blocks[2] = blk_1073741827_1003, numBytes=512
```
*After Restart (file now has 33 blocks from concat, all non-null):*
```
2025-12-01 12:44:15,034 [IPC Server handler 0] INFO BlockManager -
RESTART_DEBUG: createLocatedBlockList called with blocks.length=33, offset=0,
length=5120
blocks[0] = blk_1073741825_1001, numBytes=512
blocks[1] = blk_1073741826_1002, numBytes=512
blocks[2] = blk_1073741827_1003, numBytes=512
blocks[3] = blk_1073741855_1031, numBytes=512
... (all 33 blocks non-null)
blocks[32] = blk_1073741830_1006, numBytes=512
```
*Then NPE occurs* - suggesting the null block comes from a code path not
instrumented by my debug logging, or there's a race condition during BlocksMap
access.
h3. *Likely Bug Location*
*Suspect Area 1: Concat Implementation*
The concat operation (in `FSNamesystem` or `FSDirectory`) may not properly
handle block ownership transfer during the transaction that gets persisted to
edit log. On restart:
- FSImage loading might not correctly restore all blocks to BlocksMap
- Blocks from deleted source files might be in a transitional state
*Suspect Area 2: BlockInfo Reference vs BlocksMap Inconsistency*
There may be a race or ordering issue where:
1. INodeFile's blocks array references BlockInfo objects
2. These BlockInfo objects are not yet added to (or have been removed from) the
global BlocksMap
3. When `createLocatedBlock()` tries to look up block locations, it accesses a
BlockInfo that's not in the map
*Code Path to NPE*
{code:java}
getBlockLocations()
→ createLocatedBlocks()
→ createLocatedBlockList()
→ createLocatedBlock(blocks[curBlk], ...) // blocks[curBlk] might be
problematic
→ createLocatedBlock(blk, ...)
→ blocksMap.numNodes(blk) // NPE if blk is somehow null or
invalid{code}
h2. Impact
Production Impact:
1. NameNode Crash Risk: NPE in RPC handler can crash the NameNode during client
`getBlockLocations()` calls
2. Data Availability: Files created via `concat()` become unreadable after
NameNode restart
3. Silent Corruption: The concat operation appears to succeed, but the file is
broken after restart
Affected Operations
- Any `getBlockLocations()` call on concatenated files after restart
- File reads (since clients call getBlockLocations)
- MapReduce/Spark jobs reading concatenated files
- Backup/replication tools accessing these files
h2.
Recommended Fix
*Immediate Mitigation*
Add null-safety check in `BlocksMap.numNodes()`:
{code:java}
int numNodes(Block b) {
if (b == null) {
LOG.error("Null block passed to numNodes()!", new Exception("Stack trace"));
return 0; // Or throw IOException
}
BlockInfo info = blocks.get(b);
return info == null ? 0 : info.numNodes();
} {code}
And in `BlockManager.createLocatedBlock()`:
{code:java}
private LocatedBlock createLocatedBlock(LocatedBlockBuilder locatedBlocks,
final BlockInfo blk, final long pos, final AccessMode mode)
throws IOException {
if (blk == null) {
LOG.error("Null block in createLocatedBlock at pos=" + pos, new
Exception());
throw new IOException("Null block reference in file's block list");
}
// ... rest of method
} {code}
*Root Cause Fix*
{*}{{*}}Requires deeper investigation:{{*}}{*}
1. {*}{{*}}Audit concat implementation{{*}}{*} to ensure all blocks are
properly:
- Added to target file's INode
- Registered in BlocksMap
- Persisted correctly in edit log
- Loaded correctly from FSImage/EditLog on restart
2. {*}{{*}}Check FSImage/EditLog loading{{*}}{*} for concat transactions:
- Verify blocks from concatenated files are added to BlocksMap
- Ensure proper ordering of operations during replay
- Check for race conditions in block map population
3. {*}{{*}}Add consistency checks{{*}}{*} during NameNode startup:
- Verify all blocks referenced by INodes exist in BlocksMap
- Log warnings for orphaned block references
- Option to auto-repair or fail-safe mode
—
#
## Related Code
### Files to Investigate
1. `BlocksMap.java` - Block storage map
2. `BlockManager.java` - Block management and location services
3. `FSNamesystem.java` / `FSDirConcatOp.java` - Concat operation implementation
4. `FSImageFormat.java` / `FSEditLog.java` - Persistence and loading
5. `INodeFile.java` - File inode and block array management
—
> NPE in BlocksMap After NameNode Concat and Restart
> --------------------------------------------------
>
> Key: HDFS-17860
> URL: https://issues.apache.org/jira/browse/HDFS-17860
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.3.5
> Reporter: ConfX
> Priority: Major
> Attachments: reproduce.sh, restart.patch
>
>
> h2. Overview
> A NullPointerException occurs in `BlocksMap.numNodes()` when calling
> `getBlockLocations()` on a file that has been created via `concat()`
> operation, after a NameNode restart. This is a critical production bug that
> could crash the NameNode during normal file read operations.
> h2. Reproduction
> h3. Quick Start with Bash Script
> Use the provided `reproduce.sh` script to automatically reproduce this bug:
> This script automates the entire reproduction process by cloning Hadoop
> 3.3.5, applying the test patch, building the project, and running the failing
> test case.
>
> {code:java}
> cd bug1
> ./reproduce.sh{code}
>
> *What the script does:*
> 1. Clones the Hadoop repository (release 3.3.5 branch)
> 2. Applies the test patch (`restart.patch`) that adds the reproduction test
> 3. Builds the Hadoop HDFS module
> 4. Runs the test case `TestHDFSConcat#testConcatWithRestart` which
> demonstrates the NullPointerException
>
> The bug is confirmed if the test fails with a `NullPointerException` in
> `BlocksMap.numNodes()`.
>
>
> *Manual Reproduction Steps:*
> If you prefer to run the test manually:
>
> {code:java}
> mvn surefire:test
> -Dtest=TestHDFSConcat_RestartInjected#testConcat_AfterConcat_NN_Crash{code}
>
>
> *Test Scenario*
> 1. Create a target file (`/trg`) with 3 blocks (512 bytes each)
> 2. Create 10 source files, each with 3 blocks
> 3. Call `dfs.concat(trgPath, files)` to concatenate all source files into
> target
> 4. Restart the NameNode
> 5. Call `nn.getBlockLocations(trg, 0, trgLen)` on the concatenated file
> 6. NPE occurs at `BlocksMap.numNodes()`
>
> *Stack Trace*
>
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.numNodes(BlocksMap.java:172)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1420)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1382)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlockList(BlockManager.java:1353)
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1503)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:179)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2124)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:769){code}
>
> h2. Root Cause Analysis
> Code Location
> File: `BlocksMap.java:172`
> {code:java}
> int numNodes(Block b) {
> BlockInfo info = blocks.get(b); // LINE 172 - NPE HERE
> return info == null ? 0 : info.numNodes();
> } {code}
>
>
> *Why the NPE Occurs*
>
> The NPE happens because a null `Block` parameter `b` is being passed to
> `numNodes()`. When `b` is null, calling `blocks.get(null)` throws a
> `NullPointerException` because:
> 1. The `blocks` map is likely a `LightWeightGSet` or similar hash map
> implementation
> 2. The `get()` method calls `b.hashCode()` for lookup
> 3. Calling `hashCode()` on null throws NPE
>
> *How Null Blocks Enter the System*
>
> *Debug Investigation*
> I added debug logging to `BlockManager.createLocatedBlockList()` to inspect
> the blocks array:
>
>
> {code:java}
> LOG.info("RESTART_DEBUG: createLocatedBlockList called with blocks.length=" +
> blocks.length);
> for (int i = 0; i < blocks.length; i++) {
> if (blocks[i] == null) {
> LOG.error("RESTART_DEBUG: blocks[" + i + "] is NULL!");
> }
> }{code}
>
> {*}Finding{*}: The blocks array itself does NOT contain null elements after
> restart. All 33 blocks in the concatenated file's blocks array are non-null
> BlockInfo objects.
>
> Hypothesis: Stale Block References
>
> After concat, the file's INode contains blocks from the original target file
> PLUS all blocks from the concatenated source files. The test logs show:
> - Before restart: Each source file has 3 blocks (10 files × 3 = 30 blocks)
> - After concat: Target file should have 3 + 30 = 33 blocks
> - After restart: `blocks.length=33` - all blocks present
>
> However, based on the stack trace and NPE location, the issue likely stems
> from:
>
> 1. Concat operation moves blocks from source files to target file
> 2. Source files are deleted after concat
> 3. Block metadata for source files may be marked for deletion/invalidation
> 4. After restart, when NameNode reloads FSImage/EditLog:
> - The concatenated file's INode correctly references all 33 blocks
> - BUT some blocks may have been removed from the `BlocksMap` during
> fsimage load
> - Or blocks from deleted source files were never added to BlocksMap
> 5. When `createLocatedBlock()` is called with one of these "ghost" block
> references:
> - The BlockInfo object exists in the file's blocks array
> - But the actual Block/BlockInfo lookup in BlocksMap fails or returns
> inconsistent state
> - A null block reference propagates to `numNodes()`
> h3. Evidence
>
> Test Log Analysis
>
> *Before Restart (all blocks non-null):*
> ```
> 2025-12-01 12:44:06,136 [Time-limited test] INFO BlockManager -
> RESTART_DEBUG: createLocatedBlockList called with blocks.length=3,
> offset=0, length=1536
> blocks[0] = blk_1073741825_1001, numBytes=512
> blocks[1] = blk_1073741826_1002, numBytes=512
> blocks[2] = blk_1073741827_1003, numBytes=512
> ```
>
> *After Restart (file now has 33 blocks from concat, all non-null):*
> ```
> 2025-12-01 12:44:15,034 [IPC Server handler 0] INFO BlockManager -
> RESTART_DEBUG: createLocatedBlockList called with blocks.length=33,
> offset=0, length=5120
> blocks[0] = blk_1073741825_1001, numBytes=512
> blocks[1] = blk_1073741826_1002, numBytes=512
> blocks[2] = blk_1073741827_1003, numBytes=512
> blocks[3] = blk_1073741855_1031, numBytes=512
> ... (all 33 blocks non-null)
> blocks[32] = blk_1073741830_1006, numBytes=512
> ```
>
> *Then NPE occurs* - suggesting the null block comes from a code path not
> instrumented by my debug logging, or there's a race condition during
> BlocksMap access.
>
> h3. *Likely Bug Location*
>
> *Suspect Area 1: Concat Implementation*
>
> The concat operation (in `FSNamesystem` or `FSDirectory`) may not properly
> handle block ownership transfer during the transaction that gets persisted to
> edit log. On restart:
> - FSImage loading might not correctly restore all blocks to BlocksMap
> - Blocks from deleted source files might be in a transitional state
>
> *Suspect Area 2: BlockInfo Reference vs BlocksMap Inconsistency*
>
> There may be a race or ordering issue where:
> 1. INodeFile's blocks array references BlockInfo objects
> 2. These BlockInfo objects are not yet added to (or have been removed from)
> the global BlocksMap
> 3. When `createLocatedBlock()` tries to look up block locations, it accesses
> a BlockInfo that's not in the map
>
> *Code Path to NPE*
>
> {code:java}
> getBlockLocations()
> → createLocatedBlocks()
> → createLocatedBlockList()
> → createLocatedBlock(blocks[curBlk], ...) // blocks[curBlk] might be
> problematic
> → createLocatedBlock(blk, ...)
> → blocksMap.numNodes(blk) // NPE if blk is somehow null or
> invalid{code}
>
> h2. Impact
> Production Impact:
> 1. NameNode Crash Risk: NPE in RPC handler can crash the NameNode during
> client `getBlockLocations()` calls
> 2. Data Availability: Files created via `concat()` become unreadable after
> NameNode restart
> 3. Silent Corruption: The concat operation appears to succeed, but the file
> is broken after restart
>
> Affected Operations
> - Any `getBlockLocations()` call on concatenated files after restart
> - File reads (since clients call getBlockLocations)
> - MapReduce/Spark jobs reading concatenated files
> - Backup/replication tools accessing these files
> h2.
> Recommended Fix
> *Immediate Mitigation*
> Add null-safety check in `BlocksMap.numNodes()`:
> {code:java}
> int numNodes(Block b) {
> if (b == null) {
> LOG.error("Null block passed to numNodes()!", new Exception("Stack
> trace"));
> return 0; // Or throw IOException
> }
> BlockInfo info = blocks.get(b);
> return info == null ? 0 : info.numNodes();
> } {code}
> And in `BlockManager.createLocatedBlock()`:
> {code:java}
> private LocatedBlock createLocatedBlock(LocatedBlockBuilder locatedBlocks,
> final BlockInfo blk, final long pos, final AccessMode mode)
> throws IOException {
> if (blk == null) {
> LOG.error("Null block in createLocatedBlock at pos=" + pos, new
> Exception());
> throw new IOException("Null block reference in file's block list");
> }
> // ... rest of method
> } {code}
>
> *Root Cause Fix*
>
> Requires deeper investigation:
>
> 1. Audit concat implementation to ensure all blocks are properly:
> - Added to target file's INode
> - Registered in BlocksMap
> - Persisted correctly in edit log
> - Loaded correctly from FSImage/EditLog on restart
>
> 2. Check FSImage/EditLog loading for concat transactions:
> - Verify blocks from concatenated files are added to BlocksMap
> - Ensure proper ordering of operations during replay
> - Check for race conditions in block map population
>
> 3. Add consistency checks during NameNode startup:
> - Verify all blocks referenced by INodes exist in BlocksMap
> - Log warnings for orphaned block references
> - Option to auto-repair or fail-safe mode
>
>
> *Related Code Files to Investigate*
> 1. `BlocksMap.java` - Block storage map
> 2. `BlockManager.java` - Block management and location services
> 3. `FSNamesystem.java` / `FSDirConcatOp.java` - Concat operation
> implementation
> 4. `FSImageFormat.java` / `FSEditLog.java` - Persistence and loading
> 5. `INodeFile.java` - File inode and block array management
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
