Bram Schuur created HDFS-17984:
----------------------------------
Summary: Block recovery deletes the block when a restarting
DataNode reports no replica before its replicas are loaded
Key: HDFS-17984
URL: https://issues.apache.org/jira/browse/HDFS-17984
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, hdfs
Affects Versions: 3.5.0
Environment: Hadoop hdfs 3.5.0
Reporter: Bram Schuur
When a DataNode starts, {{DataNode#initBlockPool}} creates the FsDataset
({{{}initStorage{}}}) and only afterwards loads the block pool's replicas
({{{}FsDatasetImpl#addBlockPool{}}}), which first registers an empty replica
map ({{{}volumeMap.initBlockPool{}}}) and then fills it
({{{}getAllVolumesMap{}}}). The IPC server is already running at that point. An
{{initReplicaRecovery}} call from a block recovery primary in this window
passes {{{}checkStorageState{}}}, finds no replica in the not yet filled map
and returns null, i.e. "this DataNode does not have the block".
{{BlockRecoveryWorker#recover}} does not count a null answer as an error. If
the other replicas fail, the sync list is empty and {{syncBlock}} commits with
{{{}deleteBlock=true{}}}: the block is removed and the file is closed without
it, although the restarting DataNode holds a valid replica, which it then
deletes as an orphan once it has loaded.
Observed on 3.5.0, writing an HBase WAL (replication 2) during a rolling node
restart. dn0 restarted at 11:22:43; dn1 (primary) ran the recovery at 11:22:46
and dn0 answered without an error:
{noformat}
11:22:43 [dn0] STARTUP_MSG: Starting DataNode
11:22:46,494 [dn1] java.io.IOException: Block length mismatch, len=7308711 but
r=ReplicaBeingWritten, blk_1074458450_718705, RBW
11:22:46,494 [dn1] BlockRecoveryWorker: block=BP-...:blk_1074458450_718705
(length=0), isTruncateRecovery=false, syncList=[]
11:22:46,495 [nn]
commitBlockSynchronization(oldBlock=BP-...:blk_1074458450_718705,
newgenerationstamp=718982, newlength=0, newtargets=[], closeFile=true,
deleteBlock=true)
11:22:55,864 [dn0] Scheduling blk_1074458450_718705 replica
ReplicaWaitingToBeRecovered, blk_1074458450_718705, RWR ... for deletion
{noformat}
A few hundred milliseconds earlier (11:22:46.19) dn0 answered another recovery
with "Storage not yet initialized for initReplicaRecovery"; that case is
already an error. The gap is between {{initStorage}} and the end of
{{{}addBlockPool{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]