[
https://issues.apache.org/jira/browse/HDFS-7842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336302#comment-14336302
]
J.Andreina commented on HDFS-7842:
----------------------------------
Observation:
===========
Logs after Step 5
{noformat}
Namenode Log:
=============
15/02/25 13:10:59 INFO hdfs.StateChange: BLOCK* allocate
blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-da5955d6-d021-4576-aa43-6caf70fcfd17:NORMAL:XXXXXXXXXXX:50010|RBW]]}
for /File_1._COPYING_
15/02/25 13:10:59 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: XXXXXXXXXXX:50010 is added to blk_1073741830_1006{UCState=COMMITTED,
primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-da5955d6-d021-4576-aa43-6caf70fcfd17:NORMAL:XXXXXXXXXXX:50010|RBW]]}
size 11526
15/02/25 13:10:59 INFO hdfs.StateChange: DIR* completeFile: /File_1._COPYING_
is closed by DFSClient_NONMAPREDUCE_-1004187273_1
Datanode Log:
=============
2015-02-25 13:10:59,222 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1954121396-XXXXXXXXXXX-1424840820188:blk_1073741830_1006 src:
/XXXXXXXXXXX:34363 dest: /XXXXXXXXXXX:50010
2015-02-25 13:10:59,295 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1954121396-XXXXXXXXXXX-1424840820188:blk_1073741830_1006,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
{noformat}
Logs after step 12
{noformat}
Namenode Log:
============
15/02/25 13:15:51 INFO BlockStateChange: BLOCK* InvalidateBlocks: add
blk_1073741830_1006 to XXXXXXXXXXX:50010
15/02/25 13:16:04 INFO hdfs.StateChange: BLOCK* allocate
blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-f560cc10-74e8-4ea8-a8d9-6959fe5c1104:NORMAL:XXXXXXXXXXX:50010|RBW]]}
for /File_2._COPYING_
15/02/25 13:16:05 INFO BlockStateChange: BLOCK* addStoredBlock: blockMap
updated: XXXXXXXXXXX:50010 is added to
blk_1073741830_1006{UCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUC[[DISK]DS-da5955d6-d021-4576-aa43-6caf70fcfd17:NORMAL:XXXXXXXXXXX:50010|FINALIZED]]}
size 0
15/02/25 13:16:05 INFO hdfs.StateChange: DIR* completeFile: /File_2._COPYING_
is closed by DFSClient_NONMAPREDUCE_-1317707332_1
Datanode Log:
=============
2015-02-25 13:15:51,831 INFO org.apache.hadoop.hdfs.server.common.Storage:
Enabled trash for bpid BP-1954121396-XXXXXXXXXXX-1424840820188
2015-02-25 13:15:54,801 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Scheduling blk_1073741830_1006 file
/mnt/tmp1/current/BP-1954121396-XXXXXXXXXXX-1424840820188/current/finalized/subdir0/subdir0/blk_1073741830
for deletion
2015-02-25 13:15:54,805 INFO
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService:
Deleted BP-1954121396-XXXXXXXXXXX-1424840820188 blk_1073741830_1006 file
/mnt/tmp1/current/BP-1954121396-XXXXXXXXXXX-1424840820188/current/finalized/subdir0/subdir0/blk_1073741830
2015-02-25 13:16:05,074 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Receiving BP-1954121396-XXXXXXXXXXX-1424840820188:blk_1073741830_1006 src:
/XXXXXXXXXXX:34528 dest: /XXXXXXXXXXX:50010
2015-02-25 13:16:05,138 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src:
/XXXXXXXXXXX:34528, dest: /XXXXXXXXXXX:50010, bytes: 6324, op: HDFS_WRITE,
cliID: DFSClient_NONMAPREDUCE_-1317707332_1, offset: 0, srvID:
e33b81ce-8820-4343-955f-8726965d1917, blockid:
BP-1954121396-XXXXXXXXXXX-1424840820188:blk_1073741830_1006, duration: 50371413
2015-02-25 13:16:05,141 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
PacketResponder: BP-1954121396-XXXXXXXXXXX-1424840820188:blk_1073741830_1006,
type=LAST_IN_PIPELINE, downstreams=0:[] terminating
{noformat}
Log after Step 14
{noformat}
Datanode Log:
=============
2015-02-25 13:18:06,796 INFO org.apache.hadoop.hdfs.server.common.Storage:
Restoring
/mnt/tmp1/current/BP-1954121396-XXXXXXXXXXX-1424840820188/trash/finalized/subdir0/subdir0/blk_1073741832_1008.meta
to
/mnt/tmp1/current/BP-1954121396-XXXXXXXXXXX-1424840820188/current/finalized/subdir0/subdir0
2015-02-25 13:18:06,797 INFO org.apache.hadoop.hdfs.server.common.Storage:
Restored 4 block files from trash.
Namenode Log:
============
15/02/25 13:18:07 INFO BlockStateChange: BLOCK
NameSystem.addToCorruptReplicasMap: blk_1073741830 added as corrupt on
XXXXXXXXXXX:50010 by host-10-177-112-123/XXXXXXXXXXX because block is COMPLETE
and reported length 11526 does not match length in block map 6324
15/02/25 13:18:07 INFO BlockStateChange: BLOCK* processReport: from storage
DS-da5955d6-d021-4576-aa43-6caf70fcfd17 node DatanodeRegistration(XXXXXXXXXXX,
datanodeUuid=e33b81ce-8820-4343-955f-8726965d1917, infoPort=50075,
infoSecurePort=0, ipcPort=50020,
storageInfo=lv=-56;cid=CID-dd48fb1f-1d88-4d65-90c3-a7535053f4e1;nsid=2021392782;c=0),
blocks: 5, hasStaleStorage: false, processing time: 0 msecs
{noformat}
Suggession :
===========
Restoring blocks from trash after downgrade can be avoided.
Please review and give your opinion on this issue. If it sounds good ill give a
patch on this.
> Blocks missed while performing downgrade immediately after rolling back the
> cluster.
> ------------------------------------------------------------------------------------
>
> Key: HDFS-7842
> URL: https://issues.apache.org/jira/browse/HDFS-7842
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: J.Andreina
> Assignee: J.Andreina
> Priority: Critical
>
> Performing downgrade immediately after rolling back the cluster , will
> replace the blocks from trash
> Since the block id for the files created before rollback will be same as the
> file created before downgrade, namenode will get into safemode , as the block
> size reported from Datanode will be different from the one in block map
> (corrupted blocks) .
> Steps to Reproduce
> {noformat}
> Step 1: Prepare rolling upgrade using "hdfs dfsadmin -rollingUpgrade prepare"
> Step 2: Shutdown SNN and NN
> Step 3: Start NN with the "hdfs namenode -rollingUpgrade started" option.
> Step 4: Executed "hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT>
> upgrade" and restarted Datanode
> Step 5: Create File_1 of size 11526
> Step 6: Shutdown both NN and DN
> Step 7: Start NNs with the "hdfs namenode -rollingUpgrade rollback" option.
> Start DNs with the "-rollback" option.
> Step 8: Prepare rolling upgrade using "hdfs dfsadmin -rollingUpgrade prepare"
> Step 9: Shutdown SNN and NN
> Step 10: Start NN with the "hdfs namenode -rollingUpgrade started" option .
> Step 11: Executed "hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT>
> upgrade" and restarted Datanode
> step 12: Add file File_2 with size 6324 (which has same blockid as previous
> created File_1 with block size 11526)
> Step 13: Shutdown both NN and DN
> Step 14: Start NNs with the "hdfs namenode -rollingUpgrade downgrade"
> option.Start DNs normally.
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
