[
https://issues.apache.org/jira/browse/HDFS-7443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14252863#comment-14252863
]
Colin Patrick McCabe commented on HDFS-7443:
--------------------------------------------
I added a test case where one duplicate block was shorter than the other, and
one duplicate block had an older genstamp than the other.
{code}
cmccabe@keter:~> find | egrep 'blk_[0-9]*$' | awk -F '/' '{print $NF}' | sort |
uniq -c | grep -v ' 1'
2 blk_1073742010
2 blk_1073742407
cmccabe@keter:~> find -name 'blk_1073742010*' -o -name 'blk_1073742407' | xargs
-l ls -l
-rw-r--r-- 1 cmccabe users 11 Dec 18 19:30
./dfs/data/current/BP-1455098086-127.0.1.1-1404855087848/current/finalized/subdir6/blk_1073742010_1185.meta
-rw-r--r-- 2 cmccabe users 512 Jul 8 14:34
./dfs/data/current/BP-1455098086-127.0.1.1-1404855087848/current/finalized/subdir6/blk_1073742010
-rw-r--r-- 1 cmccabe users 510 Dec 18 19:37
./dfs/data/current/BP-1455098086-127.0.1.1-1404855087848/current/finalized/subdir60/blk_1073742407
-rw-r--r-- 1 cmccabe users 512 Dec 18 19:36
./dfs/data/current/BP-1455098086-127.0.1.1-1404855087848/current/finalized/subdir3/blk_1073742407
-rw-r--r-- 2 cmccabe users 512 Jul 8 14:34
./dfs/data/current/BP-1455098086-127.0.1.1-1404855087848/current/finalized/subdir53/blk_1073742010
-rw-r--r-- 1 cmccabe users 11 Jul 8 14:34
./dfs/data/current/BP-1455098086-127.0.1.1-1404855087848/current/finalized/subdir53/blk_1073742010_1186.meta
{code}
> Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are
> present in the same volume
> ------------------------------------------------------------------------------------------------------
>
> Key: HDFS-7443
> URL: https://issues.apache.org/jira/browse/HDFS-7443
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Kihwal Lee
> Assignee: Colin Patrick McCabe
> Priority: Blocker
> Attachments: HDFS-7443.001.patch
>
>
> When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of
> datanodes were not coming up. They treid data file layout upgrade for
> BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.
> All failures were caused by {{NativeIO.link()}} throwing IOException saying
> {{EEXIST}}. The data nodes didn't die right away, but the upgrade was soon
> retried when the block pool initialization was retried whenever
> {{BPServiceActor}} was registering with the namenode. After many retries,
> datenodes terminated. This would leave {{previous.tmp}} and {{current}} with
> no {{VERSION}} file in the block pool slice storage directory.
> Although {{previous.tmp}} contained the old {{VERSION}} file, the content was
> in the new layout and the subdirs were all newly created ones. This
> shouldn't have happened because the upgrade-recovery logic in {{Storage}}
> removes {{current}} and renames {{previous.tmp}} to {{current}} before
> retrying. All successfully upgraded volumes had old state preserved in their
> {{previous}} directory.
> In summary there were two observed issues.
> - Upgrade failure with {{link()}} failing with {{EEXIST}}
> - {{previous.tmp}} contained not the content of original {{current}}, but
> half-upgraded one.
> We did not see this in smaller scale test clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
