Kihwal Lee created HDFS-7443:
--------------------------------

             Summary: Datanode upgrade to BLOCKID_BASED_LAYOUT sometimes fails
                 Key: HDFS-7443
                 URL: https://issues.apache.org/jira/browse/HDFS-7443
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 2.6.0
            Reporter: Kihwal Lee
            Priority: Blocker


When we did an upgrade from 2.5 to 2.6 in a medium size cluster, about 4% of 
datanodes were not coming up.  They treid data file layout upgrade for 
BLOCKID_BASED_LAYOUT introduced in HDFS-6482, but failed.

All failures were caused by {{NativeIO.link()}} throwing IOException saying 
{{EEXIST}}.  The data nodes didn't die right away, but the upgrade was soon 
retried when the block pool initialization was retried whenever 
{{BPServiceActor}} was registering with the namenode.  After many retries, 
datenodes terminated.  This would leave {{previous.tmp}} and {{current}} with 
no {{VERSION}} file in the block pool slice storage directory.  

Although {{previous.tmp}} contained the old {{VERSION}} file, the content was 
in the new layout and the subdirs were all newly created ones.  This shouldn't 
have happened because the upgrade-recovery logic in {{Storage}} removes 
{{current}} and renames {{previous.tmp}} to {{current}} before retrying.  All 
successfully upgraded volumes had old state preserved in their {{previous}} 
directory.

In summary there were two observed issues.
- Upgrade failure with {{link()}} failing with {{EEXIST}}
- {{previous.tmp}} contained not the content of original {{current}}, but 
half-upgraded one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to