[ https://issues.apache.org/jira/browse/HDFS-9949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198022#comment-15198022 ]
Colin Patrick McCabe commented on HDFS-9949: -------------------------------------------- Thanks, [~qwertymaniac]. Great find, and great root-cause analysis. {code} 106 while (!cluster.getDataNodes().get(0).isDatanodeFullyStarted()) { 106 Thread.sleep(500); 107 } {code} Can we have a sleep for 50 ms here instead, just to take advantage of cases where the registration is quicker? +1 once that's addressed. > Testcase for catching DN UUID regeneration regression > ----------------------------------------------------- > > Key: HDFS-9949 > URL: https://issues.apache.org/jira/browse/HDFS-9949 > Project: Hadoop HDFS > Issue Type: Test > Affects Versions: 2.6.0 > Reporter: Harsh J > Assignee: Harsh J > Priority: Minor > Attachments: HDFS-9949.000.branch-2.7.not-for-commit.patch, > HDFS-9949.000.patch > > > In the following scenario, in releases without HDFS-8211, the DN may > regenerate its UUIDs unintentionally. > 0. Consider a DN with two disks {{/data1/dfs/dn,/data2/dfs/dn}} > 1. Stop DN > 2. Unmount the second disk, {{/data2/dfs/dn}} > 3. Create (in the scenario, this was an accident) /data2/dfs/dn on the root > path > 4. Start DN > 5. DN now considers /data2/dfs/dn empty so formats it, but during the format > it uses {{datanode.getDatanodeUuid()}} which is null until register() is > called. > 6. As a result, after the directory loading, {{datanode.checkDatanodUuid()}} > gets called with successful condition, and it causes a new generation of UUID > which is written to all disks {{/data1/dfs/dn/current/VERSION}} and > {{/data2/dfs/dn/current/VERSION}}. > 7. Stop DN (in the scenario, this was when the mistake of unmounted disk was > realised) > 8. Mount the second disk back again {{/data2/dfs/dn}}, causing the > {{VERSION}} file to be the original one again on it (mounting masks the root > path that we last generated upon). > 9. DN fails to start up cause it finds mismatched UUID between the two disks > The DN should not generate a new UUID if one of the storage disks already > have the older one. > HDFS-8211 unintentionally fixes this by changing the > {{datanode.getDatanodeUuid()}} function to rely on the {{DataStorage}} > representation of the UUID vs. the {{DatanodeID}} object which only gets > available (non-null) _after_ the registration. > It'd still be good to add a direct test case to the above scenario that > passes on trunk and branch-2, but fails on branch-2.7 and lower, so we can > catch a regression around this in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)