[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.001.patch This patch resolves the entries in the hosts files to their first IP addresses before de-duplicating everything. when creating a new {{DatanodeInfo}} for them, it uses {{getCanonicalAddress}}. This fixes the "invisible node" problem where due to lacking a hostname, the NameNode web UI would show an entry like {{:50010}} in its lists (note missing hostname). It also fixes the problem where we put two hostnames which refer to the same IP address in a host file, or a hostname and an IP which both turn out to map to the same hostname. > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Andy Isaacson >Priority: Minor > Attachments: HDFS-3934.001.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.002.patch * fix test that relies on unresolvable hostname * fix behavior for hosts files that have lines of the form : rather than just > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Andy Isaacson >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Status: Patch Available (was: Open) > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Andy Isaacson >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.003.patch * add a test for duplicate hosts entries. * Add test timeouts for tests in {{TestDecommission}}. * In {{DatanodeManager#getDatanodeListForReport}}, de-dupe by ip address + port, rather than by ip address alone. (since multiple DNs can run on the same node with different ports). > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Andy Isaacson >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.004.patch when the port is not specified in the hosts file, use a default from the configuration, rather than a compile-time default. > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Andy Isaacson >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.005.patch here is a new version that is smarter about DNS resolution. it resolves the DNS only during the refreshNodes operation. > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Andy Isaacson >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.006.patch * use immutablemap * reduce synchronization > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Andy Isaacson >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.007.patch * add license header * use a HashMap rather than ImmutableMap.Builder, to handle duplicates > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Andy Isaacson >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.008.patch * If the port is missing, treat the include/exclude entry as covering any Datanode on the host. * grammar fix in comment. > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.010.patch bq. It'd be nice to continue to use the HostsFileReader and post-process the result. Otherwise it's a consistency/maintenance to copy-n-paste any new parsing functionality. OK, I'll use the {{HostsFileReader}} parsing code. bq. Why does the reader need to instantiate dummy DatanodeID? You're right. Re-using {{DatanodeID}} for this purpose doesn't reall ymake sense. I created a new type called {{HostFileManager#Entry}} to represent host file entries. bq. It appears to be for repeatedly making the somewhat fragile assumption that xferAddr is ipAddr+port? If that relationship changes, we've got a problem... Fixed to use getIpAddr() + ":" + getXferPort() in all cases. bq. Patch appears to have dropped supported for the node's registration name. Eli Collins wanted me to maintain that feature in HDFS-3990. If we need to keep it, doing a lookup and a canonical lookup (can trigger another dns lookup) isn't compatible with supporting the reg name. Thanks for pointing this out. I talked to Eli and he explained the distinction between registration names and hostnames to me. I added back support for "registration names" and added a unit test to ensure this works properly. bq. Doing a lookup followed by getCanonicalName is a bad idea. It does 2 more lookups: hostname -> PTR -> A so it can resolve CNAMES to IP to hostname. With this change I think it will cause 3 lookups per host. One key feature of this change is that all the lookups happen when the include and exclude files are read. *No* lookups happen during {{DatanodeManager#getDatanodeListForReport}}, or any of the other cases where we check the host file entries. On the advice of Eli, I removed the call to {{getCanonicalName}}. We can just use the name the user specified in the hosts file; that should be fine. bq. Question about "// If no transfer port was specified, we take a guess". Why needed, and what are the ramifications for getting this wrong? Just a display issue? We just don't have the information. If the datanode is dead, we only know what the entry says in the hosts file(s). If the entries don't have the port, we have to guess. I don't see any way around this. It might be more elegant if the web UI could understand the concept of "port is unknown," but adding that seems out of scope. In addition to the unit tests, I did some manual testing on this and verified that it got rid of the double-counting of nodes in the web UI for me. > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.011.patch rebase on trunk > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.012.patch > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.013.patch fix javadoc warning, etc > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: (was: HDFS-3934.013.patch) > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.013.patch > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch, > HDFS-3934.013.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.014.patch removed some stray log messages > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch, > HDFS-3934.013.patch, HDFS-3934.014.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.015.patch > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch, > HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.016.patch * Fix a place in {{TestDatanodeRegistration}} where we created a mock datanode that returned 0 for its xferPort. Since we're asserting on this now, we need to return a legal port value here. * Looks like {{DFS_DATANODE_IPC_ADDRESS_DEFAULT}} has been wrong all along (the colon is missing), but we never noticed because we never tried to parse it as a host:port until now. So let's fix {{DFS_DATANODE_IPC_ADDRESS_DEFAULT}}. > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch, > HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch, > HDFS-3934.016.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Attachment: HDFS-3934.017.patch rebased on trunk > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch, > HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch, > HDFS-3934.016.patch, HDFS-3934.017.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Fix Version/s: 3.0.0 > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch, > HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch, > HDFS-3934.016.patch, HDFS-3934.017.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Fix Version/s: 2.3.0 2.1.0-beta > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 3.0.0, 2.1.0-beta, 2.3.0 > > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch, > HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch, > HDFS-3934.016.patch, HDFS-3934.017.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3934) duplicative dfs_hosts entries handled wrong
[ https://issues.apache.org/jira/browse/HDFS-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3934: --- Resolution: Fixed Status: Resolved (was: Patch Available) committed to branch-2.1-beta and branch-2 > duplicative dfs_hosts entries handled wrong > --- > > Key: HDFS-3934 > URL: https://issues.apache.org/jira/browse/HDFS-3934 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.1-alpha >Reporter: Andy Isaacson >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 3.0.0, 2.1.0-beta, 2.3.0 > > Attachments: HDFS-3934.001.patch, HDFS-3934.002.patch, > HDFS-3934.003.patch, HDFS-3934.004.patch, HDFS-3934.005.patch, > HDFS-3934.006.patch, HDFS-3934.007.patch, HDFS-3934.008.patch, > HDFS-3934.010.patch, HDFS-3934.011.patch, HDFS-3934.012.patch, > HDFS-3934.013.patch, HDFS-3934.014.patch, HDFS-3934.015.patch, > HDFS-3934.016.patch, HDFS-3934.017.patch > > > A dead DN listed in dfs_hosts_allow.txt by IP and in dfs_hosts_exclude.txt by > hostname ends up being displayed twice in {{dfsnodelist.jsp?whatNodes=DEAD}} > after the NN restarts because {{getDatanodeListForReport}} does not handle > such a "pseudo-duplicate" correctly: > # the "Remove any nodes we know about from the map" loop no longer has the > knowledge to remove the spurious entries > # the "The remaining nodes are ones that are referenced by the hosts files" > loop does not do hostname lookups, so does not know that the IP and hostname > refer to the same host. > Relatedly, such an IP-based dfs_hosts entry results in a cosmetic problem in > the JSP output: The *Node* column shows ":50010" as the nodename, with HTML > markup {{ href="http://:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F&nnaddr=172.29.97.196:8020"; > title="172.29.97.216:50010">:50010}}. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira