[
https://issues.apache.org/jira/browse/HDFS-13433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460072#comment-16460072
]
Arpit Agarwal edited comment on HDFS-13433 at 5/1/18 9:09 PM:
--
bq. In your example, nn3/nn4 start up with a config containing defaultFS set to
ns1? I would say that is a misconfig. Isn't it supposed to be hdfs://ns2 for
nn3/nn4?
Hi [~kihwal], we configure a given cluster with the same {{fs.defaultFS}}
setting for all nodes, so without viewfs all namenodes would see fs.defaultFS
pointing to the same nameservice, not their own nameservice. I think this
follows the spirit of the original design for HA/federation configs e.g. from
our HA docs:
{code:none}
Similar to Federation configuration, HA configuration is backward compatible
and allows existing single NameNode configurations to work without change. The
new configuration is designed such that all the nodes in the cluster may have
the same configuration without the need for deploying different configuration
files to different machines based on the type of the node.
{code}
[~daryn] gave me some details about your IP failover setup offline. iiuc your
fs.defaultFS points to a hostname that resolves to the floating IP and this is
not a valid nameservice. So you always want the redirect to be sent to
fs.defaultFS and not the nameservice. Is that correct? How do you setup
federated clusters - do you start federated NNs with different {{fs.defaultFS}}
values?
I don't see an easy way that can address both of these situations without
additional config parameters, but let me think about it some more.
was (Author: arpitagarwal):
bq. In your example, nn3/nn4 start up with a config containing defaultFS set to
ns1? I would say that is a misconfig. Isn't it supposed to be hdfs://ns2 for
nn3/nn4?
Hi [~kihwal], our clusters are configured with the same {{fs.defaultFS}}
setting for all nodes, so without viewfs all namenodes would see fs.defaultFS
pointing to the same nameservice, not their own nameservice. I think this
follows the spirit of the original design for HA/federation configs e.g. from
our HA docs:
{code:none}
Similar to Federation configuration, HA configuration is backward compatible
and allows existing single NameNode configurations to work without change. The
new configuration is designed such that all the nodes in the cluster may have
the same configuration without the need for deploying different configuration
files to different machines based on the type of the node.
{code}
[~daryn] gave me some details about your IP failover setup offline. iiuc your
fs.defaultFS points to a hostname that resolves to the floating IP and this is
not a valid nameservice. So you always want the redirect to be sent to
fs.defaultFS and not the nameservice. Is that correct? How do you setup
federated clusters - do you start federated NNs with different {{fs.defaultFS}}
values?
I don't see an easy way that can address both of these situations without
additional config parameters, but let me think about it some more.
> webhdfs requests can be routed incorrectly in federated cluster
> ---
>
> Key: HDFS-13433
> URL: https://issues.apache.org/jira/browse/HDFS-13433
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
>Priority: Critical
> Fix For: 3.1.1, 3.0.3
>
> Attachments: HDFS-13433.01.patch, HDFS-13433.02.patch,
> HDFS-13433.03.patch, HDFS-13433.04.patch
>
>
> In the following HA+Federated setup with two nameservices ns1 and ns2:
> # ns1 -> namenodes nn1, nn2
> # ns2 -> namenodes nn3, nn4
> # fs.defaultFS is {{hdfs://ns1}}.
> A webhdfs request issued to nn3/nn4 will be routed to ns1. This is because
> {{setClientNamenodeAddress}} initializes {{NameNode#clientNamenodeAddress}}
> using fs.defaultFS before the config is overriden.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
