[jira] [Comment Edited] (HDFS-13214) RBF: Configuration on Router conflicts with client side configuration
[
https://issues.apache.org/jira/browse/HDFS-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389026#comment-16389026
]
Tao Jie edited comment on HDFS-13214 at 3/7/18 4:56 AM:
Sorry for replying late and thank you [~linyiqun] [~elgoiri] for working on
this JIRA.
It clear to me now:)
Some other minor suggestion about the document:
1, Rebalancing data across subclusters mentioned in the document of
2.9.0/3.0.0GA is not ready today, right? We'd better avoid misleading users
when the function is not available (I have tried to find out the way of
rebalancing for a while :) ).
2, The diagram of the diagram of Architecture implies that the subclusters are
independent HDFS clusters. Actually subclusters could also be federation
cluster or a mixed cluster with federation and independent cluster. We could
mention it explicitly in the document.
I am ok to handle this in another jira.
+1 for the current patch.
was (Author: tao jie):
Sorry for replying late and thank you [~linyiqun] [~elgoiri] for working on
this JIRA.
It clear to me now:)
Some other minor suggestion about the document:
1, Rebalancing data across subclusters mentioned in the document of
2.9.0/3.0.0GA is not ready today, right? We'd better avoid misleading users
when the function is not available (I have tried to find out the way of
rebalancing for a while :) ).
2, The diagram of the diagram of Architecture implies that the subclusters are
independent HDFS clusters. Actually subclusters could also be federation
cluster or a mixed cluster with federation and independent cluster. We could
mention it explicitly in the document.
I'am ok to handle this in another jira.
+1 for the current patch.
> RBF: Configuration on Router conflicts with client side configuration
> -
>
> Key: HDFS-13214
> URL: https://issues.apache.org/jira/browse/HDFS-13214
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13214.001.patch, HDFS-13214.002.patch,
> HDFS-13214.003.patch, HDFS-13214.004.patch
>
>
> In a typical router-based federation cluster, hdfs-site.xml is supposed to be:
> {code}
>
> dfs.nameservices
> ns1,ns2,ns-fed
>
>
> dfs.ha.namenodes.ns-fed
> r1,r2
>
>
> dfs.namenode.rpc-address.ns1
> host1:8020
>
>
> dfs.namenode.rpc-address.ns2
> host2:8020
>
>
> dfs.namenode.rpc-address.ns-fed.r1
> host1:
>
>
> dfs.namenode.rpc-address.ns-fed.r2
> host2:
>
> {code}
> {{dfs.ha.namenodes.ns-fed}} here is used for client to access the Router.
> However with this configuration on server node, Router fails to start with
> error:
> {code}
> org.apache.hadoop.HadoopIllegalArgumentException: Configuration has multiple
> addresses that match local node's address. Please configure the system with
> dfs.nameservice.id and dfs.ha.namenode.id
> at org.apache.hadoop.hdfs.DFSUtil.getSuffixIDs(DFSUtil.java:1198)
> at org.apache.hadoop.hdfs.DFSUtil.getNameServiceId(DFSUtil.java:1131)
> at
> org.apache.hadoop.hdfs.DFSUtil.getNamenodeNameServiceId(DFSUtil.java:1086)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createLocalNamenodeHearbeatService(Router.java:466)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createNamenodeHearbeatServices(Router.java:423)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:199)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> 2018-03-01 18:05:56,208 ERROR
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
> router
> {code}
> Then the router tries to find the local namenode, multiple properties:
> {{dfs.namenode.rpc-address.ns1}}, {{dfs.namenode.rpc-address.ns-fed.r1}}
> match the local address.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13214) RBF: Configuration on Router conflicts with client side configuration
[
https://issues.apache.org/jira/browse/HDFS-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387211#comment-16387211
]
Yiqun Lin edited comment on HDFS-13214 at 3/6/18 3:27 AM:
--
Attach the updated patch to update the doc.
was (Author: linyiqun):
Attach the updated to update the doc.
> RBF: Configuration on Router conflicts with client side configuration
> -
>
> Key: HDFS-13214
> URL: https://issues.apache.org/jira/browse/HDFS-13214
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13214.001.patch, HDFS-13214.002.patch,
> HDFS-13214.003.patch, HDFS-13214.004.patch
>
>
> In a typical router-based federation cluster, hdfs-site.xml is supposed to be:
> {code}
>
> dfs.nameservices
> ns1,ns2,ns-fed
>
>
> dfs.ha.namenodes.ns-fed
> r1,r2
>
>
> dfs.namenode.rpc-address.ns1
> host1:8020
>
>
> dfs.namenode.rpc-address.ns2
> host2:8020
>
>
> dfs.namenode.rpc-address.ns-fed.r1
> host1:
>
>
> dfs.namenode.rpc-address.ns-fed.r2
> host2:
>
> {code}
> {{dfs.ha.namenodes.ns-fed}} here is used for client to access the Router.
> However with this configuration on server node, Router fails to start with
> error:
> {code}
> org.apache.hadoop.HadoopIllegalArgumentException: Configuration has multiple
> addresses that match local node's address. Please configure the system with
> dfs.nameservice.id and dfs.ha.namenode.id
> at org.apache.hadoop.hdfs.DFSUtil.getSuffixIDs(DFSUtil.java:1198)
> at org.apache.hadoop.hdfs.DFSUtil.getNameServiceId(DFSUtil.java:1131)
> at
> org.apache.hadoop.hdfs.DFSUtil.getNamenodeNameServiceId(DFSUtil.java:1086)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createLocalNamenodeHearbeatService(Router.java:466)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createNamenodeHearbeatServices(Router.java:423)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:199)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> 2018-03-01 18:05:56,208 ERROR
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
> router
> {code}
> Then the router tries to find the local namenode, multiple properties:
> {{dfs.namenode.rpc-address.ns1}}, {{dfs.namenode.rpc-address.ns-fed.r1}}
> match the local address.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13214) RBF: Configuration on Router conflicts with client side configuration
[
https://issues.apache.org/jira/browse/HDFS-13214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383371#comment-16383371
]
Yiqun Lin edited comment on HDFS-13214 at 3/2/18 9:00 AM:
--
Hi [~Tao Jie],
{quote}
So I think we should set dfs.nameservice.id to ns1 or ns2 rather than ns-fed.
{quote}
Yes, this should be right.
{quote}
Today property dfs.nameservice.id is not a necessary one in a federation
cluster (HA or non-HA), right?
{quote}
>From the function {{DFSUtil.getNameServiceId(Configuration conf, String
>addressKey)}} which used in the Router, {{dfs.nameservice.id}} doesn't must be
>specified. If this key isn't set, then it find the nameservice Id by matching
>the addressKey with the the address of the local node. If multiple addresses
>that matched, then the error will be thrown.
To avoid some potential problems happening in setting up Router, we can
recommend users to configure the {{dfs.nameservice.id}} which help directly
find the local node. If the local node is in a HA mode, {{dfs.ha.namenode.id}}
is also recommended to configure. Otherwise, it will also found this by
matching way. This can be documented in the doc of RBF. Any other thoughts?
was (Author: linyiqun):
Hi [~Tao Jie],
{quote}
So I think we should set dfs.nameservice.id to ns1 or ns2 rather than ns-fed.
{quote}
Yes, this should be right.
{quote}
Today property dfs.nameservice.id is not a necessary one in a federation
cluster (HA or non-HA), right?
{quote}
>From the function {{DFSUtil.getNameServiceId(Configuration conf, String
>addressKey)}} which used in the Router, {{dfs.nameservice.id}} doesn't must be
>specified. If this key isn't set, then it find the nameservice Id by matching
>the addressKey with the the address of the local node. If multiple addresses
>that matched, then the error will be thrown.
To avoid some potential problems happening in setting up Router, we can
recommend users to configure the {{dfs.nameservice.id}}. If the local node is
in a HA mode, {{dfs.ha.namenode.id}} is also recommended to configure.
Otherwise, it will also found this by matching way. This can be documented in
the doc of RBF. Any other thoughts?
> RBF: Configuration on Router conflicts with client side configuration
> -
>
> Key: HDFS-13214
> URL: https://issues.apache.org/jira/browse/HDFS-13214
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 2.9.0
>Reporter: Tao Jie
>Priority: Major
>
> In a typical router-based federation cluster, hdfs-site.xml is supposed to be:
> {code}
>
> dfs.nameservices
> ns1,ns2,ns-fed
>
>
> dfs.ha.namenodes.ns-fed
> r1,r2
>
>
> dfs.namenode.rpc-address.ns1
> host1:8020
>
>
> dfs.namenode.rpc-address.ns2
> host2:8020
>
>
> dfs.namenode.rpc-address.ns-fed.r1
> host1:
>
>
> dfs.namenode.rpc-address.ns-fed.r2
> host2:
>
> {code}
> {{dfs.ha.namenodes.ns-fed}} here is used for client to access the Router.
> However with this configuration on server node, Router fails to start with
> error:
> {code}
> org.apache.hadoop.HadoopIllegalArgumentException: Configuration has multiple
> addresses that match local node's address. Please configure the system with
> dfs.nameservice.id and dfs.ha.namenode.id
> at org.apache.hadoop.hdfs.DFSUtil.getSuffixIDs(DFSUtil.java:1198)
> at org.apache.hadoop.hdfs.DFSUtil.getNameServiceId(DFSUtil.java:1131)
> at
> org.apache.hadoop.hdfs.DFSUtil.getNamenodeNameServiceId(DFSUtil.java:1086)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createLocalNamenodeHearbeatService(Router.java:466)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.createNamenodeHearbeatServices(Router.java:423)
> at
> org.apache.hadoop.hdfs.server.federation.router.Router.serviceInit(Router.java:199)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter.main(DFSRouter.java:69)
> 2018-03-01 18:05:56,208 ERROR
> org.apache.hadoop.hdfs.server.federation.router.DFSRouter: Failed to start
> router
> {code}
> Then the router tries to find the local namenode, multiple properties:
> {{dfs.namenode.rpc-address.ns1}}, {{dfs.namenode.rpc-address.ns-fed.r1}}
> match the local address.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
