[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049139#comment-17049139 ] runzhiwang commented on HDDS-1933: -- [~msingh] Hi, I have set "dfs.datanode.use.datanode.hostname" to true, but it does not work. Have you test it ? I want to fix this bug. > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049773#comment-17049773 ] runzhiwang commented on HDDS-1933: -- I'm working on it > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[
https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058268#comment-17058268
]
Attila Doroszlai commented on HDDS-1933:
Setting {{dfs.datanode.use.datanode.hostname=true}} is not enough, because
Ozone also [uses IP for Raft peer
address|https://github.com/apache/hadoop-ozone/blob/a144ab871e5e7af0416eb0cb5143b02183336d0f/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java#L93],
which is stored in Raft group config.
The problem can be reproduced (most of the time) even in docker-compose
environment:
{code}
cd hadoop-ozone/dist/target/ozone-0.5.0-SNAPSHOT/compose/ozone
OZONE_REPLICATION_FACTOR=3 ./run.sh
OZONE_REPLICATION_FACTOR=3 ./run.sh
{code}
If IP addresses are shuffled by Docker during restart, datanodes will not be
able to elect leader.
> Datanode should use hostname in place of ip addresses to allow DN's to work
> when ipaddress change
> -
>
> Key: HDDS-1933
> URL: https://issues.apache.org/jira/browse/HDDS-1933
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode, SCM
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Priority: Blocker
>
> This was noticed by [~elek] while deploying Ozone on Kubernetes based
> environment.
> When the datanode ip address change on restart, the Datanode details cease to
> be correct for the datanode. and this prevents the cluster from functioning
> after a restart.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058274#comment-17058274 ] Siddharth Wagle commented on HDDS-1933: --- [~adoroszlai] But what if you set _dfs.datanode.use.datanode.hostname_? Does it still fail? > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058297#comment-17058297 ] runzhiwang commented on HDDS-1933: -- [~swagle] set dfs.datanode.use.datanode.hostname cannot work. The reason just as [~adoroszlai] said, I have test it. > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058301#comment-17058301 ] Siddharth Wagle commented on HDDS-1933: --- [~yjxxtd] are you making a change so that this config is respected? I think that should suffice, what do you think? > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058302#comment-17058302 ] runzhiwang commented on HDDS-1933: -- [~swagle] Hi, I'm sorry, but what do you mean "making a change so that this config is respected" ? > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058304#comment-17058304 ] Siddharth Wagle commented on HDDS-1933: --- DatanodeDetails are identified by unique UUID already, the problem according to me is that NodeManager has a Map DatanodeDetails> which can be Map DatanodeDetails> which should make it ip independent based on my previous analysis, although I have tested this, just looked at the code and pointed out where we use this config property to decide what should be the key in the Map. > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058305#comment-17058305 ] runzhiwang commented on HDDS-1933: -- [~swagle] Okay, I see. I will check the code and find out how to make it work. > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058476#comment-17058476 ] runzhiwang commented on HDDS-1933: -- [~swagle] Hi, I think the following code use the ipaddress of datanode and the code [~adoroszlai] has found out were the root cause. I will submit a PR. https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java#L173 > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[ https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058543#comment-17058543 ] runzhiwang commented on HDDS-1933: -- I change getIpAddress at the following links to getHostName, it works fine. 1. https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientGrpc.java#L173. 2. https://github.com/apache/hadoop-ozone/blob/master/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/ratis/RatisHelper.java#L93. (pointed by [~adoroszlai]) But I'm not sure which of the following two fixes is better: 1. replace the two fileds ipAddress and hostName in DatanodeDetails with only one field dnsName: dnsName = if (dfs.datanode.use.datanode.hostname) hostName else ipAddress, this fix is thorough but has to change a lot of code and has a high risk; 2. pass the parameter dfs.datanode.use.datanode.hostname to methods which need to difference between hostName and ipAddress, this fix is simple but if someone forget to do this in the case need to difference between hostName and ipAddress, it will produce a bug. So what do you think ? [~Sammi][~xyao][~adoroszlai][~swagle][~msingh] > Datanode should use hostname in place of ip addresses to allow DN's to work > when ipaddress change > - > > Key: HDDS-1933 > URL: https://issues.apache.org/jira/browse/HDDS-1933 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Priority: Blocker > > This was noticed by [~elek] while deploying Ozone on Kubernetes based > environment. > When the datanode ip address change on restart, the Datanode details cease to > be correct for the datanode. and this prevents the cluster from functioning > after a restart. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1933) Datanode should use hostname in place of ip addresses to allow DN's to work when ipaddress change
[
https://issues.apache.org/jira/browse/HDDS-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17058681#comment-17058681
]
Attila Doroszlai commented on HDDS-1933:
One more note: our docker-compose samples won't work with hostnames because
Java's {{URI}} validation rejects underscores ({{_}}), which compose uses to
generate network and host names. A workaround is to set custom {{hostname}}
and {{container_name}} in {{docker-compose.yaml}} for each service (and define
custom network), but that prevents {{--scale}}-ing services to more than 1
instance.
> Datanode should use hostname in place of ip addresses to allow DN's to work
> when ipaddress change
> -
>
> Key: HDDS-1933
> URL: https://issues.apache.org/jira/browse/HDDS-1933
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode, SCM
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Priority: Blocker
>
> This was noticed by [~elek] while deploying Ozone on Kubernetes based
> environment.
> When the datanode ip address change on restart, the Datanode details cease to
> be correct for the datanode. and this prevents the cluster from functioning
> after a restart.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: ozone-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: ozone-issues-h...@hadoop.apache.org
