HDFS federation Configuration
Guys, I need some tutorials to configure fedration. Can you pls suggest me some? Thanks, Manickam P
HDFS federation configuration
Hi, We tried setting up HDFS name node federation set up with 2 name nodes. I am facing few issues. Can any one help me in understanding below points? 1) how can we configure different namespaces to different name node? Where exactly we need to configure this? 2) After formatting each NN with one cluster id, Do we need to set this cluster id in hdfs-site.xml? 3) I am getting exception like, data dir already locked by one of the NN, But when don't specify data.dir, then it's not showing exception. So what could be the issue? Thanks & Regards, B Anil Kumar.
Re: HDFS federation Configuration
Have you looked at - http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html Let me know if the document is not clear or needs improvements. Regards, Suresh On Thu, Sep 19, 2013 at 12:01 PM, Manickam P wrote: > Guys, > > I need some tutorials to configure fedration. Can you pls suggest me some? > > > > > Thanks, > Manickam P > -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: HDFS federation Configuration
Hi Suresh, I'm not able to follow the page completely. Can you pls help me to get some clear step by step or little bit more details in the configuration side? I am trying to follow a 3 node machines with 2.1.0 beta version of hadoop. My intention is having 2 federated name node and 2 data nodes. Thanks, Manickam P Date: Thu, 19 Sep 2013 12:28:04 -0700 Subject: Re: HDFS federation Configuration From: sur...@hortonworks.com To: user@hadoop.apache.org Have you looked at - http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html Let me know if the document is not clear or needs improvements. Regards,Suresh On Thu, Sep 19, 2013 at 12:01 PM, Manickam P wrote: Guys, I need some tutorials to configure fedration. Can you pls suggest me some? Thanks, Manickam P -- http://hortonworks.com/download/ CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: HDFS federation Configuration
Hi, Thanks. I have done that. Thanks, Manickam P Date: Mon, 23 Sep 2013 10:02:42 -0700 Subject: Re: HDFS federation Configuration From: sur...@hortonworks.com To: user@hadoop.apache.org I'm not able to follow the page completely. Can you pls help me to get some clear step by step or little bit more details in the configuration side? Have you setup a non-federated cluster before. If you have, the page should be easy to follow. If you have not setup a non-federated cluster before, I suggest doing so, before looking at this document. I think the document already has step by step instructions.I CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
RE: HDFS federation Configuration
Hi, I got couple of doubts while doing this. Please help me to understand that. How to generate or find out the the cluster id for the below step? I saw that for one node we can format without this cluster id and we need to pass same cluster id which got generated for the other name node while formatting or else we could not get the federated cluster. $HADOOP_PREFIX_HOME/bin/hdfs namenode -format -clusterId In my case i want to have two name nodes and 3 data nodes. Do i need to mention all the data nodes ip address in the slave files in two name nodes? Thanks,Manickam P Date: Mon, 23 Sep 2013 10:02:42 -0700 Subject: Re: HDFS federation Configuration From: sur...@hortonworks.com To: user@hadoop.apache.org I'm not able to follow the page completely. Can you pls help me to get some clear step by step or little bit more details in the configuration side? Have you setup a non-federated cluster before. If you have, the page should be easy to follow. If you have not setup a non-federated cluster before, I suggest doing so, before looking at this document. I think the document already has step by step instructions.I CONFIDENTIALITY NOTICENOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: HDFS federation Configuration
> > I'm not able to follow the page completely. > Can you pls help me to get some clear step by step or little bit more > details in the configuration side? > Have you setup a non-federated cluster before. If you have, the page should be easy to follow. If you have not setup a non-federated cluster before, I suggest doing so, before looking at this document. I think the document already has step by step instructions. > I > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: HDFS federation configuration
Have you looked at - http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/Federation.html ? On Thu, Jan 23, 2014 at 9:35 AM, AnilKumar B wrote: > Hi, > > We tried setting up HDFS name node federation set up with 2 name nodes. I > am facing few issues. > > Can any one help me in understanding below points? > > 1) how can we configure different namespaces to different name node? Where > exactly we need to configure this? > See the documentation. If it is not clear, please open a jira. > > 2) After formatting each NN with one cluster id, Do we need to set this > cluster id in hdfs-site.xml? > There is no need to set the cluster id in hdfs-site.xml > > 3) I am getting exception like, data dir already locked by one of the NN, > But when don't specify data.dir, then it's not showing exception. So what > could be the issue? > Are you running the two namenode processes on the same machine? > > Thanks & Regards, > B Anil Kumar. > -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: HDFS federation configuration
Thanks Suresh. I followed the link it's clear now. But client side configuration is not covered on the doc. Thanks & Regards, B Anil Kumar. On Thu, Jan 23, 2014 at 11:44 PM, Suresh Srinivas wrote: > Have you looked at - > http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/Federation.html > ? > > > > On Thu, Jan 23, 2014 at 9:35 AM, AnilKumar B wrote: > >> Hi, >> >> We tried setting up HDFS name node federation set up with 2 name nodes. I >> am facing few issues. >> >> Can any one help me in understanding below points? >> >> 1) how can we configure different namespaces to different name node? >> Where exactly we need to configure this? >> > See the documentation. If it is not clear, please open a jira. > > >> >> 2) After formatting each NN with one cluster id, Do we need to set this >> cluster id in hdfs-site.xml? >> > There is no need to set the cluster id in hdfs-site.xml > > >> >> 3) I am getting exception like, data dir already locked by one of the NN, >> But when don't specify data.dir, then it's not showing exception. So what >> could be the issue? >> > > Are you running the two namenode processes on the same machine? > >> >> Thanks & Regards, >> B Anil Kumar. >> > > > > -- > http://hortonworks.com/download/ > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
