Hi, I was just wondering if it is okay to exclude the NN IP.
I hope it won't create any "side effects". This is because I don't want to actually decommission the NN node, but just make it stop working as a DataNode. However, if you are sure there are no problems, I will go ahead and make the changes. Thanks & regards Arko On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <donta...@gmail.com> wrote: > It should. > > BTW what's the problem with decommissioning?It's the correct way of removing > the DN, as said by Charles earlier. Create the exclude file and add the > property in the hdfs-site.xml file. > > Warm Regards, > Tariq > https://mtariq.jux.com/ > cloudfront.blogspot.com > > > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee > <arkoprovomukher...@gmail.com> wrote: >> >> Hi, >> >> I see that the NameNode IP is present in conf/slaves file. >> >> Will removing the IP of the NN from the conf/slaves file and >> restarting hadoop help? >> >> Thanks & regards >> Arko >> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee >> <arkoprovomukher...@gmail.com> wrote: >> > Hi, >> > >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes >> > (running DN and Task Tracker). >> > >> > My issue is that the my NN is getting filled up pretty fast as the NN >> > acts as a DN. >> > >> > I want my NN not to function as the DN. >> > >> > Does including the IP for the NN in the "dfs.hosts.exclude" and >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem? >> > >> > Thanks & regards >> > Arko >> > >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <donta...@gmail.com> >> > wrote: >> >> Stopping DN on one machine means excluding it. What exactly is the >> >> requirement? How many nodes do you have right now? >> >> >> >> Warm Regards, >> >> Tariq >> >> https://mtariq.jux.com/ >> >> cloudfront.blogspot.com >> >> >> >> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee >> >> <arkoprovomukher...@gmail.com> wrote: >> >>> >> >>> Hi, >> >>> >> >>> I am not sure the provided link matches my situation. >> >>> >> >>> That is because, I am not trying to exclude any node but just trying >> >>> to stop the NN act as a DataNode. >> >>> >> >>> Thanks & regards >> >>> Arko >> >>> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee >> >>> <arkoprovomukher...@gmail.com> wrote: >> >>> > Thanks! Let me study and find out if it works for me. >> >>> > Thanks & regards >> >>> > Arko >> >>> > >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee >> >>> > <arkoprovomukher...@gmail.com> wrote: >> >>> >> Hi, >> >>> >> >> >>> >> But the new location will also be the same NN physical server. >> >>> >> So how does that help? >> >>> >> >> >>> >> I would like to clear up the space in the master by removing all >> >>> >> the >> >>> >> HDFS data to the slave nodes. >> >>> >> >> >>> >> Thanks & regards >> >>> >> Arko >> >>> >> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq >> >>> >> <donta...@gmail.com> >> >>> >> wrote: >> >>> >>> No, copy this directory to the new location of your choice. >> >>> >>> >> >>> >>> Warm Regards, >> >>> >>> Tariq >> >>> >>> https://mtariq.jux.com/ >> >>> >>> cloudfront.blogspot.com >> >>> >>> >> >>> >>> >> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee >> >>> >>> <arkoprovomukher...@gmail.com> wrote: >> >>> >>>> >> >>> >>>> Thanks! >> >>> >>>> However, in that case, how do I handle the current data that is >> >>> >>>> already in >> >>> >>>> NN? >> >>> >>>> Do I have to reformat HDFS? >> >>> >>>> Thanks & regards >> >>> >>>> Arko >> >>> >>>> >> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq >> >>> >>>> <donta...@gmail.com> >> >>> >>>> wrote: >> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate that >> >>> >>>> > machine >> >>> >>>> > to >> >>> >>>> > NN. >> >>> >>>> > Remove the hostname of the NN machine from the 'slaves' file. >> >>> >>>> > >> >>> >>>> > Warm Regards, >> >>> >>>> > Tariq >> >>> >>>> > https://mtariq.jux.com/ >> >>> >>>> > cloudfront.blogspot.com >> >>> >>>> > >> >>> >>>> > >> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee >> >>> >>>> > <arkoprovomukher...@gmail.com> wrote: >> >>> >>>> >> >> >>> >>>> >> Hi, >> >>> >>>> >> >> >>> >>>> >> Thanks for the help! >> >>> >>>> >> >> >>> >>>> >> However, I am still unsure about how to "turn off" the >> >>> >>>> >> datanode >> >>> >>>> >> feature on the NN. >> >>> >>>> >> >> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is >> >>> >>>> >> pointed >> >>> >>>> >> to >> >>> >>>> >> a directory. >> >>> >>>> >> >> >>> >>>> >> Should I just comment out that property? What would happen to >> >>> >>>> >> the >> >>> >>>> >> current data that there is in the Master? Will it get removed >> >>> >>>> >> automatically? >> >>> >>>> >> >> >>> >>>> >> Thanks & regards >> >>> >>>> >> Arko >> >>> >>>> >> >> >>> >>>> >> >> >>> >>>> >> >> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq >> >>> >>>> >> <donta...@gmail.com> >> >>> >>>> >> wrote: >> >>> >>>> >> > You can specify the logging level as specified by Charles. >> >>> >>>> >> > But >> >>> >>>> >> > turning >> >>> >>>> >> > logs >> >>> >>>> >> > off is never a good idea. Logs are really helpful in problem >> >>> >>>> >> > diagnosis, >> >>> >>>> >> > which are eventual. >> >>> >>>> >> > >> >>> >>>> >> > Warm Regards, >> >>> >>>> >> > Tariq >> >>> >>>> >> > https://mtariq.jux.com/ >> >>> >>>> >> > cloudfront.blogspot.com >> >>> >>>> >> > >> >>> >>>> >> > >> >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee >> >>> >>>> >> > <arkoprovomukher...@gmail.com> wrote: >> >>> >>>> >> >> >> >>> >>>> >> >> Hi, >> >>> >>>> >> >> >> >>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode. >> >>> >>>> >> >> >> >>> >>>> >> >> I would like to "turn off" this feature. >> >>> >>>> >> >> >> >>> >>>> >> >> Request help regarding the same. >> >>> >>>> >> >> >> >>> >>>> >> >> Thanks & regards >> >>> >>>> >> >> Arko >> >>> >>>> >> >> >> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker >> >>> >>>> >> >> <cba...@sdl.com> >> >>> >>>> >> >> wrote: >> >>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the >> >>> >>>> >> >> > NameNode >> >>> >>>> >> >> > which >> >>> >>>> >> >> > is >> >>> >>>> >> >> > not recommended practice. Normally, the only files the NN >> >>> >>>> >> >> > stores >> >>> >>>> >> >> > are >> >>> >>>> >> >> > the >> >>> >>>> >> >> > image and edits files. It does not store any actual HDFS >> >>> >>>> >> >> > data. If >> >>> >>>> >> >> > you >> >>> >>>> >> >> > must >> >>> >>>> >> >> > run a DN on the NN, try turning down the logging in >> >>> >>>> >> >> > /conf/log4j.properties: >> >>> >>>> >> >> > >> >>> >>>> >> >> > #hadoop.root.logger=INFO,console >> >>> >>>> >> >> > #hadoop.root.logger=WARN,console >> >>> >>>> >> >> > hadoop.root.logger=ERROR,console >> >>> >>>> >> >> > >> >>> >>>> >> >> > Depending on the logging information you require, of >> >>> >>>> >> >> > course. >> >>> >>>> >> >> > >> >>> >>>> >> >> > -Chuck >> >>> >>>> >> >> > >> >>> >>>> >> >> > >> >>> >>>> >> >> > -----Original Message----- >> >>> >>>> >> >> > From: Arko Provo Mukherjee >> >>> >>>> >> >> > [mailto:arkoprovomukher...@gmail.com] >> >>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM >> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org >> >>> >>>> >> >> > Subject: Managing space in Master Node >> >>> >>>> >> >> > >> >>> >>>> >> >> > Hello Gurus, >> >>> >>>> >> >> > >> >>> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments. >> >>> >>>> >> >> > >> >>> >>>> >> >> > The issue I am continuously facing is that the Master >> >>> >>>> >> >> > Node >> >>> >>>> >> >> > runs >> >>> >>>> >> >> > out >> >>> >>>> >> >> > of >> >>> >>>> >> >> > disk >> >>> >>>> >> >> > space due to logs and data files. >> >>> >>>> >> >> > >> >>> >>>> >> >> > I can monitor and delete log files. However, I cannot >> >>> >>>> >> >> > delete >> >>> >>>> >> >> > the >> >>> >>>> >> >> > HDFS >> >>> >>>> >> >> > data. >> >>> >>>> >> >> > >> >>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS >> >>> >>>> >> >> > data in >> >>> >>>> >> >> > the >> >>> >>>> >> >> > Master >> >>> >>>> >> >> > Node? >> >>> >>>> >> >> > >> >>> >>>> >> >> > Then I can use my master to handle the metadata only and >> >>> >>>> >> >> > store the >> >>> >>>> >> >> > logs. >> >>> >>>> >> >> > >> >>> >>>> >> >> > Thanks & regards >> >>> >>>> >> >> > Arko >> >>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. >> >>> >>>> >> >> > The >> >>> >>>> >> >> > information contained in this email may be confidential >> >>> >>>> >> >> > and/or >> >>> >>>> >> >> > legally >> >>> >>>> >> >> > privileged. It has been sent for the sole use of the >> >>> >>>> >> >> > intended >> >>> >>>> >> >> > recipient(s). >> >>> >>>> >> >> > If you are not the intended recipient of this mail, you >> >>> >>>> >> >> > are >> >>> >>>> >> >> > hereby >> >>> >>>> >> >> > notified >> >>> >>>> >> >> > that any unauthorized review, use, disclosure, >> >>> >>>> >> >> > dissemination, >> >>> >>>> >> >> > distribution, >> >>> >>>> >> >> > or copying of this communication, or any of its contents, >> >>> >>>> >> >> > is >> >>> >>>> >> >> > strictly >> >>> >>>> >> >> > prohibited. If you have received this communication in >> >>> >>>> >> >> > error, >> >>> >>>> >> >> > please >> >>> >>>> >> >> > reply >> >>> >>>> >> >> > to the sender and destroy all copies of the message. >> >>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225, >> >>> >>>> >> >> > Wakefield, MA >> >>> >>>> >> >> > 01880, >> >>> >>>> >> >> > USA >> >>> >>>> >> >> > >> >>> >>>> >> > >> >>> >>>> >> > >> >>> >>>> > >> >>> >>>> > >> >>> >>> >> >>> >>> >> >> >> >> > >
