Secondary Namenode on hadoop 0.20.205 ?
Hey people, How can we setup another machine in the cluster as Secondary Namenode in hadoop 0.20.205 ? Can a DN also act as SNN, any pros and cons of having this configuration ? Thanks, Praveenesh
RE: Secondary Namenode on hadoop 0.20.205 ?
Hey Praveenesh, You can start secondary namenode also by just giving the option ./hadoop secondarynamenode DN can not act as seconday namenode. The basic work for seconday namenode is to do checkpointing and getting the edits insync with Namenode till last checkpointing period. DN is to store the real data blocks physically. you need to configure correct namenode http address also for the secondaryNN, so that it can connect NN for checkpointing operations. http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode You can configure secondary node IP in masters file, start-dfs.sh itself will start the SNN automatically as it starts DN and NN as well. also you can see http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/ Regards, Uma From: praveenesh kumar [praveen...@gmail.com] Sent: Monday, December 26, 2011 5:05 PM To: common-user@hadoop.apache.org Subject: Secondary Namenode on hadoop 0.20.205 ? Hey people, How can we setup another machine in the cluster as Secondary Namenode in hadoop 0.20.205 ? Can a DN also act as SNN, any pros and cons of having this configuration ? Thanks, Praveenesh
Re: Secondary Namenode on hadoop 0.20.205 ?
(Answering beyond Uma's reply) > Can a DN also act as SNN, any pros and cons of having this configuration ? You can run SNN on a regular slave box if you can't have a dedicate a box, it shouldn't be an issue for small clusters -- Do ensure its disk configuration is proper, and its allocated near to the same heap as the NameNode is. For large clusters where the fsimage and periodic edits file sizes are larger, it would be worth placing it on a separate box given SNN's interactions. On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote: > Hey Praveenesh, > > You can start secondary namenode also by just giving the option ./hadoop > secondarynamenode > > DN can not act as seconday namenode. The basic work for seconday namenode is > to do checkpointing and getting the edits insync with Namenode till last > checkpointing period. DN is to store the real data blocks physically. > you need to configure correct namenode http address also for the > secondaryNN, so that it can connect NN for checkpointing operations. > http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode > You can configure secondary node IP in masters file, start-dfs.sh itself will > start the SNN automatically as it starts DN and NN as well. > > also you can see > http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/ > > Regards, > Uma > > From: praveenesh kumar [praveen...@gmail.com] > Sent: Monday, December 26, 2011 5:05 PM > To: common-user@hadoop.apache.org > Subject: Secondary Namenode on hadoop 0.20.205 ? > > Hey people, > > How can we setup another machine in the cluster as Secondary Namenode > in hadoop 0.20.205 ? > Can a DN also act as SNN, any pros and cons of having this configuration ? > > Thanks, > Praveenesh
Re: Secondary Namenode on hadoop 0.20.205 ?
Thanks..But, my 1st question is still unanswered. I have a 8 DN/TT machines and 1 NN machine. I want to set one of my DN/TT machine as SNN. How I have to configure my conf/*.xml files to achieve this ? Thanks, Praveenesh On Mon, Dec 26, 2011 at 8:44 PM, Harsh J wrote: > (Answering beyond Uma's reply) > >> Can a DN also act as SNN, any pros and cons of having this configuration ? > > You can run SNN on a regular slave box if you can't have a dedicate a box, it > shouldn't be an issue for small clusters -- Do ensure its disk configuration > is proper, and its allocated near to the same heap as the NameNode is. > > For large clusters where the fsimage and periodic edits file sizes are > larger, it would be worth placing it on a separate box given SNN's > interactions. > > On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote: > >> Hey Praveenesh, >> >> You can start secondary namenode also by just giving the option ./hadoop >> secondarynamenode >> >> DN can not act as seconday namenode. The basic work for seconday namenode is >> to do checkpointing and getting the edits insync with Namenode till last >> checkpointing period. DN is to store the real data blocks physically. >> you need to configure correct namenode http address also for the >> secondaryNN, so that it can connect NN for checkpointing operations. >> http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode >> You can configure secondary node IP in masters file, start-dfs.sh itself >> will start the SNN automatically as it starts DN and NN as well. >> >> also you can see >> http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/ >> >> Regards, >> Uma >> ____________ >> From: praveenesh kumar [praveen...@gmail.com] >> Sent: Monday, December 26, 2011 5:05 PM >> To: common-user@hadoop.apache.org >> Subject: Secondary Namenode on hadoop 0.20.205 ? >> >> Hey people, >> >> How can we setup another machine in the cluster as Secondary Namenode >> in hadoop 0.20.205 ? >> Can a DN also act as SNN, any pros and cons of having this configuration ? >> >> Thanks, >> Praveenesh >
Re: Secondary Namenode on hadoop 0.20.205 ?
The link Uma passed already covered that question: http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/ [dfs.http.address in hdfs-site.xml pointing to NN_HOST:50070 should do.] Also, if you are using the tarball start/stop scripts, putting in the hostname for SNN in the conf/masters list is sufficient to get it auto-started there. On 27-Dec-2011, at 11:36 AM, praveenesh kumar wrote: > Thanks..But, my 1st question is still unanswered. > I have a 8 DN/TT machines and 1 NN machine. > I want to set one of my DN/TT machine as SNN. > How I have to configure my conf/*.xml files to achieve this ? > > Thanks, > Praveenesh > > On Mon, Dec 26, 2011 at 8:44 PM, Harsh J wrote: >> (Answering beyond Uma's reply) >> >>> Can a DN also act as SNN, any pros and cons of having this configuration ? >> >> You can run SNN on a regular slave box if you can't have a dedicate a box, >> it shouldn't be an issue for small clusters -- Do ensure its disk >> configuration is proper, and its allocated near to the same heap as the >> NameNode is. >> >> For large clusters where the fsimage and periodic edits file sizes are >> larger, it would be worth placing it on a separate box given SNN's >> interactions. >> >> On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote: >> >>> Hey Praveenesh, >>> >>> You can start secondary namenode also by just giving the option ./hadoop >>> secondarynamenode >>> >>> DN can not act as seconday namenode. The basic work for seconday namenode >>> is to do checkpointing and getting the edits insync with Namenode till last >>> checkpointing period. DN is to store the real data blocks physically. >>> you need to configure correct namenode http address also for the >>> secondaryNN, so that it can connect NN for checkpointing operations. >>> http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode >>> You can configure secondary node IP in masters file, start-dfs.sh itself >>> will start the SNN automatically as it starts DN and NN as well. >>> >>> also you can see >>> http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/ >>> >>> Regards, >>> Uma >>> >>> From: praveenesh kumar [praveen...@gmail.com] >>> Sent: Monday, December 26, 2011 5:05 PM >>> To: common-user@hadoop.apache.org >>> Subject: Secondary Namenode on hadoop 0.20.205 ? >>> >>> Hey people, >>> >>> How can we setup another machine in the cluster as Secondary Namenode >>> in hadoop 0.20.205 ? >>> Can a DN also act as SNN, any pros and cons of having this configuration ? >>> >>> Thanks, >>> Praveenesh >>
Re: Secondary Namenode on hadoop 0.20.205 ?
Cool. I just did that.. So now I am seeing my fsimage file on SNN's hadoop.tmp.dir... So incase my NN went down.. I can take this image file from SNN and paste it at NN's *dfs.name.dir/current/fsimage * and I can have NN up based on last snapshot that SNN had, right ? Thanks, Praveenesh On Tue, Dec 27, 2011 at 12:20 PM, Harsh J wrote: > The link Uma passed already covered that question: http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/[dfs.http.address in hdfs-site.xml pointing to NN_HOST:50070 should d.] > > Also, if you are using the tarball start/stop scripts, putting in the hostname for SNN in the conf/masters list is sufficient to get it auto-started there. > > On 27-Dec-2011, at 11:36 AM, praveenesh kumar wrote: > >> Thanks..But, my 1st question is still unanswered. >> I have a 8 DN/TT machines and 1 NN machine. >> I want to set one of my DN/TT machine as SNN. >> How I have to configure my conf/*.xml files to achieve this ? >> >> Thanks, >> Praveenesh >> >> On Mon, Dec 26, 2011 at 8:44 PM, Harsh J wrote: >>> (Answering beyond Uma's reply) >>> >>>> Can a DN also act as SNN, any pros and cons of having this configuration ? >>> >>> You can run SNN on a regular slave box if you can't have a dedicate a box, it shouldn't be an issue for small clusters -- Do ensure its disk configuration is proper, and its allocated near to the same heap as the NameNode is. >>> >>> For large clusters where the fsimage and periodic edits file sizes are larger, it would be worth placing it on a separate box given SNN's interactions. >>> >>> On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote: >>> >>>> Hey Praveenesh, >>>> >>>> You can start secondary namenode also by just giving the option ./hadoop secondarynamenode >>>> >>>> DN can not act as seconday namenode. The basic work for seconday namenode is to do checkpointing and getting the edits insync with Namenode till last checkpointing period. DN is to store the real data blocks physically. >>>> you need to configure correct namenode http address also for the secondaryNN, so that it can connect NN for checkpointing operations. >>>> http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode >>>> You can configure secondary node IP in masters file, start-dfs.sh itself will start the SNN automatically as it starts DN and NN as well. >>>> >>>> also you can see http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/ >>>> >>>> Regards, >>>> Uma >>>> >>>> From: praveenesh kumar [praveen...@gmail.com] >>>> Sent: Monday, December 26, 2011 5:05 PM >>>> To: common-user@hadoop.apache.org >>>> Subject: Secondary Namenode on hadoop 0.20.205 ? >>>> >>>> Hey people, >>>> >>>> How can we setup another machine in the cluster as Secondary Namenode >>>> in hadoop 0.20.205 ? >>>> Can a DN also act as SNN, any pros and cons of having this configuration ? >>>> >>>> Thanks, >>>> Praveenesh >>> >
Re: Secondary Namenode on hadoop 0.20.205 ?
Yes, checkpoints are helpful when your original NN image goes corrupt (very very rare, if you use dual or more dfs.name.dir points to be safe). On 27-Dec-2011, at 12:33 PM, praveenesh kumar wrote: > Cool. > I just did that.. > So now I am seeing my fsimage file on SNN's hadoop.tmp.dir... > So incase my NN went down.. I can take this image file from SNN and paste > it at NN's *dfs.name.dir/current/fsimage * > and I can have NN up based on last snapshot that SNN had, right ? > > Thanks, > Praveenesh > > On Tue, Dec 27, 2011 at 12:20 PM, Harsh J wrote: >> The link Uma passed already covered that question: > http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/[dfs.http.address > in hdfs-site.xml pointing to NN_HOST:50070 should d.] >> >> Also, if you are using the tarball start/stop scripts, putting in the > hostname for SNN in the conf/masters list is sufficient to get it > auto-started there. >> >> On 27-Dec-2011, at 11:36 AM, praveenesh kumar wrote: >> >>> Thanks..But, my 1st question is still unanswered. >>> I have a 8 DN/TT machines and 1 NN machine. >>> I want to set one of my DN/TT machine as SNN. >>> How I have to configure my conf/*.xml files to achieve this ? >>> >>> Thanks, >>> Praveenesh >>> >>> On Mon, Dec 26, 2011 at 8:44 PM, Harsh J wrote: >>>> (Answering beyond Uma's reply) >>>> >>>>> Can a DN also act as SNN, any pros and cons of having this > configuration ? >>>> >>>> You can run SNN on a regular slave box if you can't have a dedicate a > box, it shouldn't be an issue for small clusters -- Do ensure its disk > configuration is proper, and its allocated near to the same heap as the > NameNode is. >>>> >>>> For large clusters where the fsimage and periodic edits file sizes are > larger, it would be worth placing it on a separate box given SNN's > interactions. >>>> >>>> On 26-Dec-2011, at 7:53 PM, Uma Maheswara Rao G wrote: >>>> >>>>> Hey Praveenesh, >>>>> >>>>> You can start secondary namenode also by just giving the option > ./hadoop secondarynamenode >>>>> >>>>> DN can not act as seconday namenode. The basic work for seconday > namenode is to do checkpointing and getting the edits insync with Namenode > till last checkpointing period. DN is to store the real data blocks > physically. >>>>> you need to configure correct namenode http address also for the > secondaryNN, so that it can connect NN for checkpointing operations. >>>>> > http://hadoop.apache.org/common/docs/current/hdfs_user_guide.html#Secondary+NameNode >>>>> You can configure secondary node IP in masters file, start-dfs.sh > itself will start the SNN automatically as it starts DN and NN as well. >>>>> >>>>> also you can see > http://www.cloudera.com/blog/2009/02/multi-host-secondarynamenode-configuration/ >>>>> >>>>> Regards, >>>>> Uma >>>>> >>>>> From: praveenesh kumar [praveen...@gmail.com] >>>>> Sent: Monday, December 26, 2011 5:05 PM >>>>> To: common-user@hadoop.apache.org >>>>> Subject: Secondary Namenode on hadoop 0.20.205 ? >>>>> >>>>> Hey people, >>>>> >>>>> How can we setup another machine in the cluster as Secondary Namenode >>>>> in hadoop 0.20.205 ? >>>>> Can a DN also act as SNN, any pros and cons of having this > configuration ? >>>>> >>>>> Thanks, >>>>> Praveenesh >>>> >>