Re: dfs.name.dir capacity for namenode backup?
On Mon, May 17, 2010 at 5:10 PM, jiang licht licht_ji...@yahoo.com wrote: I am considering to use a machine to save a redundant copy of HDFS metadata through setting dfs.name.dir in hdfs-site.xml like this (as in YDN): property namedfs.name.dir/name value/home/hadoop/dfs/name,/mnt/namenode-backup/value finaltrue/final /property where the two folders are on different machines so that /mnt/namenode-backup keeps a copy of hdfs file system information and its machine can be used to replace the first machine that fails as namenode. So, my question is how big this hdfs metatdata will consume? I guess it is proportional to the hdfs capacity. What ratio is that or what size will be for 150TB hdfs? On the order of a few GB, max (you really need double the size of your image, so it has tmp space when downloading a checkpoint or performing an upgrade). But on any disk you can buy these days you'll have plenty of space. -Todd -- Todd Lipcon Software Engineer, Cloudera
Re: dfs.name.dir capacity for namenode backup?
Sorry to hijack but after following this thread, I had a related question to the secondary location of dfs.name.dir. Is the approach outlined below the preferred/suggested way to do this? Is this people mean when they say, stick it on NFS ? Thanks! On May 17, 2010, at 11:14 PM, Todd Lipcon wrote: On Mon, May 17, 2010 at 5:10 PM, jiang licht licht_ji...@yahoo.com wrote: I am considering to use a machine to save a redundant copy of HDFS metadata through setting dfs.name.dir in hdfs-site.xml like this (as in YDN): property namedfs.name.dir/name value/home/hadoop/dfs/name,/mnt/namenode-backup/value finaltrue/final /property where the two folders are on different machines so that /mnt/namenode-backup keeps a copy of hdfs file system information and its machine can be used to replace the first machine that fails as namenode. So, my question is how big this hdfs metatdata will consume? I guess it is proportional to the hdfs capacity. What ratio is that or what size will be for 150TB hdfs? On the order of a few GB, max (you really need double the size of your image, so it has tmp space when downloading a checkpoint or performing an upgrade). But on any disk you can buy these days you'll have plenty of space. -Todd -- Todd Lipcon Software Engineer, Cloudera
Re: dfs.name.dir capacity for namenode backup?
Yes, we recommend at least one local directory and one NFS directory for dfs.name.dir in production environments. This allows an up-to-date recovery of NN metadata if the NN should fail. In future versions the BackupNode functionality will move us one step closer to not needing NFS for production deployments. Note that the NFS directory does not need to be anything fancy - you can simply use an NFS mount on another normal Linux box. -Todd On Tue, May 18, 2010 at 11:19 AM, Andrew Nguyen and...@ucsfcti.org wrote: Sorry to hijack but after following this thread, I had a related question to the secondary location of dfs.name.dir. Is the approach outlined below the preferred/suggested way to do this? Is this people mean when they say, stick it on NFS ? Thanks! On May 17, 2010, at 11:14 PM, Todd Lipcon wrote: On Mon, May 17, 2010 at 5:10 PM, jiang licht licht_ji...@yahoo.com wrote: I am considering to use a machine to save a redundant copy of HDFS metadata through setting dfs.name.dir in hdfs-site.xml like this (as in YDN): property namedfs.name.dir/name value/home/hadoop/dfs/name,/mnt/namenode-backup/value finaltrue/final /property where the two folders are on different machines so that /mnt/namenode-backup keeps a copy of hdfs file system information and its machine can be used to replace the first machine that fails as namenode. So, my question is how big this hdfs metatdata will consume? I guess it is proportional to the hdfs capacity. What ratio is that or what size will be for 150TB hdfs? On the order of a few GB, max (you really need double the size of your image, so it has tmp space when downloading a checkpoint or performing an upgrade). But on any disk you can buy these days you'll have plenty of space. -Todd -- Todd Lipcon Software Engineer, Cloudera -- Todd Lipcon Software Engineer, Cloudera
dfs.name.dir capacity for namenode backup?
I am considering to use a machine to save a redundant copy of HDFS metadata through setting dfs.name.dir in hdfs-site.xml like this (as in YDN): property namedfs.name.dir/name value/home/hadoop/dfs/name,/mnt/namenode-backup/value finaltrue/final /property where the two folders are on different machines so that /mnt/namenode-backup keeps a copy of hdfs file system information and its machine can be used to replace the first machine that fails as namenode. So, my question is how big this hdfs metatdata will consume? I guess it is proportional to the hdfs capacity. What ratio is that or what size will be for 150TB hdfs? Thanks, Michael
