Re: Adding hard-disks to an existing HDFS cluster

That's on my list; is it perhaps as simple as adjusting the config file (ex: /etc/hadoop/conf/core-site.xml) by adding another value to the dfs.data.dir property? (Ref ) If so, afterward, it may or may not be necessary to rebalance; it can

Re: Adding hard-disks to an existing HDFS cluster

On Sun, Feb 28, 2010 at 5:14 PM, Eli Collins wrote: > You don't need to format to to add disks to DNs. Just format them and > add the directories to dfs.data.dir in the config file, and restart > the DN Nice; I was hoping it was that simple! :) I presume it makes no sense to try to spread the

Re: Adding hard-disks to an existing HDFS cluster

On Sun, Feb 28, 2010 at 5:27 PM, Eli Collins wrote: > dfs.name.dir (where the NN > stores its metadata) should have multiple directories on different > disks to guard against the failure of any single disk. Many people > also use RAIDed disks and include an NFS mount in dfs.name.dir to have > add

Re: Adding hard-disks to an existing HDFS cluster

On Mon, Mar 1, 2010 at 5:48 AM, Steve Loughran wrote: > Best of all: a secondary namenode to get the streamed event log, as that > will mean your cluster restarts faster. You do not want to lose your NN > data. > If the NN data is lost, all the HDFS data is functionally lost, right? - Marc

Re: Adding hard-disks to an existing HDFS cluster

On Mon, Mar 1, 2010 at 2:00 PM, Eli Collins wrote: > Yes, it's good to have multiple directories as well as may each or at > least some of the directories reliable, eg below > /data//dfs/namenode are local disks and /mnt/filer-hdfs is a > reliable NFS filer. > > dfs.name.dir > > /data/1/dfs/namen

Re: Adding hard-disks to an existing HDFS cluster

On Mon, Mar 1, 2010 at 2:02 PM, Eli Collins wrote: > Yes, it is important that the NN metadata (the fsimage and edits log) > are stored reliably. There's a great chapter on HDFS administration > in the book Hadoop: The Definitive Guide. Ah; I'd discounted it, because the hadoop ecosystem seems

Re: Adding hard-disks to an existing HDFS cluster

On Mon, Mar 1, 2010 at 7:20 PM, Allen Wittenauer wrote: > You might find my preso I did on Hadoop at Apachecon EU last year handy... Terrific; thanks! - Marc