For dfs.datanode.scan.period.hours, why isn't it documented here http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
Also, once these settings are in effect, how can I see that they are active? is there a JSON page I can login to see them? On Mon, Oct 7, 2013 at 10:50 AM, Ravi Prakash <ravi...@ymail.com> wrote: > Rita! > > 14-16 Tb is perhaps a big node. Even then the scalability limits of the > Namenode in your case would depend on how many files (more accurately how > many blocks) there are on HDFS. > > In any case, if you want the datanodes to be marked dead quickly when > their heartbeats are lost, you should reduce the two parameters I told you > about. > > The datanode block scanner is unfortunately hard coded to use a maximum of > 8Mb/s and a minimum of 1 Mb/s. The only thing you can change is > dfs.datanode.scan.period.hours > > HTH > Ravi > > > ________________________________ > From: Rita <rmorgan...@gmail.com> > To: "<common-user@hadoop.apache.org>" <common-user@hadoop.apache.org>; > Ravi Prakash <ravi...@ymail.com> > Sent: Monday, October 7, 2013 5:55 AM > Subject: Re: datanode tuning > > > Thanks Ravi. The number of nodes isn't a lot but the size is rather large. > Each data node has about 14-16T (560-640T). > > For the datanode block scanner, how can increase its "Current scan rate > limit KBps" ? > > > > > > On Sun, Oct 6, 2013 at 11:09 PM, Ravi Prakash <ravi...@ymail.com> wrote: > > > Please look at dfs.heartbeat.interval and > > dfs.namenode.heartbeat.recheck-interval > > > > 40 datanodes is not a large cluster IMHO and the Namenode is capable of > > managing 100 times more datanodes. > > > > > > > > > > ________________________________ > > From: Rita <rmorgan...@gmail.com> > > To: "<common-user@hadoop.apache.org>" <common-user@hadoop.apache.org> > > Sent: Sunday, October 6, 2013 9:49 AM > > Subject: datanode tuning > > > > > > I would like my 40 data nodes to aggressively report to namenode if they > > are alive or not therefore I think I need to change these params > > > > dfs.block.access.token.lifetime : Default is 600 seconds. Can I decrease > > this to 60? > > > > > > dfs.block.access.key.update.interval: Default is 600 seconds. Can I > > decrease this to 60? > > > > Also, what are some other turnings people do for datanodes in a > relatively > > large cluster? > > > > > > > > -- > > --- Get your facts first, then you can distort them as you please.-- > > > > > > -- > --- Get your facts first, then you can distort them as you please.-- > -- --- Get your facts first, then you can distort them as you please.--
