Re: One NameNode keeps Rolling Edit Log on HDFS Federation

2013-02-24 Thread Harsh J
Could you send us the log messages (with timestamps) which you think is behind the behavior you see? The new edit log format is different than the one in 1.x, and uses several smaller edit log files. This new format is described in PDFs attached at https://issues.apache.org/jira/browse/HDFS-1073, a

Re: One NameNode keeps Rolling Edit Log on HDFS Federation

2013-02-24 Thread YouPeng Yang
Hi Azuryy,Harsh It is strange that the NameNode rolling edits log rolls edits logs all the times,even i do nothing on the NameNode vi client. Any futher help will be appreciated. regard. 2013/2/25 YouPeng Yang > Hi Azuryy > >Yes.that is what is. >Thank you for your reply.

Re: One NameNode keeps Rolling Edit Log on HDFS Federation

2013-02-24 Thread YouPeng Yang
Hi Azuryy Yes.that is what is. Thank you for your reply. I am making effort to make clear about federation and HA. Regards. 2013/2/25 Azuryy Yu > I think you mixed federation with HA. am I right? > > If another name node hasn't changes, then It doesn't do any edit log > rolling. fede

Re: One NameNode keeps Rolling Edit Log on HDFS Federation

2013-02-24 Thread Azuryy Yu
I think you mixed federation with HA. am I right? If another name node hasn't changes, then It doesn't do any edit log rolling. federated NNs don't keep concurrency( I think you want say keep sync-able?) On Sun, Feb 24, 2013 at 11:09 PM, YouPeng Yang wrote: > Hi All > > I'm testing the HDFS Fe

Re: One NameNode keeps Rolling Edit Log on HDFS Federation

2013-02-24 Thread YouPeng Yang
Hi Mike,Harsh Thank you for your reply. I need to time to digest the above stuff. Thanks Regards 2013/2/25 Harsh J > Mike, > > I don't see how SPOF comes into the picture when HA is already present > in the releases that also carry Federation and each NN (federated or > non) can b

Re: One NameNode keeps Rolling Edit Log on HDFS Federation

2013-02-24 Thread Harsh J
Mike, I don't see how SPOF comes into the picture when HA is already present in the releases that also carry Federation and each NN (federated or non) can be assigned further Standby-NN roles. At this point we can stop using the word "SPOF" completely for HDFS. Would do great good for avoiding fur

Re: One NameNode keeps Rolling Edit Log on HDFS Federation

2013-02-24 Thread Michel Segel
I think part of the confusion stems from the fact that federation of name nodes only splits the very large cluster in to smaller portions of the same cluster. If you lose a federated name node, you only lose a portion of the cluster not the whole thing. So now instead of one SPOF, you have two S

Re: One NameNode keeps Rolling Edit Log on HDFS Federation

2013-02-24 Thread Harsh J
Hi, Federated namenodes are independent of one another (except that they both get reports from all the/common DNs in the cluster). It is natural to see one roll its edit logs based on its own rate of metadata growth, as compared to the other. Their edits, image, etc. everything is independent - th