Re: backup node question
Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J
Re: backup node question
Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J
Re: backup node question
Yes, it need not keep an edits (transactions) stream locally cause those are passed synchronously to the BackupNameNode, which persists it on its behalf. On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote: Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J -- Harsh J
Re: backup node question
Thanks for answering my question, Harsh. regards, Lin On Sun, Apr 7, 2013 at 4:05 PM, Harsh J ha...@cloudera.com wrote: Yes, it need not keep an edits (transactions) stream locally cause those are passed synchronously to the BackupNameNode, which persists it on its behalf. On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote: Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J -- Harsh J
Re: backup node question
Hi Harsh, Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x? On Sun, Apr 7, 2013 at 4:05 PM, Harsh J ha...@cloudera.com wrote: Yes, it need not keep an edits (transactions) stream locally cause those are passed synchronously to the BackupNameNode, which persists it on its behalf. On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote: Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J -- Harsh J
Re: backup node question
BackupNameNode is not present in the maintenance 1.x releases, it is a feature added to a higher version; you can try it out in 2.x today if you wish to. On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu azury...@gmail.com wrote: Hi Harsh, Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x? On Sun, Apr 7, 2013 at 4:05 PM, Harsh J ha...@cloudera.com wrote: Yes, it need not keep an edits (transactions) stream locally cause those are passed synchronously to the BackupNameNode, which persists it on its behalf. On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote: Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J -- Harsh J -- Harsh J
Re: backup node question
I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats Standby Namenode? --Send from my Sony mobile. On Apr 7, 2013 9:03 PM, Harsh J ha...@cloudera.com wrote: BackupNameNode is not present in the maintenance 1.x releases, it is a feature added to a higher version; you can try it out in 2.x today if you wish to. On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu azury...@gmail.com wrote: Hi Harsh, Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x? On Sun, Apr 7, 2013 at 4:05 PM, Harsh J ha...@cloudera.com wrote: Yes, it need not keep an edits (transactions) stream locally cause those are passed synchronously to the BackupNameNode, which persists it on its behalf. On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote: Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J -- Harsh J -- Harsh J
Re: backup node question
SNN=secondary name node in my last mail. --Send from my Sony mobile. On Apr 7, 2013 10:01 PM, Azuryy Yu azury...@gmail.com wrote: I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats Standby Namenode? --Send from my Sony mobile. On Apr 7, 2013 9:03 PM, Harsh J ha...@cloudera.com wrote: BackupNameNode is not present in the maintenance 1.x releases, it is a feature added to a higher version; you can try it out in 2.x today if you wish to. On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu azury...@gmail.com wrote: Hi Harsh, Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x? On Sun, Apr 7, 2013 at 4:05 PM, Harsh J ha...@cloudera.com wrote: Yes, it need not keep an edits (transactions) stream locally cause those are passed synchronously to the BackupNameNode, which persists it on its behalf. On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote: Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J -- Harsh J -- Harsh J
Re: backup node question
StandbyNameNode is the term we use to refer to a NameNode in HA that is currently not the active one (i.e. its state is 'Standby'). Its not a special type of daemon (i.e. it just runs the NameNode service), just a naming convention. On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu azury...@gmail.com wrote: I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats Standby Namenode? --Send from my Sony mobile. On Apr 7, 2013 9:03 PM, Harsh J ha...@cloudera.com wrote: BackupNameNode is not present in the maintenance 1.x releases, it is a feature added to a higher version; you can try it out in 2.x today if you wish to. On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu azury...@gmail.com wrote: Hi Harsh, Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x? On Sun, Apr 7, 2013 at 4:05 PM, Harsh J ha...@cloudera.com wrote: Yes, it need not keep an edits (transactions) stream locally cause those are passed synchronously to the BackupNameNode, which persists it on its behalf. On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote: Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J -- Harsh J -- Harsh J -- Harsh J
Re: backup node question
oh, got it. you are a good guy. --Send from my Sony mobile. On Apr 7, 2013 10:11 PM, Harsh J ha...@cloudera.com wrote: StandbyNameNode is the term we use to refer to a NameNode in HA that is currently not the active one (i.e. its state is 'Standby'). Its not a special type of daemon (i.e. it just runs the NameNode service), just a naming convention. On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu azury...@gmail.com wrote: I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats Standby Namenode? --Send from my Sony mobile. On Apr 7, 2013 9:03 PM, Harsh J ha...@cloudera.com wrote: BackupNameNode is not present in the maintenance 1.x releases, it is a feature added to a higher version; you can try it out in 2.x today if you wish to. On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu azury...@gmail.com wrote: Hi Harsh, Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x? On Sun, Apr 7, 2013 at 4:05 PM, Harsh J ha...@cloudera.com wrote: Yes, it need not keep an edits (transactions) stream locally cause those are passed synchronously to the BackupNameNode, which persists it on its behalf. On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma lin...@gmail.com wrote: Thanks Harsh, For your comments, What it means is that the NameNode need not store anything locally, you mean Primary Name Node do not need to store checkpoint/journal locally, and only need to keep memory image up-to-date for edits? regards, Lin On Sun, Apr 7, 2013 at 3:31 PM, Harsh J ha...@cloudera.com wrote: Hi Lin, My reply inline. On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma lin...@gmail.com wrote: Hi guys, I am reading from this paper to learn about backup nodes (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf ), It is mentioned, It contains all file system metadata information except for block locations. It can perform all operations of the regular NameNode that do not involve modification of the namespace or knowledge of block locations. , what kinds of operations do not need knowledge of block locations? Operations that do not involve data reads or writes would not require knowledge of block locations. Applying also the restriction of no namespace mutation, an example would be listing directories and looking up file information via FileStatus objects (perhaps the only examples - its like a safemode but no reads either). It is also mentioned, Use of a BackupNode provides the option of running the NameNode without persistent storage, delegating responsibility for the namespace state persisting to the BackupNode., what means running the NameNode without persistent storage and delegating responsibility for the namespace state persisting? What it means is that the NameNode need not store anything locally, but can rely on the edits being stored at the BackupNameNode which would continuously be receiving it. When restarted, it can grab a current checkpoint from the BNN and boot up anywhere, since there's no local storage requirement. -- Harsh J -- Harsh J -- Harsh J -- Harsh J
