Re: SecondaryNameNode on separate machine
Now SecondaryNameNode connects to the NameNode (after I configured dfs.http.address to the NN's http server - NN hostname on port 50070) and creates(transfers) edits and fsimage from NameNode. It didn't work for me - I get an error: java.io.FileNotFoundException: http://192.168.30.5:50070/getimage?putimage=1port=50090machine=127.0.0.1token=-16:1173009257:0:1226503705000:1226503705207 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1168) at org.apache.hadoop.dfs.TransferFsImage.getFileClient(TransferFsImage.java:150) at org.apache.hadoop.dfs.SecondaryNameNode.putFSImage(SecondaryNameNode.java:271) at org.apache.hadoop.dfs.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:311) at org.apache.hadoop.dfs.SecondaryNameNode.run(SecondaryNameNode.java:216) at java.lang.Thread.run(Thread.java:595) And when I run the http request directly (in the browser) , I receive this : GetImage failed. java.io.IOException: Namenode is not expecting an new image UPLOAD_START at org.apache.hadoop.dfs.FSImage.validateCheckpointUpload(FSImage.java:1193) at org.apache.hadoop.dfs.GetImageServlet.doGet(GetImageServlet.java:57) .. If it is a mundane thing (i.e not need to check point now ) why does it throw an Error? What is the UPLOAD_START at the end of the message? (if it failed , how come it starts?) - but more importantly - how do I get rid of it? Thanks! -- View this message in context: http://www.nabble.com/SecondaryNameNode-on-separate-machine-tp20207482p20463349.html Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: SecondaryNameNode on separate machine
Konstantin, it works, thanks a lot! Tomislav On Mon, 2008-11-03 at 11:13 -0800, Konstantin Shvachko wrote: You can either do what you just described with dfs.name.dir = dirX or you can start name-node with -importCheckpoint option. This is an automation for copying image files from secondary to primary. See here: http://hadoop.apache.org/core/docs/current/commands_manual.html#namenode http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Secondary+NameNode http://issues.apache.org/jira/browse/HADOOP-2585#action_12584755 --Konstantin Tomislav Poljak wrote: Hi, Thank you all for your time and your answers! Now SecondaryNameNode connects to the NameNode (after I configured dfs.http.address to the NN's http server - NN hostname on port 50070) and creates(transfers) edits and fsimage from NameNode. Can you explain me a little bit more how NameNode failover should work now? For example, SecondaryNameNode now stores fsimage and edits to (SNN's) dirX and let's say NameNode goes down (disk becomes unreadable). Now I create/dedicate a new machine for NameNode (also change DNS to point to this new NameNode machine as nameNode host) and take the data dirX from SNN and copy it to new NameNode. How do I configure new NameNode to use data from dirX (do I configure dfs.name.dir to point to dirX and start new NameNode)? Thanks, Tomislav On Fri, 2008-10-31 at 11:38 -0700, Konstantin Shvachko wrote: True, dfs.http.address is the NN Web UI address. This where the NN http server runs. Besides the Web UI there also a servlet running on that server which is used to transfer image and edits from NN to the secondary using http get. So SNN uses both addresses fs.default.name and dfs.http.address. When SNN finishes the checkpoint the primary needs to transfer the resulting image back. This is done via the http server running on SNN. Answering Tomislav's question: The difference between fs.default.name and dfs.http.address is that fs.default.name is the name-node's PRC address, where clients and data-nodes connect to, while dfs.http.address is the NN's http server address where our browsers connect to, but it is also used for transferring image and edits files. --Konstantin Otis Gospodnetic wrote: Konstantin Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN. The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. hdfs://) in this URL. Is this correct? Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Tomislav Poljak [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Thursday, October 30, 2008 1:52:18 PM Subject: Re: SecondaryNameNode on separate machine Hi, can you, please, explain the difference between fs.default.name and dfs.http.address (like how and when is SecondaryNameNode using fs.default.name and how/when dfs.http.address). I have set them both to same (namenode's) hostname:port. Is this correct (or dfs.http.address needs some other port)? Thanks, Tomislav On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote: SecondaryNameNode uses http protocol to transfer the image and the edits from the primary name-node and vise versa. So the secondary does not access local files on the primary directly. The primary NN should know the secondary's http address. And the secondary NN need to know both fs.default.name and dfs.http.address of the primary. In general we usually create one configuration file hadoop-site.xml and copy it to all other machines. So you don't need to set up different values for all servers. Regards, --Konstantin Tomislav Poljak wrote: Hi, I'm not clear on how does SecondaryNameNode communicates with NameNode (if deployed on separate machine). Does SecondaryNameNode uses direct connection (over some port and protocol) or is it enough for SecondaryNameNode to have access to data which NameNode writes locally on disk? Tomislav On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote: I think a lot of the confusion comes from this thread : http://www.nabble.com/NameNode-failover-procedure-td11711842.html Particularly because the wiki was updated with wrong information, not maliciously I'm sure. This information is now gone for good. Otis, your solution is pretty much like the one given by Dhruba Borthakur and augmented by Konstantin Shvachko later in the thread but I never did it myself. One thing should be clear though
Re: SecondaryNameNode on separate machine
You can either do what you just described with dfs.name.dir = dirX or you can start name-node with -importCheckpoint option. This is an automation for copying image files from secondary to primary. See here: http://hadoop.apache.org/core/docs/current/commands_manual.html#namenode http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Secondary+NameNode http://issues.apache.org/jira/browse/HADOOP-2585#action_12584755 --Konstantin Tomislav Poljak wrote: Hi, Thank you all for your time and your answers! Now SecondaryNameNode connects to the NameNode (after I configured dfs.http.address to the NN's http server - NN hostname on port 50070) and creates(transfers) edits and fsimage from NameNode. Can you explain me a little bit more how NameNode failover should work now? For example, SecondaryNameNode now stores fsimage and edits to (SNN's) dirX and let's say NameNode goes down (disk becomes unreadable). Now I create/dedicate a new machine for NameNode (also change DNS to point to this new NameNode machine as nameNode host) and take the data dirX from SNN and copy it to new NameNode. How do I configure new NameNode to use data from dirX (do I configure dfs.name.dir to point to dirX and start new NameNode)? Thanks, Tomislav On Fri, 2008-10-31 at 11:38 -0700, Konstantin Shvachko wrote: True, dfs.http.address is the NN Web UI address. This where the NN http server runs. Besides the Web UI there also a servlet running on that server which is used to transfer image and edits from NN to the secondary using http get. So SNN uses both addresses fs.default.name and dfs.http.address. When SNN finishes the checkpoint the primary needs to transfer the resulting image back. This is done via the http server running on SNN. Answering Tomislav's question: The difference between fs.default.name and dfs.http.address is that fs.default.name is the name-node's PRC address, where clients and data-nodes connect to, while dfs.http.address is the NN's http server address where our browsers connect to, but it is also used for transferring image and edits files. --Konstantin Otis Gospodnetic wrote: Konstantin Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN. The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. hdfs://) in this URL. Is this correct? Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Tomislav Poljak [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Thursday, October 30, 2008 1:52:18 PM Subject: Re: SecondaryNameNode on separate machine Hi, can you, please, explain the difference between fs.default.name and dfs.http.address (like how and when is SecondaryNameNode using fs.default.name and how/when dfs.http.address). I have set them both to same (namenode's) hostname:port. Is this correct (or dfs.http.address needs some other port)? Thanks, Tomislav On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote: SecondaryNameNode uses http protocol to transfer the image and the edits from the primary name-node and vise versa. So the secondary does not access local files on the primary directly. The primary NN should know the secondary's http address. And the secondary NN need to know both fs.default.name and dfs.http.address of the primary. In general we usually create one configuration file hadoop-site.xml and copy it to all other machines. So you don't need to set up different values for all servers. Regards, --Konstantin Tomislav Poljak wrote: Hi, I'm not clear on how does SecondaryNameNode communicates with NameNode (if deployed on separate machine). Does SecondaryNameNode uses direct connection (over some port and protocol) or is it enough for SecondaryNameNode to have access to data which NameNode writes locally on disk? Tomislav On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote: I think a lot of the confusion comes from this thread : http://www.nabble.com/NameNode-failover-procedure-td11711842.html Particularly because the wiki was updated with wrong information, not maliciously I'm sure. This information is now gone for good. Otis, your solution is pretty much like the one given by Dhruba Borthakur and augmented by Konstantin Shvachko later in the thread but I never did it myself. One thing should be clear though, the NN is and will remain a SPOF (just like HBase's Master) as long as a distributed manager service (like Zookeeper) is not plugged into Hadoop to help with failover. J-D On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic [EMAIL PROTECTED] wrote: Hi, So what is the recipe for avoiding NN SPOF using only what comes with Hadoop
Re: SecondaryNameNode on separate machine
Konstantin Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN. The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. hdfs://) in this URL. Is this correct? Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Tomislav Poljak [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Thursday, October 30, 2008 1:52:18 PM Subject: Re: SecondaryNameNode on separate machine Hi, can you, please, explain the difference between fs.default.name and dfs.http.address (like how and when is SecondaryNameNode using fs.default.name and how/when dfs.http.address). I have set them both to same (namenode's) hostname:port. Is this correct (or dfs.http.address needs some other port)? Thanks, Tomislav On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote: SecondaryNameNode uses http protocol to transfer the image and the edits from the primary name-node and vise versa. So the secondary does not access local files on the primary directly. The primary NN should know the secondary's http address. And the secondary NN need to know both fs.default.name and dfs.http.address of the primary. In general we usually create one configuration file hadoop-site.xml and copy it to all other machines. So you don't need to set up different values for all servers. Regards, --Konstantin Tomislav Poljak wrote: Hi, I'm not clear on how does SecondaryNameNode communicates with NameNode (if deployed on separate machine). Does SecondaryNameNode uses direct connection (over some port and protocol) or is it enough for SecondaryNameNode to have access to data which NameNode writes locally on disk? Tomislav On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote: I think a lot of the confusion comes from this thread : http://www.nabble.com/NameNode-failover-procedure-td11711842.html Particularly because the wiki was updated with wrong information, not maliciously I'm sure. This information is now gone for good. Otis, your solution is pretty much like the one given by Dhruba Borthakur and augmented by Konstantin Shvachko later in the thread but I never did it myself. One thing should be clear though, the NN is and will remain a SPOF (just like HBase's Master) as long as a distributed manager service (like Zookeeper) is not plugged into Hadoop to help with failover. J-D On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic [EMAIL PROTECTED] wrote: Hi, So what is the recipe for avoiding NN SPOF using only what comes with Hadoop? From what I can tell, I think one has to do the following two things: 1) configure primary NN to save namespace and xa logs to multiple dirs, one of which is actually on a remotely mounted disk, so that the data actually lives on a separate disk on a separate box. This saves namespace and xa logs on multiple boxes in case of primary NN hardware failure. 2) configure secondary NN to periodically merge fsimage+edits and create the fsimage checkpoint. This really is a second NN process running on another box. It sounds like this secondary NN has to somehow have access to fsimage edits files from the primary NN server. http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to give secondary NN access to primary NN's fsimage and edits files. Should one mount a disk from the primary NN box to the secondary NN box to get access to those files? Or is there a simpler way? In any case, this checkpoint is just a merge of fsimage+edits files and again is there in case the box with the primary NN dies. That's what's described on http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less. Is this sufficient, or are there other things one has to do to eliminate NN SPOF? Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Jean-Daniel Cryans To: core-user@hadoop.apache.org Sent: Tuesday, October 28, 2008 8:14:44 PM Subject: Re: SecondaryNameNode on separate machine Tomislav. Contrary to popular belief the secondary namenode does not provide failover, it's only used to do what is described here : http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode So the term secondary does not mean a second one
Re: SecondaryNameNode on separate machine
Otis Gospodnetic wrote: Konstantin Co, please correct me if I'm wrong, but looking at hadoop-default.xml makes me think that dfs.http.address is only the URL for the NN *Web UI*. In other words, this is where we people go look at the NN. The secondary NN must then be using only the Primary NN URL specified in fs.default.name. This URL looks like hdfs://name-node-hostname-here/. Something in Hadoop then knows the exact port for the Primary NN based on the URI schema (e.g. hdfs://) in this URL. Is this correct? Yes. The default port for an HDFS URI is 8020 (NameNode.DEFAULT_PORT). The value of fs.default.name is used by HDFS. When starting the namenode or datanodes, this must be an HDFS URI. If this names an explicit port, then that will be used, otherwise the default, 8020 will be used. The default port for HTTP URIs is 80, but the namenode typically runs its web UI on 50070 (the default for dfs.http.address). Doug
Re: SecondaryNameNode on separate machine
Hi, can you, please, explain the difference between fs.default.name and dfs.http.address (like how and when is SecondaryNameNode using fs.default.name and how/when dfs.http.address). I have set them both to same (namenode's) hostname:port. Is this correct (or dfs.http.address needs some other port)? Thanks, Tomislav On Wed, 2008-10-29 at 16:10 -0700, Konstantin Shvachko wrote: SecondaryNameNode uses http protocol to transfer the image and the edits from the primary name-node and vise versa. So the secondary does not access local files on the primary directly. The primary NN should know the secondary's http address. And the secondary NN need to know both fs.default.name and dfs.http.address of the primary. In general we usually create one configuration file hadoop-site.xml and copy it to all other machines. So you don't need to set up different values for all servers. Regards, --Konstantin Tomislav Poljak wrote: Hi, I'm not clear on how does SecondaryNameNode communicates with NameNode (if deployed on separate machine). Does SecondaryNameNode uses direct connection (over some port and protocol) or is it enough for SecondaryNameNode to have access to data which NameNode writes locally on disk? Tomislav On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote: I think a lot of the confusion comes from this thread : http://www.nabble.com/NameNode-failover-procedure-td11711842.html Particularly because the wiki was updated with wrong information, not maliciously I'm sure. This information is now gone for good. Otis, your solution is pretty much like the one given by Dhruba Borthakur and augmented by Konstantin Shvachko later in the thread but I never did it myself. One thing should be clear though, the NN is and will remain a SPOF (just like HBase's Master) as long as a distributed manager service (like Zookeeper) is not plugged into Hadoop to help with failover. J-D On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic [EMAIL PROTECTED] wrote: Hi, So what is the recipe for avoiding NN SPOF using only what comes with Hadoop? From what I can tell, I think one has to do the following two things: 1) configure primary NN to save namespace and xa logs to multiple dirs, one of which is actually on a remotely mounted disk, so that the data actually lives on a separate disk on a separate box. This saves namespace and xa logs on multiple boxes in case of primary NN hardware failure. 2) configure secondary NN to periodically merge fsimage+edits and create the fsimage checkpoint. This really is a second NN process running on another box. It sounds like this secondary NN has to somehow have access to fsimage edits files from the primary NN server. http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to give secondary NN access to primary NN's fsimage and edits files. Should one mount a disk from the primary NN box to the secondary NN box to get access to those files? Or is there a simpler way? In any case, this checkpoint is just a merge of fsimage+edits files and again is there in case the box with the primary NN dies. That's what's described on http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less. Is this sufficient, or are there other things one has to do to eliminate NN SPOF? Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Jean-Daniel Cryans [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Tuesday, October 28, 2008 8:14:44 PM Subject: Re: SecondaryNameNode on separate machine Tomislav. Contrary to popular belief the secondary namenode does not provide failover, it's only used to do what is described here : http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode So the term secondary does not mean a second one but is more like a second part of. J-D On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote: Hi, I'm trying to implement NameNode failover (or at least NameNode local data backup), but it is hard since there is no official documentation. Pages on this subject are created, but still empty: http://wiki.apache.org/hadoop/NameNodeFailover http://wiki.apache.org/hadoop/SecondaryNameNode I have been browsing the web and hadoop mailing list to see how this should be implemented, but I got even more confused. People are asking do we even need SecondaryNameNode etc. (since NameNode can write local data to multiple locations, so one of those locations can be a mounted disk from other machine). I think I understand the motivation for SecondaryNameNode (to create a snapshoot of NameNode data every n seconds/hours), but setting (deploying and running
Re: SecondaryNameNode on separate machine
I think a lot of the confusion comes from this thread : http://www.nabble.com/NameNode-failover-procedure-td11711842.html Particularly because the wiki was updated with wrong information, not maliciously I'm sure. This information is now gone for good. Otis, your solution is pretty much like the one given by Dhruba Borthakur and augmented by Konstantin Shvachko later in the thread but I never did it myself. One thing should be clear though, the NN is and will remain a SPOF (just like HBase's Master) as long as a distributed manager service (like Zookeeper) is not plugged into Hadoop to help with failover. J-D On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic [EMAIL PROTECTED] wrote: Hi, So what is the recipe for avoiding NN SPOF using only what comes with Hadoop? From what I can tell, I think one has to do the following two things: 1) configure primary NN to save namespace and xa logs to multiple dirs, one of which is actually on a remotely mounted disk, so that the data actually lives on a separate disk on a separate box. This saves namespace and xa logs on multiple boxes in case of primary NN hardware failure. 2) configure secondary NN to periodically merge fsimage+edits and create the fsimage checkpoint. This really is a second NN process running on another box. It sounds like this secondary NN has to somehow have access to fsimage edits files from the primary NN server. http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to give secondary NN access to primary NN's fsimage and edits files. Should one mount a disk from the primary NN box to the secondary NN box to get access to those files? Or is there a simpler way? In any case, this checkpoint is just a merge of fsimage+edits files and again is there in case the box with the primary NN dies. That's what's described on http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less. Is this sufficient, or are there other things one has to do to eliminate NN SPOF? Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Jean-Daniel Cryans [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Tuesday, October 28, 2008 8:14:44 PM Subject: Re: SecondaryNameNode on separate machine Tomislav. Contrary to popular belief the secondary namenode does not provide failover, it's only used to do what is described here : http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode So the term secondary does not mean a second one but is more like a second part of. J-D On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote: Hi, I'm trying to implement NameNode failover (or at least NameNode local data backup), but it is hard since there is no official documentation. Pages on this subject are created, but still empty: http://wiki.apache.org/hadoop/NameNodeFailover http://wiki.apache.org/hadoop/SecondaryNameNode I have been browsing the web and hadoop mailing list to see how this should be implemented, but I got even more confused. People are asking do we even need SecondaryNameNode etc. (since NameNode can write local data to multiple locations, so one of those locations can be a mounted disk from other machine). I think I understand the motivation for SecondaryNameNode (to create a snapshoot of NameNode data every n seconds/hours), but setting (deploying and running) SecondaryNameNode on different machine than NameNode is not as trivial as I expected. First I found that if I need to run SecondaryNameNode on other machine than NameNode I should change masters file on NameNode (change localhost to SecondaryNameNode host) and set some properties in hadoop-site.xml on SecondaryNameNode (fs.default.name, fs.checkpoint.dir, fs.checkpoint.period etc.) This was enough to start SecondaryNameNode when starting NameNode with bin/start-dfs.sh , but it didn't create image on SecondaryNameNode. Then I found that I need to set dfs.http.address on NameNode address (so now I have NameNode address in both fs.default.name and dfs.http.address). Now I get following exception: 2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in doCheckpoint: 2008-10-28 09:18:00,098 ERROR NameNode.Secondary - java.net.SocketException: Unexpected end of file from server My questions are following: How to resolve this problem (this exception)? Do I need additional property in SecondaryNameNode's hadoop-site.xml or NameNode's hadoop-site.xml? How should NameNode failover work ideally? Is it like this: SecondaryNameNode runs on separate machine than NameNode and stores NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir. When NameNode machine crashes, we
Re: SecondaryNameNode on separate machine
Hi, I'm not clear on how does SecondaryNameNode communicates with NameNode (if deployed on separate machine). Does SecondaryNameNode uses direct connection (over some port and protocol) or is it enough for SecondaryNameNode to have access to data which NameNode writes locally on disk? Tomislav On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote: I think a lot of the confusion comes from this thread : http://www.nabble.com/NameNode-failover-procedure-td11711842.html Particularly because the wiki was updated with wrong information, not maliciously I'm sure. This information is now gone for good. Otis, your solution is pretty much like the one given by Dhruba Borthakur and augmented by Konstantin Shvachko later in the thread but I never did it myself. One thing should be clear though, the NN is and will remain a SPOF (just like HBase's Master) as long as a distributed manager service (like Zookeeper) is not plugged into Hadoop to help with failover. J-D On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic [EMAIL PROTECTED] wrote: Hi, So what is the recipe for avoiding NN SPOF using only what comes with Hadoop? From what I can tell, I think one has to do the following two things: 1) configure primary NN to save namespace and xa logs to multiple dirs, one of which is actually on a remotely mounted disk, so that the data actually lives on a separate disk on a separate box. This saves namespace and xa logs on multiple boxes in case of primary NN hardware failure. 2) configure secondary NN to periodically merge fsimage+edits and create the fsimage checkpoint. This really is a second NN process running on another box. It sounds like this secondary NN has to somehow have access to fsimage edits files from the primary NN server. http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to give secondary NN access to primary NN's fsimage and edits files. Should one mount a disk from the primary NN box to the secondary NN box to get access to those files? Or is there a simpler way? In any case, this checkpoint is just a merge of fsimage+edits files and again is there in case the box with the primary NN dies. That's what's described on http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less. Is this sufficient, or are there other things one has to do to eliminate NN SPOF? Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Jean-Daniel Cryans [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Tuesday, October 28, 2008 8:14:44 PM Subject: Re: SecondaryNameNode on separate machine Tomislav. Contrary to popular belief the secondary namenode does not provide failover, it's only used to do what is described here : http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode So the term secondary does not mean a second one but is more like a second part of. J-D On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote: Hi, I'm trying to implement NameNode failover (or at least NameNode local data backup), but it is hard since there is no official documentation. Pages on this subject are created, but still empty: http://wiki.apache.org/hadoop/NameNodeFailover http://wiki.apache.org/hadoop/SecondaryNameNode I have been browsing the web and hadoop mailing list to see how this should be implemented, but I got even more confused. People are asking do we even need SecondaryNameNode etc. (since NameNode can write local data to multiple locations, so one of those locations can be a mounted disk from other machine). I think I understand the motivation for SecondaryNameNode (to create a snapshoot of NameNode data every n seconds/hours), but setting (deploying and running) SecondaryNameNode on different machine than NameNode is not as trivial as I expected. First I found that if I need to run SecondaryNameNode on other machine than NameNode I should change masters file on NameNode (change localhost to SecondaryNameNode host) and set some properties in hadoop-site.xml on SecondaryNameNode (fs.default.name, fs.checkpoint.dir, fs.checkpoint.period etc.) This was enough to start SecondaryNameNode when starting NameNode with bin/start-dfs.sh , but it didn't create image on SecondaryNameNode. Then I found that I need to set dfs.http.address on NameNode address (so now I have NameNode address in both fs.default.name and dfs.http.address). Now I get following exception: 2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in doCheckpoint: 2008-10-28 09:18:00,098 ERROR NameNode.Secondary - java.net.SocketException
Re: SecondaryNameNode on separate machine
SecondaryNameNode uses http protocol to transfer the image and the edits from the primary name-node and vise versa. So the secondary does not access local files on the primary directly. The primary NN should know the secondary's http address. And the secondary NN need to know both fs.default.name and dfs.http.address of the primary. In general we usually create one configuration file hadoop-site.xml and copy it to all other machines. So you don't need to set up different values for all servers. Regards, --Konstantin Tomislav Poljak wrote: Hi, I'm not clear on how does SecondaryNameNode communicates with NameNode (if deployed on separate machine). Does SecondaryNameNode uses direct connection (over some port and protocol) or is it enough for SecondaryNameNode to have access to data which NameNode writes locally on disk? Tomislav On Wed, 2008-10-29 at 09:08 -0400, Jean-Daniel Cryans wrote: I think a lot of the confusion comes from this thread : http://www.nabble.com/NameNode-failover-procedure-td11711842.html Particularly because the wiki was updated with wrong information, not maliciously I'm sure. This information is now gone for good. Otis, your solution is pretty much like the one given by Dhruba Borthakur and augmented by Konstantin Shvachko later in the thread but I never did it myself. One thing should be clear though, the NN is and will remain a SPOF (just like HBase's Master) as long as a distributed manager service (like Zookeeper) is not plugged into Hadoop to help with failover. J-D On Wed, Oct 29, 2008 at 2:12 AM, Otis Gospodnetic [EMAIL PROTECTED] wrote: Hi, So what is the recipe for avoiding NN SPOF using only what comes with Hadoop? From what I can tell, I think one has to do the following two things: 1) configure primary NN to save namespace and xa logs to multiple dirs, one of which is actually on a remotely mounted disk, so that the data actually lives on a separate disk on a separate box. This saves namespace and xa logs on multiple boxes in case of primary NN hardware failure. 2) configure secondary NN to periodically merge fsimage+edits and create the fsimage checkpoint. This really is a second NN process running on another box. It sounds like this secondary NN has to somehow have access to fsimage edits files from the primary NN server. http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodedoes not describe the best practise around that - the recommended way to give secondary NN access to primary NN's fsimage and edits files. Should one mount a disk from the primary NN box to the secondary NN box to get access to those files? Or is there a simpler way? In any case, this checkpoint is just a merge of fsimage+edits files and again is there in case the box with the primary NN dies. That's what's described on http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNodemore or less. Is this sufficient, or are there other things one has to do to eliminate NN SPOF? Thanks, Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Jean-Daniel Cryans [EMAIL PROTECTED] To: core-user@hadoop.apache.org Sent: Tuesday, October 28, 2008 8:14:44 PM Subject: Re: SecondaryNameNode on separate machine Tomislav. Contrary to popular belief the secondary namenode does not provide failover, it's only used to do what is described here : http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode So the term secondary does not mean a second one but is more like a second part of. J-D On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak wrote: Hi, I'm trying to implement NameNode failover (or at least NameNode local data backup), but it is hard since there is no official documentation. Pages on this subject are created, but still empty: http://wiki.apache.org/hadoop/NameNodeFailover http://wiki.apache.org/hadoop/SecondaryNameNode I have been browsing the web and hadoop mailing list to see how this should be implemented, but I got even more confused. People are asking do we even need SecondaryNameNode etc. (since NameNode can write local data to multiple locations, so one of those locations can be a mounted disk from other machine). I think I understand the motivation for SecondaryNameNode (to create a snapshoot of NameNode data every n seconds/hours), but setting (deploying and running) SecondaryNameNode on different machine than NameNode is not as trivial as I expected. First I found that if I need to run SecondaryNameNode on other machine than NameNode I should change masters file on NameNode (change localhost to SecondaryNameNode host) and set some properties in hadoop-site.xml on SecondaryNameNode (fs.default.name, fs.checkpoint.dir, fs.checkpoint.period etc.) This was enough to start SecondaryNameNode when starting NameNode with bin/start-dfs.sh , but it didn't create image on SecondaryNameNode. Then I found that I need to set
Re: SecondaryNameNode on separate machine
Tomislav. Contrary to popular belief the secondary namenode does not provide failover, it's only used to do what is described here : http://hadoop.apache.org/core/docs/r0.18.1/hdfs_user_guide.html#Secondary+NameNode So the term secondary does not mean a second one but is more like a second part of. J-D On Tue, Oct 28, 2008 at 9:44 AM, Tomislav Poljak [EMAIL PROTECTED] wrote: Hi, I'm trying to implement NameNode failover (or at least NameNode local data backup), but it is hard since there is no official documentation. Pages on this subject are created, but still empty: http://wiki.apache.org/hadoop/NameNodeFailover http://wiki.apache.org/hadoop/SecondaryNameNode I have been browsing the web and hadoop mailing list to see how this should be implemented, but I got even more confused. People are asking do we even need SecondaryNameNode etc. (since NameNode can write local data to multiple locations, so one of those locations can be a mounted disk from other machine). I think I understand the motivation for SecondaryNameNode (to create a snapshoot of NameNode data every n seconds/hours), but setting (deploying and running) SecondaryNameNode on different machine than NameNode is not as trivial as I expected. First I found that if I need to run SecondaryNameNode on other machine than NameNode I should change masters file on NameNode (change localhost to SecondaryNameNode host) and set some properties in hadoop-site.xml on SecondaryNameNode (fs.default.name, fs.checkpoint.dir, fs.checkpoint.period etc.) This was enough to start SecondaryNameNode when starting NameNode with bin/start-dfs.sh , but it didn't create image on SecondaryNameNode. Then I found that I need to set dfs.http.address on NameNode address (so now I have NameNode address in both fs.default.name and dfs.http.address). Now I get following exception: 2008-10-28 09:18:00,098 ERROR NameNode.Secondary - Exception in doCheckpoint: 2008-10-28 09:18:00,098 ERROR NameNode.Secondary - java.net.SocketException: Unexpected end of file from server My questions are following: How to resolve this problem (this exception)? Do I need additional property in SecondaryNameNode's hadoop-site.xml or NameNode's hadoop-site.xml? How should NameNode failover work ideally? Is it like this: SecondaryNameNode runs on separate machine than NameNode and stores NameNode's data (fsimage and fsiedits) locally in fs.checkpoint.dir. When NameNode machine crashes, we start NameNode on machine where SecondaryNameNode was running and we set dfs.name.dir to fs.checkpoint.dir. Also we need to change how DNS resolves NameNode hostname (change from the primary to the secondary). Is this correct ? Tomislav
