how can we know the statistics of namenode and datanode using API's
Hi All, I have been trying to access the statistics of FSNameSystem using FSNameSystemMetrics , but i have not been able to do it yet. Am I doing it right, if not kindly guide me. I am stuck. Thanks, Vidur -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
[jira] Created: (HDFS-1200) The namenode could remember the last good location of a missing block
The namenode could remember the last good location of a missing block - Key: HDFS-1200 URL: https://issues.apache.org/jira/browse/HDFS-1200 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: dhruba borthakur There are times when datanodes die and all replicas of a block are lost. An fsck on the HDFS reports these as "MISSING" blocks in the filesystem. The administrator has to go mine through lots of namenode logs to find out which datanode(s) had a replica of that block. Then the administrator can try to reboot that particular datanode to retrieve the missing block. This case can me made easier if the NN remembered the last known good replica location of a block. An fsck command can report it and the it will help the administrator to determine if that particular dead datanode can be brought back to life. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Integrating Lustre and HDFS
Thanks for the replies. If I have fs.default.name = file://my_lustre_mount_point , then only the lustre filesystem will be used. I would like to have something like fs.default.name=file://my_lustre_mount_point , hdfs://localhost:9123 so that both local filesystem and lustre are in use. Kindly correct me if I am missing something here. Thanks, Vikas On Thu, Jun 10, 2010 at 8:09 PM, Owen O'Malley wrote: > > Your local storage should get used for MR. Use Lustre via file:// > (LocalFileSystem, iirc) > > instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default > file system type. > > If Lustre has integrated checksums, you'll want to use the > RawLocalFileSystem instead of LocalFileSystem. You'll want to make it > accessible via: > > fs.raw.impl = org.apache.hadoop.fs.RawLocalFileSystem > > so that urls like raw:///my/path won't go through the Hadoop checksum code. > > -- Owen >
Re: Is it safe enough to write edit log in buffer before writing to disk ?
Hi Jeff, All of the FSNamesystem methods call logSync() before returning to the client. So, if the edit is lost, it also will not have returned a success to the client. -Todd On Thu, Jun 10, 2010 at 6:29 PM, Jeff Zhang wrote: > Hi all, > > I check the source code of EditLogFileOutputStream, it seems hadoop > will first write edit log to buffer, then flush to disk. I know that > it will improve performance, but in the other hand it will cause the > edit log in buffer lost when the name node is down. So I wonder is it > possible and necessary to let users do configuration in the conf file > to decide whether use buffer or not? Thanks. > > > > -- > Best Regards > > Jeff Zhang > -- Todd Lipcon Software Engineer, Cloudera
Is it safe enough to write edit log in buffer before writing to disk ?
Hi all, I check the source code of EditLogFileOutputStream, it seems hadoop will first write edit log to buffer, then flush to disk. I know that it will improve performance, but in the other hand it will cause the edit log in buffer lost when the name node is down. So I wonder is it possible and necessary to let users do configuration in the conf file to decide whether use buffer or not? Thanks. -- Best Regards Jeff Zhang
[jira] Resolved: (HDFS-1198) Resolving cross-realm principals
[ https://issues.apache.org/jira/browse/HDFS-1198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HDFS-1198. --- Fix Version/s: 0.22.0 Resolution: Fixed I've committed this. Thanks, Jitendra. Resolving as fixed. > Resolving cross-realm principals > > > Key: HDFS-1198 > URL: https://issues.apache.org/jira/browse/HDFS-1198 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.22.0 > > Attachments: HDFS-1198.1.patch > > > This jira covers hdfs changes corresponding to HADOOP-6603. Kerberos has bug > in resolving cross realm principals. This jira provides a work-around for > that. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Newbie point to start
You can checkout hadoop-20 branch from http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20/ , and build and run following the steps on the wiki. On 6/10/10 2:01 PM, "Alberich de megres" wrote: Thanks! Can i compile just the source at the repo and use it just as is? I mean, without having any hadoop source code (except the hdfs code at the web i told you). Or without the need to integrate it with a hadoop compiiled code? Just as if a diferent or standalone project. On Thu, Jun 10, 2010 at 10:50 PM, Jitendra Nath Pandey wrote: > You can test hdfs without setting up map-reduce cluster if that's what you > mean. > > Instead of bin/start-all.sh , use bin/start-dfs.sh and you can skip > configurations related to mapreduce. > > To test it, use DFS command line "bin/hadoop dfs". > > > On 6/10/10 1:16 PM, "Alberich de megres" wrote: > > Thanks for the quick reply, > > But I'm talking about just hdfs.. is it posible to test it separately? > with source code available at: > http://github.com/apache/hadoop-hdfs > > I compiled it, and now i want to test it. (aside from hadoop) > > > On Thu, Jun 10, 2010 at 9:37 PM, Jitendra Nath Pandey > wrote: >> This link should help. >>http://wiki.apache.org/hadoop/QuickStart >> >> >> On 6/10/10 12:20 PM, "Alberich de megres" wrote: >> >> Hello! >> >> I'm new on HDFS, i just downloaded the source code and compiled it. >> >> Now I want to excecure it on 2 machines.. but i don't know how to start >> servers. >> >> Is there any web/doc or someone can point me some light on how to start? >> >> Thanks!! >> Alberich >> >> > >
[jira] Created: (HDFS-1199) Extract a subset of tests for smoke (DOA) validation.
Extract a subset of tests for smoke (DOA) validation. - Key: HDFS-1199 URL: https://issues.apache.org/jira/browse/HDFS-1199 Project: Hadoop HDFS Issue Type: Improvement Components: build Affects Versions: 0.21.0 Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Attachments: HDFS-1199.patch Similar to that of HADOOP-6810 for HDFS. Adds an ability to run up to 30 minutes of the tests to 'smoke' HDFS build i.e. find possible issues faster than the full test cycle does). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Newbie point to start
Thanks! Can i compile just the source at the repo and use it just as is? I mean, without having any hadoop source code (except the hdfs code at the web i told you). Or without the need to integrate it with a hadoop compiiled code? Just as if a diferent or standalone project. On Thu, Jun 10, 2010 at 10:50 PM, Jitendra Nath Pandey wrote: > You can test hdfs without setting up map-reduce cluster if that's what you > mean. > > Instead of bin/start-all.sh , use bin/start-dfs.sh and you can skip > configurations related to mapreduce. > > To test it, use DFS command line "bin/hadoop dfs". > > > On 6/10/10 1:16 PM, "Alberich de megres" wrote: > > Thanks for the quick reply, > > But I'm talking about just hdfs.. is it posible to test it separately? > with source code available at: > http://github.com/apache/hadoop-hdfs > > I compiled it, and now i want to test it. (aside from hadoop) > > > On Thu, Jun 10, 2010 at 9:37 PM, Jitendra Nath Pandey > wrote: >> This link should help. >> http://wiki.apache.org/hadoop/QuickStart >> >> >> On 6/10/10 12:20 PM, "Alberich de megres" wrote: >> >> Hello! >> >> I'm new on HDFS, i just downloaded the source code and compiled it. >> >> Now I want to excecure it on 2 machines.. but i don't know how to start >> servers. >> >> Is there any web/doc or someone can point me some light on how to start? >> >> Thanks!! >> Alberich >> >> > >
Re: Newbie point to start
Thanks for the quick reply, But I'm talking about just hdfs.. is it posible to test it separately? with source code available at: http://github.com/apache/hadoop-hdfs I compiled it, and now i want to test it. (aside from hadoop) On Thu, Jun 10, 2010 at 9:37 PM, Jitendra Nath Pandey wrote: > This link should help. > http://wiki.apache.org/hadoop/QuickStart > > > On 6/10/10 12:20 PM, "Alberich de megres" wrote: > > Hello! > > I'm new on HDFS, i just downloaded the source code and compiled it. > > Now I want to excecure it on 2 machines.. but i don't know how to start > servers. > > Is there any web/doc or someone can point me some light on how to start? > > Thanks!! > Alberich > >
Re: Newbie point to start
You can test hdfs without setting up map-reduce cluster if that's what you mean. Instead of bin/start-all.sh , use bin/start-dfs.sh and you can skip configurations related to mapreduce. To test it, use DFS command line "bin/hadoop dfs". On 6/10/10 1:16 PM, "Alberich de megres" wrote: Thanks for the quick reply, But I'm talking about just hdfs.. is it posible to test it separately? with source code available at: http://github.com/apache/hadoop-hdfs I compiled it, and now i want to test it. (aside from hadoop) On Thu, Jun 10, 2010 at 9:37 PM, Jitendra Nath Pandey wrote: > This link should help. >http://wiki.apache.org/hadoop/QuickStart > > > On 6/10/10 12:20 PM, "Alberich de megres" wrote: > > Hello! > > I'm new on HDFS, i just downloaded the source code and compiled it. > > Now I want to excecure it on 2 machines.. but i don't know how to start > servers. > > Is there any web/doc or someone can point me some light on how to start? > > Thanks!! > Alberich > >
Newbie point to start
Hello! I'm new on HDFS, i just downloaded the source code and compiled it. Now I want to excecure it on 2 machines.. but i don't know how to start servers. Is there any web/doc or someone can point me some light on how to start? Thanks!! Alberich
Re: Newbie point to start
This link should help. http://wiki.apache.org/hadoop/QuickStart On 6/10/10 12:20 PM, "Alberich de megres" wrote: Hello! I'm new on HDFS, i just downloaded the source code and compiled it. Now I want to excecure it on 2 machines.. but i don't know how to start servers. Is there any web/doc or someone can point me some light on how to start? Thanks!! Alberich
[jira] Created: (HDFS-1198) Resolving cross-realm principals
Resolving cross-realm principals Key: HDFS-1198 URL: https://issues.apache.org/jira/browse/HDFS-1198 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey This jira covers hdfs changes corresponding to HADOOP-6603. Kerberos has bug in resolving cross realm principals. This jira provides a work-around for that. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1197) 0.20: TestFileAppend3.testTC2 failure
0.20: TestFileAppend3.testTC2 failure - Key: HDFS-1197 URL: https://issues.apache.org/jira/browse/HDFS-1197 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client, name-node Affects Versions: 0.20-append Reporter: Todd Lipcon I saw this failure once on my internal Hudson job that runs the append tests 48 times a day: junit.framework.AssertionFailedError: expected:<114688> but was:<98304> at org.apache.hadoop.hdfs.AppendTestUtil.check(AppendTestUtil.java:112) at org.apache.hadoop.hdfs.TestFileAppend3.testTC2(TestFileAppend3.java:116) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Integrating Lustre and HDFS
> Your local storage should get used for MR. Use Lustre via file:// > (LocalFileSystem, iirc) > instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file > system type. If Lustre has integrated checksums, you'll want to use the RawLocalFileSystem instead of LocalFileSystem. You'll want to make it accessible via: fs.raw.impl = org.apache.hadoop.fs.RawLocalFileSystem so that urls like raw:///my/path won't go through the Hadoop checksum code. -- Owen
Re: Integrating Lustre and HDFS
On Jun 10, 2010, at 2:37 AM, Vikas Ashok Patil wrote: > In my cluster I have the lustre parallel file system which mainly exposes > storage over a network. Also there is some local space on each node of the > cluster. This space is not part of the lustre file system. My hadoop > installation currently makes use of this local file system. However, I would > like to make use of the space available through lustre(exposed over the > network). Hence I was thinking of a way to integrate HDFS and lustre, where > HDFS would manage the local storage and lustre would provide the storage > over the network. > > Please let me know your thoughts on this. Your local storage should get used for MR. Use Lustre via file:// (LocalFileSystem, iirc) instead of HDFS via hdfs:// (DistributedFileSystem, irrc) as the default file system type.
Integrating Lustre and HDFS
Hello All, I would like to try out a hadoop configuration involving both lustre and hdfs. Hence I would like to know any thoughts/criticisms on the idea. In my cluster I have the lustre parallel file system which mainly exposes storage over a network. Also there is some local space on each node of the cluster. This space is not part of the lustre file system. My hadoop installation currently makes use of this local file system. However, I would like to make use of the space available through lustre(exposed over the network). Hence I was thinking of a way to integrate HDFS and lustre, where HDFS would manage the local storage and lustre would provide the storage over the network. Please let me know your thoughts on this. Thanks, Vikas A Patil
