Re: hdfs fsck -locations
Got it, Harsh, now I see it all. Thank you, Mark On Fri, Jan 24, 2014 at 7:49 PM, Harsh J ha...@cloudera.com wrote: The right syntax is to use -files -blocks -locations, so it drills down all the way. You are not missing a feature - this has existed for as long as I've known HDFS. In Rodrigo's output, he's seeing a BlockPool ID, which is not equivalent to a location, but just carries an IP in it for identification purposes. On Sat, Jan 25, 2014 at 3:53 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Yes, Rodrigo, that's what I was looking for. So in my install I somehow don't have it at all. Was asked by my students, so I got the answer. Mark On Fri, Jan 24, 2014 at 4:00 PM, Nascimento, Rodrigo rodrigo.nascime...@netapp.com wrote: Mark, there we go ;-) Rodrigo Nascimento Systems Engineer @ Brazil Mobile +55 11 991.873.810 Sent from my iPhone Begin forwarded message: From: Nascimento, Rodrigo rodrigo.nascime...@netapp.com Date: 24 de janeiro de 2014 15:59:33 BRST To: user@hadoop.apache.org user@hadoop.apache.org Subject: RE: hdfs fsck -locations Hi Mark, It is a sample from my sandbox. Your question is about the part that is in RED at the output below, right? [root@sandbox ~]# hdfs fsck /user/ambari-qa/passwd -locations Connecting to namenode via http://sandbox.hortonworks.com:50070 FSCK started by root (auth:SIMPLE) from /172.16.13.30 for path /user/ambari-qa/passwd at Fri Jan 24 09:53:43 PST 2014 . /user/ambari-qa/passwd: Under replicated BP-1578958328-10.0.2.15-1382306880516:blk_1073742464_1640. Target Replicas is 3 but found 1 replica(s). Status: HEALTHY Total size:1708 B Total dirs:0 Total files:1 Total symlinks:0 Total blocks (validated):1 (avg. block size 1708 B) Minimally replicated blocks:1 (100.0 %) Over-replicated blocks:0 (0.0 %) Under-replicated blocks:1 (100.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor:3 Average block replication:1.0 Corrupt blocks:0 Missing replicas:2 (66.64 %) Number of data-nodes:1 Number of racks:1 FSCK ended at Fri Jan 24 09:53:43 PST 2014 in 1 milliseconds The filesystem under path '/user/ambari-qa/passwd' is HEALTHY [root@sandbox ~]# Rod Nascimento From: Nascimento, Rodrigo [rodrigo.nascime...@netapp.com] Sent: Friday, January 24, 2014 3:34 PM To: user@hadoop.apache.org Subject: RE: hdfs fsck -locations I’m not seeing locations flag yet. Rod Nascimento Systems Engineer @ Brazil People don’t buy WHAT you do. They buy WHY you do it. From: Mark Kerzner [mailto:mark.kerz...@shmsoft.com] Sent: Friday, January 24, 2014 3:16 PM To: Hadoop User Subject: Re: hdfs fsck -locations Sorry, did not copy the full command hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated):1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks:0 (0.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY On Fri, Jan 24, 2014 at 11:08 AM, Harsh J ha...@cloudera.com wrote: Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24
Re: hdfs fsck -locations
Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Hi, hdfs fsck -locations is supposed to show every block with its location? Is location the ip of the datanode? Thank you, Mark -- Harsh J
Re: hdfs fsck -locations
Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Hi, hdfs fsck -locations is supposed to show every block with its location? Is location the ip of the datanode? Thank you, Mark -- Harsh J
Re: hdfs fsck -locations
Sorry, did not copy the full command hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY On Fri, Jan 24, 2014 at 11:08 AM, Harsh J ha...@cloudera.com wrote: Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Hi, hdfs fsck -locations is supposed to show every block with its location? Is location the ip of the datanode? Thank you, Mark -- Harsh J
RE: hdfs fsck -locations
I'm not seeing locations flag yet. Rod Nascimento Systems Engineer @ Brazil People don't buy WHAT you do. They buy WHY you do it. From: Mark Kerzner [mailto:mark.kerz...@shmsoft.com] Sent: Friday, January 24, 2014 3:16 PM To: Hadoop User Subject: Re: hdfs fsck -locations Sorry, did not copy the full command hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232http://192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated):1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks:0 (0.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY On Fri, Jan 24, 2014 at 11:08 AM, Harsh J ha...@cloudera.commailto:ha...@cloudera.com wrote: Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.commailto:mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232http://192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.commailto:ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.commailto:mark.kerz...@shmsoft.com wrote: Hi, hdfs fsck -locations is supposed to show every block with its location? Is location the ip of the datanode? Thank you, Mark -- Harsh J
RE: hdfs fsck -locations
Hi Mark, It is a sample from my sandbox. Your question is about the part that is in RED at the output below, right? [root@sandbox ~]# hdfs fsck /user/ambari-qa/passwd -locations Connecting to namenode via http://sandbox.hortonworks.com:50070 FSCK started by root (auth:SIMPLE) from /172.16.13.30 for path /user/ambari-qa/passwd at Fri Jan 24 09:53:43 PST 2014 . /user/ambari-qa/passwd: Under replicated BP-1578958328-10.0.2.15-1382306880516:blk_1073742464_1640. Target Replicas is 3 but found 1 replica(s). Status: HEALTHY Total size:1708 B Total dirs:0 Total files:1 Total symlinks:0 Total blocks (validated):1 (avg. block size 1708 B) Minimally replicated blocks:1 (100.0 %) Over-replicated blocks:0 (0.0 %) Under-replicated blocks:1 (100.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor:3 Average block replication:1.0 Corrupt blocks:0 Missing replicas:2 (66.64 %) Number of data-nodes:1 Number of racks:1 FSCK ended at Fri Jan 24 09:53:43 PST 2014 in 1 milliseconds The filesystem under path '/user/ambari-qa/passwd' is HEALTHY [root@sandbox ~]# Rod Nascimento From: Nascimento, Rodrigo [rodrigo.nascime...@netapp.com] Sent: Friday, January 24, 2014 3:34 PM To: user@hadoop.apache.org Subject: RE: hdfs fsck -locations I’m not seeing locations flag yet. Rod Nascimento Systems Engineer @ Brazil People don’t buy WHAT you do. They buy WHY you do it. From: Mark Kerzner [mailto:mark.kerz...@shmsoft.com] Sent: Friday, January 24, 2014 3:16 PM To: Hadoop User Subject: Re: hdfs fsck -locations Sorry, did not copy the full command hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232http://192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated):1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks:0 (0.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY On Fri, Jan 24, 2014 at 11:08 AM, Harsh J ha...@cloudera.commailto:ha...@cloudera.com wrote: Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.commailto:mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232http://192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.commailto:ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.commailto:mark.kerz...@shmsoft.com wrote: Hi, hdfs fsck -locations is supposed to show every block with its location? Is location the ip of the datanode? Thank you, Mark -- Harsh J
Re: hdfs fsck -locations
hdfs fsck /user/mark/data/word_count.csv *-locations* On Fri, Jan 24, 2014 at 11:34 AM, Nascimento, Rodrigo rodrigo.nascime...@netapp.com wrote: I’m not seeing locations flag yet. *Rod Nascimento* *Systems Engineer @ Brazil* *People **don’t** buy **WHAT** you do. They buy **WHY** you do it.* *From:* Mark Kerzner [mailto:mark.kerz...@shmsoft.com] *Sent:* Friday, January 24, 2014 3:16 PM *To:* Hadoop User *Subject:* Re: hdfs fsck -locations Sorry, did not copy the full command hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated):1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks:0 (0.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY On Fri, Jan 24, 2014 at 11:08 AM, Harsh J ha...@cloudera.com wrote: Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Hi, hdfs fsck -locations is supposed to show every block with its location? Is location the ip of the datanode? Thank you, Mark -- Harsh J
Re: hdfs fsck -locations
Can you send me your output? hadoop version Hadoop 2.0.0-cdh4.5.0 Subversion git://ubuntu64-12-04-mk1/var/lib/jenkins/workspace/generic-package-ubuntu64-12-04/CDH4.5.0-Packaging-Hadoop-2013-11-20_14-31-53/hadoop-2.0.0+1518-1.cdh4.5.0.p0.24~precise/src/hadoop-common-project/hadoop-common -r 8e266e052e423af592871e2dfe09d54c03f6a0e8 Compiled by jenkins on Wed Nov 20 15:10:35 PST 2013 From source with checksum 9848b0f85b461913ed63fa19c2b79ccc This command was run using /usr/lib/hadoop/hadoop-common-2.0.0-cdh4.5.0.jar On Fri, Jan 24, 2014 at 3:14 PM, Nascimento, Rodrigo rodrigo.nascime...@netapp.com wrote: Mark, Did you see that your output is different from mine? which Is your hadoop versioks Rodrigo Nascimento Systems Engineer @ Brazil Mobile +55 11 991.873.810 Sent from my iPhone On 24/01/2014, at 18:31, Mark Kerzner mark.kerz...@shmsoft.com wrote: HI, Rodrigo, I am fine thank you :) Here is the _complete_ output mark@mark-7:~$ hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 14:30:23 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 14:30:23 CST 2014 in 0 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY mark@mark-7:~$ On Fri, Jan 24, 2014 at 1:35 PM, Nascimento, Rodrigo rodrigo.nascime...@netapp.com wrote: Hi Mark, How are you? at your output from word_count.csv, the portion related to the nlock locations is missing. This is the reason why I told you in my previous message that I'm not seeing locations yet. please, take a look at my last e-mail. All the best, Rodrigo Nascimento Systems Engineer @ Brazil Sent from my iPhone On 24/01/2014, at 16:40, Mark Kerzner mark.kerz...@shmsoft.com wrote: hdfs fsck /user/mark/data/word_count.csv *-locations* On Fri, Jan 24, 2014 at 11:34 AM, Nascimento, Rodrigo rodrigo.nascime...@netapp.com wrote: I’m not seeing locations flag yet. *Rod Nascimento* *Systems Engineer @ Brazil* *People **don’t** buy **WHAT** you do. They buy **WHY** you do it.* *From:* Mark Kerzner [mailto:mark.kerz...@shmsoft.com] *Sent:* Friday, January 24, 2014 3:16 PM *To:* Hadoop User *Subject:* Re: hdfs fsck -locations Sorry, did not copy the full command hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated):1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks:0 (0.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY On Fri, Jan 24, 2014 at 11:08 AM, Harsh J ha...@cloudera.com wrote: Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote
Re: hdfs fsck -locations
Yes, Rodrigo, that's what I was looking for. So in my install I somehow don't have it at all. Was asked by my students, so I got the answer. Mark On Fri, Jan 24, 2014 at 4:00 PM, Nascimento, Rodrigo rodrigo.nascime...@netapp.com wrote: Mark, there we go ;-) Rodrigo Nascimento Systems Engineer @ Brazil Mobile +55 11 991.873.810 Sent from my iPhone Begin forwarded message: *From:* Nascimento, Rodrigo rodrigo.nascime...@netapp.com *Date:* 24 de janeiro de 2014 15:59:33 BRST *To:* user@hadoop.apache.org user@hadoop.apache.org *Subject:* *RE: hdfs fsck -locations* Hi Mark, It is a sample from my sandbox. Your question is about the part that is in RED at the output below, right? [root@sandbox ~]# hdfs fsck /user/ambari-qa/passwd -locations Connecting to namenode via http://sandbox.hortonworks.com:50070 FSCK started by root (auth:SIMPLE) from /172.16.13.30 for path /user/ambari-qa/passwd at Fri Jan 24 09:53:43 PST 2014 . */user/ambari-qa/passwd: Under replicated BP-1578958328-10.0.2.15-1382306880516:blk_1073742464_1640. Target Replicas is 3 but found 1 replica(s).* Status: HEALTHY Total size:1708 B Total dirs:0 Total files:1 Total symlinks:0 Total blocks (validated):1 (avg. block size 1708 B) Minimally replicated blocks:1 (100.0 %) Over-replicated blocks:0 (0.0 %) Under-replicated blocks:1 (100.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor:3 Average block replication:1.0 Corrupt blocks:0 Missing replicas:2 (66.64 %) Number of data-nodes:1 Number of racks:1 FSCK ended at Fri Jan 24 09:53:43 PST 2014 in 1 milliseconds The filesystem under path '/user/ambari-qa/passwd' is HEALTHY [root@sandbox ~]# Rod Nascimento -- *From:* Nascimento, Rodrigo [rodrigo.nascime...@netapp.com] *Sent:* Friday, January 24, 2014 3:34 PM *To:* user@hadoop.apache.org *Subject:* RE: hdfs fsck -locations I’m not seeing locations flag yet. *Rod Nascimento* *Systems Engineer @ Brazil* *People **don’t** buy **WHAT** you do. They buy **WHY** you do it.* *From:* Mark Kerzner [mailto:mark.kerz...@shmsoft.commark.kerz...@shmsoft.com] *Sent:* Friday, January 24, 2014 3:16 PM *To:* Hadoop User *Subject:* Re: hdfs fsck -locations Sorry, did not copy the full command hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated):1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks:0 (0.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY On Fri, Jan 24, 2014 at 11:08 AM, Harsh J ha...@cloudera.com wrote: Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Hi, hdfs fsck -locations is supposed to show every block with its location? Is location the ip of the datanode? Thank you, Mark -- Harsh J
Re: hdfs fsck -locations
The right syntax is to use -files -blocks -locations, so it drills down all the way. You are not missing a feature - this has existed for as long as I've known HDFS. In Rodrigo's output, he's seeing a BlockPool ID, which is not equivalent to a location, but just carries an IP in it for identification purposes. On Sat, Jan 25, 2014 at 3:53 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Yes, Rodrigo, that's what I was looking for. So in my install I somehow don't have it at all. Was asked by my students, so I got the answer. Mark On Fri, Jan 24, 2014 at 4:00 PM, Nascimento, Rodrigo rodrigo.nascime...@netapp.com wrote: Mark, there we go ;-) Rodrigo Nascimento Systems Engineer @ Brazil Mobile +55 11 991.873.810 Sent from my iPhone Begin forwarded message: From: Nascimento, Rodrigo rodrigo.nascime...@netapp.com Date: 24 de janeiro de 2014 15:59:33 BRST To: user@hadoop.apache.org user@hadoop.apache.org Subject: RE: hdfs fsck -locations Hi Mark, It is a sample from my sandbox. Your question is about the part that is in RED at the output below, right? [root@sandbox ~]# hdfs fsck /user/ambari-qa/passwd -locations Connecting to namenode via http://sandbox.hortonworks.com:50070 FSCK started by root (auth:SIMPLE) from /172.16.13.30 for path /user/ambari-qa/passwd at Fri Jan 24 09:53:43 PST 2014 . /user/ambari-qa/passwd: Under replicated BP-1578958328-10.0.2.15-1382306880516:blk_1073742464_1640. Target Replicas is 3 but found 1 replica(s). Status: HEALTHY Total size:1708 B Total dirs:0 Total files:1 Total symlinks:0 Total blocks (validated):1 (avg. block size 1708 B) Minimally replicated blocks:1 (100.0 %) Over-replicated blocks:0 (0.0 %) Under-replicated blocks:1 (100.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor:3 Average block replication:1.0 Corrupt blocks:0 Missing replicas:2 (66.64 %) Number of data-nodes:1 Number of racks:1 FSCK ended at Fri Jan 24 09:53:43 PST 2014 in 1 milliseconds The filesystem under path '/user/ambari-qa/passwd' is HEALTHY [root@sandbox ~]# Rod Nascimento From: Nascimento, Rodrigo [rodrigo.nascime...@netapp.com] Sent: Friday, January 24, 2014 3:34 PM To: user@hadoop.apache.org Subject: RE: hdfs fsck -locations I’m not seeing locations flag yet. Rod Nascimento Systems Engineer @ Brazil People don’t buy WHAT you do. They buy WHY you do it. From: Mark Kerzner [mailto:mark.kerz...@shmsoft.com] Sent: Friday, January 24, 2014 3:16 PM To: Hadoop User Subject: Re: hdfs fsck -locations Sorry, did not copy the full command hdfs fsck /user/mark/data/word_count.csv -locations Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 11:15:17 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated):1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks:0 (0.0 %) Mis-replicated blocks:0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 11:15:17 CST 2014 in 1 milliseconds The filesystem under path '/user/mark/data/word_count.csv' is HEALTHY On Fri, Jan 24, 2014 at 11:08 AM, Harsh J ha...@cloudera.com wrote: Sorry, but what was the question? I also do not see a locations option flag. On Jan 24, 2014 7:17 PM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Here is an example hdfs fsck /user/mark/data/word_count.csv Connecting to namenode via http://mark-7:50070 FSCK started by mark (auth:SIMPLE) from /192.168.1.232 for path /user/mark/data/word_count.csv at Fri Jan 24 07:45:24 CST 2014 .Status: HEALTHY Total size: 7217 B Total dirs: 0 Total files: 1 Total blocks (validated): 1 (avg. block size 7217 B) Minimally replicated blocks: 1 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 0 (0.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 1 Average block replication: 1.0 Corrupt blocks: 0 Missing replicas: 0 (0.0 %) Number of data-nodes: 1 Number of racks: 1 FSCK ended at Fri Jan 24 07:45:24 CST 2014 in 0 milliseconds On Fri, Jan 24, 2014 at 4:34 AM, Harsh J ha...@cloudera.com wrote: Hi Mark, Yes, the locations are shown as IP. On Fri, Jan 24, 2014 at 12:09 AM, Mark Kerzner mark.kerz...@shmsoft.com wrote: Hi, hdfs fsck -locations is supposed to show every block with its location
