Re: HDFS block physical location
nohup hadoop fsck / -files -blocks -locations cat nohup.out | grep [your block name] Hope this helps. On Wed, Jul 25, 2012 at 5:17 PM, 20seco...@web.de wrote: Hi, just a short question. Is there any way to figure out the physical storage location of a given block? I don't mean just a list of hostnames (which I know how to obtain), but actually the file where it is being stored in. We use several hard disks for hdfs data on each node, and I would need to know which block ends up on which harddisk. Thanks! JS
Aw: Re: HDFS block physical location
Thanks, but that just gives me the hostnames or am I overlooking something? I actually need the filename/harddisk on the node. JS Gesendet: Mittwoch, 25. Juli 2012 um 23:33 Uhr Von: Chen He airb...@gmail.com An: common-user@hadoop.apache.org Betreff: Re: HDFS block physical location nohup hadoop fsck / -files -blocks -locations cat nohup.out | grep [your block name] Hope this helps. On Wed, Jul 25, 2012 at 5:17 PM, 20seco...@web.de wrote: Hi, just a short question. Is there any way to figure out the physical storage location of a given block? I don't mean just a list of hostnames (which I know how to obtain), but actually the file where it is being stored in. We use several hard disks for hdfs data on each node, and I would need to know which block ends up on which harddisk. Thanks! JS
Re: Re: HDFS block physical location
For block to filename mapping, you can get from my previous answer. For block to harddisk mapping, you may need to traverse all the directory that used for HDFS, I am sure your OS has the information about which hard drive is mounted to which directory. with these two types of information, you can write a small Perl or Python script to get what you want. Or Take look of the namenode.java and see where and how it saves the table of block information. Please correct me if there is any mistake. Chen On Wed, Jul 25, 2012 at 6:10 PM, 20seco...@web.de wrote: Thanks, but that just gives me the hostnames or am I overlooking something? I actually need the filename/harddisk on the node. JS Gesendet: Mittwoch, 25. Juli 2012 um 23:33 Uhr Von: Chen He airb...@gmail.com An: common-user@hadoop.apache.org Betreff: Re: HDFS block physical location nohup hadoop fsck / -files -blocks -locations cat nohup.out | grep [your block name] Hope this helps. On Wed, Jul 25, 2012 at 5:17 PM, 20seco...@web.de wrote: Hi, just a short question. Is there any way to figure out the physical storage location of a given block? I don't mean just a list of hostnames (which I know how to obtain), but actually the file where it is being stored in. We use several hard disks for hdfs data on each node, and I would need to know which block ends up on which harddisk. Thanks! JS
Re: Re: HDFS block physical location
Hi JS, You may also be interested in the following JIRA which proposes an API for block-disk mapping information: https://issues.apache.org/jira/browse/HDFS-3672 There has been some discussion about potential use cases on this JIRA. If you can describe what your use case is for this information on the JIRA, we'd really appreciate the input. -Todd On Wed, Jul 25, 2012 at 3:20 PM, Chen He airb...@gmail.com wrote: For block to filename mapping, you can get from my previous answer. For block to harddisk mapping, you may need to traverse all the directory that used for HDFS, I am sure your OS has the information about which hard drive is mounted to which directory. with these two types of information, you can write a small Perl or Python script to get what you want. Or Take look of the namenode.java and see where and how it saves the table of block information. Please correct me if there is any mistake. Chen On Wed, Jul 25, 2012 at 6:10 PM, 20seco...@web.de wrote: Thanks, but that just gives me the hostnames or am I overlooking something? I actually need the filename/harddisk on the node. JS Gesendet: Mittwoch, 25. Juli 2012 um 23:33 Uhr Von: Chen He airb...@gmail.com An: common-user@hadoop.apache.org Betreff: Re: HDFS block physical location nohup hadoop fsck / -files -blocks -locations cat nohup.out | grep [your block name] Hope this helps. On Wed, Jul 25, 2012 at 5:17 PM, 20seco...@web.de wrote: Hi, just a short question. Is there any way to figure out the physical storage location of a given block? I don't mean just a list of hostnames (which I know how to obtain), but actually the file where it is being stored in. We use several hard disks for hdfs data on each node, and I would need to know which block ends up on which harddisk. Thanks! JS -- Todd Lipcon Software Engineer, Cloudera
