Question about Data Node configuration

[A Levine]

I am trying to configure a large install and I have a question about
the configuration of Data Nodes.  Each data node has multiple drives.
Each drive is 1TB in size.  In the hdfs-site.xml, I can have multiple
directories (which will be mounted drives) specified as shown by:

  
dfs.data.dir
/mount1,/mount2,/mount3,
true
  

For the drive that has the OS, only 100G will be used for the OS.  Is
it good practice to have a partition on the drive that has the OS used
for the dfs.data.dir?  Will this slow things down?  Will the size
difference available to each directory be a problem?  Also, if it is
not a good idea to use the OS drive, then how about pointing logs to
that drive?

andrew

Re: Listing subdirectories in Hadoop

[A Levine]

If you were talking about looking at directories within a Java
program, here is what has worked for me.

FileSystem fs;
FileStatus[] fileStat;
Path[] fileList;
SequenceFile.Reader reader = null;
try{
 // connect to the file system
 fs = FileSystem.get(conf);

 // get the stat on all files in the source directory
 fileStat = fs.listStatus(sourceDir);

 // get paths to the files in the source directory
 fileList = FileUtil.stat2Paths(fileStat);

// then you can do something like
for(int x = 0; x < fileList.length; x++){
 System.out.println(x + " " + fileList[x]);
}
} catch(IOException ioe){
// do something
}

Hope this helps.

andrew

--

On Tue, Mar 30, 2010 at 11:54 AM, Ted Yu  wrote:
> Does this get what you want ?
> hadoop dfs -ls  | grep drwx
>
> On Tue, Mar 30, 2010 at 8:24 AM, Santiago Pérez  wrote:
>
>>
>> Hej
>>
>> I've checking the API and on internet but I have not found any method for
>> listing the subdirectories of a given directory in the HDFS.
>>
>> Can anybody show me how to get the list of subdirectories or even how to
>> implement the method? (I guess that it should be possible and not very
>> hard).
>>
>> Thanks in advance ;)
>> --
>> View this message in context:
>> http://old.nabble.com/Listing-subdirectories-in-Hadoop-tp28084164p28084164.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
>