Re: org.apache.hadoop.io.MapFile.Reader.Reader(FileSystem fs, String dirName, Configuration conf) constructor ignores the FileSystem

2012-05-18 Thread Subroto
Thanks Harsh….. :-) Cheers, Subroto Sanyal On May 18, 2012, at 4:30 PM, Harsh J wrote: > With 0.23/2.x onwards, to have better behavior, its your Path that > will determine the FS in SequenceFile readers/writers. Hence, always > make sure your Path carries the proper URI if it is supposed to be

Re: org.apache.hadoop.io.MapFile.Reader.Reader(FileSystem fs, String dirName, Configuration conf) constructor ignores the FileSystem

2012-05-18 Thread Harsh J
With 0.23/2.x onwards, to have better behavior, its your Path that will determine the FS in SequenceFile readers/writers. Hence, always make sure your Path carries the proper URI if it is supposed to belong to a different FS. You can look at one of the many Path constructors to help you get such an

org.apache.hadoop.io.MapFile.Reader.Reader(FileSystem fs, String dirName, Configuration conf) constructor ignores the FileSystem

2012-05-18 Thread Subroto
Hi, The constructor of Reader class ignores the FileSystem parameter provided in the constructor parameter. This results in creation of Path on the basis of default FileSystem mentioned in the configuration even though user wants to pass LocalFileSystem. The behavior has been verified with cdh4

Need help for writing map reduce functions in hadoop-1.0.1 java

2012-05-18 Thread Ravi Joshi
Yeah, finally i get the exact place for my question. Hi, I am newbie in Hadoop. I have successfully installed Hadoop-1.0.1 on my Ubuntu10.04 LTS and i am using Eclipse Indigo for designing Hadoop MapReduce application. I am writing my own map and reduce method for implementing K Means algorithm

Re: How HDFS divides Files into block

2012-05-18 Thread Harsh J
Utkarsh, This question has been asked several times before. I've myself previously answered the same question at: http://www.mail-archive.com/mapreduce-user@hadoop.apache.org/msg04282.html If HDFS says its block size is 64M, then that is what the block size is. HDFS is a filesystem, and writes on

How HDFS divides Files into block

2012-05-18 Thread Utkarsh Gupta
Hi, I have a doubt about HDFS which may be a very trivial thing but I am not able to understand it. Since hdfs keeps the files in block of 64/128 MB how does HDFS splits files? The problem which I see is that suppose I have a long string in my input file as: 672364,423746273,4234234,2,342,34,2