Also, I just want to clear... the delay seems to at the intial
(read = in.read(buf))
after the first time into the loop it flies...
Ananth T Sarathy
On Wed, Aug 19, 2009 at 1:58 PM, Raghu Angadi <rang...@yahoo-inc.com> wrote:
> Edward Capriolo wrote:
>
>> On Wed, Aug 19, 2009 at 11:11 AM, Edward Capriolo <edlinuxg...@gmail.com
>>> >wrote:
>>>
>>> It would be as fast as underlying filesystem goes.
>>>>>>
>>>>> I would not agree with that statement. There is overhead.
>>>>
>>>
> You might be misinterpreting my comment. There is of course some over head
> (at the least the procedure calls).. depending on you underlying filesystem,
> there could be extra buffer copies and CRC overhead. But none of that
> explains transfer as slow as 1 MBps (if my interpretation of of results is
> correct).
>
> Raghu.
>
>
> In some testing I did writing a small file can
>>>> take 30-300 ms. So if you have 9000 small files (like I did) and you
>>>> are single threaded this takes a long time.
>>>>
>>>> If you orchestrate your task to use FSDataInput and FSDataOutput in
>>>> the map or reduce phase then each mapper or reducer is writing a file
>>>> at a time. Now that is fast.
>>>>
>>>> Ananth, are you doing your r/w inside a map/reduce job or are you just
>>>> using FS* in a top down program?
>>>>
>>>>
>>>>
>>>> On Wed, Aug 19, 2009 at 1:26 AM, Raghu Angadi<rang...@yahoo-inc.com>
>>>> wrote:
>>>>
>>>>> Ananth T. Sarathy wrote:
>>>>>
>>>>>> I am trying to download binary files stored in Hadoop but there is
>>>>>> like
>>>>>>
>>>>> a
>>>>
>>>>> 2
>>>>>> minute wait on a 20mb file when I try to execute the in.read(buf).
>>>>>>
>>>>> What does this mean : 2 min to pipe 20mb or one or your one of the
>>>>>
>>>> in.read()
>>>>
>>>>> calls took 2 minutes? Your code actually measures team for read and
>>>>>
>>>> write.
>>>>
>>>>> There is nothing in FSInputstream to cause this slow down. Do you think
>>>>> anyone would use Hadoop otherwise? It would be as fast as underlying
>>>>> filesystem goes.
>>>>>
>>>>> Raghu.
>>>>>
>>>>> is there a better way to be doing this?
>>>>>>
>>>>>> private void pipe(InputStream in, OutputStream out) throws
>>>>>>
>>>>> IOException
>>>>
>>>>> { System.out.println(System.currentTimeMillis()+" Starting to Pipe
>>>>>> Data");
>>>>>> byte[] buf = new byte[1024];
>>>>>> int read = 0;
>>>>>> while ((read = in.read(buf)) >= 0)
>>>>>> {
>>>>>> out.write(buf, 0, read);
>>>>>> System.out.println(System.currentTimeMillis()+" Piping
>>>>>>
>>>>> Data");
>>>>
>>>>> }
>>>>>> out.flush();
>>>>>> System.out.println(System.currentTimeMillis()+" Finished Piping
>>>>>> Data");
>>>>>>
>>>>>> }
>>>>>>
>>>>>> public void readFile(String fileToRead, OutputStream out)
>>>>>> throws IOException
>>>>>> {
>>>>>> System.out.println(System.currentTimeMillis()+" Start Read
>>>>>>
>>>>> File");
>>>>
>>>>> Path inFile = new Path(fileToRead);
>>>>>> System.out.println(System.currentTimeMillis()+" Set Path");
>>>>>> // Validate the input/output paths before reading/writing.
>>>>>>
>>>>>> if (!fs.exists(inFile))
>>>>>> {
>>>>>> throw new HadoopFileException("Specified file " +
>>>>>> fileToRead
>>>>>> + " not found.");
>>>>>> }
>>>>>> if (!fs.isFile(inFile))
>>>>>> {
>>>>>> throw new HadoopFileException("Specified file " +
>>>>>> fileToRead
>>>>>> + " not found.");
>>>>>> }
>>>>>> // Open inFile for reading.
>>>>>> System.out.println(System.currentTimeMillis()+" Opening Data
>>>>>> Stream");
>>>>>> FSDataInputStream in = fs.open(inFile);
>>>>>>
>>>>>> System.out.println(System.currentTimeMillis()+" Opened Data
>>>>>> Stream");
>>>>>> // Open outFile for writing.
>>>>>>
>>>>>> // Read from input stream and write to output stream until EOF.
>>>>>> pipe(in, out);
>>>>>>
>>>>>> // Close the streams when done.
>>>>>> out.close();
>>>>>> in.close();
>>>>>> }
>>>>>> Ananth T Sarathy
>>>>>>
>>>>>>
>>>>>
>
