Subject: Re: hadoop input buffer size
To: common-user@hadoop.apache.org
Hi,
Hadoop neither read one line each time, nor fetching
dfs.block.size of lines
into a buffer,
Actually, for the TextInputFormat, it read io.file.buffer.size
bytes of text
into a buffer each time,
this can be seen
here.
Regards,
Uma
- Original Message -
From: Yang Xiaoliang yangxiaoliang2...@gmail.com
Date: Wednesday, October 5, 2011 4:27 pm
Subject: Re: hadoop input buffer size
To: common-user@hadoop.apache.org
Hi,
Hadoop neither read one line each time, nor fetching
dfs.block.size
Hello,
Correct me if I'm wrong, but when a program opens n-files at the same time
to read from, and start reading from each file at a time 1 line at a time.
Isn't hadoop actually fetching dfs.block.size of lines into a buffer? and
not actually one line.
If this is correct, I set up my
Hi,
Hadoop neither read one line each time, nor fetching dfs.block.size of lines
into a buffer,
Actually, for the TextInputFormat, it read io.file.buffer.size bytes of text
into a buffer each time,
this can be seen from the hadoop source file LineReader.java
2011/10/5 Mark question
