Hi Gang,
Yes PathFilters work only on file paths. I meant you can include such type of
logic at split level.
The input format's getSplits() method is responsible for computing and adding
splits to a list container, for which JT initializes mapper tasks. You can
override the getSplits() method to add only a few , say, based on the location
or offset, to the list. Here's the reference :
while (((double) bytesRemaining)/splitSize > SPLIT_SLOP) {
int blkIndex = getBlockIndex(blkLocations, length-bytesRemaining);
splits.add(new FileSplit(path, length-bytesRemaining, splitSize,
blkLocations[blkIndex].getHosts()));
bytesRemaining -= splitSize;
}
if (bytesRemaining != 0) {
splits.add(new FileSplit(path, length-bytesRemaining, bytesRemaining,
blkLocations[blkLocations.length-1].getHosts()));
Before splits.add you can use your logic for discarding. However, you need to
ensure your record reader takes care of incomplete records at boundaries.
To get the block locations to load separately, the FileSystem class APIs expose
few methods like getBlockLocations etc ..
Hope this helps.
Amogh
On 1/28/10 7:26 PM, "Gang Luo" <lgpub...@yahoo.com.cn> wrote:
Thanks Amogh.
For the second part of my question, I actually mean loading block separately
from HDFS. I don't know whether it is realistic. Anyway, for my goal is to
process different division of a file separately, to do that at split level is
OK. But even I can get the splits from inputformat, how to "add only a few
splits you need to mapper and discard the others"? (pathfilters only works for
file, but not block, I think).
Thanks.
-Gang
