hi all,
Regarding CompositeInputFormat, my experience is that when giving a
directory as an input, the entries from the files in the directory do
not join. Entries join as expected when giving each individual file as
an input. Is this the expected behavior? I would expect both join
expressions below to give the same result.
Path dirPath = new Path("hdfs://some/dir");
Path[] list = allFilesInDir(dirPath);
// This join expression does not join entries:
CompositeInputFormat.compose("outer", SequenceFileInputFormat.class, new
Path[]{dirPath});
// Does join entries:
CompositeInputFormat.compose("outer", SequenceFileInputFormat.class, list);
regards,
Brien
