[
https://issues.apache.org/jira/browse/HIVE-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793675#action_12793675
]
Dave Lerman commented on HIVE-1007:
-----------------------------------
Sorry about the non-compiling patch - it requires a few changes to get working
and I thought I'd check on the approach before I do that.
I believe the underlying issue is in Hadoop. FileInputFormat.getSplits actually
creates an empty InputSplit for each empty input file:
{code}
if (length != 0) {
...
} else {
//Create empty hosts array for zero length files
splits.add(new FileSplit(path, 0, length, new String[0]));
}
{code}
so if you pass all empty files, you'll still get a mapper per input file.
However, with CombineFileInputFormat.getSplits, it looks at the list of blocks
and creates splits made up of all the blocks it finds, so if you pass all empty
files, it returns an empty list which hangs Hadoop.
I'm using Cloudera's hadoop-0.20.1+152, so maybe this is fixed in a more recent
Hadoop, but if not, I can file a bug in Hadoop and/or could add a workaround in
Hive which creates an empty split if CombineFileInputFormat.getSplits returns
an empty list.
> CombinedHiveInputFormat fails with empty input
> ----------------------------------------------
>
> Key: HIVE-1007
> URL: https://issues.apache.org/jira/browse/HIVE-1007
> Project: Hadoop Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.4.1
> Reporter: Dave Lerman
> Attachments: hive.1007.1.patch
>
>
> In a multi-stage query, when one stage returns no data (resulting in a bunch
> of output files with size 0), the next stage creates a job with 0 mappers
> which just sits in the Hadoop task track forever and hangs the query at 0%.
> The issue is that CombineHiveInputFormat looks for blocks to populate splits,
> find nones (since input is all 0 bytes), and then returns an empty array from
> getSplits.
> There may be good a way to just skip that job altogether, but as a quick hack
> to get it working, when there are no splits, I just create a single empty one
> using the first path so that the job doesn't hang.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.