[
https://issues.apache.org/jira/browse/DRILL-4476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15271035#comment-15271035
]
Sean Hsuan-Yi Chu commented on DRILL-4476:
------------------------------------------
The logic in this patch can cause problems where Union-all operator makes wrong
judgement regarding whether a record-batch is coming from an empty file/table.
To be more specific, this patch uses the number of rows to determine whether
the record-batch from the {color:red} left-side {color} was produced by an
empty file/table.
(Actually, this logic was applied to the {color:red} right-side {color} of
Union-all in DRILL-2288 patch.)
However, the number of rows is not sufficient to make a correct judgement. For
instance, a record-batch which had passed by a "false filter" / "limit 0" will
carry 0 row.
For another example (DRILL-4510), when multiple Union-alls are running in
parallel, one of the Union-all might just happen to not receive 0 row (due to
data partition).
Then, this Union-all will infer an output schema which is different from others.
Unless the record-batch can somehow capture the information regarding its
source (whether this record-batch is from an empty file), Union-all cannot be
make a correct judgement.
Thus, I think we have to back out this patch. And do not start to support empty
files until there is a way to carry the empty file information with the
record-batch (possibly by having a new type to represent non-existent column?).
> Enhance Union-All operator for dealing with empty left input or empty both
> inputs
> ---------------------------------------------------------------------------------
>
> Key: DRILL-4476
> URL: https://issues.apache.org/jira/browse/DRILL-4476
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Reporter: Sean Hsuan-Yi Chu
> Assignee: Sean Hsuan-Yi Chu
> Fix For: 1.7.0
>
>
> Union-All operator does not deal with the situation where left side comes
> from empty source.
> Due to DRILL-2288's enhancement for empty sources, Union-All operator now can
> be allowed to support this scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
