[
https://issues.apache.org/jira/browse/HIVE-7541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080920#comment-14080920
]
Xuefu Zhang commented on HIVE-7541:
-----------------------------------
[~nyang] Thanks for working on this. This task is fairly large, and I think
breaking the task into smaller ones would help in sharing the load and tracking
progress. Could you please create smaller JIRAs for this? I'd image that
supporting union would require work in the following area:
1. SparkCompiler changes: generate a SparkWork that contains UnionWork from
logical operator tree.
2. SparkPlan modeling: represent the spark job in terms of a graph (rather
than) list of SparkTran instances. We may need to enhance SparkTran interface.
3. SparkPlanGenerator: need to generate a plan from SparkWork, which needs to
use Spark's union transformation to achieve the functionality..
4. other earas.
Tez can be a good reference point
Please feel free to create JIRAs for those or other areas.
> Support union all on Spark
> --------------------------
>
> Key: HIVE-7541
> URL: https://issues.apache.org/jira/browse/HIVE-7541
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Na Yang
>
> For union all operator, we will use Spark's union transformation. Refer to
> the design doc on wiki for more information.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
