[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977619#comment-15977619
]
Xuefu Zhang edited comment on HIVE-11133 at 4/20/17 9:53 PM:
-------------------------------------------------------------
[~stakiar], In spark, one non-root stage may have a vertex dependency. For
instance
{code}
select sum(hash(a.k1,a.v1,a.k2, a.v2))
from (
select src1.key as k1, src1.value as v1,
src2.key as k2, src2.value as v2 FROM
(select * FROM src WHERE src.key < 10) src1
JOIN
(select src.key, sum(src.value) FROM src WHERE src.key < 10 group by src.key)
src2
SORT BY k1, v1, k2, v2
) a
{code}
Basically, the stage that generates the hash table used for mapjoin can have
vertex dependencies (map -> reduce). I'm not sure if this is the only case,
though.
was (Author: xuefuz):
[~stakiar], In spark, any stage may have a vertex dependency. For instance
{code}
select sum(hash(a.k1,a.v1,a.k2, a.v2))
from (
select src1.key as k1, src1.value as v1,
src2.key as k2, src2.value as v2 FROM
(select * FROM src WHERE src.key < 10) src1
JOIN
(select src.key, sum(src.value) FROM src WHERE src.key < 10 group by src.key)
src2
SORT BY k1, v1, k2, v2
) a
{code}
Basically, the stage that generates the hash table used for mapjoin can have
vertex dependencies (map -> reduce). I'm not sure if this is the only case,
though.
> Support hive.explain.user for Spark
> -----------------------------------
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Mohit Sabharwal
> Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
