[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009998#comment-16009998
]
Lefty Leverenz commented on HIVE-11133:
---
Thanks for the docs, Sahil. I moved *hive.spark.explain.user* into the Spark
section and added a reference from *hive.explain.user*. Here are direct links
to both configs:
* [Configuration Properties -- hive.spark.explain.user |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.spark.explain.user]
* [Configuration Properties -- hive.explain.user |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.explain.user]
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.10.patch, HIVE-11133.1.patch,
> HIVE-11133.2.patch, HIVE-11133.3.patch, HIVE-11133.4.patch,
> HIVE-11133.5.patch, HIVE-11133.6.patch, HIVE-11133.7.patch,
> HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1611#comment-1611
]
Sahil Takiar commented on HIVE-11133:
-
I updated the config descriptions and updated the Explain Wiki:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain#LanguageManualExplain-User-levelExplainOutput
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.10.patch, HIVE-11133.1.patch,
> HIVE-11133.2.patch, HIVE-11133.3.patch, HIVE-11133.4.patch,
> HIVE-11133.5.patch, HIVE-11133.6.patch, HIVE-11133.7.patch,
> HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996539#comment-15996539
]
Lefty Leverenz commented on HIVE-11133:
---
Doc note: The documentation for *hive.explain.user* needs to be updated in the
wiki for release 3.0.0, at least in Configuration Properties and perhaps also
(but probably not) in Hive on Spark: Getting Started.
* [Configuration Properties -- hive.explain.user |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.explain.user]
* [Hive on Spark: Getting Started -- Configuring Hive |
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive]
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.10.patch, HIVE-11133.1.patch,
> HIVE-11133.2.patch, HIVE-11133.3.patch, HIVE-11133.4.patch,
> HIVE-11133.5.patch, HIVE-11133.6.patch, HIVE-11133.7.patch,
> HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993365#comment-15993365
]
Sahil Takiar commented on HIVE-11133:
-
Thanks [~xuefuz]! Sorry for all the hassle in getting this merged.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.10.patch, HIVE-11133.1.patch,
> HIVE-11133.2.patch, HIVE-11133.3.patch, HIVE-11133.4.patch,
> HIVE-11133.5.patch, HIVE-11133.6.patch, HIVE-11133.7.patch,
> HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993332#comment-15993332
]
Xuefu Zhang commented on HIVE-11133:
Patch #9 worked just fine. I guessed it needed java8 to compile.
Committed #9 to master. Thanks again for the contribution, [~stakiar].
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.10.patch, HIVE-11133.1.patch,
> HIVE-11133.2.patch, HIVE-11133.3.patch, HIVE-11133.4.patch,
> HIVE-11133.5.patch, HIVE-11133.6.patch, HIVE-11133.7.patch,
> HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992341#comment-15992341
]
Xuefu Zhang commented on HIVE-11133:
It didn't work. Please play it and make sure the patch works for you and then
provide instructions. Thanks.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992320#comment-15992320
]
Sahil Takiar commented on HIVE-11133:
-
Hey [~xuefuz] sorry about that, {{git apply [path-to-patch-file]}} should work
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992303#comment-15992303
]
Xuefu Zhang commented on HIVE-11133:
Sorry for breaking the build. I should have built locally before push it.
[~stakiar], could you please provide instructions on how to commit your patch?
I used "patch -p1 < HIVE-11133.9.patch" and it appeared not working. Thanks.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992118#comment-15992118
]
Sahil Takiar commented on HIVE-11133:
-
Hey [~xuefuz] I think the the file rename in the patch wasn't applied properly:
{code}
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Connection.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Connection.java
similarity index 95%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Connection.java
rename to
common/src/java/org/apache/hadoop/hive/common/jsonexplain/Connection.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/DagJsonParserUtils.java
similarity index 92%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/TezJsonParserUtils.java
rename to
common/src/java/org/apache/hadoop/hive/common/jsonexplain/DagJsonParserUtils.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Op.java
similarity index 90%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Op.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Op.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Printer.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Printer.java
similarity index 95%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Printer.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Printer.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Stage.java
similarity index 94%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Stage.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Stage.java
diff --git
a/common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java
b/common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
similarity index 81%
rename from
common/src/java/org/apache/hadoop/hive/common/jsonexplain/tez/Vertex.java
rename to common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java
{code}
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Labels: TODOC3.0
> Fix For: 3.0.0
>
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991861#comment-15991861
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12865832/HIVE-11133.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10631 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=234)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4971/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4971/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4971/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12865832 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch, HIVE-11133.9.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991509#comment-15991509
]
Xuefu Zhang commented on HIVE-11133:
Unfortunately, the patch doesn't apply clean on master. [~stakiar], could you
please rebase? Thanks.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991209#comment-15991209
]
Sahil Takiar commented on HIVE-11133:
-
[~lirui], [~xuefuz] can this be merged? Once this is merged I'll start working
on the follow up JIRAs.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984125#comment-15984125
]
Rui Li commented on HIVE-11133:
---
OK I'm +1 to commit this and fix the bugs in follow-ups. [~xuefuz], [~pxiong]
do you have any further comments?
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981683#comment-15981683
]
Sahil Takiar commented on HIVE-11133:
-
[~lirui] no it isn't truncated. This could be another bug. I filed HIVE-16521
as another follow up item. I suspect it has something to do with the UNION
operator, the user level plans for HoS vs. Hive-on-Tez look very different for
queries with a UNION.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980630#comment-15980630
]
Rui Li commented on HIVE-11133:
---
Hi [~stakiar], in the query plan of your
[comment|https://issues.apache.org/jira/browse/HIVE-11133?focusedCommentId=15979385&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15979385],
why it only displays Reducer 11 for Stage-2? Is the output truncated?
{noformat}
Stage-2
Reducer 11
{noformat}
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979922#comment-15979922
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12864584/HIVE-11133.8.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10626 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_index]
(batchId=225)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4840/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4840/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4840/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12864584 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979487#comment-15979487
]
Sahil Takiar commented on HIVE-11133:
-
Thanks [~xuefuz] - addressed the comments from the RB, updated the RB, attached
an updated patch. Created HIVE-16507 to track the naming issue about vertex
dependencies. Created HIVE-16508 and HIVE-16509 as additional follow up tasks
for this JIRA.
[~pxiong] yeah its odd. I created HIVE-16507 as a follow up item to address the
issue. It has all the details.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch, HIVE-11133.8.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979394#comment-15979394
]
Xuefu Zhang commented on HIVE-11133:
Looks good. Two things:
1. Please update the patch based on the review comments. It's mostly cosmetic,
so we don't have to wait for another round of test run.
2. Create a JIRA to track the naming issue about vertex dependencies.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979395#comment-15979395
]
Pengcheng Xiong commented on HIVE-11133:
Why it outputs "Vertex dependency in root stage" twice? Sounds weird...
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979385#comment-15979385
]
Sahil Takiar commented on HIVE-11133:
-
[~xuefuz]
The query:
{code}
set hive.optimize.ppd=true;
set hive.ppd.remove.duplicatefilters=true;
set hive.spark.dynamic.partition.pruning=true;
set hive.optimize.metadataonly=false;
set hive.optimize.index.filter=true;
set hive.strict.checks.cartesian.product=false;
set hive.spark.explain.user=true;
set hive.spark.dynamic.partition.pruning=true;
EXPLAIN select count(*) from srcpart where srcpart.ds in (select
max(srcpart.ds) from srcpart union all select min(srcpart.ds) from srcpart);
{code}
Prints
{code}
Plan optimized by CBO.
Vertex dependency in root stage
Reducer 10 <- Map 9 (GROUP)
Reducer 11 <- Reducer 10 (GROUP), Reducer 13 (GROUP)
Reducer 13 <- Map 12 (GROUP)
Vertex dependency in root stage
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT), Reducer 6 (PARTITION-LEVEL SORT)
Reducer 3 <- Reducer 2 (GROUP)
Reducer 5 <- Map 4 (GROUP)
Reducer 6 <- Reducer 5 (GROUP), Reducer 8 (GROUP)
Reducer 8 <- Map 7 (GROUP)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 3
File Output Operator [FS_34]
Group By Operator [GBY_32] (rows=1 width=8)
Output:["_col0"],aggregations:["count(VALUE._col0)"]
<-Reducer 2 [GROUP]
GROUP [RS_31]
Group By Operator [GBY_30] (rows=1 width=8)
Output:["_col0"],aggregations:["count()"]
Join Operator [JOIN_28] (rows=2200 width=10)
condition
map:[{"":"{\"type\":\"Inner\",\"left\":0,\"right\":1}"}],keys:{"0":"_col0","1":"_col0"}
<-Map 1 [PARTITION-LEVEL SORT]
PARTITION-LEVEL SORT [RS_26]
PartitionCols:_col0
Select Operator [SEL_2] (rows=2000 width=10)
Output:["_col0"]
TableScan [TS_0] (rows=2000 width=10)
default@srcpart,srcpart,Tbl:COMPLETE,Col:NONE
<-Reducer 6 [PARTITION-LEVEL SORT]
PARTITION-LEVEL SORT [RS_27]
PartitionCols:_col0
Group By Operator [GBY_24] (rows=1 width=184)
Output:["_col0"],keys:KEY._col0
<-Reducer 5 [GROUP]
GROUP [RS_23]
PartitionCols:_col0
Group By Operator [GBY_22] (rows=2 width=184)
Output:["_col0"],keys:_col0
Filter Operator [FIL_9] (rows=1 width=184)
predicate:_col0 is not null
Group By Operator [GBY_7] (rows=1 width=184)
Output:["_col0"],aggregations:["max(VALUE._col0)"]
<-Map 4 [GROUP]
GROUP [RS_6]
Group By Operator [GBY_5] (rows=1 width=184)
Output:["_col0"],aggregations:["max(ds)"]
Select Operator [SEL_4] (rows=2000 width=10)
Output:["ds"]
TableScan [TS_3] (rows=2000 width=10)
default@srcpart,srcpart,Tbl:COMPLETE,Col:NONE
<-Reducer 8 [GROUP]
GROUP [RS_23]
PartitionCols:_col0
Group By Operator [GBY_22] (rows=2 width=184)
Output:["_col0"],keys:_col0
Filter Operator [FIL_17] (rows=1 width=184)
predicate:_col0 is not null
Group By Operator [GBY_15] (rows=1 width=184)
Output:["_col0"],aggregations:["min(VALUE._col0)"]
<-Map 7 [GROUP]
GROUP [RS_14]
Group By Operator [GBY_13] (rows=1 width=184)
Output:["_col0"],aggregations:["min(ds)"]
Select Operator [SEL_12] (rows=2000 width=10)
Output:["ds"]
TableScan [TS_11] (rows=2000 width=10)
default@srcpart,srcpart,Tbl:COMPLETE,Col:NONE
Stage-2
Reducer 11
{code}
So there are two sections that say {{Vertex dependency in root stage}}. I
haven't checked to see if this is possible with Hive-on-Tez, but it looks like
an existing bug in the user-level explain code.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: H
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979287#comment-15979287
]
Xuefu Zhang commented on HIVE-11133:
[~stakiar], could you please provide an example where non-root stage vertex
dependency is shown? Thanks.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979255#comment-15979255
]
Sahil Takiar commented on HIVE-11133:
-
[~lirui] this may be an existing bug in the user-level explain plan, or maybe
something that needs more documentation. From what I can tell, the section
labeled {{Vertex dependency in root stage}} basically iterates through all the
stages, and prints out the edges between the vertices. It doesn't seem to be
limited to the vertices of the root stage. The code that controls this is
{{DagJsonParser#print}} - specifically the section that contains
{{printer.println("Vertex dependency in root stage");}}
It does seem confusing to me that the section says {{root stage}} but iterates
through all the stages. I can file a separate JIRA for this.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977939#comment-15977939
]
Rui Li commented on HIVE-11133:
---
Another example of non-root stage having vertex dependency is conditional task.
Suppose you disable {{hive.auto.convert.join}} and enable
{{hive.optimize.skewjoin}}. The following query will generate conditional task
to join skewed data:
{code}
select A.key from A join B on A.key=B.key group by A.key;
{code}
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977931#comment-15977931
]
Rui Li commented on HIVE-11133:
---
[~stakiar], the plan in your last comment doesn't seem correct. Looking at the
"raw query plan", Stage-2 is the root stage which only contains Map 4. However,
in the explain output, the vertex dependency is for Stage-1, although it says
it's for root stage.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977912#comment-15977912
]
Sahil Takiar commented on HIVE-11133:
-
The qtest in the patch has a very similar query:
{code}
select sum(hash(a.k1,a.v1,a.k2, a.v2))
from (
select src1.key as k1, src1.value as v1,
src2.key as k2, src2.value as v2 FROM
(select * FROM src WHERE src.key < 10) src1
JOIN
(select * FROM src WHERE src.key < 10) src2
SORT BY k1, v1, k2, v2
) a
{code}
It's also a mapjoin. The user-level explain output is:
{code}
Plan not optimized by CBO.
Vertex dependency in root stage
Reducer 2 <- Map 1 (PARTITION-LEVEL SORT)
Reducer 3 <- Reducer 2 (GROUP)
Stage-0
Fetch Operator
limit:-1
Stage-1
Reducer 3
File Output Operator [FS_17]
Group By Operator [GBY_15] (rows=1 width=8)
Output:["_col0"],aggregations:["sum(VALUE._col0)"]
<-Reducer 2 [GROUP]
GROUP [RS_14]
Group By Operator [GBY_13] (rows=1 width=8)
Output:["_col0"],aggregations:["sum(hash(_col0,_col1,_col2,_col3))"]
Select Operator [SEL_11] (rows=27556 width=22)
Output:["_col0","_col1","_col2","_col3"]
<-Map 1 [PARTITION-LEVEL SORT]
PARTITION-LEVEL SORT [RS_10]
Map Join Operator [MAPJOIN_20] (rows=27556 width=22)
Conds:(Inner),Output:["_col0","_col1","_col2","_col3"]
<-Select Operator [SEL_2] (rows=166 width=10)
Output:["_col0","_col1"]
Filter Operator [FIL_18] (rows=166 width=10)
predicate:(key < 10)
TableScan [TS_0] (rows=500 width=10)
default@src,src,Tbl:COMPLETE,Col:NONE,Output:["key","value"]
Map Reduce Local Work
Stage-2
Map 4
keys: [HASHTABLESINK_22]
Select Operator [SEL_5] (rows=166 width=10)
Output:["_col0","_col1"]
Filter Operator [FIL_19] (rows=166 width=10)
predicate:(key < 10)
TableScan [TS_3] (rows=500 width=10)
default@src,src,Tbl:COMPLETE,Col:NONE,Output:["key","value"]
Map Reduce Local Work
{code}
The raw query plan looks like:
{code}
{
"STAGE DEPENDENCIES": {
"Stage-2": {
"ROOT STAGE": "TRUE"
},
"Stage-1": {
"DEPENDENT STAGES": "Stage-2"
},
"Stage-0": {
"DEPENDENT STAGES": "Stage-1"
}
},
"STAGE PLANS": {
"Stage-2": {
"Spark": {
"Vertices:": {
"Map 2": {
"Map Operator Tree:": [
{
"TableScan": {
"Output:": [
"key",
"value"
],
"_empty_": "default@myinput1,b,Tbl:COMPLETE,Col:NONE",
"Statistics:": "rows=3 width=8",
"OperatorId:": "TS_1",
"children": {
"keys:": {
"0": "key",
"1": "value",
"OperatorId:": "HASHTABLESINK_10"
}
}
}
}
],
"Local Work:": {
"Map Reduce Local Work": {
}
},
"tag:": "0"
}
}
}
},
"Stage-1": {
"Spark": {
"Vertices:": {
"Map 1": {
"Map Operator Tree:": [
{
"TableScan": {
"Output:": [
"key",
"value"
],
"_empty_": "default@myinput1,a,Tbl:COMPLETE,Col:NONE",
"Statistics:": "rows=3 width=8",
"OperatorId:": "TS_0",
"children": {
"Map Join Operator": {
"condition map:": [
{
"_empty_":
"{\"type\":\"Inner\",\"left\":0,\"right\":1}"
}
],
"input vertices:": {
"1": "Map 2"
},
"keys:": {
"0": "key",
"1": "value"
},
"Output:": [
"_col0",
"_col1",
"_col5",
"_col6"
],
"Statistics:": "rows=3 width=9",
"OperatorId:": "MAPJOIN_7",
"children": {
"Select Operator": {
"Output:": [
"_col0",
"_col1",
"
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977619#comment-15977619
]
Xuefu Zhang commented on HIVE-11133:
[~stakiar], In spark, any stage may have a vertex dependency. For instance
{code}
select sum(hash(a.k1,a.v1,a.k2, a.v2))
from (
select src1.key as k1, src1.value as v1,
src2.key as k2, src2.value as v2 FROM
(select * FROM src WHERE src.key < 10) src1
JOIN
(select src.key, sum(src.value) FROM src WHERE src.key < 10 group by src.key)
src2
SORT BY k1, v1, k2, v2
) a
{code}
Basically, the stage that generates the hash table used for mapjoin can have
vertex dependencies (map -> reduce). I'm not sure if this is the only case,
though.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977392#comment-15977392
]
Pengcheng Xiong commented on HIVE-11133:
[~stakiar], thanks for reaching out to me. The patch looks good to me as well.
Glad that most of the code works for spark as well after your refactoring.
Thanks. :)
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977241#comment-15977241
]
Sahil Takiar commented on HIVE-11133:
-
CC: [~pxiong] this refactors a lot of the code for User-Level Explain, tagging
you in case you want to take a look as I believe you have been working on this
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15977217#comment-15977217
]
Sahil Takiar commented on HIVE-11133:
-
[~lirui] thanks for taking a look. Do you have a HoS query in mind that has
dependencies between non-root stages? I can test it and find out.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975956#comment-15975956
]
Rui Li commented on HIVE-11133:
---
I haven't used the explain user feature before. Got one question, will the
vertex dependency only be printed for root stage? If the plan has multiple
stages, I suppose users expect to see the dependency for all of them.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975728#comment-15975728
]
Xuefu Zhang commented on HIVE-11133:
[~stakiar], Thanks for working on this. I left some minor comments on RB.
Overall, very nice refactoring.
> Support hive.explain.user for Spark
> ---
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974826#comment-15974826
]
Sahil Takiar commented on HIVE-11133:
-
[~xuefuz] could you take a look? This patch basically adds support for
User-Level Explain plans to HoS. Previously, it only worked for Tez. User-Level
Explain plans were added in HIVE-9780 and the documentation can be found
[here|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain#LanguageManualExplain-User-levelExplainOutput].
My comment [here|#comment-15966675] has an explanation of the changes. And I
have made a few more changes since then:
* Added a new config called {{hive.spark.explain.user}} that is an equivalent
of {{hive.explain.user}} but is Spark specific. {{hive.explain.user}} is true
by default, {{hive.spark.explain.user}} is false by default
* I copied {{explainuser_1.q}} to {{spark_explainuser_1.q}} and generated a
{{spark_explainuser_1.q.out}} file - I confirmed that the new explain plans
look correct
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15974163#comment-15974163
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863946/HIVE-11133.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10581 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=28)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4752/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4752/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4752/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863946 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch,
> HIVE-11133.6.patch, HIVE-11133.7.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973818#comment-15973818
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863888/HIVE-11133.6.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10581 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=28)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.ql.parse.TestParseNegativeDriver.testCliDriver[wrong_distinct2]
(batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4744/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4744/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4744/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863888 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch, HIVE-11133.6.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972079#comment-15972079
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863712/HIVE-11133.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10580 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_order_null]
(batchId=27)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explainuser_1]
(batchId=113)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4718/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4718/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4718/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863712 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch, HIVE-11133.5.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966748#comment-15966748
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863139/HIVE-11133.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10572 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4662/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4662/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4662/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863139 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966675#comment-15966675
]
Sahil Takiar commented on HIVE-11133:
-
RB: https://reviews.apache.org/r/58402/
* Added a {{DagJsonParser}} which is an abstract class and has two
implementations: {{TezJsonParser}} and {{SparkJsonParser}}
** Almost all of the logic in {{TezJsonParser}} was moved to {{DagJsonParser}}
* Had to disable the {{CorrelationOptimizer}} for Spark, seems that it doesn't
work - should be ok since the {{CorrelationOptimizer}} is disabled by default
anyways
* Had to add a few explicit checks for JSON objects to avoid some
NullPointerExceptions
* Most of the other changes are just re-factoring
* Turned {{hive.explain.user}} off for all the Spark qtests so that we don't
have to updated all the qtests
* {{hive.explain.user}} is on by default, but we could create a separate config
for Spark and turn it off by default?
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965504#comment-15965504
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862972/HIVE-11133.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10572 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=167)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=222)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4653/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4653/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4653/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862972 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15965295#comment-15965295
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862945/HIVE-11133.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 327 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucket6]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_semijoin]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[gen_udf_example_add10]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[index_bitmap3]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[index_bitmap_auto]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_map_operators]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[infer_bucket_sort_num_buckets]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[parallel_orderby]
(batchId=167)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[add_part_multiple]
(batchId=127)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[annotate_stats_join]
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join10]
(batchId=113)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join11]
(batchId=102)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join12]
(batchId=108)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join13]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join14]
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join15]
(batchId=105)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join16]
(batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join17]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18]
(batchId=103)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join19]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join20]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join22]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join23]
(batchId=106)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join24]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join26]
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join27]
(batchId=136)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join28]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join2]
(batchId=125)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join31]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join3]
(batchId=133)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join4]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join5]
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join6]
(batchId=134)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join7]
(batchId=110)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join8]
(batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join9]
(batchId=131)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_stats2]
(batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_stats]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_join_without_localtask]
(batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[auto_smb_mapjoin_14]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=100)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=109)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_6]
(batchId=101)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketsortoptimize_insert_7]
(batchId=120)
org.apache
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15963794#comment-15963794
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862766/HIVE-11133.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData (batchId=222)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4635/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4635/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4635/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862766 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614035#comment-14614035
]
Mohit Sabharwal commented on HIVE-11133:
Currently only supported for Tez.
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
