[
https://issues.apache.org/jira/browse/HIVE-18148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319959#comment-16319959
]
Rui Li edited comment on HIVE-18148 at 1/10/18 9:45 AM:
--------------------------------------------------------
Hi [~stakiar],
bq. Is there any way to move the code changes into SplitOpTreeForDPP?
The added code runs during OP tree optimization, which is before
SplitOpTreeForDPP. So we won't generate the malformed tree. I put it there as
another kind of DPP to be removed, together with cyclic DPPs, too big DPPs, etc.
bq. I don't think this is an issue with map-joins correct?
Yeah the issue is not related to map join. And we won't remove nested DPP sink
if it's with map join, because SplitOpTreeForDPP doesn't split the tree in this
case.
was (Author: lirui):
Hi [~stakiar],
bq. Is there any way to move the code changes into SplitOpTreeForDPP?
The added code runs during OP tree optimization, which is before
SplitOpTreeForDPP. So we won't generate the malformed tree.
bq. I don't think this is an issue with map-joins correct?
The issue is not related to map join. And we won't remove nested DPP sink if
it's with map join, because SplitOpTreeForDPP doesn't split the tree in this
case.
> NPE in SparkDynamicPartitionPruningResolver
> -------------------------------------------
>
> Key: HIVE-18148
> URL: https://issues.apache.org/jira/browse/HIVE-18148
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Rui Li
> Assignee: Rui Li
> Attachments: HIVE-18148.1.patch, HIVE-18148.2.patch,
> HIVE-18148.3.patch
>
>
> The stack trace is:
> {noformat}
> 2017-11-27T10:32:38,752 ERROR [e6c8aab5-ddd2-461d-b185-a7597c3e7519 main]
> ql.Driver: FAILED: NullPointerException null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.optimizer.physical.SparkDynamicPartitionPruningResolver$SparkDynamicPartitionPruningDispatcher.dispatch(SparkDynamicPartitionPruningResolver.java:100)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
> at
> org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
> at
> org.apache.hadoop.hive.ql.optimizer.physical.SparkDynamicPartitionPruningResolver.resolve(SparkDynamicPartitionPruningResolver.java:74)
> at
> org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeTaskPlan(SparkCompiler.java:568)
> {noformat}
> At this stage, there shouldn't be a DPP sink whose target map work is null.
> The root cause seems to be a malformed operator tree generated by
> SplitOpTreeForDPP.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
