[
https://issues.apache.org/jira/browse/PIG-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947177#comment-15947177
]
Nandor Kollar commented on PIG-5163:
------------------------------------
I did some more investigation, and it seems that the problem is related to
PIG-4675. We maintain the operator keys between spark operators, and in this
case, after multiquery optimization and before join group optimization, we have
a mapping in SparkOperator[scope-30] multiQueryOptimizeConnectionMap: scope-22
-> scope-8. But after join group optimization the local rearrange in scope-22
is deleted, and replaced with POJoinGroupSpark with same operator key as the
global rearrange. I think replacing the mapping in the
multiQueryOptimizeConnectionMap would fix the problem (see attached patch).
However, I'm not sure why do we need this map?
> MultiQuery_Streaming_1 is failing with spark exec type
> ------------------------------------------------------
>
> Key: PIG-5163
> URL: https://issues.apache.org/jira/browse/PIG-5163
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Nandor Kollar
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-5163_1.patch
>
>
> 2nd output was empty, looks like pig on spark didn't generate any data.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
