[ https://issues.apache.org/jira/browse/PIG-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947177#comment-15947177 ]
Nandor Kollar commented on PIG-5163: ------------------------------------ I did some more investigation, and it seems that the problem is related to PIG-4675. We maintain the operator keys between spark operators, and in this case, after multiquery optimization and before join group optimization, we have a mapping in SparkOperator[scope-30] multiQueryOptimizeConnectionMap: scope-22 -> scope-8. But after join group optimization the local rearrange in scope-22 is deleted, and replaced with POJoinGroupSpark with same operator key as the global rearrange. I think replacing the mapping in the multiQueryOptimizeConnectionMap would fix the problem (see attached patch). However, I'm not sure why do we need this map? > MultiQuery_Streaming_1 is failing with spark exec type > ------------------------------------------------------ > > Key: PIG-5163 > URL: https://issues.apache.org/jira/browse/PIG-5163 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: Nandor Kollar > Assignee: liyunzhang_intel > Fix For: spark-branch > > Attachments: PIG-5163_1.patch > > > 2nd output was empty, looks like pig on spark didn't generate any data. -- This message was sent by Atlassian JIRA (v6.3.15#6346)