[
https://issues.apache.org/jira/browse/PIG-5163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15948015#comment-15948015
]
Nandor Kollar edited comment on PIG-5163 at 3/29/17 10:42 PM:
--------------------------------------------------------------
[~kellyzly] thanks you for the review! MapReduceLauncher was modified, because
I think instead of hardcoded strings it is better to use existing constants,
but reverting that change is fine, it wasn't related to this fix.
I've one more question regarding multiquery optimization: do we need
multiQueryOptimizeConnectionMap? The plan after the optimization looked
strange, I don't understand why do we connect the two spark operators via a map
instead of replacing scope-34 (loading of temp file) with scope-8 (POStream) in
the optimized plan. My patch fixes the e2e test, and if it doesn't break any
other test case (I'll execute the entire e2e test suite to make sure it
doesn't) then it is fine, though we might have to reconsider the need of
multiQueryOptimizeConnectionMap later.
was (Author: nkollar):
[~kellyzly] thank you for the review! MapReduceLauncher was modified, because I
think instead of hardcoded strings it is better to use existing constants, but
reverting that change is fine, it wasn't related to this fix.
I've one more question regarding multiquery optimization: do we need
multiQueryOptimizeConnectionMap? The plan after the optimization looked
strange, I don't understand why do we connect the two spark operators via a map
instead of replacing scope-34 (loading of temp file) with scope-8 (POStream) in
the optimized plan. My patch fixes the e2e test, and if it doesn't break any
other test case (I'll execute the entire e2e test suite to make sure it
doesn't) then it is fine, though we might have to reconsider the need of
multiQueryOptimizeConnectionMap later.
> MultiQuery_Streaming_1 is failing with spark exec type
> ------------------------------------------------------
>
> Key: PIG-5163
> URL: https://issues.apache.org/jira/browse/PIG-5163
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Fix For: spark-branch
>
> Attachments: PIG-5163_1.patch
>
>
> 2nd output was empty, looks like pig on spark didn't generate any data.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
