[
https://issues.apache.org/jira/browse/PIG-4615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606893#comment-14606893
]
Mohit Sabharwal commented on PIG-4615:
--------------------------------------
Thanks, [~kellyzly]! LGTM.
+1 (non-binding)
> Fix null keys join in SkewedJoin in spark mode
> ----------------------------------------------
>
> Key: PIG-4615
> URL: https://issues.apache.org/jira/browse/PIG-4615
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4615.patch
>
>
> Let's use an example to explain the problem:
> testSkewedJoinNullKeys.pig:
> {code}
> A = LOAD './SkewedJoinInput5.txt' as (id,name);
> B = LOAD './SkewedJoinInput5.txt' as (id,name);
> C = join A by id, B by id using 'skewed';
> store C into './testSkewedJoinNullKeys.out';
> {code}
> cat SkewedJoinInput5.txt
> {code}
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> 100 apple2
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> 100
> {code}
> the result of mr:
> {code}
> 100 apple2 100 apple2
> 100 apple2 100
> 100 100 apple2
> 100 100
> {code}
> The result of spark:
> {code}
> cat testSkewedJoinNullKeys.out.spark/part-r-00000
> 100 apple2 100 apple2
> 100 apple2 100
> 100 100 apple2
> 100 100
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 apple1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1 orange1
> apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 apple1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1 orange1
> orange1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> apple1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> orange1
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
