liyunzhang_intel created PIG-4615:
-------------------------------------
Summary: Fix null keys join in SkewedJoin in spark mode
Key: PIG-4615
URL: https://issues.apache.org/jira/browse/PIG-4615
Project: Pig
Issue Type: Sub-task
Reporter: liyunzhang_intel
Assignee: liyunzhang_intel
Let's use an example to explain the problem:
testSkewedJoinNullKeys.pig:
{code}
A = LOAD './SkewedJoinInput5.txt' as (id,name);
B = LOAD './SkewedJoinInput5.txt' as (id,name);
C = join A by id, B by id using 'skewed';
store C into './testSkewedJoinNullKeys.out';
{code}
cat SkewedJoinInput5.txt
{code}
apple1
apple1
apple1
apple1
apple1
apple1
apple1
apple1
apple1
apple1
100 apple2
orange1
orange1
orange1
orange1
orange1
orange1
orange1
orange1
orange1
orange1
100
{code}
the result of mr:
{code}
100 apple2 100 apple2
100 apple2 100
100 100 apple2
100 100
{code}
The result of spark:
{code}
cat testSkewedJoinNullKeys.out.spark/part-r-00000
100 apple2 100 apple2
100 apple2 100
100 100 apple2
100 100
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 apple1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1 orange1
apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 apple1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1 orange1
orange1
apple1
apple1
apple1
apple1
apple1
apple1
apple1
apple1
apple1
apple1
orange1
orange1
orange1
orange1
orange1
orange1
orange1
orange1
orange1
orange1
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
