[
https://issues.apache.org/jira/browse/PIG-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520963#comment-14520963
]
Mohit Sabharwal commented on PIG-4421:
--------------------------------------
Thanks, [~kellyzly]. Could you please create a review board for this?
Could you rebase this patch so that it compatible with the "single spark
operator" change in PIG-4518 ?
I know Skew Join only allows 2 tables. But would {{joinPlans.keySet()}} always
be of size 2 (and correspond to the 2 POForeach operators in the physical plan)
? If you could add some comments about this, it'd be great.
{code}
-----------------------------------------------
# Physical Plan:
#-----------------------------------------------
X: Store
|
|---X: SkewedJoin[tuple] - scope-21
| |
|
|---A: New For Each(false,false,false)[bag] - scope-10
| | |
| |---A:
Load(/Users/mohit/pig/left.txt:org.apache.pig.builtin.PigStorage) - scope-0
|
|---B: New For Each(false,false)[bag] - scope-18
| |
|
|---B: Load(/Users/mohit/pig/right.txt
{code}
> implement visitSkewedJoin in SparkCompiler
> ------------------------------------------
>
> Key: PIG-4421
> URL: https://issues.apache.org/jira/browse/PIG-4421
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4421.patch, PIG-4421_2.patch
>
>
> If visitSkewedJoin is not implemented, following unittests will fail.
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinWithGroup
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinMapKey
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinManyReducers
> org.apache.pig.test.TestSkewedJoin.testNonExistingInputPathInSkewJoin
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinOneValue
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinWithNoProperties
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinEmptyInput
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinNullKeys
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinOuter
> org.apache.pig.test.TestSkewedJoin.testRecursiveFileListing
> org.apache.pig.test.TestSkewedJoin.testSkewedJoinReducers
> org.apache.pig.test.TestJoinSmoke.testSkewedJoinWithGroup
> org.apache.pig.test.TestJoinSmoke.testSkewedJoinOuter
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
