[ https://issues.apache.org/jira/browse/PIG-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527970#comment-14527970 ]
Mohit Sabharwal commented on PIG-4421: -------------------------------------- Thanks [~kellyzly]. LGTM. Could you confirm if all skew join related failures are fixed ? - Is TestSkewedJoin.testSkewedJoinKeyPartition failure also resolved with this patch ? - Could you also check if TestJoinSmoke.testSkewedJoinWithGroup is resolved with this patch ? - Also, using {{ant spark-test-local}} TestPigServerLocal#testNonExistingSecondDirectoryInSkewJoin We should fix all Skewed Join test failures ( or disable them if we are not going to fix them ) +1 if failures are all fixed. > implement visitSkewedJoin in SparkCompiler > ------------------------------------------ > > Key: PIG-4421 > URL: https://issues.apache.org/jira/browse/PIG-4421 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: liyunzhang_intel > Assignee: liyunzhang_intel > Fix For: spark-branch > > Attachments: PIG-4421.patch, PIG-4421_2.patch, PIG-4421_3.patch, > PIG-4421_4.patch, PIG-4421_5.patch > > > If visitSkewedJoin is not implemented, following unittests will fail. > org.apache.pig.test.TestSkewedJoin.testSkewedJoinWithGroup > org.apache.pig.test.TestSkewedJoin.testSkewedJoinMapKey > org.apache.pig.test.TestSkewedJoin.testSkewedJoinManyReducers > org.apache.pig.test.TestSkewedJoin.testNonExistingInputPathInSkewJoin > org.apache.pig.test.TestSkewedJoin.testSkewedJoinOneValue > org.apache.pig.test.TestSkewedJoin.testSkewedJoinWithNoProperties > org.apache.pig.test.TestSkewedJoin.testSkewedJoinEmptyInput > org.apache.pig.test.TestSkewedJoin.testSkewedJoinNullKeys > org.apache.pig.test.TestSkewedJoin.testSkewedJoinOuter > org.apache.pig.test.TestSkewedJoin.testRecursiveFileListing > org.apache.pig.test.TestSkewedJoin.testSkewedJoinReducers > org.apache.pig.test.TestJoinSmoke.testSkewedJoinWithGroup > org.apache.pig.test.TestJoinSmoke.testSkewedJoinOuter -- This message was sent by Atlassian JIRA (v6.3.4#6332)