[ https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276747#comment-16276747 ]
Nandor Kollar commented on PIG-5318: ------------------------------------ Attached PIG-5318_4.patch, it looks like the way I wanted to tell Spark version doesn't work on Spark 1.x, using SparkContext#version instead. > Unit test failures on Pig on Spark with Spark 2.2 > ------------------------------------------------- > > Key: PIG-5318 > URL: https://issues.apache.org/jira/browse/PIG-5318 > Project: Pig > Issue Type: Bug > Components: spark > Reporter: Nandor Kollar > Assignee: Nandor Kollar > Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch, > PIG-5318_4.patch > > > There are several failing cases when executing the unit tests with Spark 2.2: > {code} > org.apache.pig.test.TestAssert#testNegativeWithoutFetch > org.apache.pig.test.TestAssert#testNegative > org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch > org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput > org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore > org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication > org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore > {code} > All of these are related to fixes/changes in Spark. > TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed > by asserting on the message of the exception's root cause, looks like on > Spark 2.2 the exception is wrapped into an additional layer. > TestStore and TestStoreLocal failure are also a test related problems: looks > like SPARK-7953 is fixed in Spark 2.2 > The root cause of TestStoreInstances is yet to be found out. -- This message was sent by Atlassian JIRA (v6.4.14#64029)