[
https://issues.apache.org/jira/browse/PIG-4613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14611425#comment-14611425
]
liyunzhang_intel commented on PIG-4613:
---------------------------------------
[~kexianda] and [~mohitsabharwal]:
the root cause of TestAssert.testNegativeWithoutFetch and
TestAssert.testNegative is the difference of
org.apache.pig.tools.pigstats.JobStats#exception is null in mr mode and
org.apache.pig.tools.pigstats.JobStats#exception is not null in spark mode.
in mr, it sets org.apache.pig.tools.pigstats.JobStats#setBackendException in
https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java#L460.
It seems that backendException is set in the [catch
case|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java#L454]
when [{{MRJobID ==
null}}|https://github.com/apache/pig/blob/trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/MapReduceLauncher.java#L784].
in spark mode, it sets
org.apache.pig.tools.pigstats.JobStats#setBackendException in
https://github.com/apache/pig/blob/spark/src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java#L83.
Now there are two ways to fix this bug:
1. remove
[SparkPigStats.java#L82|https://github.com/apache/pig/blob/spark/src/org/apache/pig/tools/pigstats/spark/SparkPigStats.java#L82]-L84:
this way will make the exception info of TestAssert#testNegative in spark same
with mr and tez but not set
org.apache.pig.tools.pigstats.JobStats#setBackendException even when the spark
job fails.
2.modify the code of TestAssert#testNegative like following:
{code}
public void testNegative() throws Exception {
PigServer pigServer = new PigServer(Util.getLocalTestMode());
Data data = resetData(pigServer);
data.set("foo",
tuple(1),
tuple(2),
tuple(3)
);
pigServer.registerQuery("A = LOAD 'foo' USING mock.Storage() AS
(i:int);");
pigServer.registerQuery("ASSERT A BY i > 1 , 'i should be greater than
1';");
try {
pigServer.openIterator("A");
} catch (FrontendException fe) {
if( !Util.isSparkExecType(Util.getLocalTestMode())) {
Assert.assertTrue(fe.getCause().getMessage().contains(
"Job terminated with anomalous status FAILED"));
} else{
Assert.assertTrue(fe.getCause().getMessage().contains(
"i should be greater than 1"));
}
}
}
{code}
this way will set org.apache.pig.tools.pigstats.JobStats#setBackendException
when the spark job fails but the exception info of TestAssert#testNegative in
spark mode is not same with mr and tez.
Which one is better?
> Fix unit test failures about TestAssert
> ---------------------------------------
>
> Key: PIG-4613
> URL: https://issues.apache.org/jira/browse/PIG-4613
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: kexianda
> Assignee: kexianda
> Fix For: spark-branch
>
> Attachments: PIG-4613.patch
>
>
> UT failed at following cases:
> org.apache.pig.test.TestAssert.testNegativeWithoutFetch
> org.apache.pig.test.TestAssert.testNegative
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
