[
https://issues.apache.org/jira/browse/PIG-5416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17428481#comment-17428481
]
Koji Noguchi commented on PIG-5416:
-----------------------------------
Not sure what's happening since the code is going through the below method
before hitting the error...
{code:title=SparkStatsUtil.java}
45 public static void waitForJobAddStats(int jobID,
46 POStore poStore, SparkOperator
sparkOperator,
47 JobStatisticCollector
jobStatisticCollector,
48 JavaSparkContext sparkContext,
49 SparkPigStats sparkPigStats)
50 throws InterruptedException {
51 // Even though we are not making any async calls to spark,
52 // the SparkStatusTracker can still return RUNNING status
53 // for a finished job.
54 // Looks like there is a race condition between spark
55 // "event bus" thread updating it's internal listener and
56 // this driver thread calling SparkStatusTracker.
57 // To workaround this, we will wait for this job to "finish".
58 jobStatisticCollector.waitForJobToEnd(jobID);
********************************************************
59 sparkPigStats.addJobStats(poStore, sparkOperator, jobID,
jobStatisticCollector,
60 sparkContext);
{code}
> Spark unit tests failing randomly with "java.lang.RuntimeException:
> Unexpected job execution status RUNNING"
> ------------------------------------------------------------------------------------------------------------
>
> Key: PIG-5416
> URL: https://issues.apache.org/jira/browse/PIG-5416
> Project: Pig
> Issue Type: Bug
> Components: spark
> Reporter: Koji Noguchi
> Priority: Minor
>
> Spark unit tests fail randomly with same errors.
> Sample stack trace showing "Caused by: java.lang.RuntimeException:
> Unexpected job execution status RUNNING".
> {noformat:title=TestBuiltInBagToTupleOrString.testPigScriptForBagToTupleUDF}
> Unable to store alias B
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1002: Unable to
> store alias B
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1783)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:708)
> at org.apache.pig.PigServer.registerQuery(PigServer.java:721)
> at
> org.apache.pig.test.TestBuiltInBagToTupleOrString.testPigScriptForBagToTupleUDF(TestBuiltInBagToTupleOrString.java:429)
> Caused by: org.apache.pig.impl.plan.VisitorException: ERROR 0: fail to get
> the rdds of this spark operator:
> at
> org.apache.pig.backend.hadoop.executionengine.spark.JobGraphBuilder.visitSparkOp(JobGraphBuilder.java:115)
> at
> org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkOperator.visit(SparkOperator.java:140)
> at
> org.apache.pig.backend.hadoop.executionengine.spark.plan.SparkOperator.visit(SparkOperator.java:37)
> at
> org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:87)
> at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:46)
> at
> org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:240)
> at
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:290)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1479)
> at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1464)
> at org.apache.pig.PigServer.execute(PigServer.java:1453)
> at org.apache.pig.PigServer.access$500(PigServer.java:119)
> at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1778)
> Caused by: java.lang.RuntimeException: Unexpected job execution status RUNNING
> at
> org.apache.pig.tools.pigstats.spark.SparkStatsUtil.isJobSuccess(SparkStatsUtil.java:138)
> at
> org.apache.pig.tools.pigstats.spark.SparkPigStats.addJobStats(SparkPigStats.java:75)
> at
> org.apache.pig.tools.pigstats.spark.SparkStatsUtil.waitForJobAddStats(SparkStatsUtil.java:59)
> at
> org.apache.pig.backend.hadoop.executionengine.spark.JobGraphBuilder.sparkOperToRDD(JobGraphBuilder.java:225)
> at
> org.apache.pig.backend.hadoop.executionengine.spark.JobGraphBuilder.visitSparkOp(JobGraphBuilder.java:112)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
