Sam Brougher created LIVY-586: --------------------------------- Summary: When a batch fails on startup, Livy continues to report the batch as "starting", even though it has failed Key: LIVY-586 URL: https://issues.apache.org/jira/browse/LIVY-586 Project: Livy Issue Type: Bug Components: Batch Affects Versions: 0.5.0 Environment: AWS EMR, Livy submits batches to YARN in cluster mode Reporter: Sam Brougher
When starting a Livy batch, I accidentally provided it a jar location in S3 that did not exist. Livy then continued to report that the job was "starting", even though it had clearly failed. stdout: {code:java} 2019-04-05 11:24:18,149 [main] WARN org.apache.hadoop.util.NativeCodeLoader [appName=] [jobId=] [clusterId=] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Warning: Skip remote jar s3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar. 2019-04-05 11:24:19,152 [main] INFO org.apache.hadoop.yarn.client.RMProxy [appName=] [jobId=] [clusterId=] - Connecting to ResourceManager at ip-10-25-30-127.dev.cainc.internal/10.25.30.127:8032 2019-04-05 11:24:19,453 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Requesting a new application from cluster with 6 NodeManagers 2019-04-05 11:24:19,532 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Verifying our application has not requested more than the maximum memory capability of the cluster (54272 MB per container) 2019-04-05 11:24:19,533 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Will allocate AM container, with 9011 MB memory including 819 MB overhead 2019-04-05 11:24:19,534 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Setting up container launch context for our AM 2019-04-05 11:24:19,537 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Setting up the launch environment for our AM container 2019-04-05 11:24:19,549 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Preparing resources for our AM container 2019-04-05 11:24:21,059 [main] WARN org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 2019-04-05 11:24:23,790 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Uploading resource file:/mnt/tmp/spark-b4e4a760-77a3-4554-a3f3-c3f82675d865/__spark_libs__3639879082942366045.zip -> hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222/__spark_libs__3639879082942366045.zip 2019-04-05 11:24:26,817 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Uploading resource s3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar -> hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar 2019-04-05 11:24:26,940 [main] INFO org.apache.spark.deploy.yarn.Client [appName=] [jobId=] [clusterId=] - Deleted staging directory hdfs://ip-10-25-30-127.dev.cainc.internal:8020/user/livy/.sparkStaging/application_1554234858331_0222 Exception in thread "main" java.io.FileNotFoundException: No such file or directory 's3://dev-dp-local/jars/develop-fix/ap5-app-transform-0.2-thread-pool-SNAPSHOT.jar' at com.amazon.ws.emr.hadoop.fs.s3n.S3NativeFileSystem.getFileStatus(S3NativeFileSystem.java:805) at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.getFileStatus(EmrFileSystem.java:536) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:340) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:292) at org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:356) at org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10.apply(Client.scala:577) at org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$10.apply(Client.scala:576) at scala.Option.foreach(Option.scala:257) at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:576) at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:869) at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:169) at org.apache.spark.deploy.yarn.Client.run(Client.scala:1152) at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1520) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:894) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 2019-04-05 11:24:26,964 [pool-1-thread-1] INFO org.apache.spark.util.ShutdownHookManager [appName=] [jobId=] [clusterId=] - Shutdown hook called 2019-04-05 11:24:26,965 [pool-1-thread-1] INFO org.apache.spark.util.ShutdownHookManager [appName=] [jobId=] [clusterId=] - Deleting directory /mnt/tmp/spark-aa8e8eff-ca2c-4358-a24f-19eb3863ef8f 2019-04-05 11:24:26,966 [pool-1-thread-1] INFO org.apache.spark.util.ShutdownHookManager [appName=] [jobId=] [clusterId=] - Deleting directory /mnt/tmp/spark-b4e4a760-77a3-4554-a3f3-c3f82675d865 {code} stderror is empty YARN Diagnostics eventually warns that the tag for the batch can't be found after 900 seconds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)