[ https://issues.apache.org/jira/browse/SPARK-22708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16281232#comment-16281232 ]
jinchen commented on SPARK-22708: --------------------------------- This happened when I do : {code:java} df.createOrReplaceTempView {code} How to catch this error and set yarn exit code? > spark on yarn error but Final app status: SUCCEEDED, exitCode: 0 > ---------------------------------------------------------------- > > Key: SPARK-22708 > URL: https://issues.apache.org/jira/browse/SPARK-22708 > Project: Spark > Issue Type: Question > Components: YARN > Affects Versions: 2.2.0 > Reporter: jinchen > > I got log : > {code:java} > 17/12/06 18:14:59 INFO state.StateStoreCoordinatorRef: Registered > StateStoreCoordinator endpoint > 17/12/06 18:15:01 INFO util.Version: Elasticsearch Hadoop v6.0.0 [8b59a8f82d] > 17/12/06 18:15:02 INFO httpclient.HttpMethodDirector: I/O exception > (java.net.ConnectException) caught when processing request: Connection > refused (Connection refused) > 17/12/06 18:15:02 INFO httpclient.HttpMethodDirector: Retrying request > 17/12/06 18:15:02 INFO httpclient.HttpMethodDirector: I/O exception > (java.net.ConnectException) caught when processing request: Connection > refused (Connection refused) > 17/12/06 18:15:02 INFO httpclient.HttpMethodDirector: Retrying request > 17/12/06 18:15:02 INFO httpclient.HttpMethodDirector: I/O exception > (java.net.ConnectException) caught when processing request: Connection > refused (Connection refused) > 17/12/06 18:15:02 INFO httpclient.HttpMethodDirector: Retrying request > 17/12/06 18:15:02 ERROR rest.NetworkClient: Node [192.168.200.154:9200] > failed (Connection refused (Connection refused)); no other nodes left - > aborting... > 17/12/06 18:15:02 ERROR dispatcher.StrategyDispatcher: 调用链路异常 > org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES > version - typically this happens if the network/Elasticsearch cluster is not > accessible or when targeting a WAN/Cloud instance without the proper setting > 'es.nodes.wan.only' > at > org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:327) > at > org.elasticsearch.spark.sql.SchemaUtils$.discoverMappingAndGeoFields(SchemaUtils.scala:98) > at > org.elasticsearch.spark.sql.SchemaUtils$.discoverMapping(SchemaUtils.scala:91) > at > org.elasticsearch.spark.sql.ElasticsearchRelation.lazySchema$lzycompute(DefaultSource.scala:196) > at > org.elasticsearch.spark.sql.ElasticsearchRelation.lazySchema(DefaultSource.scala:196) > at > org.elasticsearch.spark.sql.ElasticsearchRelation$$anonfun$schema$1.apply(DefaultSource.scala:200) > at > org.elasticsearch.spark.sql.ElasticsearchRelation$$anonfun$schema$1.apply(DefaultSource.scala:200) > at scala.Option.getOrElse(Option.scala:121) > at > org.elasticsearch.spark.sql.ElasticsearchRelation.schema(DefaultSource.scala:200) > at > org.apache.spark.sql.execution.datasources.LogicalRelation$.apply(LogicalRelation.scala:77) > at > org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:415) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:172) > at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:156) > at > streaming.core.compositor.spark.source.MultiSQLSourceCompositor$$anonfun$result$1.apply(MultiSQLSourceCompositor.scala:37) > at > streaming.core.compositor.spark.source.MultiSQLSourceCompositor$$anonfun$result$1.apply(MultiSQLSourceCompositor.scala:27) > at scala.collection.Iterator$class.foreach(Iterator.scala:893) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at > streaming.core.compositor.spark.source.MultiSQLSourceCompositor.result(MultiSQLSourceCompositor.scala:27) > at > streaming.core.strategy.SparkStreamingStrategy.result(SparkStreamingStrategy.scala:52) > at > serviceframework.dispatcher.StrategyDispatcher$$anonfun$dispatch$2.apply(StrategyDispatcher.scala:65) > at > serviceframework.dispatcher.StrategyDispatcher$$anonfun$dispatch$2.apply(StrategyDispatcher.scala:63) > at scala.collection.immutable.List.flatMap(List.scala:327) > at > serviceframework.dispatcher.StrategyDispatcher.dispatch(StrategyDispatcher.scala:62) > at > streaming.core.strategy.platform.PlatformManager$$anonfun$run$3.apply(PlatformManager.scala:120) > at > streaming.core.strategy.platform.PlatformManager$$anonfun$run$3.apply(PlatformManager.scala:118) > at > scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) > at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) > at > streaming.core.strategy.platform.PlatformManager.run(PlatformManager.scala:117) > at streaming.core.StreamingApp$.main(StreamingApp.scala:14) > at streaming.core.StreamingApp.main(StreamingApp.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635) > Caused by: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: > Connection error (check network and/or proxy settings)- all nodes failed; > tried [[192.168.200.154:9200]] > at > org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:149) > at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:466) > at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:430) > at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:434) > at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:155) > at > org.elasticsearch.hadoop.rest.RestClient.remoteEsVersion(RestClient.java:660) > at > org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:320) > ... 36 more > 17/12/06 18:15:02 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, > exitCode: 0 > 17/12/06 18:15:02 INFO spark.SparkContext: Invoking stop() from shutdown hook > 17/12/06 18:15:02 INFO ui.SparkUI: Stopped Spark web UI at > http://192.168.13.150:45073 > 17/12/06 18:15:02 INFO yarn.YarnAllocator: Driver requested a total number of > 0 executor(s). > 17/12/06 18:15:02 INFO cluster.YarnClusterSchedulerBackend: Shutting down all > executors > 17/12/06 18:15:02 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: > Asking each executor to shut down > 17/12/06 18:15:02 INFO cluster.SchedulerExtensionServices: Stopping > SchedulerExtensionServices > (serviceOption=None, > services=List(), > started=false) > 17/12/06 18:15:02 INFO spark.MapOutputTrackerMasterEndpoint: > MapOutputTrackerMasterEndpoint stopped! > 17/12/06 18:15:02 INFO memory.MemoryStore: MemoryStore cleared > 17/12/06 18:15:02 INFO storage.BlockManager: BlockManager stopped > 17/12/06 18:15:02 INFO storage.BlockManagerMaster: BlockManagerMaster stopped > 17/12/06 18:15:02 INFO > scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: > OutputCommitCoordinator stopped! > 17/12/06 18:15:02 INFO spark.SparkContext: Successfully stopped SparkContext > 17/12/06 18:15:02 INFO yarn.ApplicationMaster: Unregistering > ApplicationMaster with SUCCEEDED > 17/12/06 18:15:02 INFO impl.AMRMClientImpl: Waiting for application to be > successfully unregistered. > 17/12/06 18:15:02 INFO yarn.ApplicationMaster: Deleting staging directory > hdfs://nameservice1/user/spark2/.sparkStaging/application_1511515104748_13554 > 17/12/06 18:15:02 INFO util.ShutdownHookManager: Shutdown hook called > {code} > But > {code:java} > 17/12/06 10:45:15 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, > exitCode: 0 > {code} > spark version 2.2.0 yarn version 2.6 -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org