when driver pod throws exception , driver pod still running ?
kubectl logs wordcount-e3141c7834d3dd68-driver 21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to 'value1 Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`value1`' given input columns: [key, value]; 'Aggregate [key#6], [key#6, count('value1) AS cnt#14] +- Project [(id#4 % 5) AS key#6, (id#4 % 10) AS value#7] +- Project [value#1 AS id#4] +- LocalRelation [value#1] at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155) at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152) at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341) at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73) kubectl get pods wordcount-e3141c7834d3dd68-driver NAME READY STATUS RESTARTS AGE wordcount-e3141c7834d3dd68-driver 1/1 Running 0 2m58s On 03/12/2021 05:42,Attila Zsolt Piros<piros.attila.zs...@gmail.com> wrote: > but the spark-submit log still running Set the "spark.kubernetes.submission.waitAppCompletion" config to false to change that. As the doc says: "spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether to wait for the application to finish before exiting the launcher process. When changed to false, the launcher has a "fire-and-forget" behavior when launching the Spark job. On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros <piros.attila.zs...@gmail.com> wrote: For getting the logs please read Accessing Logs part of the Running Spark on Kubernetes page. For stopping and generic management of the spark application please read the Spark Application Management, where you find the example: $ spark-submit --kill spark:spark-pi*--master k8s://https://192.168.2.8:8443 On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <yxl040840...@126.com> wrote: when run the code in k8s , driver pod throw AnalysisException , but the spark-submit log still running , then how to get the exception and stop pods ? val spark = SparkSession.builder().getOrCreate() import spark.implicits._ val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", "id%10 as value") .groupBy("key").agg(count("value1").as("cnt")) df.show() spark.stop() bin/spark-submit \ --master k8s://https://localhost:9443 \ --deploy-mode cluster \ --name wordcount \ --class k8s.WordCount \ --conf spark.kubernetes.container.image=rspark:v3.1.1 \ --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \ /data/spark-example-1.0.0.jar