Sure, that is expected, see the "How it works" section in "Running Spark on Kubernetes" page <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#how-it-works>, quote:
When the application completes, the executor pods terminate and are cleaned > up, but the driver pod persists logs and remains in “completed” state in > the Kubernetes API until it’s eventually garbage collected or manually > cleaned up. On Mon, Mar 15, 2021 at 8:45 AM 040840219 <yxl040840...@126.com> wrote: > > when driver pod throws exception , driver pod still running ? > > kubectl logs wordcount-e3141c7834d3dd68-driver > > 21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to > 'value1 > Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot > resolve '`value1`' given input columns: [key, value]; > 'Aggregate [key#6], [key#6, count('value1) AS cnt#14] > +- Project [(id#4 % 5) AS key#6, (id#4 % 10) AS value#7] > +- Project [value#1 AS id#4] > +- LocalRelation [value#1] > > at > org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155) > at > org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152) > at > org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341) > at > org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73) > > kubectl get pods wordcount-e3141c7834d3dd68-driver > > NAME READY STATUS RESTARTS AGE > wordcount-e3141c7834d3dd68-driver 1/1 Running 0 2m58s > > On 03/12/2021 05:42,Attila Zsolt Piros<piros.attila.zs...@gmail.com> > <piros.attila.zs...@gmail.com> wrote: > > > but the spark-submit log still running > > Set the "spark.kubernetes.submission.waitAppCompletion" config to false to > change that. As the doc says: > > "spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether > to wait for the application to finish before exiting the launcher process. > When changed to false, the launcher has a "fire-and-forget" behavior when > launching the Spark job. > > On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros < > piros.attila.zs...@gmail.com> wrote: > >> >> For getting the logs please read Accessing Logs >> <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#accessing-logs> >> part >> of the *Running Spark on Kubernetes* page. >> >> For stopping and generic management of the spark application please read >> the Spark Application Management >> <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#spark-application-management>, >> where you find the example: >> >> $ spark-submit --kill spark:spark-pi* --master >> k8s://https://192.168.2.8:8443 >> >> >> >> On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <yxl040840...@126.com> >> wrote: >> >>> >>> >>> >>> when run the code in k8s , driver pod throw AnalysisException , but >>> the spark-submit log still running , then how to get the exception and >>> stop pods ? >>> >>> val spark = SparkSession.builder().getOrCreate() >>> import spark.implicits._ >>> val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key", >>> "id%10 as value") >>> .groupBy("key").agg(count("value1").as("cnt")) >>> df.show() >>> spark.stop() >>> >>> bin/spark-submit \ >>> --master k8s://https://localhost:9443 \ >>> --deploy-mode cluster \ >>> --name wordcount \ >>> --class k8s.WordCount \ >>> --conf spark.kubernetes.container.image=rspark:v3.1.1 \ >>> --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \ >>> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ >>> --conf >>> spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \ >>> /data/spark-example-1.0.0.jar >>> >>