Sure, that is expected, see the "How it works" section in "Running Spark on
Kubernetes" page
<https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#how-it-works>,
quote:
When the application completes, the executor pods terminate and are cleaned
> up, but the driver pod persists logs and remains in “completed” state in
> the Kubernetes API until it’s eventually garbage collected or manually
> cleaned up.
On Mon, Mar 15, 2021 at 8:45 AM 040840219 <yxl040840...@126.com> wrote:
>
> when driver pod throws exception , driver pod still running ？
>
> kubectl logs wordcount-e3141c7834d3dd68-driver
>
> 21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to
> 'value1
> Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot
> resolve '`value1`' given input columns: [key, value];
> 'Aggregate [key#6], [key#6, count('value1) AS cnt#14]
> +- Project [(id#4 % 5) AS key#6, (id#4 % 10) AS value#7]
> +- Project [value#1 AS id#4]
> +- LocalRelation [value#1]
>
> at
> org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:155)
> at
> org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$$nestedInanonfun$checkAnalysis$1$2.applyOrElse(CheckAnalysis.scala:152)
> at
> org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformUp$2(TreeNode.scala:341)
> at
> org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:73)
>
> kubectl get pods wordcount-e3141c7834d3dd68-driver
>
> NAME READY STATUS RESTARTS AGE
> wordcount-e3141c7834d3dd68-driver 1/1 Running 0 2m58s
>
> On 03/12/2021 05:42，Attila Zsolt Piros<piros.attila.zs...@gmail.com>
> <piros.attila.zs...@gmail.com> wrote：
>
> > but the spark-submit log still running
>
> Set the "spark.kubernetes.submission.waitAppCompletion" config to false to
> change that. As the doc says:
>
> "spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether
> to wait for the application to finish before exiting the launcher process.
> When changed to false, the launcher has a "fire-and-forget" behavior when
> launching the Spark job.
>
> On Thu, Mar 11, 2021 at 10:05 PM Attila Zsolt Piros <
> piros.attila.zs...@gmail.com> wrote:
>
>>
>> For getting the logs please read Accessing Logs
>> <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#accessing-logs>
>> part
>> of the *Running Spark on Kubernetes* page.
>>
>> For stopping and generic management of the spark application please read
>> the Spark Application Management
>> <https://spark.apache.org/docs/3.1.1/running-on-kubernetes.html#spark-application-management>,
>> where you find the example:
>>
>> $ spark-submit --kill spark:spark-pi* --master
>> k8s://https://192.168.2.8:8443
>>
>>
>>
>> On Thu, Mar 11, 2021 at 1:07 PM yxl040840219 <yxl040840...@126.com>
>> wrote:
>>
>>>
>>>
>>>
>>> when run the code in k8s , driver pod throw AnalysisException , but
>>> the spark-submit log still running , then how to get the exception and
>>> stop pods ?
>>>
>>> val spark = SparkSession.builder().getOrCreate()
>>> import spark.implicits._
>>> val df = (0 until 100000).toDF("id").selectExpr("id % 5 as key",
>>> "id%10 as value")
>>> .groupBy("key").agg(count("value1").as("cnt"))
>>> df.show()
>>> spark.stop()
>>>
>>> bin/spark-submit \
>>> --master k8s://https://localhost:9443 \
>>> --deploy-mode cluster \
>>> --name wordcount \
>>> --class k8s.WordCount \
>>> --conf spark.kubernetes.container.image=rspark:v3.1.1 \
>>> --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
>>> --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
>>> --conf
>>> spark.kubernetes.file.upload.path=hdfs://localhost:8020/data/spark \
>>> /data/spark-example-1.0.0.jar
>>>
>>
