Madhukar525722 commented on PR #6700:
URL: https://github.com/apache/kyuubi/pull/6700#issuecomment-2356788324
Hi @pan3793, As per my understanding I was expecting the same behaviour, cm
to be deleted by spark.
But what I am observing that, driver configs are getting deleted, not
executor configs.
While launching the kyuubi-server I have defined,
spark.submit.deployMode=cluster
This is the engine launch submit command:
/opt/spark/bin/spark-submit \
--class org.apache.kyuubi.engine.spark.SparkSQLEngine \
--conf spark.hive.server2.thrift.resultset.default.fetch.size=1000 \
--conf spark.kyuubi.client.ipAddress=10.4.180.32 \
--conf
spark.kyuubi.engine.engineLog.path=/opt/kyuubi/work/madlnu/kyuubi-spark-sql-engine.log.3
\
--conf spark.kyuubi.engine.submit.time=1726597018098 \
--conf spark.kyuubi.ha.addresses=10.4.132.79:2181 \
--conf
spark.kyuubi.ha.engine.ref.id=1218c38d-f742-402e-84bf-161b89552eb6 \
--conf
spark.kyuubi.ha.namespace=/kyuubi_1.9.1-SNAPSHOT_USER_SPARK_SQL/madlnu/default \
--conf spark.kyuubi.ha.zookeeper.auth.type=NONE \
--conf spark.kyuubi.kubernetes.master.address=$MASTER \
--conf spark.kyuubi.kubernetes.namespace=scaas \
--conf spark.kyuubi.server.ipAddress=0.0.0.0 \
--conf spark.kyuubi.session.connection.url=0.0.0.0:10009 \
--conf spark.kyuubi.session.engine.initialize.timeout=PT10M \
--conf spark.kyuubi.session.real.user=madlnu \
--conf spark.kyuubi.zookeeper.embedded.client.port=2181 \
--conf
spark.app.name=kyuubi_USER_SPARK_SQL_madlnu_default_1218c38d-f742-402e-84bf-161b89552eb6
\
--conf spark.driver.extraJavaOptions=-Divy.home=/tmp \
--conf spark.driver.port=7078 \
--conf spark.eventLog.enabled=true \
--conf spark.hadoop.scaas.skipDeleteOnTerminationValidation=true \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.authenticate.serviceAccountName=spark \
--conf spark.kubernetes.container.image=$IMAGE \
--conf
spark.kubernetes.driver.label.kyuubi-unique-tag=1218c38d-f742-402e-84bf-161b89552eb6
\
--conf
spark.kubernetes.driver.pod.name=kyuubi-user-spark-sql-madlnu-default-1218c38d-f742-402e-84bf-161b89552eb6-driver
\
--conf spark.kubernetes.executor.deleteOnTermination=false \
--conf
spark.kubernetes.executor.podNamePrefix=kyuubi-user-spark-sql-madlnu-default-1218c38d-f742-402e-84bf-161b89552eb6
\
--conf spark.kubernetes.namespace=scaas \
--conf spark.rpc.askTimeout=300 \
--conf spark.security.credentials.hbase.enabled=false \
--conf spark.submit.deployMode=cluster \
--conf spark.kubernetes.driverEnv.SPARK_USER_NAME=madlnu \
--conf spark.executorEnv.SPARK_USER_NAME=madlnu \
--proxy-user madlnu
/opt/kyuubi/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.9.1-SNAPSHOT.jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
