dongjoon-hyun commented on code in PR #58343: URL: https://github.com/apache/spark/pull/58343#discussion_r4017587690
########## docs/core-migration-guide.md: ########## @@ -42,6 +42,8 @@ license: | - Since Spark 4.3, a positive `spark.executor.pyspark.memory` allocation that is too small to give each concurrent task slot at least 1 MiB fails the Python task with an error instead of silently running the workers without any memory limit. Setting `spark.executor.pyspark.memory=0` still disables the limit. To restore a working memory limit, increase `spark.executor.pyspark.memory` or reduce the executor's concurrent task capacity. +- Since Spark 4.3, an executor pod template that names a service account in `serviceAccountName` keeps it: Spark applies `spark.kubernetes.authenticate.executor.serviceAccountName`, or the driver's account as a fallback, only when the template names no account in either `serviceAccount` or `serviceAccountName`. Earlier versions decided by reading the deprecated `serviceAccount` field alone, so a template that named the account in `serviceAccountName` had it overwritten. Spark logs a warning when `spark.kubernetes.authenticate.executor.serviceAccountName` named an account the template displaced. Review Comment: Maybe, Spark 4.4? It's too late for Spark 4.3 because it's almost RC2 status. WDYT, @LuciferYang ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
