Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/19840 I think in YARN we have several different ways to set `PYSPARK_PYTHON`, I guess your issue is that which one should take priority? Can you please: 1. Define a consistent ordering for such envs, which one should take priority (spark.yarn.appMasterEnv.XXX or XXX), and document it. 2. Check if it works as expected for `spark.yarn.appMasterEnv.PYSPARK_PYTHON` and `spark.executorEnv.PYSPARK_PYTHON`. I guess for other envs, it will also suffer from this problem, am I right?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org