Tian Gao created SPARK-54701:
--------------------------------
Summary: Make `runner_conf` a method for python runners
Key: SPARK-54701
URL: https://issues.apache.org/jira/browse/SPARK-54701
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 4.2.0
Reporter: Tian Gao
In SPARK-54615, we made some changes to always pass runner_conf to worker side.
However, the framework is not super flexible. For example, if a certain python
worker (arrow worker maybe) needs to pass some extra runner configs, it needs
to restructure the whole code.
The main reason is that we do not inherit runnerConf well. We should combine
all the parent configs, instead of overwriting them.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
