Sudhanva Huruli created SPARK-62027:
---------------------------------------
Summary: spark-submit in Spark Connect mode drops settings beyond
the first group of ten, such as spark.jars
Key: SPARK-62027
URL: https://issues.apache.org/jira/browse/SPARK-62027
Project: Spark
Issue Type: Bug
Components: Declarative Pipelines
Affects Versions: 4.3.0
Reporter: Sudhanva Huruli
When spark-submit runs a Python application in Spark Connect mode, the launcher
passes its settings to Python through environment variables, split into groups
of ten. The Python side reads every group but keeps only the last one, so
settings in the other groups, such as spark.jars or --conf values, are missing
from the session and the Spark Connect server. Reproduction: run a Python
script with spark-submit in Spark Connect mode, with --jars pointing at a local
jar file and about a dozen --conf flags, and have the script print spark.jars.
Expected: every setting passed on the command line is visible to the session.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]