Sudhanva Huruli created SPARK-61825:
---------------------------------------

             Summary: Static or core Spark settings in a pipeline's 
configuration fail every flow with CANNOT_MODIFY_STATIC_CONFIG
                 Key: SPARK-61825
                 URL: https://issues.apache.org/jira/browse/SPARK-61825
             Project: Spark
          Issue Type: Bug
          Components: Declarative Pipelines
    Affects Versions: 4.3.0
            Reporter: Sudhanva Huruli


The spark-pipelines CLI sets the spec's configuration on the session builder 
and also sends the same map to the pipeline graph as the default SQL 
configuration, which is applied to every flow. Setting a static SQL 
configuration (for example spark.sql.catalogImplementation) or a core setting 
(for example spark.driver.memory) on a running session throws, so every flow 
fails with CANNOT_MODIFY_STATIC_CONFIG or CANNOT_MODIFY_CONFIG even when the 
value matches the session. The server should skip settings that cannot change 
at run time when it applies per-flow configuration, and log a warning when the 
requested value differs.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to