[
https://issues.apache.org/jira/browse/SPARK-52856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18038207#comment-18038207
]
Dongjoon Hyun commented on SPARK-52856:
---------------------------------------
Hi, [~jackywangdb] and [~sandyr]. How do you want to proceed the remaining
SPARK-54019 ? For me, I believe we can handle it during the QA period as a kind
of bug fixes.
> Prevent session state mutation from within Pipelines Python definition files
> ----------------------------------------------------------------------------
>
> Key: SPARK-52856
> URL: https://issues.apache.org/jira/browse/SPARK-52856
> Project: Spark
> Issue Type: Umbrella
> Components: Declarative Pipelines
> Affects Versions: 4.1.0
> Reporter: Sandy Ryza
> Priority: Major
>
> Mutating Spark session state is an imperative construct that can cause
> friction and unexpected behavior from within a pipeline declaration. E.g. it
> makes pipeline behavior sensitive to the order that Python files are imported
> in, which can be unpredictable.
> For most elements of session state, there are already existing declarative
> ways to specify these within pipelines.
> Raising an error when someone tries to directly mutate session state would
> help avoid this unpredictable behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
