[
https://issues.apache.org/jira/browse/SPARK-45136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Herman van Hövell resolved SPARK-45136.
---------------------------------------
Fix Version/s: 4.0.0
Resolution: Fixed
> Improve ClosureCleaner to support closures defined in Ammonite REPL
> -------------------------------------------------------------------
>
> Key: SPARK-45136
> URL: https://issues.apache.org/jira/browse/SPARK-45136
> Project: Spark
> Issue Type: Improvement
> Components: Connect
> Affects Versions: 4.0.0, 3.5.1
> Reporter: Vsevolod Stepanov
> Assignee: Vsevolod Stepanov
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> ConnectRepl uses Ammonite REPL with CodeClassWrapper to run Scala code. It
> means that each code cell is wrapped into a separate object. If there are
> multiple variables defined in the same cell / code block it will lead to
> capturing extra variables, increasing serialized UDF payload size or making
> it non-serializable.
> For example, this code
> {code:java}
> // cell 1
> {
> val x = 100
> val y = new NonSerializable
> }
> // cell 2
> spark.range(10).map(i => i + x).agg(sum("value")).collect(){code}
> will fail because lambda will capture both `x` and `y` as they're defined in
> the same wrapper object
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
