[
https://issues.apache.org/jira/browse/SPARK-62004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62004:
------------------------------------
Assignee: Hyukjin Kwon
> Pickle DataFrames of failing plans and table arguments, keep the schema that
> a copy requests, rebuild same-session deep copies, raise frame-bound errors
> at the build, and report Column._expr to Python's GC, as PySpark
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62004
> URL: https://issues.apache.org/jira/browse/SPARK-62004
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> From a review of the documented differences of DataFrames, and of the GC
> helpers of SPARK-61971, against PySpark 4.3.0:
> - A DataFrame whose plan fails with an AttributeError (a join on a tuple, or
> on conditions of a Row) refuses to pickle, where PySpark pickles it and its
> unpickle raises NOT_ITERABLE.
> - A copy, deep copy or unpickle of a DataFrame requests its schema, as
> PySpark's does, but does not keep it: PySpark's schema, printSchema, toArrow,
> the set operations and numeric aggregates of the copy read the schema that
> the copy requested, where the drop-in's read the cached one, so a view
> replaced in between gives other results.
> - A TableArg (df.asTable()) does not pickle, where PySpark's pickles and
> cloudpickles.
> - A deep copy of a DataFrame into the same session builds its relation, so
> later requests lose the warnings and origins that PySpark gives at each build.
> - The errors of the frame bounds of Window.rowsBetween and rangeBetween
> (VALUE_NOT_BETWEEN) are raised at the call, where PySpark raises them when
> the plan is built.
> - Column._expr and TableArg._subquery_expr are not reported to Python's
> garbage collector, so a reference cycle through a UDF of them leaks, and they
> do not copy (TypeError: cannot pickle '_ColumnExpression' object).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]