Hyukjin Kwon created SPARK-62041:
------------------------------------

             Summary: Warn of a window in turn with the picklings of UDFs, send 
the clauses of a merge as they were built, keep the execution info of a failed 
toPandas, and pickle subquery Columns and Column._expr, as PySpark
                 Key: SPARK-62041
                 URL: https://issues.apache.org/jira/browse/SPARK-62041
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: 5.0.0
            Reporter: Hyukjin Kwon


>From a review of the documented differences of UDFs and pickles against 
>PySpark 4.3.0:
- A window of no partitions warns before a pickling error of a UDF that its 
plan holds, where PySpark's build pickles the UDF first and raises its error 
without the warning.
- The merge command of mergeInto carries the pickles of its clauses made when 
the merge runs, where PySpark's carries the ones of the clauses, which its 
_build_merge_action makes when each clause is added; a window in a clause 
warned twice.
- When toPandas fails after its execution, in the conversion or the copy, 
PySpark leaves df.executionInfo as it was (None the first time); the drop-in 
has already kept the execution's.
- A pickled Column of a subquery (df.scalar(), df.exists()) loses the plan of 
its subquery: its requests fail on the server with INTERNAL_ERROR, where 
PySpark's unpickled Column works, also on a Python worker of the server.
- pickle.dumps(c._expr) raises "TypeError: cannot pickle '_ColumnExpression' 
object", where PySpark pickles the expression.
- The pickle of a Column of a UDF or of a UDTF of a return type that PySpark 
does not convert raises at the pickle, where PySpark pickles it and fails its 
request.
- The reasons of the documented differences of the GC of shared functions and 
of pickled Columns are incomplete.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to