Hyukjin Kwon created SPARK-61875:
------------------------------------
Summary: Keep the cached schema of deferred-error DataFrames, the
instance attributes of copies and pickles, PySpark's Python origins, and the
window warning of no partitions
Key: SPARK-61875
URL: https://issues.apache.org/jira/browse/SPARK-61875
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Four plan-time gaps against PySpark 4.3.0:
- The DataFrames whose counts or seeds fail when their plan is built
(SPARK-61766) drop the cached schema of their parent, where PySpark's limit,
coalesce, repartition, repartitionByRange, repartitionById, sample and
randomSplit copy _cached_schema.
- Copies and pickles of df.na, df.stat, spark.tvf, spark.dataSource,
PandasCogroupedOps, StreamingQuery, StreamingQueryManager, RuntimeConf and the
merge builder drop their instance attributes; the merge actions and
SparkSession.Builder have no __dict__; and the pickle errors of spark.streams,
the conf and the merge builder differ from PySpark's.
- The Python origins of groupBy().count(), the numeric aggregates, agg of a
dict, repartitionByRange of names and the conditions of a join of a list of
Columns differ from PySpark's, which applies col and asc_nulls_first to the
columns only, aliases the count, and makes the & of the join's conditions when
it builds the plan.
- PySpark warns "WARN WindowExpression: No Partition Defined for Window
operation!" at each plan build of a window of no partitions; the drop-in never
warns.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]