Tim Meehan created SPARK-61414:
----------------------------------
Summary: Send a limit of one row over an empty projection in
DataFrame::is_empty of the Rust API, as PySpark does
Key: SPARK-61414
URL: https://issues.apache.org/jira/browse/SPARK-61414
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Tim Meehan
DataFrame::is_empty of the Rust API, and so DataFrame.isEmpty of the Python
client of spark-connect-rust, sends a global count aggregate over the limit:
Aggregate(count(1)) over Limit(1). PySpark's isEmpty is
{{len(self.select().take(1)) == 0}}: it sends Limit(1) over a Project with no
expressions and checks whether a row comes back. The answers are the same; the
plan, and so what the server runs and returns, is not.
{code:python}
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
print(spark.range(3).isEmpty())
{code}
Against an OSS Spark 4.3.0 Connect server on loopback, the plan of the request:
* pyspark-client 4.3.0: Limit(limit=1, input=Project(expressions=[],
input=Range(0, 3, 1))). The result is False, from the row that comes back.
* spark-connect-rust master 125654b: Aggregate(group_type=Groupby,
aggregate_expressions=[count(1)], input=Limit(limit=1, input=Range(0, 3, 1))).
The result is False, from the count.
>From source, at 125654b: DataFrame::is_empty is
>{{self.limit(1).count().map(|c| c == 0)}}
>(crates/spark-connect/src/dataframe.rs:1315-1317), and DataFrame::count pushes
>a {{count(1)}} aggregate (crates/spark-connect/src/dataframe.rs:1052-1070).
>The Python client calls DataFrame::is_empty
>(crates/pyspark-rs/src/dataframe.rs:2631-2635). PySpark's isEmpty is in
>python/pyspark/sql/connect/dataframe.py.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]