Tim Meehan created SPARK-61414:
----------------------------------

             Summary: Send a limit of one row over an empty projection in 
DataFrame::is_empty of the Rust API, as PySpark does
                 Key: SPARK-61414
                 URL: https://issues.apache.org/jira/browse/SPARK-61414
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Tim Meehan


DataFrame::is_empty of the Rust API, and so DataFrame.isEmpty of the Python 
client of spark-connect-rust, sends a global count aggregate over the limit: 
Aggregate(count(1)) over Limit(1). PySpark's isEmpty is 
{{len(self.select().take(1)) == 0}}: it sends Limit(1) over a Project with no 
expressions and checks whether a row comes back. The answers are the same; the 
plan, and so what the server runs and returns, is not.

{code:python}
from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
print(spark.range(3).isEmpty())
{code}

Against an OSS Spark 4.3.0 Connect server on loopback, the plan of the request:
* pyspark-client 4.3.0: Limit(limit=1, input=Project(expressions=[], 
input=Range(0, 3, 1))). The result is False, from the row that comes back.
* spark-connect-rust master 125654b: Aggregate(group_type=Groupby, 
aggregate_expressions=[count(1)], input=Limit(limit=1, input=Range(0, 3, 1))). 
The result is False, from the count.

>From source, at 125654b: DataFrame::is_empty is 
>{{self.limit(1).count().map(|c| c == 0)}} 
>(crates/spark-connect/src/dataframe.rs:1315-1317), and DataFrame::count pushes 
>a {{count(1)}} aggregate (crates/spark-connect/src/dataframe.rs:1052-1070). 
>The Python client calls DataFrame::is_empty 
>(crates/pyspark-rs/src/dataframe.rs:2631-2635). PySpark's isEmpty is in 
>python/pyspark/sql/connect/dataframe.py.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to