Uros Stankovic created SPARK-52655:
--------------------------------------
Summary: Name isPartiallyPushed methods differently in DSv2 scan
builder interfaces
Key: SPARK-52655
URL: https://issues.apache.org/jira/browse/SPARK-52655
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.0
Reporter: Uros Stankovic
`SupportsPushDownLimit` and `SupportsPushDownTopN` have the same method
(isPartiallyPushed) that means that certain implementation of `ScanBuilder`
cannot have different implementation of these two methods.
In real world usage, it is usually fine to have same implementation since the
logic for limit pushdown and topN pushdown should be similar, but it is safer
to allow different implementation and more granular logic if it is needed for
certain implementation.
[https://github.com/apache/spark/blob/a1e628574b7d9cdf89472fa550ecc41f8a871b98/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownLimit.java#L41C19-L41C36]
https://github.com/apache/spark/blob/a1e628574b7d9cdf89472fa550ecc41f8a871b98/sql/catalyst/src/main/java/org/apache/spark/sql/connector/read/SupportsPushDownTopN.java#L42
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
