hemanthboyina opened a new pull request, #58272: URL: https://github.com/apache/spark/pull/58272
**What changes were proposed in this pull request?**
Introduces SupportsReportCatalogStatistics, a new mix-in interface for
Table in the DSv2 connector API. When a connector implements this interface,
DataSourceV2Relation.computeStats() uses the returned
catalog statistics (size, row count) directly from metadata — without
building a scan.
- New interface:
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsReportCatalogStatistics.java
- Rewrites DataSourceV2RelationBase.computeStats() to check for the
interface; falls back to defaultSizeInBytes when not implemented
- Adds unit tests in DataSourceV2RelationSuite
**Why are the changes needed?**
PushDownLeftSemiAntiJoin (optimizer batch ~166) calls
plan.stats.sizeInBytes on a DataSourceV2Relation before V2ScanRelationPushDown
(batch ~251) has converted it to a DataSourceV2ScanRelation. At this
point no scan exists, so computeStats() has nothing to delegate to.
In test/assert mode this throws:
BUG: computeStats called before pushdown on DSv2 relation
In production it silently builds a throwaway full-table scan just to
extract stats — wasteful and returns pre-pushdown (unfiltered) stats.
The fix adds a lightweight pre-pushdown stats path via a new Table-level
interface. Connectors like Iceberg can implement it by reading from the
snapshot summary (already in memory, zero I/O). Tables
that don't implement it get the existing defaultSizeInBytes behavior
unchanged.
**Does this PR introduce any user-facing change?**
Yes. LEFT SEMI and LEFT ANTI joins over DSv2 tables (e.g. Iceberg) no
longer crash in assert mode. Connectors implementing
SupportsReportCatalogStatistics will provide more accurate join-type selection
for broadcast decisions on these join types.
**How was this patch tested?**
Added 3 unit tests to DataSourceV2RelationSuite:
- Returns catalog stats when the table implements
SupportsReportCatalogStatistics
- Falls back to defaultSizeInBytes when the interface is not implemented
- Falls back to defaultSizeInBytes when catalog stats return empty
OptionalLong
**Was this patch authored or co-authored using generative AI tooling?**
Yes, Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
