felipepessoto commented on PR #12218: URL: https://github.com/apache/gluten/pull/12218#issuecomment-5074966376
@malinjawi I can see the Pushed Filters, but there is a difference, the expected filter contains a `*`, do you know what this mean? ``` ...PushedFilters: [IsNotNull(`id with space`), LessThan(`id with space`,5)], ReadSchema: struct<col-215ff29d-abc0-48ea-92fc-0c09b73e9f40:bigint,col-c6bb9720-4fc4-4181-b423-bf2050fbb9dc:b... NativeFilters: [isnotnull(col-c6bb9720-4fc4-4181-b423-bf2050fbb9dc#1159077L),(col-c6bb9720-4fc4-4181-b423-bf2050fbb9dc#1159077L < 5)] [info] )" did not contain "PushedFilters: [*IsNotNull(id with space), *LessThan(id with space,5)]" (DeltaCDCColumnMappingSuite.scala:671) ``` I asked AI to explain, but I'm not familiar with this and I can't confirm, I hope you know more: > What the * means > > In a scan's PushedFilters metadata, a * prefix marks a filter the data source guarantees to fully evaluate itself, so Spark drops its own redundant Filter above the scan. It's emitted only by RowDataSourceScanExec ( DataSourceScanExec.scala:168 ): > > if (handledFilters.contains(filter)) s"*$filter" else s"$filter" > > handledFilters = pushed filters minus the source's unhandledFilters(...) . A filter without * is offered to the source but Spark still re-applies it (best-effort, for skipping only). > > Why the offloaded plan has no * > > It's a different scan node, not a "less pushed-down" one. > > Before the PR — readChangeFeed produces DeltaCDFRelation , which is BaseRelation with CatalystScan and declares unhandledFilters = Array.empty ( CDCReaderBase.scala:109 ). Spark plans it as a RowDataSourceScanExec ; since nothing is unhandled, every filter is "handled" and gets a * : > > PushedFilters: [*IsNotNull(id), *LessThan(id,5)] ✅ test matches > > After the PR — DeltaCDFScanStrategy expands that relation into the underlying Parquet plan, which Gluten offloads to DeltaScanTransformer (a FileSourceScanLike ). Its metadata comes from FileSourceScanLike.metadata ( DataSourceScanExec.scala:444 ), which never marks filters: > > "PushedFilters" -> seqToString(pushedFiltersForDisplay) // no "*" > > PushedFilters: [IsNotNull(id), LessThan(id,5)] ... NativeFilters: [...] ❌ no "*" > > So the exact substring [*IsNotNull(id), *LessThan(id,5)] disappears and the assertion fails. > > The key point: the filter is still pushed down — it's in DeltaScanTransformer.dataFilters (your own test asserts exactly that). File-source scans (Parquet, hence Velox) never emit * , because Parquet pushdown is best-effort row-group skipping and Spark always keeps a post-scan filter for correctness. The * is a CatalystScan / PrunedFilteredScan "I own this filter" marker that a file scan structurally cannot reproduce. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
