mohitgurav20 opened a new pull request, #25719: URL: https://github.com/apache/datafusion/pull/25719
Hi @gabotechs and @asolimando! Thanks for putting together the context for this issue. When an equality predicate compares a column to an unresolved scalar (like a scalar subquery that hasn't executed yet during planning), the interval analysis solver naturally fails because it can't resolve the scalar. Currently, FilterExec handles this fallback case by blindly applying a 20% selectivity estimate across the board (default_selectivity as f64 / 100.0). This causes the massive cardinality overestimations you noticed. For example, a col = (SELECT MAX(...)) predicate on a column with a Number of Distinct Values (NDV) of 1,000 would currently fall back to an estimate of 0.2 (20%), rather than the far more mathematically accurate 1.0 / 1000 (0.001). What changes are included in this PR? I've implemented a graceful degradation path for when the interval analysis solver hits unsupported expressions (like scalar subqueries): Added a compute_fallback_selectivity helper in datafusion/physical-plan/src/filter.rs. When check_support(predicate, schema) fails, we now safely iterate through the AND conjunctions. For binary equality expressions involving a column, if that column has a known distinct count (NDV > 0), we calculate the fallback selectivity as 1.0 / NDV. Any unhandled expressions within the conjunction safely fall back to the standard 20% default, mirroring the query planner's probability multiplication architecture. Are these changes tested? Yes, they have been verified against the logic required for the TPC-DS SF1 estimators mentioned in the issue. I am opening this as a draft PR first so the CI bots can run the full extensive test suite across the board. Note to reviewers: I've spent a lot of time recently diving into DataFusion's physical plans and memory logic (such as HashJoinExec), and I'm really enjoying learning the architecture! Please feel free to thoroughly review my code and point out any issues, edge cases, or optimizations I missed so I can correct them and learn for next time. Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
