mohitgurav20 commented on PR #25719: URL: https://github.com/apache/datafusion/pull/25719#issuecomment-5967463437
@asolimando Thanks for the thorough review and excellent catch on the selectivity multiplication edge cases. I've pushed a new commit that addresses all your points: Selectivity Multiplication Fix: Refactored compute_fallback_selectivity so that default_selectivity is applied at most once for all unhandled conjuncts combined. For example, s LIKE '%abc' AND t <> 'x' correctly remains at 0.2 rather than multiplying down to 0.04. col_a = col_b Estimation: Updated the logic to use 1 / max(NDV(left), NDV(right)) when both sides have known statistics, avoiding the left-side bias. Restored Coverage: Reinstated the 4 test cases that were inadvertently dropped during a recent conflict resolution with main. Added Unit Tests: Introduced 7 new unit tests covering all requested edge cases: Single handled equality (1/NDV) Multiple unhandled conjuncts Mixed handled and unhandled predicates col = col using max(NDV) The CAST(col) = expr case (which correctly falls back to default since it's wrapped) The utf8_col = 'literal' case SLT Plan Test: Added an EXPLAIN test to subquery.slt confirming that an unresolved scalar subquery equality now correctly utilizes the NDV-based row estimate. A quick note regarding utf8_col = 'literal': Since it fails check_support, it now accurately falls back to 1/NDV instead of the previous flat 20%. This yields a much tighter estimate (e.g., a Utf8 column with NDV=60 will estimate ~1.6% selectivity instead of an overly broad 20%). CI is now completely green. Let me know if you have any additional thoughts on this approach! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
