shrivasshankar commented on PR #25790: URL: https://github.com/apache/datafusion/pull/25790#issuecomment-6319636220
> @shrivasshankar, > > Thanks for working on decimal filter selectivity. The approach is appropriate, and the focused tests cover important decimal boundaries and cast behavior. > > Keeping decimal support specific to statistics estimation, rather than expanding the existing symmetric hash join eligibility gate, is the right architectural boundary. The shared expression traversal also avoids unnecessary duplication. However, enabling decimal types must not admit arithmetic propagation that violates source column type invariants. > > I'm requesting changes because valid decimal arithmetic predicates can now fail filter construction or produce column statistics with incorrect decimal types. These are correctness regressions that need to be addressed before merging. The additional SQL-level and cast-boundary tests are useful but non-blocking suggestions. Thanks @kosiew, good catches! I went with excluding decimal arithmetic from statistics estimation, so predicates like `a * 2.00 > 5.0000` now fall back to the default selectivity. Comparisons, casts and negation still get estimates. Fixing the types through arithmetic seemed like a bigger change, so I'd leave that for a follow-up. Added tests for your multiplication and subtraction cases, plus both of your suggestions: a sqllogictest with Q6-style fractional literals on a parquet table, and more cast boundary cases (cross-width, negative scale, near the precision limits). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
