pingzh opened a new issue, #7143: URL: https://github.com/apache/datafusion-comet/issues/7143
### Describe the bug The scalar parent guard added in #7117 conservatively rejects a `make_date` filter behind Spark's inferred NULL checks. This preserves correctness but unnecessarily loses native filter execution for the direct-column shape below. This assessment is based on source inspection; runtime performance impact has not been measured. ### Steps to reproduce Use the materialized `make_date_boundaries` Parquet table in `spark/src/test/resources/sql-tests/expressions/datetime/make_date_overflow.sql`: ```sql SELECT y, m, d FROM make_date_boundaries WHERE make_date(y, m, d) > DATE '2000-01-01'; ``` Spark's constraint inference prepends checks equivalent to: ```sql isnotnull(y) AND isnotnull(m) AND isnotnull(d) AND make_date(y, m, d) > DATE '2000-01-01' ``` `CometFilterExec` serializes the entire condition, and `CometMakeDate.unsupportedParentReason` rejects the constructor under `And`. For direct column arguments, the rows rejected by these checks already return NULL from the native constructor without validating the date, including `(6000000, NULL, 1)`. ### Expected behavior - Keep this proven-safe filter native and add a `query expect_native(make_date)` regression to the existing fixture, covering both ANSI modes and its NULL-argument rows. - Recognize the redundant NULL-check shape narrowly. Allowing only the eager left operand of `And`/`Or` is insufficient: Spark constructs `And(newFilters.reduce(And), condition)`, so the date comparison remains on the right. Account for filter predicate ordering when proving eager evaluation safe. - Preserve fallback for general predicates that skip a non-NULL constructor that would overflow, and preserve errors when an overflowing date is actually evaluated. ### Additional context Follow-up to [the review on #7117](https://github.com/apache/datafusion-comet/pull/7117#discussion_r4332013632). `spark.comet.exec.makeDate.operatorFallback.enabled=false` controls operator-level safeguards and does not disable this scalar guard. This is a performance follow-up to the correctness fix, with no measured critical regression established yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
