xudong963 opened a new issue, #26131:
URL: https://github.com/apache/datafusion/issues/26131
### Is your feature request related to a problem or challenge?
Follow-up to #25207 and #5830: simplify a disjunction using range
constraints supplied by surrounding conjunctions.
The motivating query is a `UNION ALL` of a primary table and a historical
backfill table. The backfill branch explicitly restricts a timestamp column to
two disjoint periods. A client applies a time range to the union, possibly
followed by `ORDER BY ... LIMIT`. When the client range overlaps neither
backfill period, the goal is to eliminate that branch during logical
optimization, before calling its table provider's `scan()` method. This can
avoid planning-time file listing or index access as well as execution-time
reads.
An integer-only example isolates the predicate shape:
```sql
CREATE TABLE t AS
SELECT * FROM (VALUES (1), (6), (11)) AS v(x);
SET datafusion.explain.format = 'indent';
EXPLAIN
SELECT x
FROM t
WHERE x >= 10
AND (
(x >= 1 AND x < 3)
OR
(x >= 5 AND x < 7)
);
```
No value can satisfy this predicate, regardless of the table contents. Under
the outer `x >= 10` condition, both disjuncts are impossible.
#25207 handles contradictions among direct column comparisons in a
conjunction. In the current [`simplify_predicates`
implementation](https://github.com/apache/datafusion/blob/97c7593f609b277f65c086a7868a92c24ab5f8f2/datafusion/optimizer/src/simplify_expressions/simplify_predicates.rs),
an `OR` expression falls into `other_predicates` and is retained without
comparing its branches against the surrounding range constraints. This request
is specifically for that contextual range reasoning.
### Describe the solution you'd like
Recognize that the example's filter can never be true and produce an
`EmptyRelation` with the correct output schema. In the corresponding `UNION
ALL` query, propagate that empty relation and remove the impossible branch
before physical planning invokes its provider.
One possible approach is to collect supported range constraints from the
outer conjunction and use them to simplify the remaining expression
recursively. `ExprSimplifier::with_guarantees()` may provide reusable
machinery. Conceptually:
```text
P AND Q
-> P AND simplify(Q, assuming ranges established by P)
x >= 10 AND (first_interval OR second_interval)
-> x >= 10 AND (FALSE OR FALSE)
-> FALSE
```
The predicates supplying the assumptions must be retained unless a separate
proof establishes that they are redundant. Fully distributing arbitrary AND/OR
expressions into DNF should not be necessary; analysis can have a work/depth
budget and conservatively keep expressions it cannot prove.
Useful regression coverage:
- Both disjuncts contradict the outer range: produce an empty relation.
- Only one disjunct contradicts it: retain the satisfiable branch.
- The requested range falls in the gap between the two intervals.
- Inclusive/exclusive endpoints, nullable columns, and timestamp
precision/time zones.
- An unsupported or potentially satisfiable disjunct must prevent
incorrectly declaring the whole OR impossible.
- A `UNION ALL` integration test with a mock provider confirms the
eliminated branch's `scan()` is not called, including an outer sort/limit.
This should preserve WHERE-clause semantics; replacing an expression that
can be NULL with FALSE is not generally valid in projection expressions.
### Describe alternatives you've considered
- Add a redundant enclosing interval alongside the exact OR predicate. This
exposes a simple contradiction for requests outside the overall interval, but
cannot identify requests that fall in the gap.
- Split the disjoint periods into separate `UNION ALL` branches. This
exposes direct conjunctions but changes the query structure and may increase
planning overhead for overlapping requests.
- Rely on partition/file pruning or physical empty-branch elimination. These
can reduce reads, but may happen after the provider has already performed
planning-time I/O.
### Additional context
Source inspected at upstream `main` commit
`97c7593f609b277f65c086a7868a92c24ab5f8f2`.
The SQL above was executed locally on a `datafusion-cli 55.0.0` build and
retained this logical plan:
```text
Filter: t.x >= Int64(10) AND (t.x >= Int64(1) AND t.x < Int64(3) OR t.x >=
Int64(5) AND t.x < Int64(7))
TableScan: t projection=[x]
```
That local build predates #25207; this output is not presented as a
reproduction on current `main`. The proposed extension beyond #25207 is based
on the source inspection linked above. I have not run the full current-main
optimizer pipeline for this example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
