colinmarc opened a new pull request, #91:
URL: https://github.com/apache/datafusion-iceberg/pull/91
The purpose of `expr_to_predicate` is to take an expr and return a valid,
equivalent push-down predicate. The existing behavior of just dropping the cast
ignored many boundary conditions that would lead to incorrectly excluding
partitions.
There was already a carveout for dates, because that is the most obvious
example. The following expressions are obviously not equivalent:
arrow_cast(x, 'timestamp(ns)') < '2026-01-22T12:00' x < '2026-01-22T12:00'
Whether it's a bug depends on the type of `x`; if x is a date, or broadly
more granular than the literal, then dropping the cast creates a bad predicate
(specifically: the partition '2026-01-22' would be dropped, even though the
original expression should keep it).
Once I started looking, though (using an LLM), I found a few more weird
cases:
```sql
-- x is int32. '10' < '5' (lexicographically)
arrow_cast(x, 'utf8') < '5'
-- Drop the cast and Datum::to('5', int32) gives you:
x < 5
```
And here's another:
```sql
-- x is f32. 0.1f32 > 0.1f64 (0.100000001490116 > 0.100000000000000005)
arrow_cast(x, 'float64') > 0.1
-- Dropping the cast, plus coercion to f32:
x > 0.1f32
```
It seems like there might be more, any time two conditions are fulfilled:
- The cast is to a coarser type
- The literal would be coerced in a lossy way by `Datum::to`
# Context
I discovered this because with
https://github.com/apache/iceberg-rust/pull/3574, the following query goes from
an error to silently dropping partitions:
```sql
SELECT * FROM foobar WHERE some_date_column < TIMESTAMP '2023-01-05 12:00:00'
```
However, I think that PR is correct, at least in mimicking iceberg-java
behavior. It's just that there's also a latent bug here.
The bad news is that this locks us out of partition pushdown for some
queries today. The good news is that there's lots of work upstream in DF to
simplify casts out anyway. I think that's the right place to push the
complexity.
You can read more about that work here:
https://github.com/apache/datafusion/issues/19946
https://github.com/apache/datafusion/issues/26289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
