wombatu-kun opened a new issue, #20227:
URL: https://github.com/apache/hudi/issues/20227
### Bug Description
**What happened:**
With the column stats index enabled, a `lower(col)` or `upper(col)` filter
can prune a file that holds a matching row. Data skipping treats
`lower`/`upper` as order-preserving and applies them to the file's min/max
(`BaseHoodieCatalystExpressionUtils`), but case folding does not preserve the
binary ordering of strings: a file holding `'Zebra'` and `'apple'` has min
`'Zebra'` and max `'apple'`, and `lower('apple')` sorts before `'zebra'`, so
the file is pruned.
**What you expected:**
The matching rows returned.
**Steps to reproduce:**
```sql
CREATE TABLE t (id INT, name STRING) USING hudi
TBLPROPERTIES (primaryKey = 'id',
'hoodie.metadata.index.column.stats.enable' = 'true');
INSERT INTO t VALUES (1, 'Zebra'), (2, 'apple');
SELECT id, name FROM t WHERE lower(name) = 'zebra';
-- expected: 1, Zebra
-- actual: no rows
SELECT id, name FROM t WHERE upper(name) = 'APPLE';
-- expected: 2, apple
-- actual: no rows
```
Expression indexes on `lower(col)`/`upper(col)` are not affected: their
filters are rewritten to the indexed column before translation.
### Environment
**Hudi version:** master (1.3.0-SNAPSHOT)
**Query engine:** Spark 3.5
**Relevant configs:** `hoodie.metadata.index.column.stats.enable=true`,
`hoodie.enable.data.skipping=true` (default)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
