wombatu-kun opened a new issue, #20520:
URL: https://github.com/apache/hudi/issues/20520

   ### Bug Description
   
   **What happened:**
   
   With the column stats index enabled, a filter on `date_format(ts, fmt)` or 
`from_unixtime(sec, fmt)` can prune a file that holds matching rows when `fmt` 
has a time-of-day or offset field and the session time zone changes its offset, 
as it does at daylight saving transitions. Data skipping treats both functions 
as order-preserving for any pattern and zone 
(`BaseHoodieCatalystExpressionUtils`) and re-applies them to the file's min/max 
stats, but they render the time in the session time zone, where local time runs 
backwards across a fall-back: in `America/New_York`, 05:30Z renders as `01:30` 
and the later 06:10Z as `01:10`. Date-only patterns such as `yyyy-MM-dd` are 
unaffected under current rules, since the local date only runs backwards where 
the clocks fall back across midnight; tzdb last records that in 2010 
(`America/St_Johns` at 00:01), and that historical case, which `CAST(ts AS 
DATE)` shares, is not covered here. The partition stats index skips these 
functions, so only file 
 pruning is affected.
   
   Part of #20449 (time-zone conversions across DST transitions), next to 
#20518. Patterns that do not sort chronologically, such as `MM/dd/yyyy`, are a 
separate problem tracked in #19481.
   
   **What you expected:**
   
   The matching row returned.
   
   **Steps to reproduce:**
   
   ```sql
   SET spark.sql.session.timeZone = UTC;
   
   CREATE TABLE t (id INT, ts TIMESTAMP, sec BIGINT) USING hudi
   TBLPROPERTIES (primaryKey = 'id', 
'hoodie.metadata.index.column.stats.enable' = 'true');
   
   INSERT INTO t VALUES (1, TIMESTAMP'2026-11-01 05:30:00', 1793511000), (2, 
TIMESTAMP'2026-11-01 06:10:00', 1793513400);
   
   SET spark.sql.session.timeZone = America/New_York;
   
   SELECT id FROM t WHERE date_format(ts, 'yyyy-MM-dd HH:mm') > '2026-11-01 
01:20';
   -- expected: 1 row (id = 1)
   -- actual: no rows
   
   SELECT id FROM t WHERE from_unixtime(sec, 'yyyy-MM-dd HH:mm') > '2026-11-01 
01:20';
   -- expected: 1 row (id = 1)
   -- actual: no rows
   ```
   
   With `hoodie.enable.data.skipping=false`, both queries return the row.
   
   ### Environment
   
   **Hudi version:** master (1.3.0-SNAPSHOT); `release-1.2.1` and 
`release-1.1.2` carry the same matcher arms
   **Query engine:** Spark 3.5
   **Relevant configs:** `hoodie.metadata.index.column.stats.enable=true`, 
`hoodie.enable.data.skipping=true` (default), a `spark.sql.session.timeZone` 
with DST
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to