Indian-boult opened a new issue, #20743:
URL: https://github.com/apache/hudi/issues/20743

   ### Bug Description
   
   **What happened:**
   
   Data skipping drops files and partitions that hold matching rows when a 
TIMESTAMP column has values before 1582-10-15. The query returns too few rows, 
even for a plain filter on the column.
   
   Column stats index version 2 (used from table version 9) stores each 
timestamp bound as an `Instant`. When the stats are loaded for data skipping, 
`SparkValueMetadataUtils#convertJavaTypeToSparkType` (hudi-spark-client) turns 
the `Instant` into a `java.sql.Timestamp` with `Timestamp.from(instant)` 
whenever `spark.sql.datetime.java8API.enabled` is false, which is the Spark 
default. Spark then reads the `java.sql.Timestamp` with 
`DateTimeUtils.fromJavaTimestamp`, which takes it to be in the hybrid Julian / 
Gregorian calendar. Before 1582-10-15 the two calendars differ by days, so the 
bounds that data skipping compares move away from the stored ones:
   
   | stored bound (UTC) | bound data skipping compares, JVM zone UTC |
   | --- | --- |
   | 0001-01-01 00:00:00 | 0001-01-03 00:00:00 |
   | 1500-03-05 00:00:00 | 1500-02-25 00:00:00 |
   | 1582-10-04 12:00:00 | 1582-09-24 12:00:00 |
   | 1582-10-15 12:00:00 and later | unchanged |
   
   In a JVM zone with a local mean time offset the shift is not whole days and 
reaches a little past 1582-10-15 (for ex, 1408 s at 1582-10-15 12:00 UTC in 
Asia/Kolkata, 422 s in America/Los_Angeles). Column stats and partition stats 
both load through this code. The index version 1 decoder 
(`ColumnStatsIndexSupport#deserialize`) already uses 
`DateTimeUtils.toJavaTimestamp`, and with the Java 8 API on the `Instant` is 
passed through, so neither is affected. DATE stats are not affected, except the 
10 days 1582-10-05 to 1582-10-14, which a `java.sql.Date` cannot hold. The same 
helper builds the rows of Hudi Streamer JSON sources 
(`MercifulJsonToRowConverter`), so a JSON timestamp before 1582-10-15 is also 
written days off (171392 s for 0000-01-01T00:00Z in an Asia/Kolkata JVM).
   
   **What you expected:**
   
   Every file and partition that holds a matching row stays a candidate, 
whatever the year of the timestamp.
   
   **Steps to reproduce:**
   
   ```sql
   set spark.sql.session.timeZone = UTC;
   -- Spark 3 refuses to write or read datetimes before 1582-10-15 without these
   set spark.sql.parquet.datetimeRebaseModeInWrite = CORRECTED;
   set spark.sql.parquet.datetimeRebaseModeInRead = CORRECTED;
   set spark.sql.avro.datetimeRebaseModeInWrite = CORRECTED;
   set spark.sql.avro.datetimeRebaseModeInRead = CORRECTED;
   set hoodie.metadata.index.column.stats.enable = true;
   set hoodie.metadata.index.column.stats.column.list = ts;
   create table t (id int, ts timestamp, ord long, dt string) using hudi
     tblproperties (primaryKey = 'id', orderingFields = 'ord', type = 'cow')
     partitioned by (dt);
   insert into t values (1, timestamp'0001-01-01 00:00:00', 1, 'par1'), (2, 
timestamp'0001-01-05 00:00:00', 1, 'par1');
   insert into t values (3, timestamp'1500-03-01 00:00:00', 1, 'par2'), (4, 
timestamp'1500-03-05 00:00:00', 1, 'par2');
   insert into t values (5, timestamp'2024-01-01 00:00:00', 1, 'par3'), (6, 
timestamp'2024-01-05 00:00:00', 1, 'par3');
   set hoodie.enable.data.skipping = true;
   select id from t where ts = timestamp'0001-01-01 00:00:00';  -- expected 1, 
returns nothing
   select id from t where ts = timestamp'1500-03-05 00:00:00';  -- expected 4, 
returns nothing
   select id from t where ts in (timestamp'0001-01-01 00:00:00', 
timestamp'2024-01-05 00:00:00');  -- expected 1 and 6, returns 6
   ```
   
   With `hoodie.enable.data.skipping = false`, or with 
`spark.sql.datetime.java8API.enabled = true`, each query returns its rows. The 
result is the same with partition stats on or off.
   
   This is one of the extreme-value cases in the gaps listed in #20449.
   
   ### Environment
   
   - Hudi version: master (1.3.0-SNAPSHOT), table version 9, column stats index 
version 2
   - Query engine: Spark 3.5
   - Relevant configs: `hoodie.enable.data.skipping=true`, column stats on the 
TIMESTAMP column, `spark.sql.datetime.java8API.enabled=false` (default)
   
   ### Logs and Stack Trace
   
   No error. The files are pruned and the rows are missing.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to