wombatu-kun opened a new issue, #19387: URL: https://github.com/apache/hudi/issues/19387
Sub-task of #18780 (RFC-105 Trino connector migration). Distinct from #19381, which is about resolving *data* columns through the internal schema; this one bites without `hoodie.schema.on.read.enable` at all. ### Conditions All of these must hold: 1. `hudi.parquet.use-column-names=false` (the default is `true`); 2. the split has no log files - that is the only path on which `HudiPageSourceProvider` enables predicate pushdown; 3. a predicate or dynamic filter on a column whose metastore ordinal differs from its physical ordinal in the base file; 4. the column physically sitting at that stale ordinal is also part of the projection. Condition 3 needs no schema evolution: hive sync with `hoodie.datasource.hive_sync.omit_metadata_fields=true` drops the five meta columns from the metastore, so every data column's ordinal ends up 5 lower than its physical position. Dropping or reordering a column under full schema evolution produces the same divergence. ### Wrong behavior Rows go missing, with no error. The domain is attached to whichever column occupies the stale ordinal, and row groups are then pruned on that column's statistics. When those statistics exclude the domain, every row group is dropped and the query returns nothing. When condition 4 is not met the domain resolves to no descriptor and is discarded instead. Results stay correct, but predicate pushdown silently does nothing. ### Cause In `HudiPageSourceProvider.createPageSource`, `remapColumnIndicesToPhysical` rebuilds the physical indices of the projection, but `getParquetTupleDomain` is fed `getCombinedPredicate(hudiSplit, dynamicFilter)`, whose handles come from `HudiTableHandle.getRegularPredicates()` and from the dynamic filter and are never remapped. With `useParquetColumnNames == false`, trino-hive's `getBaseColumnParquetType` resolves a predicate column as `fileSchema.getType(handle.getBaseHiveColumnIndex())` - whatever field happens to sit at that ordinal. ### Reproduced Measured on a real parquet file laid out like a Hudi base file (5 meta columns, then `c0..c9`), read back through the same four Trino APIs `createPageSource` uses on the base-file-only path, with handles carrying the ordinals of a metastore that omits the meta columns. `c7` carries the predicate and grows with the row index; `c2` sits at c7's stale ordinal and is always in 0..9. All three reads must return the 99 rows matching `c7 > 900`: ``` stale ordinal, shadowed column projected domain pushed to c2 | row groups 0/10 | rows read 0 | matching rows returned 0 <-- 99 rows lost stale ordinal, shadowed column NOT projected domain pushed to (none) | row groups 10/10 | rows read 1000 | matching rows returned 99 <-- correct, no pruning predicate handle remapped (expected behaviour) domain pushed to c7 | row groups 1/10 | rows read 100 | matching rows returned 99 <-- correct, pruned ``` The last line is the control: its only difference from the first is that the predicate handle carries the physical ordinal, which isolates the stale ordinal as the cause. Harness available on request - it is standalone rather than a `hudi-trino` test because that module needs JDK 25 and Trino 481; it ran against Trino 472, where the four APIs have the same signatures and `getBaseColumnParquetType` has the same body. ### Note on the fix Remapping the predicate keys through `remapColumnIndicesToPhysical` directly will not work: it maps every column absent from the file to the same `fileFields.size()` sentinel, and `TupleDomain.transformKeys` throws `IllegalArgumentException` when two keys collide. Columns absent from the file have to be dropped from the predicate rather than mapped to the sentinel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
