voonhous opened a new issue, #19381:
URL: https://github.com/apache/hudi/issues/19381
Sub-task of #18780 (RFC-105 Trino connector migration).
### Context
The connector resolves parquet columns by NAME on both read paths: the
default name-based path
(`hudi.parquet.use-column-names=true`) via trino-hive's
`ParquetPageSourceFactory`, and the index-based
path (`hudi.parquet.use-column-names=false`) via
`HudiPageSourceProvider.remapColumnIndicesToPhysical`,
which derives physical indices from a name lookup and maps a miss one past
the last field so the parquet
reader null-fills it (#19288).
Name resolution covers the schema-on-write evolution Hudi performs today: an
added column null-fills for
base files written before the add, which is also what the file-group reader
merge expects for those
records. It does NOT cover full schema evolution written under
`hoodie.schema.on.read.enable`:
- a RENAMED column no longer matches the old file's field name, so old base
files read as null instead of
returning the data stored under the old name;
- TYPE changes (e.g. int -> long) are not remapped against the file's
physical type.
Spark and Hive readers resolve both through the table's InternalSchema
history, mapping each query column
to the file column by column id before the parquet read. The Trino connector
currently reads neither the
internal-schema history nor the per-file schema commit id.
### Proposal
Port internal-schema resolution to the connector read path: when
`hoodie.schema.on.read.enable` is set,
load the InternalSchema history, map requested columns to file columns by
column id (rename-safe) and
apply type promotion -- on the name-based and index-based parquet paths and
the file-group reader's
log-side projection alike.
### Acceptance
- Reads of a table with a renamed column return the old files' data under
the new name.
- Promoted types read correctly from files written with the narrower type.
- Behavior without `hoodie.schema.on.read.enable` is unchanged.
Pointer:
`hudi-trino/src/main/java/io/trino/plugin/hudi/HudiPageSourceProvider.java`
(`createPageSource`,
`remapColumnIndicesToPhysical`); Spark reference:
`HoodieParquetFileFormat`'s internal-schema handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
