voonhous commented on issue #20630: URL: https://github.com/apache/hudi/issues/20630#issuecomment-6976637404
## Design note: Trino-native records for the MOR merge **Proposal.** Give the connector its own `RecordContext<TrinoRow>`, where a `TrinoRow` is a view of one position in a Trino page. The file-group reader then merges Trino rows directly, the way Spark (`InternalRow`), Flink (`RowData`) and Hive (`ArrayWritable`) already do. Avro stays only where hudi-common hands us Avro: classic Avro log blocks, and payload classes for the two records being merged. A base row with no log update becomes one small view object plus a position copy per column, which meets the second exit criterion of this issue by construction. **Depends on** typing columns from `HoodieSchema` (`hudi.schema-column-types.enabled`, branch `trino-mapper-column-types` on top of #20680). Native rows need each value's Trino type to match its schema field. On metastore types (bigint for a timestamp, varbinary for a uuid), the record context would have to guess value shapes the way `HudiAvroSerializer` does today. This lands on that gated path first; the legacy path stays as it is until the default flips. ### Today <!-- diagram: old-path.png --> - `HudiTrinoReaderContext` extends `HoodieReaderContext<IndexedRecord>` and uses hudi-common's `AvroRecordContext` (`HudiTrinoReaderContext.java:67-68, :122`). - Every base row goes through `HudiAvroSerializer.serialize()` (page row to `GenericData.Record`) and back through `buildRecordInPage()`, including rows no log record touches. - The values `serialize()` puts in the record are Trino objects (`SqlTimestamp`, `SqlDecimal`, `SqlVarbinary`), not Avro-native ones. So `appendTo` has to work out each value's shape with `instanceof`, and anything that Avro-encodes these records breaks on them. ### Proposed <!-- diagram: new-path.png --> <details> <summary>Why this fits hudi-common's contract (line references against branch trino-mapper-column-types)</summary> - **Commit-time and event-time ordering need no merger class.** `BufferedRecordMergerFactory` returns `CommitTimeRecordMerger` / `EventTimeRecordMerger` for them, which only compare ordering values read through the `RecordContext` (`BufferedRecordMergerFactory.java:78-87, :179-209`). - **Mergers are generic over the record type.** `HoodieRecordMerger.merge(BufferedRecord<T>, BufferedRecord<T>, RecordContext<T>, ...)` (`HoodieRecordMerger.java:73`). `HoodieAvroRecordMerger` converts just the two records it merges through `convertToAvroRecord` / `convertAvroRecord` (`HoodieAvroRecordMerger.java:58-60, :79`). `PayloadUpdateProcessor` does the same for log-only rows (`UpdateProcessor.java:136-147`). So payload tables work on `TrinoRow` and pay for Avro only on merged rows. - **`seal()` runs on every base row** (`FileGroupRecordBuffer.java:212`). Spark copies there because `UnsafeRow` reuses its buffer (`BaseSparkInternalRecordContext.java:222-224`). Trino blocks are immutable, so `seal()` can return the view as is. - **`toBinaryRow()` runs only when a record is retained.** That's insertion into the log record buffer (`BufferedRecord.java:68-77` via `KeyBasedFileGroupRecordBuffer.java:130`) and the LSM spill iterator (`SpillableLsmRecordIterator.java:88`). It is the one place to copy: compact the view into a single-position row so buffered log records don't pin whole log pages. - **Avro log blocks arrive through `convertAvroRecord`** (`HoodieAvroDataBlock.java:173`). The Avro-to-block half of `HoodieSchemaAvroSerializer` becomes that conversion. </details> <details> <summary>RecordContext<TrinoRow> method mapping</summary> | Method | TrinoRow implementation | |---|---| | `getValue(row, schema, field)` | Channel by name (cached per schema, dotted paths walk row blocks). Returns plain Java values (`Long`, `Integer`, `String`, `BigDecimal`, `Boolean`), so ordering values compare with the ones delete blocks carry, and `castToBoolean` / delete-marker `toString()` work. | | `getMetaFieldValue(row, pos)` | `VARCHAR` slice of the meta column, decoded to `String` (the record key the buffer map is keyed by). | | `convertValueToEngineType` | Identity, since `getValue` already returns Java values. | | `seal(schema, row)` | Returns `row` (blocks are immutable). | | `toBinaryRow(schema, row)` | Single-position copy (`Block.getSingleValueBlock` per channel). | | `projectRecord(from, to, renames)` | Reorders channels by name; a view over the same blocks, no copy. | | `constructEngineRecord(schema, values)` / `mergeWithEngineRecord` | Writes Java values into one-position blocks through the per-`HoodieSchemaType` writers (partial-update modes). | | `getDeleteRow(key)` | One-position row with the key fields set, other channels null. | | `convertAvroRecord(avro)` | Avro-to-block writers, keyed by `HoodieSchemaType`. | | `convertToAvroRecord(row, schema)` | Block-to-Avro writers, keyed by `HoodieSchemaType` (Avro-native values). | | `constructHoodieRecord` | Needed only by custom-merger paths; see the first open question. | | `getEngineRecordType` | `HoodieRecordType` has no TRINO (`HoodieRecord.java:574-575`); see the second open question. | Output: `HudiPageSource` writes each channel with `Type.appendTo(block, position, builder)`, by name as today. Partition and hidden columns still come from `PrefilledColumnValues`. </details> <details> <summary>Scope of the first cut</summary> - **Native `TrinoRow` context** when `hudi.schema-column-types.enabled` is on and the table uses commit-time or event-time ordering (including partial-update modes) or a payload class. - **Today's Avro context** (schema-typed `HoodieSchemaAvroSerializer`) for tables with a custom `HoodieRecordMerger`, because a user merger may cast `getRecord()` to `IndexedRecord`. The repo's test mergers read only through `recordContext.getValue`, so an opt-in for well-behaved mergers is a later step. - **Unchanged:** everything with the property off, and the base-file-only path, which already passes pages through. - **Spill:** `HudiRecordSerializer` from #20648 gets a `TrinoRow` variant; size estimation stays with #20627. </details> <details> <summary>Open questions</summary> 1. **`constructHoodieRecord`** for a custom merger: implement it over `TrinoRow` (no `HoodieRecord` subclass exists for Trino), or keep custom-merger tables on the Avro context for good? 2. **`getEngineRecordType`:** there is no TRINO value and no caller in hudi-common's read path (it feeds the native log delete writer). Throw from this read-only context, or add the enum value? 3. **KEEP_VALUES partial merging** goes through `recordMerger.partialMerge` (`BufferedRecordMergerFactory.java:72-75`); confirm the ordering-mode mergers can run it on `TrinoRow` (today's Avro path has the same gap, #19413). 4. **Batch pass-through:** runs of untouched base rows could be copied as a range (`copyPositions`) instead of row by row. That needs a hook in the file-group reader's iterator, so it's a follow-up. </details> <details> <summary>Plan and measurement</summary> 1. `TrinoRow` and `TrinoRecordContext`, with unit tests per method, reusing the `HoodieSchemaType` writers. 2. Route the native context from `HudiPageSourceProvider` by merge mode; `HudiPageSource` copies positions. 3. `TrinoRow` spill serializer (on top of #20648). 4. `HudiMorReadBenchmark` scan with `-Dbench.s3.latency.enabled=0`, measured on top of #20627, against this issue's `_rt`/`_ro` CPU ratio of 3x or less. Plus an allocation profile showing no per-row record object for base rows without a log entry. The merge-semantics suites (`TestHudiMor*`, custom-merger tests) run with the property on. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
