voonhous opened a new pull request, #18837:
URL: https://github.com/apache/hudi/pull/18837
### Describe the issue this Pull Request addresses
The Trino-Hudi connector used to live in the Trino repo (shipped via
`hudi-trino-bundle`). It lagged Hudi's reader / table-format APIs, so every
reader or new-type change needed a manual backport and the bundle drifted out
of sync (version skew).
RFC-105 flips the ownership:
- The connector implementation moves into the Hudi repo as
`hudi-trino-plugin`, so it tracks Hudi's reader APIs in lockstep.
- The Trino repo keeps only a thin `trino-hudi` shim that depends on the
published `org.apache.hudi:hudi-trino` artifact.
Relates to RFC-105. MoR merge-mode test coverage tracked as follow-up in
#18898.
### Summary and Changelog
What changes for users:
- New opt-in module `hudi-trino-plugin`, published as
`org.apache.hudi:hudi-trino` (regular non-shaded jar). Excluded from default
builds, behind the `hudi-trino` Maven profile. Requires JDK 25.
- The old `hudi-trino-bundle` packaging module and the three docker trino
modules are removed.
Code structure (two-repo shim model): see diagrams below.
Changelog:
- Add `hudi-trino-plugin` module. Connector code was migrated from the Trino
repo `trino-hudi` plugin (same `io.trino.plugin.hudi` package) and adapted to
current Hudi APIs.
- Target Trino SPI 481 and JDK 25 (enforced via `maven-enforcer-plugin`).
Trino SPI is `provided` scope.
- Read path adopts the builder-style `HoodieFileGroupReader`,
`HoodieReaderContext`, `HoodieSchema`, and Trino `SourcePage`.
- `HudiTrinoReaderContext` extends `HoodieReaderContext<IndexedRecord>`;
record-merger selection now dispatches on `RecordMergeMode` (event-time /
commit-time / custom) instead of a fixed merger.
- `HudiSplitSource` reuses `NativeTableMetadataFactory` for the
metadata-table vs filesystem-listing fallback.
- Column stats read via `TableMetadataReader.getColumnRanges`.
- `HudiPageSource` closes all reader resources on both the init-failure and
`close()` paths.
- Tests: shared file-op assertions extracted into `FileOperationAssertions`.
Test deps (Trino `*-tests.jar`) sit behind the `hudi-trino-tests` profile, off
by default, since three are not on Maven Central.
- Remove `hudi-trino-bundle` and the `trino{base,coordinator,worker}` docker
modules, plus their compose services, README steps, and release-validation
entries.
### Impact
- Opt-in only. Not built by default, so no impact on existing Hudi modules
or users who do not enable the profile.
- Public API: no change to core Hudi. Adds a new published artifact
`org.apache.hudi:hudi-trino`.
- New config `hudi.table.resolve-column-name-casing.enabled`, default
`false` (was `true` in the pre-migration code). Avoids an eager table-schema
load on the common lowercase-column path. Mixed-case tables opt in.
- Performance: reads go through `HoodieFileGroupReader`; base-file-only
splits skip the Avro merge path.
### Risk Level
low
- Module is off by default and isolated in the Trino server by a per-plugin
URLClassLoader, so it cannot affect other Hudi modules or the rest of a Trino
deployment.
- Connector correctness (MoR merge semantics) is covered by the smoke and
file-operation tests across COW and MoR tables. The merge-mode dispatch added
here is exercised by the existing default-payload MoR tests; delete-marker and
custom-payload coverage is tracked in #18898.
### Documentation Update
- `hudi-trino-plugin/README.md`: build, test, and IDE-setup instructions,
including why the default build skips tests.
- New config `hudi.table.resolve-column-name-casing.enabled` (default
`false`). Website connector docs to follow.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
---
## Code structure: hudi-repo and trino-repo
The substance lives in the Hudi repo. The Trino repo carries a thin shim
that pulls the published artifact and registers it through Trino's Plugin SPI.
```mermaid
flowchart LR
subgraph HUDI["hudi-repo : apache/hudi"]
direction TB
HC["hudi-common<br/>HoodieFileGroupReader<br/>HoodieReaderContext<br/>HoodieTableMetadata
/ HoodieSchema"]
subgraph PLUGIN["hudi-trino-plugin<br/>artifact
org.apache.hudi:hudi-trino (non-shaded jar)"]
direction TB
ENTRY["Plugin entry<br/>HudiPlugin > HudiConnectorFactory >
HudiConnector"]
READ["Read path<br/>HudiSplitManager >
HudiSplitSource<br/>HudiPageSourceProvider >
HudiPageSource<br/>HudiTrinoReaderContext extends HoodieReaderContext"]
end
ENTRY --> READ
READ --> HC
end
subgraph TRINO["trino-repo : trino fork"]
direction TB
SHIM["trino-hudi<br/>thin shim plugin module"]
SERVER["Trino server<br/>ServiceLoader + Plugin SPI<br/>per-plugin
URLClassLoader isolates deps"]
SERVER --> SHIM
end
SHIM -->|Maven dependency on published artifact| PLUGIN
SPI["Trino SPI<br/>provided scope"]
PLUGIN -.->|compiles against| SPI
SHIM --- SPI
```
Key point: `trino-hudi` is intentionally minimal. All connector logic, and
the dependency on Hudi reader APIs, sits in `hudi-trino-plugin` so it never
drifts from Hudi.
## Read path
```mermaid
flowchart TB
Q["Trino query"] --> SM["HudiSplitManager"]
SM -->
SS["HudiSplitSource<br/>NativeTableMetadataFactory:<br/>metadata-table or
filesystem listing"]
SS --> SP["HudiSplit (per file slice)"]
SP --> PSP["HudiPageSourceProvider"]
PSP --> BASE{"split has log files?"}
BASE -->|no| BOPS["HudiBaseFileOnlyPageSource<br/>parquet only, no merge"]
BASE -->|yes| FGR["HoodieFileGroupReader<br/>+
HudiTrinoReaderContext<br/>merge base + log records"]
FGR --> PS["HudiPageSource > SourcePage"]
BOPS --> OUT["Trino pages"]
PS --> OUT
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
