voonhous opened a new issue, #19640: URL: https://github.com/apache/hudi/issues/19640
### Problem `hudi-trino` pins a released Trino via `<trino.version>` (481 today, while Trino is at 483), and the nightly SPI compatibility job is red against trino master (#19379). Because Trino's SPI and internal libraries move every release, pin-to-pin bumps arrive as multi-release big-bang migrations (472 -> 481 -> ...), each one a large, risky PR. ### Proposal Track trinodb/trino master directly on the Hudi master branch: - The root pom pins an exact trinodb/trino commit via two properties that advance only together: `<trino.version>` (trino master's project version, e.g. `484-SNAPSHOT`) and a new `<trino.sha>`. - Trino publishes no SNAPSHOT artifacts, so all `io.trino` dependencies are built from source at the pinned commit and installed into the local m2 by a new `scripts/trino/bootstrap_trino.sh` (one command; used identically by contributors and CI). - `hudi_trino_ci.yml` restores the built trino artifacts from an actions cache keyed on `trino.sha`; only a pin advance pays a source build (the nightly job pre-seeds the cache). - The nightly compat job (the one behind #19379) is promoted from drift alarm to pin advancement: when hudi-trino compiles against trino HEAD, it saves the cache and force-pushes a `bot/trino-pin` PR bumping the pin (human-merged). On failure it keeps filing the drift issue as today. SPI fixups therefore land continuously in small deltas. - E2E keeps using the latest released `trinodb/trino` server image (new `<trino.e2e.version>` property, also feeding `trino-jdbc` for integ tests) and auto-skips with a loud notice when the pinned SHA has SPI-relevant drift relative to that release (detected via the GitHub compare API). There is no published master server image; building one from source per PR is not worth the cost. ### Accepted costs - Master's `hudi-trino` no longer resolves `io.trino` from Maven Central; the bootstrap script is a build prerequisite (~10-30 min per pin advance, cached afterwards). - E2E goes dark on master during SPI-drift windows (bounded by release-branch E2E; can be revisited with a weekly from-source server image if windows prove long). - Release branches still pin a released Trino (published artifacts cannot reference unreleased SPI); the release guide gains a pin-back subsection. ### Included in the first PR - Bootstrap script + CI/compat/E2E workflow rework + pin properties. - The current SPI fixup backlog (481 -> pinned master commit): split SPI rewrite (`ConnectorSplitBatch` removal, `getNextBatch(int, DynamicFilterSnapshot)`, `getSplits` with `Set<ColumnHandle>`), `createPageSource` with `ConnectorTableCredentials`, `TypeSignature` -> `TypeDescriptor`, blob-cache SPI (`CacheKeyProvider` returning `CacheKey`), `Hive3MinioDataLake` -> `Hive3FlociDataLake`/Floci test migration, new `jts-core` provided dependency. Relates to #19379 (that issue stays as the drift-report sink; this one covers the model change). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
