GitHub user voonhous added a comment to the discussion: Native SQL DDL support for Hudi table creation across engines (Trino, Presto etc.)
Thanks for the ping @danny0405, and thanks for writing this up @OjashKush -- the gap is real and your description of it is accurate. Trino has no `CREATE TABLE` path for Hudi today, and Danny's read of the connector is correct: `HudiMetadata` implements read-only `ConnectorMetadata`, there's no page sink, and `HudiTableProperties` exposes only `location` and `partitioned_by`. **Existing tracking** This is already filed as apache/hudi#15527 (from HUDI-5115) and upstream as trinodb/trino#14433. Both have been open since 2022 with no implementation attached. Worth linking this discussion to them so the context lives in one place rather than forking a third thread. **Unblocking you in the meantime** Separate from the long-term design, you probably don't need Spark for the specific thing you described. Table creation is engine-agnostic already: - `HoodieTableMetaClient.newTableBuilder().setTableType(..).setTableName(..).setRecordKeyFields(..).setPartitionFields(..).setTableCreateSchema(..).initTable(storageConf, basePath)` is pure `hudi-common`. No Spark engine, no session. That's roughly 15 lines in any JVM process, and it's exactly what `hudi-cli`'s `create` command calls. (Caveat so you're not surprised: the shipped `hudi-cli-with-bundle.sh` wrapper still puts Spark jars on the classpath, so if you want a genuinely Spark-free binary today, the small JVM shim is the cleaner route.) - HMS registration is `run_sync_tool.sh`, which is a plain `java -cp ... org.apache.hudi.hive.HiveSyncTool` invocation. Hive and Hadoop jars only, no Spark. That is not a substitute for native DDL and I'm not offering it as one. But if the immediate pain is "we have to stand up Spark just to bootstrap a table," those two steps remove it today, and it also lets you validate the property and schema mapping you'd eventually want Trino to emit. **The technical map, if someone picks this up** Sharing this in full because the shape of the work is not obvious from the outside, and whoever drives it shouldn't have to rediscover it. Context that changes the calculus: the connector source moved into `apache/hudi` as the `hudi-trino` module (RFC-105, #18837, merged 2026-07-26), with a thin shim on the Trino side. DDL work now lands in this repo on Hudi's cadence rather than being gated on a Trino release plus trinodb's bar for turning a read-only connector writable. That was the main structural reason #14433 sat for three years. Sensible staging: 0. **`register_table` / `unregister_table` procedures.** The Iceberg connector ships `system.register_table`. Table already exists on storage, you just want it queryable without running Hive Sync separately. Smallest useful change, and it covers a real slice of the bootstrap pain on its own. 1. **Empty `CREATE TABLE` + `DROP TABLE`.** Metaclient init plus catalog registration. 2. **`INSERT` / `CTAS` / `MERGE`.** A different order of magnitude, see below. Non-obvious parts of stage 1: - **Two schemas, not one.** `HudiMetadata#getColumnHandles` derives columns from the HMS table via `hiveColumnHandles(table, ...)`, not from the Hudi schema. So `createTable()` must write a correct Avro schema into `hoodie.table.create.schema` **and** a consistent HMS storage descriptor. #9435 is what happens when those diverge: a table registered with no HMS columns, and Trino returns `SELECT * not allowed from relation that has no columns`. Good news is `hudi-hive-sync` is already a compile-scope dependency of `hudi-trino`, so reusing `HiveSyncTool` / `HiveSchemaUtil` for the catalog side is the natural path and makes Trino-created tables register identically to Spark/Flink-synced ones. - **Trino type -> Hudi type is a new direction.** Today the connector only maps Hive/Hudi -> Trino. The ambiguous cases need explicit decisions: `TIMESTAMP(6)` vs `(3)`, `VARCHAR(n)`/`CHAR`, `DECIMAL` fixed vs bytes, `UUID`, nullability inside `ROW`/`ARRAY`/`MAP`, and the newer custom types. None of these fail at `CREATE TABLE`. They fail later, when another engine writes into the table. #19457 (pushdown on a `float` -> `double` evolved column) is a live example of how cheaply these assumptions drift. - **Pin the created table version deliberately.** Whatever `hoodie.table.version` gets initialized has to be one the connector can still read after someone else writes to it. Concretely: the connector cannot read RFC-103 native log blocks today (`HudiTrinoIOFactory#getFileFormatUtils` throws). Creating an empty MOR table at v10 from Trino would succeed, and then the first Spark or Flink delta write produces logs Trino can't read. Stage 1 should pin the version or restrict to COW until that closes. - **Small property allowlist plus an escape hatch.** Hudi has hundreds of table configs. Enumerate only what must be right at init (`table_type`, `primary_key`, `precombine_field`, `partitioned_by`, `location`, key generator, hive-style partitioning, record merge mode) and add a passthrough map for arbitrary `hoodie.*` instead of growing a bespoke Trino property per config. Aliases should match what Spark SQL accepts (`HoodieOptionConfig`: `primaryKey`, `preCombineField`, `type`) so the same table doesn't describe itself differently depending on where you run `SHOW CREATE TABLE`. - **Failure ordering.** On Danny's rollback point: init `.hoodie` first, then register. On registration failure, clean up the base path only if the connector created it, never a pre-existing directory. **On your shared-abstraction question** Yes, and most of it exists. `HoodieTableMetaClient.newTableBuilder()` + `HoodieTableConfig` is already the engine-neutral sink, and Spark's `HoodieCatalog` and Flink's `HoodieHiveCatalog` both funnel into it. What's missing is the layer directly above: **DDL options -> validated `HoodieTableConfig`**. Spark's lives in `HoodieOptionConfig` (`hudi-spark-common`, Scala), Flink's in `hudi-flink`, and neither is reusable. Lifting that normalization and validation into `hudi-common`, and standardising on `hudi-hive-sync` for registration, is what would make a Trino `createTable()` genuinely thin. Worth saying plainly: adding `createTable()` to Trino *alone*, without that, would add a fourth dialect rather than fix the consistency problem you describe. **On RFC scoping** I'd split it three ways instead of one, because they have very different costs: - `register_table` + empty `CREATE`/`DROP`: **no RFC needed.** A design sketch on #15527 and a PR against `hudi-trino` is enough. - Shared DDL normalization in `hudi-common`: **short RFC or design doc**, since it moves shared code and can change Spark and Flink behavior. - `INSERT` / `CTAS` / `MERGE` from Trino: **RFC, definitely.** This makes Trino a second full writer, so commit and rollback, markers, index selection, MDT updates, OCC and locking, key generation, precombine, file sizing, and who runs compaction/clean/clustering all need answers. The mechanics aren't hypothetical -- `hudi-trino` already uses `HoodieJavaWriteClient` at test scope to build fixtures -- but mapping Trino's sink model (workers emit fragments, coordinator commits) onto a Hudi commit needs deliberate design, and `hudi-java-client`'s index and table-service coverage relative to the Spark client is the real open question. Follow-up on process and next steps in a separate comment, to keep this one focused on the technical shape. GitHub link: https://github.com/apache/hudi/discussions/19484#discussioncomment-17904451 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
