GitHub user voonhous edited a comment on the discussion: Native SQL DDL support for Hudi table creation across engines (Trino, Presto etc.)
Point 1 concerns me a bit. Spark SQL’s **empty-table CREATE TABLE registration** builds metadata and registers it through Spark’s catalog/Hive APIs, while the DataFrame Hive-sync path normally uses `HiveSyncTool`. We already have separate metadata-building paths to keep consistent; introducing another in Trino would increase that maintenance burden. To make the scope precise, this is not a claim that Spark SQL never uses Hive sync. SQL write operations can invoke the shared sync path too. The source evidence for empty-table registration is: 1. [`CreateHoodieTableCommand.run()`](https://github.com/apache/hudi/blob/a7deb61f74263d5fa4d1b71fb32b17d70263ada9/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala#L85-L88) calls `createTableInCatalog(...)`. 2. [`createTableInCatalog()`](https://github.com/apache/hudi/blob/a7deb61f74263d5fa4d1b71fb32b17d70263ada9/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala#L201-L207) selects `createHiveDataSourceTable(...)` when using the Hive catalog. 3. [`createHiveDataSourceTable()`](https://github.com/apache/hudi/blob/a7deb61f74263d5fa4d1b71fb32b17d70263ada9/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableCommand.scala#L226-L233) builds the Hive-compatible catalog metadata, obtains Spark’s Hive client through `HiveClientUtils.getSingletonClientForMetadata(sparkSession)`, and calls `client.createTable(...)` directly. For the SQL-write qualification, [`ProvidesHoodieConfig.buildHiveSyncConfig()`](https://github.com/apache/hudi/blob/a7deb61f74263d5fa4d1b71fb32b17d70263ada9/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/ProvidesHoodieConfig.scala#L441-L454) enables sync when using the Hive catalog, and [`HoodieSparkSqlWriter`](https://github.com/apache/hudi/blob/a7deb61f74263d5fa4d1b71fb32b17d70263ada9/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieSparkSqlWriter.scala#L955-L958) invokes the configured sync tools. Using Trino’s `HiveMetastore` API makes sense given the dependency constraints, but can we first identify and extract the shared schema/property/storage-descriptor construction into an engine-neutral layer? Each integration could then translate that representation into its own API types, with tests verifying equivalent metastore semantics. I’d prefer a focused refactor here over another independently maintained implementation. GitHub link: https://github.com/apache/hudi/discussions/19484#discussioncomment-18316365 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
