nsivabalan opened a new issue, #20369:
URL: https://github.com/apache/hudi/issues/20369
### Bug Description
With `HoodieCatalog` configured as `spark_catalog`,
`df.write.format("hudi").mode("append").saveAsTable("db.t")` against a table
that already exists fails with *"Can not create the managed table … The
associated location already exists"*.
Root cause: `HoodieCatalog` is a `CatalogExtension`, so
`DataFrameWriter.saveAsTableCommand` takes the V1 route
(`saveAsV1TableCommand`), which plans a `CreateTable(..., SaveMode.Append)`.
Hudi's analysis rules rewrite that to `CreateHoodieTableAsSelectCommand`. Its
`run` handles an existing table for `ErrorIfExists` (throws) and `Ignore`
(returns) and asserts `mode != Overwrite`, but for **`Append` it falls
through** to `HoodieCatalogTable(...).initHoodieTable()`, which treats the
table as new and rejects the existing location
(`HoodieCatalogTable.parseSchemaAndConfigs`, `tableExists = false` branch).
Spark's own `CreateDataSourceTableAsSelectCommand` treats *exists + Append* as
an insert into the existing table.
Impact: any DataFrame-API append-by-name onto an existing Hudi table under
`HoodieCatalog`. This is also the exact call Spark 4.1 Declarative Pipelines
issues for materialized-view refresh (`BatchTableWrite` → `saveAsTable`,
RFC-112 / #19997), so it blocks that integration.
Expected: behave like `INSERT INTO` on the existing table (resolve to
`InsertIntoHoodieTableCommand` with the table's persisted config).
### Environment
Hudi 1.2.1 (`hudi-spark4.1-bundle_2.13`, Maven Central), Spark 4.1.1 (Spark
Connect server),
`spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog`,
`HoodieSparkSessionExtension`, local filesystem, macOS. Behaviour
cross-checked against current `master` source.
### Steps to reproduce
```scala
spark.sql("CREATE TABLE default.t (id STRING, v INT) USING hudi
TBLPROPERTIES (primaryKey='id')")
Seq(("a", 1)).toDF("id",
"v").write.format("hudi").mode("append").saveAsTable("default.t")
```
### Logs and Stack Trace
```
org.apache.spark.sql.hudi.command.exception.HoodieAnalysisException: Can not
create the managed table('spark_catalog.default.rate_by_bucket'). The
associated location('file:/…/warehouse/rate_by_bucket') already exists.
at
org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable.parseSchemaAndConfigs(HoodieCatalogTable.scala:277)
at
org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable.initHoodieTable(HoodieCatalogTable.scala:202)
at
org.apache.spark.sql.hudi.command.CreateHoodieTableAsSelectCommand.run(CreateHoodieTableAsSelectCommand.scala:101)
at
org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:117)
…
at
org.apache.spark.sql.classic.DataFrameWriter.saveAsTable(DataFrameWriter.scala:432)
```
Code pointers:
`hudi-spark-datasource/hudi-spark4-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableAsSelectCommand.scala`
(`run`, the `tableExists` block), `HoodieCatalogTable.scala:277`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
