nsivabalan commented on issue #20401:
URL: https://github.com/apache/hudi/issues/20401#issuecomment-6375889636
**Progress update on Phase 1**
- **Step 1: #20412 → #20413 → #20415** (#20400). With @danny0405 to land.
- **Step 2: #20423** (#20369). Reviewed. Two gaps on the new append path:
- Options passed with `.option(...)` are dropped: they never reach
`InsertIntoHoodieTableCommand.run` (`extraOptions` is not passed). For example,
`.option("hoodie.spark.sql.insert.into.operation", "upsert")` is ignored, and a
row whose key already exists lands as a duplicate.
- A `.partitionBy(...)` that differs from the existing table is silently
ignored. Spark rejects this for non-Hudi tables.
Both are small fixes; we'll follow up on them.
- **Step 3: #20371, the part Phase 1 needs.** Raised as #20426:
- The `Column[]` / `TableInfo` `createTable` path (used by SDP on Spark
4.1) now creates an initialized Hudi table.
- Re-sent properties with unchanged values are accepted, so repeated
pipeline runs no longer fail.
- Changes to properties persisted in `hoodie.properties` are rejected with
a clear error.
Tested on Spark 4.1, and it compiles on Spark 3.3. @danny0405, please take
it forward. The rest of #20371 stays with Phase 2.
- **Steps 4–6: #20370, #20403, #20402.** No PRs yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
