danny0405 opened a new pull request, #19631: URL: https://github.com/apache/hudi/pull/19631
### Describe the issue this Pull Request addresses Users currently cannot inspect or override Hudi Streamer checkpoints through Spark SQL stored procedures. Setting a checkpoint requires publishing checkpoint metadata on the table timeline, but constructing a minimally configured writer for that empty commit can inadvertently remove metadata-table partitions when their enablement properties are not supplied. This PR adds stored procedures that read and safely update Streamer checkpoints without requiring callers to repeat writer properties. ### Summary and Changelog - Add `get_deltastreamer_checkpoint`, addressable by catalog table name or table path, to return the latest valid V1 or V2 Streamer checkpoint. - Add `set_deltastreamer_checkpoint`, addressable by catalog table name or table path, to publish the requested checkpoint through an empty commit. - Derive writer configuration from catalog and persisted table properties, preserve the existing checkpoint version plus reset/ignore metadata, and match Streamer checkpoint lookup semantics for MOR tables. - Disable cleaning, archival, inline compaction/clustering, and metadata-partition auto-deletion for the metadata-only write. - Register both procedures and add functional coverage for COW and MOR tables, table-name and path access, unchanged table data, and preservation of existing metadata-table partitions. - No code was copied. ### Impact This adds two user-facing Spark SQL stored procedures. Setting a checkpoint adds an empty commit to the table timeline and updates enabled metadata-table partitions when applicable. There are no storage-format, configuration-default, backward-compatibility, or expected performance changes for existing write paths. ### Risk Level medium The setter exercises the standard write client and table timeline to create a metadata-only commit, including metadata-table updates. Risk is mitigated by explicitly disabling unrelated table services and metadata-partition deletion, and by `TestDeltastreamerCheckpointProcedure`, which passes for COW, MOR, path-based invocation, data preservation, and metadata-partition preservation. Validation: - `mvn -pl hudi-spark-datasource/hudi-spark -Dscala-2.12 -Dtest=org.apache.spark.sql.hudi.procedure.TestDeltastreamerCheckpointProcedure -Dsuites=org.apache.spark.sql.hudi.procedure.TestDeltastreamerCheckpointProcedure -Dcheckstyle.skip -Drat.skip test` - Result: 3 tests passed; Checkstyle and Scalastyle passed. - `git diff --check` ### Documentation Update Required. This introduces user-facing stored procedures, so the Hudi website stored-procedure reference should document their syntax and empty-commit behavior. Website sources are not part of this repository checkout and should be updated separately. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
