mahsoodebrahim opened a new issue, #19577:
URL: https://github.com/apache/hudi/issues/19577
### Feature Description
**What the feature achieves:**
Adds two new hudi-cli commands (`checkpoint get` and `checkpoint set`) that
allow operators to inspect and modify the ingestion checkpoint value stored in
a Hudi table's commit metadata — without manually parsing or editing commit
metadata JSON files.
- `checkpoint get` — Reads the latest ingestion instant from the active
timeline and displays the checkpoint value along with commit statistics
(timestamp, action, state, records/bytes/partitions/files written).
- `checkpoint set` — Safely overwrites the checkpoint value in the latest
completed ingestion commit's metadata. Supports both DeltaStreamer checkpoints
(default) and arbitrary custom checkpoint keys (e.g.
`latest-ingestion-timestamp`). Includes safety guardrails: requires explicit
`--force true` confirmation, backs up the original commit file to
`.checkpoint_backup/` before modification, verifies the write by re-reading the
timeline, and preserves the backup for manual recovery on failure.
**Why this feature is needed:**
When a Hudi ingestion pipeline (DeltaStreamer or custom) has a stuck,
corrupted, or incorrect checkpoint, operators currently have no built-in way to
inspect or correct it. The only option is to manually locate and edit commit
metadata JSON files on HDFS/cloud storage, which is error-prone and risky
during incident response. Common operational scenario include rewinding a
pipeline's checkpoint after a bad ingestion, fast-forwarding past a problematic
offset, or correcting a custom checkpoint key.
### User Experience
**How users will use this feature:**
- Configuration changes needed: None. No new configs or table properties are
introduced. The commands operate on the existing commit metadata format.
- API changes:
- New hudi-cli commands: `checkpoint get`, `checkpoint set`
- For Hudi 1.x, these could additionally be implemented as Spark SQL
stored procedures (e.g. `CALL checkpoint_get(basepath)` / `CALL
checkpoint_set(basepath, value, ...)`) consistent with the 1.x tooling
direction.
- Key parameters for `checkpoint set`:
| Parameter | Description | Default |
|---|---|---|
| `--checkpointValue` | The new checkpoint value to set | (required) |
| `--isDeltaStreamer` | Whether to target the DeltaStreamer checkpoint
key | `true` |
| `--customCheckpointKey` | Custom checkpoint key name (used when
`isDeltaStreamer=false`) | `""` |
| `--force` | Bypass the safety confirmation warning | `false` |
- Usage examples:
```bash
# Connect to table
connect --path hdfs:///path/to/table
# Inspect current checkpoint
checkpoint get
# Set DeltaStreamer checkpoint
checkpoint set --checkpointValue 20251021164840000 --force true
# Set a custom checkpoint key
checkpoint set --checkpointValue 20251021164840000 --isDeltaStreamer false
--customCheckpointKey latest-ingestion-timestamp --force true
### Hudi RFC Requirements
RFC PR link: N/A
Why RFC is/isn't needed:
- Does this change public interfaces/APIs? No — adds new CLI commands only;
no changes to existing APIs, writer paths, or reader paths.
- Does this change storage format? No — reads and rewrites existing commit
metadata files using the existing HoodieCommitMetadata serialization format. No
new file formats, table properties, or timeline changes.
- Justification: This is a self-contained operational CLI tool (two new
commands) that operates on existing commit metadata. All key dependencies
(TimelineUtils.getLatestIngestionInstant(),
StreamSync.getLatestInstantAndCommitMetadataWithValidCheckpointInfo(),
HoodieStreamer.CHECKPOINT_KEY) already exist in OSS. No RFC is needed as there
are no architectural, format, or API compatibility implications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
