hudi-agent commented on code in PR #19573: URL: https://github.com/apache/hudi/pull/19573#discussion_r3750266182
########## website/docs/hoodie_streaming_ingestion.md: ########## @@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`. +Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table +version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: - `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. +#### Resetting the checkpoint for the Hudi incremental source + +When Hudi Streamer writes a target table at table version 8 or higher using `HoodieIncrSource`, it tracks progress by +**completion time** instead of requested instant time, and records it in the commit metadata under +`streamer.checkpoint.key.v2`. A bare timestamp would be ambiguous between the two, so `--checkpoint` has to state which +one it is. This form is required from Hudi 1.0.1 onward; on 1.0.0 the prefixes are not recognised and `--checkpoint` +takes a bare completion time. + +```shell Review Comment: 🤖 This section could clarify the 1.0.0 behavior: it says on 1.0.0 `--checkpoint` "takes a bare completion time," but the PR's own reproduction states the bare value fed to every version was the instant's *requested* time, and that value was accepted on 1.0.0. That suggests 1.0.0 interprets a bare `--checkpoint` as a requested instant time (consistent with #12718's note that 1.x incremental sources were still request-time based), not a completion time. Could you double-check whether 1.0.0 treats the bare value as requested or completion time? @yihua could you confirm the pre-1.0.1 checkpoint semantics here? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> ########## website/docs/hoodie_streaming_ingestion.md: ########## @@ -312,14 +312,54 @@ Read more in depth about concurrency control in the [concurrency control concept Hudi Streamer uses checkpoints to keep track of what data has been read already so it can resume without needing to reprocess all data. When using a Kafka source, the checkpoint is the [Kafka Offset](https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management) When using a DFS source, the checkpoint is the 'last modified' timestamp of the latest file read. -Checkpoints are saved in the .hoodie commit file as `streamer.checkpoint.key`. +Checkpoints are saved in the .hoodie commit file, under `streamer.checkpoint.key.v2` for tables at table +version 8 or higher and `deltastreamer.checkpoint.key` below that. If you need to change the checkpoints for reprocessing or replaying data you can use the following options: - `--checkpoint` will set `streamer.checkpoint.reset_key` in the commit file to overwrite the current checkpoint. Format of checkpoint depends on [KAFKA_CHECKPOINT_TYPE](configurations.md#hoodiestreamersourcekafkacheckpointtype). By default (for type `string`), checkpoint should be provided as: `topicName,0:offset0,1:offset1,2:offset2`. For type `timestamp`, checkpoint should be provided as long value of desired timestamp. For type `single_offset`, we assume that topic consists of a single partition, so checkpoint should be provided as long value of desired offset. - `--source-limit` will set a maximum amount of data to read from the source. For DFS sources, this is max # of bytes read. For Kafka, this is the max # of events to read. Review Comment: 🤖 It might help to reconcile the key names here. The existing bullet says `--checkpoint` sets `streamer.checkpoint.reset_key`, while the new subsection says the value is recorded under `streamer.checkpoint.reset.key.v2` for table version 8+. A reader inspecting commit metadata may be unsure which key applies to their table. Consider noting that `reset.key.v2` is used for table version 8+ and the older `reset_key` for versions below that, mirroring the `streamer.checkpoint.key.v2` vs `deltastreamer.checkpoint.key` distinction you already added just above. <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
