rangareddy commented on issue #8522: URL: https://github.com/apache/hudi/issues/8522#issuecomment-5149492725
This has been addressed — closing. **Fixed by:** - [HUDI-6191](https://issues.apache.org/jira/browse/HUDI-6191) — Closed/Fixed, fixVersions **0.16.0** and **1.1.0** - #11686 (code, merged 2024-07-26) and #11690 (docs, merged 2024-07-27) **What changed.** You can now pass a bare offset instead of hand-building the `topic,partition:offset` string, by setting the checkpoint type: --checkpoint 106600 --hoodie-conf hoodie.streamer.source.kafka.checkpoint.type=single_offset Internally `KafkaOffsetGen` performs exactly the rewrite you were doing manually (`topicName + ",0:" + offset`). It applies to single-partition topics only, and raises a clear error if the topic has more than one partition, where a single offset would be ambiguous. The original explicit form still works and remains the right choice for multi-partition topics: --checkpoint "users.public.transactions,0:106600" **Docs.** The checkpointing section now documents all three checkpoint types and their formats: - https://hudi.apache.org/docs/hoodie_streaming_ingestion#checkpointing - https://hudi.apache.org/docs/configurations#Kafka-Source-Configs One gap remains from the original report: those pages describe the format for Kafka sources generally, but don't call out Debezium sources by name. Since the Debezium sources are Kafka sources the format applies directly — but if a Debezium-specific example would still have saved you the search, a follow-up docs issue for that would be welcome. Thanks @sydneyhoran for writing this up so precisely, and @wombatu-kun for the implementation and doc fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
