Thanks a lot James for the detailed article. This is exactly what I was
looking for!
This confirm me to use the seekToEnd method (option #5 in your post), as
I'm using Kafka 0.10.
Jean-Baptiste
On 16-07-13 05:45 PM, James Cheng wrote:
Jean-Baptiste,
I wrote a blog post recently on this exact subject.
https://logallthethings.com/2016/06/28/how-to-read-to-the-end-of-a-kafka-topic/
Let me know if you find it useful.
-James
Sent from my iPhone
On Jul 13, 2016, at 7:16 AM, [email protected] wrote:
Hi,
I'm using a compacted Kafka Topic to save the state of my application. When the
application crashes/restarts I can restore its state by reading the Kafka topic.
However I need to read it completely, especially up to most recent record, to
be sure to restore all data.
Is there a standard way to to that? I've checked the Kafka streams code and
found that the class ProcessorStateManager seems to be doing something similar.
It first gets the last offset by doing:
// calculate the end offset of the partition // TODO: this is a bit hacky to
first seek then position to get the end offset
restoreConsumer.seekToEnd(singleton(storePartition));
long endOffset = restoreConsumer.position(storePartition);
Then it polls the records until reaching the endoffset (there is also an other
limit but I think it is related to an other use case).
I guess it works, but the TODO message makes me wonder if it is a good solution
and if it will continue to work in future releases.
Thanks for your help,
Jean-Baptiste
