Jean-Baptiste, I wrote a blog post recently on this exact subject.
https://logallthethings.com/2016/06/28/how-to-read-to-the-end-of-a-kafka-topic/ Let me know if you find it useful. -James Sent from my iPhone > On Jul 13, 2016, at 7:16 AM, [email protected] wrote: > > Hi, > > > I'm using a compacted Kafka Topic to save the state of my application. When > the application crashes/restarts I can restore its state by reading the Kafka > topic. > > > > However I need to read it completely, especially up to most recent record, to > be sure to restore all data. > > > > Is there a standard way to to that? I've checked the Kafka streams code and > found that the class ProcessorStateManager seems to be doing something > similar. > > > > It first gets the last offset by doing: > > // calculate the end offset of the partition // TODO: this is a bit hacky to > first seek then position to get the end offset > > restoreConsumer.seekToEnd(singleton(storePartition)); > > long endOffset = restoreConsumer.position(storePartition); > > > > Then it polls the records until reaching the endoffset (there is also an > other limit but I think it is related to an other use case). > > > > I guess it works, but the TODO message makes me wonder if it is a good > solution and if it will continue to work in future releases. > > > > Thanks for your help, > > > > Jean-Baptiste > > > > >
