NM. 0.11 KafkaConsumer seems to have added "endOffsets" API! On Thu, Oct 12, 2017 at 3:31 PM, Manan G <manan....@gmail.com> wrote:
> For my use case, I need to figure out the lag within the Java consumer > itself that is consuming some topic. Ideally, the consumer application > would monitor the lag every minute or so and take some action on its own if > consumer falls behind (i.e. spin up more threads to process records - my > use case does not care about record order). For our purpose, it is OK if > lag information is bit stale. > > * AFAIK, Java KafkaConsumer APIs do not seem to expose any information > directly based on which I can figure out the lag within my Java consumer > application. > > * It seems that, alternatively, I can create separate new KafkaConsumer > (or possibly use existing KafkaConsumer my consumer application is using), > seek to end, and call "position()" API to figure out the end offsets for > all partitions I am interested in. Since within my consumer application, I > already know which offset consumer is at, I can figure out the lag. > However, for this one basic information, this solution is bit more involved > for our framework for various reasons. Also, it requires one to either use > separate KafkaConsumer just to figure out lag or possibly re-use same > KafkaConsumer our consumer application is using, but somehow implement the > logic of forwarding to end to find out end offset and resetting back (not > sure if it's feasible yet without issues). > > * If I am not mistaken, looking at KafkaConsumer code itself, "high > watermark" information seems to be already there in FetchResponse > (FetchResponse.PartitionData). It's just that it is not exposed. Is there > any way to retrieve this information somehow in my consumer application? > > In general, wouldn't it be useful for consumer application to have lag > information available with simple API call? > > Thanks, > M > > >
