Hello, I'm having trouble with commitAsync() not updating my partition offsets. I pre-populate 1,000,000 strings on topic 'lowercaseStrings' across 4 partitions. I check the offsets and they're fine--250K records/partition. Then I run some code that uses a single-thread with blocking queue to communicate with the (multi-threaded) outside world. This code does a poll and populates an iterator, which is bled dry by the caller before another is requested. For now I individually commit each and every ConsumerRecord I process until I understand how this works... I'll figure out batched commits later. I can confirm that I pull and process 1 million individual records. I can also confirm my OffsetCommitCallback is called 1 million times, 250K times for each of my 4 partitions. Exception is null in the callback for each of these times. Everything appears in order Now when I look at the partition offsets I see they're all over the place, typically with two having around 16K lag behind head offset, and the others a scattered few lag behind offset. That's not good. For grins I added long delays after my test run (before shutting everything down), to let things settle down--thinking maybe something still had to catch up, but this didn't change behavior. I'm also perplexed why the callback's numbers all appeared as expected but not the offsets. I would have expected these to match (i.e. callback not called until actual commit happens). Below is a gist of my consumer thread. Nothing fancy. Any thoughts or ideas appreciated.Greg Kafka 0.9.0.1 Consumer
| | | | | | | | | | | Kafka 0.9.0.1 Consumer Kafka 0.9.0.1 Consumer | | | |
