Github user jianbzhou commented on the pull request:
https://github.com/apache/storm/pull/1131#issuecomment-217781379
Just think another scenario â for example we polled 10k message and
emitted â say from 10001 to 20000, so the numUncommittedOffsets is 10000, if
the 10500 msg failed, which caused all the following message 10501~20000 will
not got commited until the 10500 message was reemitted and acked.
If a rebalance happened during this time, the offset will seek back to the
last commited offiset +1, possibly will seek back to offset 10001, then all the
message from 10001 to 2000 will be polled and emitted again â which will
cause numUncommittedOffsets be incremented by another 10000 again.
After successfully commit to kafka, numUncommittedOffsets will substract
10000 and leave 10000 value there.
Seems this will also gradually cause numUncommittedOffsets be a bigger
value than we expect, Is this a possible scenario?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
