Github user srdo commented on the issue:
https://github.com/apache/storm/pull/1679
@apiwoni Your expectation for 1.1.3 is wrong. seekToEnd doesn't seek to the
last committed offset, it seeks to the last offset. One of the fixes made in
this PR is to make doSeekRetriablePartitions seek to the last committed offset
instead of the end of the partition when findNextCommitOffset returns null.
Here's the null case code from 1.0.4
https://github.com/apache/storm/blob/v1.0.4/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpout.java#L283.
Note that the comment in that line is wrong, and it's actually seeking to the
end of the partition.
Here's the code for the same case after this PR
https://github.com/apache/storm/pull/1679/files#diff-7d7cbc8f5444fa7ada7962033fc31c5eR269.
Note that it now matches the comment.
Your understanding of OffsetResetStrategy.LATEST is also slightly off. The
OffsetResetStrategy describes the behavior of the KafkaConsumer when the
consumer requests an offset that is out of range. Offsets are out of range if
they are smaller than the beginning of the log (e.g. if parts of the partition
were deleted), or if they are larger than the offset of the largest message in
the partition. For example if you have a partition with offsets 10-500 present
and the consumer tries to seek to offset 0, you will trigger the
OffsetResetStrategy. The strategy has no effect on seekToEnd. seekToEnd always
seeks to the last offset on the partition, barring some additions made in
0.11.0.0 that have to do with transactional writes.
I think what happened in your case is what I described above, you hit one
of the bugs solved by this PR, and as a result your consumer jumped to the end
of the partition, which caused it to emit no messages.
---
