Github user srdo commented on a diff in the pull request:
https://github.com/apache/storm/pull/1924#discussion_r106685548
--- Diff:
external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/KafkaSpoutConfig.java
---
@@ -268,12 +268,12 @@ private Builder(Builder<?, ?> builder,
SerializableDeserializer<K> keyDes, Class
/**
* The maximum number of records a poll will return.
- * Will only work with Kafka 0.10.0 and above.
+ * This is limited by maxUncommittedOffsets, since it doesn't make
sense to allow larger polls than the spout is allowed to emit.
+ * Please note that when there are retriable tuples on a
partition, maxPollRecords is an upper bound for how far the spout will read
past the last committed offset on that partition.
+ * It is recommended that users set maxUncommittedOffsets and
maxPollRecords to be equal.
--- End diff --
@hmcl I agree that it is not ideal, but there's an issue with letting
maxUncommittedOffsets be larger than maxPollRecords.
From an earlier response:
"If maxPollRecords is less than maxUncommittedOffsets, there's a risk of
the spout getting stuck on some tuples for a while when it is retrying tuples.
Say there are 10 retriable tuples following the last committed offset, and
maxUncommittedOffsets is 10. If maxPollRecords is 5 and the first 5 retriable
tuples are reemitted in the first batch, the next 5 tuples can't be emitted
until (some of) the first 5 are acked. This is because the spout will seek the
consumer back to the last committed offset any time there are failed tuples,
which will lead to it getting the first 5 tuples out of the consumer, checking
that they are emitted, and skipping them. This will repeat until the last
committed offset moves. If there are other partitions with tuples available,
those tuples may get emitted, but the "blocked" partition won't progress until
some tuples are acked on it."
How about we fix this by making doSeekRetriableTopicPartitions seek to the
lowest retriable offset per partition for the partitions with failed tuples,
instead of seeking to the last committed/committable offset? It seems like
seeking to the last committed offset is likely to have some bad interactions
with maxUncommittedOffsets and maxPollRecords.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
