Stig Rohde Døssing created STORM-2914:
-----------------------------------------
Summary: Remove enable.auto.commit support from storm-kafka-client
Key: STORM-2914
URL: https://issues.apache.org/jira/browse/STORM-2914
Project: Apache Storm
Issue Type: Improvement
Components: storm-kafka-client
Affects Versions: 2.0.0, 1.2.0
Reporter: Stig Rohde Døssing
Assignee: Stig Rohde Døssing
The enable.auto.commit option causes the KafkaConsumer to periodically commit
the latest offsets it has returned from poll(). It is convenient for use cases
where messages are polled from Kafka and processed synchronously, in a loop.
Due to https://issues.apache.org/jira/browse/STORM-2913 we'd really like to
store some metadata in Kafka when the spout commits. This is not possible with
enable.auto.commit. I took at look at what that setting actually does, and it
just causes the KafkaConsumer to call commitAsync during poll (and during a few
other operations, e.g. close and assign) with some interval.
Ideally I'd like to get rid of ProcessingGuarantee.NONE, since I think
ProcessingGuarantee.AT_MOST_ONCE covers the same use cases, and is likely
almost as fast. The primary difference between them is that AT_MOST_ONCE
commits synchronously.
If we really want to keep ProcessingGuarantee.NONE, I think we should make our
ProcessingGuarantee.NONE setting cause the spout to call commitAsync before
poll, reassignment and close, and never use the enable.auto.commit option. This
allows us to include metadata in the commit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
