[
https://issues.apache.org/jira/browse/STORM-2914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351447#comment-16351447
]
Stig Rohde Døssing commented on STORM-2914:
-------------------------------------------
[~hmclouro] [~kabhwan]
I had time to do some tests. I've added a storm-perf topoloy that uses
storm-kafka-client instead of storm-kafka, here
[https://github.com/apache/storm/pull/2545.]
I ran the topology from the mentioned PR for 10 minutes, with NONE and
AT_MOST_ONCE against a Kafka instance with small messages preloaded
("testMessage"). Because the commitSync call in AT_MOST_ONCE is blocking and
thus likely affected by latency to Kafka, I tried running the topology both
against a local Kafka node, and against one set up in an Azure VM.
I've attached the raw results. [^storm-kafka-modes.ods] . Here's the averages
for spout acks/s for each test:
* NONE local: 124778.6
* AT_MOST_ONCE local: 120226.6
* NONE Azure (25-40 ms latency): 34593.6
* AT_MOST_ONCE Azure: 10374.8
As you can see it's not too expensive to use AT_MOST_ONCE locally instead of
NONE, but once latency is introduced, it looks much cheaper to use NONE if
possible. I'm a little surprised the difference is so large, I would have
expected something more like 50% throughput for AT_MOST_ONCE compared to NONE
due to doubling the number of blocking calls in nextTuple when polling.
I'm not too experienced with benchmarking, so it's possible that I've made a
mistake somewhere, or this benchmark may be too synthetic to really show
anything.
For now I think we should keep NONE. I'll do as Hugo suggests and rename it and
mark it unstable. I agree with Hugo's list, but we should consider waiting to
hide committing/loading commits behind an interface until we actually need to
store commits outside Kafka. Once we know what a second implementation needs,
it'll probably be easier to make a good generic interface, rather than one that
only fits our current use case, and I also would prefer not to add pluggability
if we end up not needing it.
> Remove enable.auto.commit support from storm-kafka-client
> ---------------------------------------------------------
>
> Key: STORM-2914
> URL: https://issues.apache.org/jira/browse/STORM-2914
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka-client
> Affects Versions: 2.0.0, 1.2.0
> Reporter: Stig Rohde Døssing
> Assignee: Stig Rohde Døssing
> Priority: Major
> Labels: pull-request-available
> Attachments: storm-kafka-modes.ods
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The enable.auto.commit option causes the KafkaConsumer to periodically commit
> the latest offsets it has returned from poll(). It is convenient for use
> cases where messages are polled from Kafka and processed synchronously, in a
> loop.
> Due to https://issues.apache.org/jira/browse/STORM-2913 we'd really like to
> store some metadata in Kafka when the spout commits. This is not possible
> with enable.auto.commit. I took at look at what that setting actually does,
> and it just causes the KafkaConsumer to call commitAsync during poll (and
> during a few other operations, e.g. close and assign) with some interval.
> Ideally I'd like to get rid of ProcessingGuarantee.NONE, since I think
> ProcessingGuarantee.AT_MOST_ONCE covers the same use cases, and is likely
> almost as fast. The primary difference between them is that AT_MOST_ONCE
> commits synchronously.
> If we really want to keep ProcessingGuarantee.NONE, I think we should make
> our ProcessingGuarantee.NONE setting cause the spout to call commitAsync
> after poll, and never use the enable.auto.commit option. This allows us to
> include metadata in the commit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
