[
https://issues.apache.org/jira/browse/FLUME-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969967#comment-15969967
]
JiYeon OH commented on FLUME-3086:
----------------------------------
Oh I missed the point that auto commit cannot ensure delivery. Thank you for
reminding me. I see why the AUTO_COMMIT_ENABLED was hard coded to false and I
think that it should stay the same.
How about having another parameter that can control the frequency of calling
commitOffsets() method? It can be set to default 1 which will commit offset
every transaction. And it can be set to some integer, for say 10, which will
commit offset every 10 transaction.
> KafkaChannel commits consume offset to Zookeeper every time causing
> performance problem
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-3086
> URL: https://issues.apache.org/jira/browse/FLUME-3086
> Project: Flume
> Issue Type: Bug
> Components: Kafka Channel
> Affects Versions: 1.7.0
> Reporter: JiYeon OH
>
> Currently, the AUTO_COMMIT_ENABLED setting is hard corded to
> false(KafkaChannel.java line 285), and the commit is done every
> transaction(KafkaChannel.java line 567). This will cause large overhead to
> Kafka Zookeeper when consuming very large amount of data because every
> consumer in every node will be writing to Zookeeper.
> I actually had this problem and affected other running consumers' offsets on
> the Zookeeper. So I suggest to change the hard coded setting
> AUTO_COMMIT_ENABLED to true, and set a default AUTO_COMMIT_INTERVAL value,
> and remove the commitOffsets() method.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
