[
https://issues.apache.org/jira/browse/FLUME-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969211#comment-15969211
]
Jeff Holoman commented on FLUME-3086:
-------------------------------------
We expect folks to be using Kafka for offset storage, and using auto-commit on
the channel would break our at least once guarantees as auto commit cannot
ensure delivery.
> KafkaChannel commits consume offset to Zookeeper every time causing
> performance problem
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-3086
> URL: https://issues.apache.org/jira/browse/FLUME-3086
> Project: Flume
> Issue Type: Bug
> Components: Kafka Channel
> Affects Versions: 1.7.0
> Reporter: JiYeon OH
>
> Currently, the AUTO_COMMIT_ENABLED setting is hard corded to
> false(KafkaChannel.java line 285), and the commit is done every
> transaction(KafkaChannel.java line 567). This will cause large overhead to
> Kafka Zookeeper when consuming very large amount of data because every
> consumer in every node will be writing to Zookeeper.
> I actually had this problem and affected other running consumers' offsets on
> the Zookeeper. So I suggest to change the hard coded setting
> AUTO_COMMIT_ENABLED to true, and set a default AUTO_COMMIT_INTERVAL value,
> and remove the commitOffsets() method.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
