[
https://issues.apache.org/jira/browse/FLUME-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15973827#comment-15973827
]
JiYeon OH commented on FLUME-3086:
----------------------------------
Sorry I misunderstood delivery guarantees. I'll recheck my config and test
again.
Thank you for the reply! I'll share if any issue occurs after config change.
> KafkaChannel commits consume offset to Zookeeper every time causing
> performance problem
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-3086
> URL: https://issues.apache.org/jira/browse/FLUME-3086
> Project: Flume
> Issue Type: Bug
> Components: Kafka Channel
> Affects Versions: 1.7.0
> Reporter: JiYeon OH
>
> Currently, the AUTO_COMMIT_ENABLED setting is hard corded to
> false(KafkaChannel.java line 285), and the commit is done every
> transaction(KafkaChannel.java line 567). This will cause large overhead to
> Kafka Zookeeper when consuming very large amount of data because every
> consumer in every node will be writing to Zookeeper.
> I actually had this problem and affected other running consumers' offsets on
> the Zookeeper. So I suggest to change the hard coded setting
> AUTO_COMMIT_ENABLED to true, and set a default AUTO_COMMIT_INTERVAL value,
> and remove the commitOffsets() method.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
