JiYeon OH created FLUME-3086:
--------------------------------
Summary: KafkaChannel commits consume offset to Zookeeper every
time causing performance problem
Key: FLUME-3086
URL: https://issues.apache.org/jira/browse/FLUME-3086
Project: Flume
Issue Type: Bug
Components: Kafka Channel
Affects Versions: 1.7.0
Reporter: JiYeon OH
Currently, the AUTO_COMMIT_ENABLED setting is hard corded to
false(KafkaChannel.java line 285), and the commit is done every
transaction(KafkaChannel.java line 567). This will cause large overhead to
Kafka Zookeeper when consuming very large amount of data because every consumer
in every node will be writing to Zookeeper.
I actually had this problem and affected other running consumers' offsets on
the Zookeeper. So I suggest to change the hard coded setting
AUTO_COMMIT_ENABLED to true, and set a default AUTO_COMMIT_INTERVAL value, and
remove the commitOffsets() method.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
