[
https://issues.apache.org/jira/browse/KAFKA-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15411376#comment-15411376
]
Ewen Cheslack-Postava commented on KAFKA-3967:
----------------------------------------------
[~Krishna82] Have you accounted for all replicas? What is the replication
factor for the topic? At 2-5MB/sec with a common 3 replica setup, you'd expect
7.2 - 18 GB/hr of traffic without any overhead. You didn't say how you're
getting the 2-5 MB/sec number, but if it is just the size of messages recorded
in your app, there is overhead both in writing messages to Kafka (e.g. fields
you may not be accounting for) and in the protocol (request/response overhead).
> Excessive Network IO between Kafka brokers
> -------------------------------------------
>
> Key: KAFKA-3967
> URL: https://issues.apache.org/jira/browse/KAFKA-3967
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8.2.2
> Reporter: Krishna
>
> Excessive Network IO between Kafka brokers running on AWS in different AZ's
> as compared to actual message volume.
> We are producing 2-5 MB /Sec message volume however kafka seems to me moving
> 20 gb /hr on network. The data volume has around 12 GB of message log on each
> nodes. Is this a natural behavior ?. I believe only the new messages will get
> replicated on non-leader nodes however here it seems that entire log is
> re-synced
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
