Thanks for your response Asaf. I have 4 brokers. These measurements are from the kafka brokers.
This measurement on this graph comes from Kafka. It is a sum across all 4 brokers of the metric: kafka.server.BrokerTopicMetrics.BytesInPerSec.1MinuteRate. But I also have a system metric which I feed independently using collectd "interface" plugin. And the bytes out and in match the ones reported by kafka fairly well. As well there is a corresponding increase in network packets sent. Also, in the SparkStreaming side, I can see that during these spikes, the number of received packets and bytes also spikes. So during the spikes, I believe that some of the fetch requests are perhaps failing and we hit a retry. I am debugging that currently and I think it's related to the STW GC which happens on spark streaming occasionally. Working on some GC tuning should alleviate this. However, even if this is the case, this would not explain though why under normal operations, the number of bytes out is 2x the number of bytes in. Since I only have 1 consumer for each topic, I would expect the numbers to be fairly close. Do you On Tue, Apr 12, 2016 at 8:31 PM, Asaf Mesika <asaf.mes...@gmail.com> wrote: > Where exactly do you get the measurement from? Your broker? Do you have > only one? Your producer? Your spark job? > On Mon, 11 Apr 2016 at 23:54 Jorge Rodriguez <jo...@bloomreach.com> wrote: > > > We are running a kafka cluster for our real-time pixel processing > > pipeline. The data is produced from our pixel servers into kafka, and > then > > consumed by a spark streaming application. Based on this, I would expect > > that the bytes in vs bytes out should be roughly equal, as each message > > should be consumed once. > > > > Under normal operations, the bytes out is a little less than 2X the bytes > > in. Does anyone know why this is? We do use a replication factor of 2. > > > > Occasionally, we get a spike in Bytes out. But bytes in remain the same > > (see image below). This correlates with a significant delay in > processing > > time in the spark streaming side. > > > > Below is a chart of kafka reported bytes out vs in. The system level > > network metrics show the same information (transferred bytes spike). > > > > Could anyone provide some tips for debugging/getting to the bottom of > this > > issue? > > > > Thanks, > > Jorge > > > > *Kafka reported Bytes in Per topic and for all topics vs Kafka bytes > out:* > > > > [image: Inline image 1] > > >
