We are running a kafka cluster for our real-time pixel processing pipeline. The data is produced from our pixel servers into kafka, and then consumed by a spark streaming application. Based on this, I would expect that the bytes in vs bytes out should be roughly equal, as each message should be consumed once.
Under normal operations, the bytes out is a little less than 2X the bytes in. Does anyone know why this is? We do use a replication factor of 2. Occasionally, we get a spike in Bytes out. But bytes in remain the same (see image below). This correlates with a significant delay in processing time in the spark streaming side. Below is a chart of kafka reported bytes out vs in. The system level network metrics show the same information (transferred bytes spike). Could anyone provide some tips for debugging/getting to the bottom of this issue? Thanks, Jorge *Kafka reported Bytes in Per topic and for all topics vs Kafka bytes out:* [image: Inline image 1]
