Where exactly do you get the measurement from? Your broker? Do you have only one? Your producer? Your spark job? On Mon, 11 Apr 2016 at 23:54 Jorge Rodriguez <[email protected]> wrote:
> We are running a kafka cluster for our real-time pixel processing > pipeline. The data is produced from our pixel servers into kafka, and then > consumed by a spark streaming application. Based on this, I would expect > that the bytes in vs bytes out should be roughly equal, as each message > should be consumed once. > > Under normal operations, the bytes out is a little less than 2X the bytes > in. Does anyone know why this is? We do use a replication factor of 2. > > Occasionally, we get a spike in Bytes out. But bytes in remain the same > (see image below). This correlates with a significant delay in processing > time in the spark streaming side. > > Below is a chart of kafka reported bytes out vs in. The system level > network metrics show the same information (transferred bytes spike). > > Could anyone provide some tips for debugging/getting to the bottom of this > issue? > > Thanks, > Jorge > > *Kafka reported Bytes in Per topic and for all topics vs Kafka bytes out:* > > [image: Inline image 1] >
