Thanks, Jun. Will check and get back..
We are converting JSON to Avro and that conversion is done by the custom serializer. Our volume calculation on the producer side is based on the AVRO generic record that is passed to the producer send method— not of the encoded data output from the serializer that is what actual got send the Broker I believe. That could be the gap and I am testing now without the customer serializer and seeing the two volumes are very close. That could be it!! Thanks, Maung On Jun 3, 2014, at 7:22 PM, Jun Rao <jun...@gmail.com> wrote: > We have a metric on msg rate in both the producer and the broker. Could you > see if they match? > > Thanks, > > Jun > > > On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_t...@apple.com> wrote: > >> Hi, >> >> We are seeing less data on the brokers than we send form the producers: >> 84 GB to 58 GB. >> >> What is the best way to ensure / detect if all data has been send properly >> to the brokers from the producers. >> >> Is there any logs that we can check on the producers? >> >> Configuration is 5 Brokers, 2 producers, no replication factor, async and >> ask is 1 and no compression. >> >> Thanks, >> Maung >>
