Thanks, Tim. We are just trying to benchmark the kafka producers and there is no issue with cluster or brokers being down in this case.
We are seeing way less data on the borers after calculating the sizes of the logs on the brokers) and there is no compression. We send 84 GB, but total logs sizes are only 58 GB on the brokers. Since replication factor is zero, can we use ack other than 1? Maung On Jun 3, 2014, at 3:00 PM, Timothy Chen <tnac...@gmail.com> wrote: > Hi Maung, > > If your required.acks is 1 then the producer only ensures that one > broker receives the data before it's sucessfully returned to the > client. > > Therefore if the broker crashes and lost all the data then you lose > data, or similarly it can happen even before the data is fsynced. > > To ensure there are more copies of your data in case of failure > scenarios you want to increase your required.acks to more than 1 to > tolerate failuries. > > Also async producer doesn't wait until the data is sent before it > returns, as it buffers and writes asynchronously. To ensure each write > that has a succesful response is written you want to use the sync > producer. > > Tim > > On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_t...@apple.com> wrote: >> Hi, >> >> We are seeing less data on the brokers than we send form the producers: 84 >> GB to 58 GB. >> >> What is the best way to ensure / detect if all data has been send properly >> to the brokers from the producers. >> >> Is there any logs that we can check on the producers? >> >> Configuration is 5 Brokers, 2 producers, no replication factor, async and >> ask is 1 and no compression. >> >> Thanks, >> Maung
