Thank you @Guozhang and @Drew for your responses. We don't see any errors in server logs but not sure if we could've run into KAFKA-1311. We are using 0.8.0 on brokers and 0.8.1 on producers to get the extra partition key and the option of not saving it on brokers. We would upgrade our producers to 0.8.1.1 though and see if the problems go away.
On Sat, Apr 26, 2014 at 1:35 PM, Drew Goya <[email protected]> wrote: > I ran into this problem while restarting brokers while running 0.8.1. This > was usually a sign that you ran into KAFKA-1311. > > I've since rolled upgrades to the latest on the 0.8.1 branch (0.8.1.1) and > my problems have gone away > > > On Thu, Apr 24, 2014 at 6:07 PM, Guozhang Wang <[email protected]> wrote: > > > Hi Sadhan, > > > > Do you see any errors on the server logs? > > > > Guozhang > > > > > > On Thu, Apr 24, 2014 at 12:57 PM, Sadhan Sood <[email protected]> > > wrote: > > > > > We are seeing some strange behavior from brokers after we we had to > > change > > > our log retention policy on brokers yesterday. We had a huge spike in > > > producer data for a small period which caused brokers to get very close > > to > > > the max disk space. Normally our retention policy is good 6-7 days but > > > since our consumers were synced up we changed the retention policy from > > > hour based to size based and cut short the size to a safe number (half > of > > > our max disk space and normal usage is around 30%). After the restart, > we > > > started seeing multiple producer side failures with FailedSends metrics > > > showing almost 10% failures and FailedProduceRequestsPerSec on the > broker > > > side a non-zero number. The traces from one of the brokers looked like > > > this: > > > > > > [KafkaApi-8] Produce request with correlation id 2050686 from client > xxx > > on > > > partition [TOPIC_NAME,18] failed due to Partition [TOPIC_NAME,18] > doesn't > > > exist on 8 (kafka.server.KafkaApis) > > > [KafkaApi-8] Produce request with correlation id 2102325 from client > xxx > > > on partition [TOPIC_NAME,28] failed due to Partition [TOPIC_NAME,28] > > > doesn't exist on 8 (kafka.server.KafkaApis) > > > > > > We checked and made sure those partitions were present on the broker. > > > Any help is appreciated. Also, is there a recommended way to purge log > > data > > > quickly out from the brokers. > > > > > > Thanks, > > > Sadhan > > > > > > > > > > > -- > > -- Guozhang > > >
