I ran into this problem while restarting brokers while running 0.8.1. This was usually a sign that you ran into KAFKA-1311.
I've since rolled upgrades to the latest on the 0.8.1 branch (0.8.1.1) and my problems have gone away On Thu, Apr 24, 2014 at 6:07 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Sadhan, > > Do you see any errors on the server logs? > > Guozhang > > > On Thu, Apr 24, 2014 at 12:57 PM, Sadhan Sood <sadhan.s...@gmail.com> > wrote: > > > We are seeing some strange behavior from brokers after we we had to > change > > our log retention policy on brokers yesterday. We had a huge spike in > > producer data for a small period which caused brokers to get very close > to > > the max disk space. Normally our retention policy is good 6-7 days but > > since our consumers were synced up we changed the retention policy from > > hour based to size based and cut short the size to a safe number (half of > > our max disk space and normal usage is around 30%). After the restart, we > > started seeing multiple producer side failures with FailedSends metrics > > showing almost 10% failures and FailedProduceRequestsPerSec on the broker > > side a non-zero number. The traces from one of the brokers looked like > > this: > > > > [KafkaApi-8] Produce request with correlation id 2050686 from client xxx > on > > partition [TOPIC_NAME,18] failed due to Partition [TOPIC_NAME,18] doesn't > > exist on 8 (kafka.server.KafkaApis) > > [KafkaApi-8] Produce request with correlation id 2102325 from client xxx > > on partition [TOPIC_NAME,28] failed due to Partition [TOPIC_NAME,28] > > doesn't exist on 8 (kafka.server.KafkaApis) > > > > We checked and made sure those partitions were present on the broker. > > Any help is appreciated. Also, is there a recommended way to purge log > data > > quickly out from the brokers. > > > > Thanks, > > Sadhan > > > > > > -- > -- Guozhang >
