Re: Broker Exceptions

We are trying to see what might have caused it. We had some questions : 1) Is this reproducible? That way we can dig deep. This looks interesting problem to solve and you might have caught a bug, but we need to verify the root cause before filing a ticket. Thanks, Mayuresh On Tue, Mar 17, 201

Re: Broker Exceptions

> What version are you running ? Version 0.8.2.0 > Your case is 2). But the only thing weird is your replica (broker 3) is > requesting for offset which is greater than the leaders log end offset. So what could be the cause? Thanks Zakee > On Mar 17, 2015, at 11:45 AM, Mayuresh Gharat > w

Re: Broker Exceptions

What version are you running ? The code for latest version says that : 1) if the log end offset of the replica is greater than the leaders log end offset, the replicas offset will be reset to logEndOffset of the leader. 2) Else if the log end offset of the replica is smaller than the leaders log

Re: Broker Exceptions

cool. On Tue, Mar 17, 2015 at 10:15 AM, Zakee wrote: > Hi Mayuresh, > > The logs are already attached and are in reverse order starting backwards > from [2015-03-14 07:46:52,517] to the time when brokers were started. > > Thanks > Zakee > > > > > On Mar 17, 2015, at 12:07 AM, Mayuresh Gharat < >

Re: Broker Exceptions

Hi Mayuresh, The logs are already attached and are in reverse order starting backwards from [2015-03-14 07:46:52,517] to the time when brokers were started. Thanks Zakee > On Mar 17, 2015, at 12:07 AM, Mayuresh Gharat > wrote: > > Hi Zakee, > > Thanks for the logs. Can you paste earlier l

Re: Broker Exceptions

Hi Zakee, Thanks for the logs. Can you paste earlier logs from broker-3 up to : [2015-03-14 07:46:52,517] ERROR [ReplicaFetcherThread-2-4], Current offset 1754769769 for partition [Topic22kv,5] out of range; reset offset to 1400864851 (kafka.server.ReplicaFetcherThread) That would help us figure

Re: Broker Exceptions

Hi Mayuresh, Here are the logs. Old School Yearbook Pics View Class Yearbooks Online Free. Search by School & Year. Look Now! http://thirdpartyoffers.netzero.net/TGL3231/5507ca8137dc94a805e6bst01vucBroker-4 [2015-03-13 17:49:40,514] IN

Re: Broker Exceptions

Hi Mayuresh, Here are the logs. Broker-4 [2015-03-13 17:49:40,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [Topic22kv,5] from 2,4,3 to 2,4 (kafka.cluster.Partition) [2015-03-13 17:49:40,514] INFO Partition [Topic22kv,5] on broker 4: Shrinking ISR for partition [To

Re: Broker Exceptions

Can you provide more logs (complete) on Broker 3 till time : *[2015-03-14 07:46:52,517*] WARN [ReplicaFetcherThread-2-4], Replica 3 for partition [Topic22kv,5] reset its fetch offset from 1400864851 to current leader 4's start offset 1400864851 (kafka.server.ReplicaFetcherThread) I would like to

Re: Broker Exceptions

log.cleanup.policy is delete not compact. log.cleaner.enable=true log.cleaner.threads=5 log.cleanup.policy=delete log.flush.scheduler.interval.ms=3000 log.retention.minutes=1440 log.segment.bytes=1073741824 (1gb) Messages are keyed but not compressed, producer async and uses kafka default parti

Re: Broker Exceptions

Is your topic log compacted? Also if it is are the messages keyed? Or are the messages compressed? Thanks, Mayuresh Sent from my iPhone > On Mar 14, 2015, at 2:02 PM, Zakee wrote: > > Thanks, Jiangjie for helping resolve the kafka controller migration driven > partition leader rebalance iss

Re: Broker Exceptions

Thanks, Jiangjie for helping resolve the kafka controller migration driven partition leader rebalance issue. The logs are much cleaner now. There are a few incidences of Out of range offset even though there is no consumers running, only producers and replica fetchers. I was trying to relate

Re: Broker Exceptions

No broker restarts. Created a kafka issue: https://issues.apache.org/jira/browse/KAFKA-2011 >> Logs for rebalance: >> [2015-03-07 16:52:48,969] INFO [Controller 2]: Resuming preferred replica >> election for partitions: (kafka.controller.Kafka

Re: Broker Exceptions

No broker restarts. Created a kafka issue: https://issues.apache.org/jira/browse/KAFKA-2011 >> Logs for rebalance: >> [2015-03-07 16:52:48,969] INFO [Controller 2]: Resuming preferred replica >> election for partitions: (kafka.controller.Kafka

Re: Broker Exceptions

Is there anything wrong with brokers around that time? E.g. Broker restart? The log you pasted are actually from replica fetchers. Could you paste the related logs in controller.log? Thanks. Jiangjie (Becket) Qin On 3/9/15, 10:32 AM, "Zakee" wrote: >Correction: Actually the rebalance happened

Re: Broker Exceptions

Correction: Actually the rebalance happened quite until 24 hours after the start, and thats where below errors were found. Ideally rebalance should not have happened at all. Thanks Zakee > On Mar 9, 2015, at 10:28 AM, Zakee wrote: > >> Hmm, that sounds like a bug. Can you paste the log of

Re: Broker Exceptions

> Hmm, that sounds like a bug. Can you paste the log of leader rebalance > here? Thanks for you suggestions. It looks like the rebalance actually happened only once soon after I started with clean cluster and data was pushed, it didn’t happen again so far, and I see the partitions leader counts

Re: Broker Exceptions

Hmm, that sounds like a bug. Can you paste the log of leader rebalance here? Some other things to check are: 1. The actual property name is auto.leader.rebalance.enable, not auto.leader.rebalance. You’ve probably known this, just to double confirm. 2. In zookeeper path, can you verify /admin/prefer

Re: Broker Exceptions

I started with clean cluster and started to push data. It still does the rebalance at random durations even though the auto.leader.relabalance is set to false. Thanks Zakee > On Mar 6, 2015, at 3:51 PM, Jiangjie Qin wrote: > > Yes, the rebalance should not happen in that case. That is a li

Re: Broker Exceptions

Thanks, Jiangjie, I will try with a clean cluster again. Thanks Zakee > On Mar 6, 2015, at 3:51 PM, Jiangjie Qin wrote: > > Yes, the rebalance should not happen in that case. That is a little bit > strange. Could you try to launch a clean Kafka cluster with > auto.leader.election disabled and

Re: Broker Exceptions

Yes, the rebalance should not happen in that case. That is a little bit strange. Could you try to launch a clean Kafka cluster with auto.leader.election disabled and try push data? When leader migration occurs, NotLeaderForPartition exception is expected. Jiangjie (Becket) Qin On 3/6/15, 3:14 PM

Re: Broker Exceptions

Yes, Jiangjie, I do see lots of these errors "Starting preferred replica leader election for partitions” in logs. I also see lot of Produce request failure warnings in with the NotLeader Exception. I tried switching off the auto.leader.relabalance to false. I am still noticing the rebalance ha

Re: Broker Exceptions

I don’t think num.replica.fetchers will help in this case. Increasing number of fetcher threads will only help in cases where you have a large amount of data coming into a broker and more replica fetcher threads will help keep up. We usually only use 1-2 for each broker. But in your case, it looks

Re: Broker Exceptions

Thanks, Jiangjie. Yes, I do see under partitions usually shooting every hour. Anythings that I could try to reduce it? How does "num.replica.fetchers" affect the replica sync? Currently have configured 7 each of 5 brokers. -Zakee On Wed, Feb 25, 2015 at 4:17 PM, Jiangjie Qin wrote: > These me

Re: Broker Exceptions

These messages are usually caused by leader migration. I think as long as you don¹t see this lasting for ever and got a bunch of under replicated partitions, it should be fine. Jiangjie (Becket) Qin On 2/25/15, 4:07 PM, "Zakee" wrote: >Need to know if I should I be worried about this or ignore

Broker Exceptions

Need to know if I should I be worried about this or ignore them. I see tons of these exceptions/warnings in the broker logs, not sure what causes them and what could be done to fix them. ERROR [ReplicaFetcherThread-3-5], Error for partition [TestTopic] to broker 5:class kafka.common.NotLeaderForP