Re: partition reassignment stuck

2015-04-21 Thread Jiangjie Qin
the new >> leader for the partition once it receive LeaderAndIsrRequest from >> controller to update the new leader information. If these messages keep >>got >> logged for long time then there might be an issue. >> Can you maybe check the timestamp around [2015-04-21

Re: partition reassignment stuck

2015-04-21 Thread Wesley Chow
rom: Wes Chow > Reply-To: "users@kafka.apache.org" > Date: Tuesday, April 21, 2015 at 1:29 PM > To: "users@kafka.apache.org" > Subject: Re: partition reassignment stuck > > > Quick clarification: you say broker 0, but do you actually mean broker 25? > 25

Re: partition reassignment stuck

2015-04-21 Thread Jiangjie Qin
ply-To: "users@kafka.apache.org<mailto:users@kafka.apache.org>" mailto:users@kafka.apache.org>> Date: Tuesday, April 21, 2015 at 1:29 PM To: "users@kafka.apache.org<mailto:users@kafka.apache.org>" mailto:users@kafka.apache.org>> Subject: Re: partition reassi

Re: partition reassignment stuck

2015-04-21 Thread Wes Chow
fka.apache.org>" mailto:users@kafka.apache.org>> Subject: Re: partition reassignment stuck Not for that particular partition, but I am seeing these errors on 28: kafka.common.NotAssignedReplicaException: Leader 28 failed to record follower 25's position 0 for partition [clic

Re: partition reassignment stuck

2015-04-21 Thread Jiangjie Qin
org<mailto:users@kafka.apache.org>" mailto:users@kafka.apache.org>> Subject: Re: partition reassignment stuck Not for that particular partition, but I am seeing these errors on 28: kafka.common.NotAssignedReplicaException: Leader 28 failed to record follower 25's position 0 f

Re: partition reassignment stuck

2015-04-21 Thread Wes Chow
Not for that particular partition, but I am seeing these errors on 28: kafka.common.NotAssignedReplicaException: Leader 28 failed to record follower 25's position 0 for partition [click_engage,116] since the replica 25 is not recognized to be one of the assigned r eplicas for partition [clic

Re: partition reassignment stuck

2015-04-21 Thread Jiangjie Qin
Those .index files are for different partitions and they should be generated if new replicas is assigned to the broker. We might want to know what caused the UnknownException. Did you see any error log on broker 28? Jiangjie (Becket) Qin On 4/21/15, 9:16 AM, "Wes Chow" wrote

partition reassignment stuck

2015-04-21 Thread Wes Chow
I started a partition reassignment (this is a 8.1.1 cluster) some time ago and it seems to be stuck. Partitions are no longer getting moved around, but it seems like the cluster is operational otherwise. The stuck nodes have a lot of .index files, and their logs show errors

Re: partition reassignment stuck on 0.8

2013-12-12 Thread David Birdsong
On Thu, Dec 12, 2013 at 9:46 PM, Jun Rao wrote: > Since we don't support delete topics yet, you would have to wipe out all ZK > and kafka logs. > > Thanks, > > Jun > > got it and done. so it sounds like i should run a number of disparate clusters to spread risk for topics since a partition is an

Re: partition reassignment stuck on 0.8

2013-12-12 Thread Jun Rao
Since we don't support delete topics yet, you would have to wipe out all ZK and kafka logs. Thanks, Jun On Thu, Dec 12, 2013 at 9:32 PM, David Birdsong wrote: > On Thu, Dec 12, 2013 at 9:28 PM, Jun Rao wrote: > > > Could you try starting from scratch again? The recent fix that we had may > >

Re: partition reassignment stuck on 0.8

2013-12-12 Thread David Birdsong
On Thu, Dec 12, 2013 at 9:28 PM, Jun Rao wrote: > Could you try starting from scratch again? The recent fix that we had may > not be able to recover a cluster already in an inconsistent state. > > Thanks, > > Jun > > > How does one start from scratch? Wipe ZK, is there some state file? I have oth

Re: partition reassignment stuck on 0.8

2013-12-12 Thread David Birdsong
On Thu, Dec 12, 2013 at 9:28 PM, Guozhang Wang wrote: > David, > > Could you try to see if this is due to > https://issues.apache.org/jira/browse/KAFKA-1178? > > Guozhang > Which node do I look for this on? Leader? ISR-candidate? Controller? > > > On Thu, Dec 12, 2013 at 8:45 PM, David Birdson

Re: partition reassignment stuck on 0.8

2013-12-12 Thread Guozhang Wang
David, Could you try to see if this is due to https://issues.apache.org/jira/browse/KAFKA-1178? Guozhang On Thu, Dec 12, 2013 at 8:45 PM, David Birdsong wrote: > I was running a 2-node kafka cluster off github trunnk at: > eedbea6526986783257ad0e025c451a8ee3d9095 > > ...for a few weeks with no

Re: partition reassignment stuck on 0.8

2013-12-12 Thread Jun Rao
Could you try starting from scratch again? The recent fix that we had may not be able to recover a cluster already in an inconsistent state. Thanks, Jun On Thu, Dec 12, 2013 at 8:45 PM, David Birdsong wrote: > I was running a 2-node kafka cluster off github trunnk at: > eedbea6526986783257ad0e

partition reassignment stuck on 0.8

2013-12-12 Thread David Birdsong
I was running a 2-node kafka cluster off github trunnk at: eedbea6526986783257ad0e025c451a8ee3d9095 ...for a few weeks with no issues. I recently downloaded the 0.8 stable version, configured/started two new brokers with 0.8. I successfully reassigned all but 1 partition from the older pair to th