Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Manikumar
Yes, rolling restart should be fine for 1.0 -> 1.0.1 We can add "unclean.leader.election.enable=true" to server.properties. This requires broker restart to take effect. On Tue, Apr 24, 2018 at 12:02 PM, Mika Linnanoja wrote: > Morning, group. > > On Mon, Apr 23, 2018 at 11:19 AM, Mika Linnanoja

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Mika Linnanoja
Morning, group. On Mon, Apr 23, 2018 at 11:19 AM, Mika Linnanoja wrote > > If nothing else, let this incident of ours serve as a warning to do > exactly as the book (upgrade guide) says, not sort of wing it. Thanks for > fast replies, lively mailing list! > > Mika > So yeah last night further 94

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Mika Linnanoja
On Mon, Apr 23, 2018 at 10:51 AM, Brett Rann wrote: > > Mostly updating version variable in our puppet config file (masterless) > and applying manually per instance. It works surprisingly well this way. > > Sure, we do the same, but with Chef. But we still follow that process. Lock > in inter bro

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Mika Linnanoja
On Mon, Apr 23, 2018 at 10:29 AM, Manikumar wrote: > > What is the replication factor? Was unclean election enabled (It enabled by > default in 0.10.0.1)? > RF is 2 for regular topics (global var). Re: unclean elections, whatever is the default was on, so I think unclean election was enabled for

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Brett Rann
> Mostly updating version variable in our puppet config file (masterless) and applying manually per instance. It works surprisingly well this way. Sure, we do the same, but with Chef. But we still follow that process. Lock in inter broker and log message format to existing version first. upgrade 1

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Mika Linnanoja
Hi, On Mon, Apr 23, 2018 at 10:25 AM, Brett Rann wrote: > Firstly, 1.0.1 is out and I'd strongly advise you to use that as the > upgrade path over 1.0.0 if you can because it contains a lot of bugfixes. > Some critical. > Yeah, it would've just meant starting the whole process from scratch in a

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Manikumar
Hi, Before Kafka 1.1.0, If the unclean leader election is enabled and if there are no ISRs, the leader is set to -1 and ISR will be empty. During upgrade, If you have single replica partitions or if all replicas goes out of ISR, then we get into this situation. >From Kafka 0.11.0.0, Unclean lead

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Brett Rann
Firstly, 1.0.1 is out and I'd strongly advise you to use that as the upgrade path over 1.0.0 if you can because it contains a lot of bugfixes. Some critical. With unclean leader elections it should have resolved itself when the affected broker came back online and all partitions were available. So

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Mika Linnanoja
On Mon, Apr 23, 2018 at 9:59 AM, Enrique Medina Montenegro < e.medin...@gmail.com> wrote: > What type of storage do you have for your setup? Ah, the most important promptly forgotten from the details! 5 TB EBS GP2 (regular ssd) volumes as kafka data directory, 20 GB EBS GP2 as root per instance

Re: Unavailable partitions after upgrade to kafka 1.0.0

2018-04-23 Thread Enrique Medina Montenegro
What type of storage do you have for your setup? En 23 de abril de 2018 8:04:46 a. m. Mika Linnanoja escribió: Hello, Last week I upgraded one relatively large kafka (EC2, 10 brokers, ~30 TB data, 100-300 Mbps in/out per instance) 0.10.0.1 cluster to 1.0, and saw some issues. Out of ~100 t

Unavailable partitions after upgrade to kafka 1.0.0

2018-04-22 Thread Mika Linnanoja
Hello, Last week I upgraded one relatively large kafka (EC2, 10 brokers, ~30 TB data, 100-300 Mbps in/out per instance) 0.10.0.1 cluster to 1.0, and saw some issues. Out of ~100 topics with 2..20 partitions each, 9 partitions in 8 topics become "unavailable" across 3 brokers. The leader was shown