Re: kafka + autoscaling groups fuckery

I think it'd be possible to avoid special-casing replacements, but it might be a bad idea network-traffic-wise, especially for rolling upgrades. My experience running Kafka on AWS is that rebalancing with multi-day retention periods can take a really long time, and can torch the cluster if you

Re: kafka + autoscaling groups fuckery

It would be cool to know if Netflix runs kafka in ASGs ... I can't find any mention of it online. (https://github.com/Netflix/suro/wiki/FAQ sorta implies that maybe they do, but it's not clear, and also old.) I've seen other people talking about running kafka in ASGs, e.g.

Re: kafka + autoscaling groups fuckery

Great talks, but not relevant to either of my problems -- the golang client not rebalancing the consumer offset topic, or autoscaling group behavior (which is I think is probably just a consequence of the first). Thanks though, there's good stuff in here. On Sun, Jul 3, 2016 at 10:23 AM, James

Re: kafka + autoscaling groups fuckery

Gwen, thanks for the response. 1.1 Your life may be a bit simpler if you have a way of starting a new > broker with the same ID as the old one - this means it will > automatically pick up the old replicas and you won't need to > rebalance. Makes life slightly easier in some cases. > Yeah, this

Re: kafka + autoscaling groups fuckery

Charity, 1. Nothing you do seems crazy to me. Kafka should be able to work with auto-scaling and we should be able to fix the issues you are running into. There are few things you should be careful about when using the method you described though: 1.1 Your life may be a bit simpler if you have a

Re: kafka + autoscaling groups fuckery

Reasons. Investigated it thoroughly, believe me. Some of the limitations that Kinesis uses to protect itself are non starters for us. Forgot to mention, we are using 0.9.0.1-0. On Tue, Jun 28, 2016 at 3:56 PM, Pradeep Gollakota wrote: > Just out of curiosity, if you

Re: kafka + autoscaling groups fuckery

Just out of curiosity, if you guys are in AWS for everything, why not use Kinesis? On Tue, Jun 28, 2016 at 3:49 PM, Charity Majors wrote: > Hi there, > > I just finished implementing kafka + autoscaling groups in a way that made > sense to me. I have a _lot_ of experience