Thanks Alex and Todd, I really appreciate the insights. Based on what you've shared: spending more time up-front to homogenize the nodes reduces some cognitive load for day-to-day support. Keeping things simple generally wins me over.
Thanks again. -Kevin On Thu, Jun 9, 2016 at 10:37 PM, Todd Palino <tpal...@gmail.com> wrote: > So as Alex noted, there’s no immediate problem to doing this. Kafka itself > doesn’t know much about the underlying hardware, so it’s not going to care. > At the same time, this means that it has no way natively to know that those > systems have more storage capacity. So they’re not going to automatically > get more partitions. > > You have some options here. > > 1. You could just ignore it, and treat everything like it’s the smaller > brokers. This is easy, but you’ll waste your extra storage > 2. You could manually assign more partitions to the larger brokers. This > requires a little bit of work, but it will more effectively use the > hardware. > > The gotcha with #2 is that you have to make sure you’re not sending too > much network traffic to the larger brokers, and you need to make sure that > you’re not exhausting the CPU as well. And, of course, you’re going to have > to keep an eye on new topics or anything like that to make sure that your > weighted cluster balance is still where you want it to be, and manually fix > it if not. > > -Todd > > > On Fri, Jun 10, 2016 at 5:26 AM, Alex Loddengaard <a...@confluent.io> > wrote: > > > Hi Kevin, > > > > If you keep the same configs on the new brokers with more storage > capacity, > > I don't foresee any issues. Although I haven't tried it myself. > > > > What may introduce headaches is if you have different configuration > options > > per broker. Or if you try to assign more partitions to the newer brokers > to > > use more of their disk space. > > > > Let's see if others notice anything I'm missing (again, I've never tried > > this before). Hope this helps. > > > > Alex > > > > On Thu, Jun 9, 2016 at 10:27 AM, Kevin A <k4m...@gmail.com> wrote: > > > > > Hi there, > > > > > > I have a couple of Kafka brokers and thinking about adding a few more. > > The > > > new broker machines would have a lot more storage available to them > than > > > the existing brokers. Am I setting myself up for operational headaches > by > > > deploying a heterogeneous (in terms of storage capacity) cluster? > > > > > > (Asked on IRC but thought I'd try here too.) > > > > > > Thanks! > > > -Kevin > > > > > > > > > -- > *Todd Palino* > Staff Site Reliability Engineer > Data Infrastructure Streaming > > > > linkedin.com/in/toddpalino >
