Hi Gwen, 1. I sent a message to this list a couple days ago with the subject "Controlled shutdown not relinquishing leadership of all partitions" describing the issue I saw. Sorry there's not a lot of detail on the controlled shutdown part, but I've had trouble reproducing outside of our specific deployment.
2. Yes, that makes sense. Sorry, I was implicitly assuming tight timeouts and at least one retry. 3. Right, my understanding is that it doesn't change the preferred leader, it just triggers a more graceful leader election than would occur if the broker were killed unexpectedly. I was basically asking if there's a way to move leadership away from a broker independently of shutting it down. That would really just be a workaround for the controlled shutdown issues we experienced. 4. Yep, we rely on exactly this behavior when replacing nodes. It's very helpful :) Thanks! Luke On Thu, Jan 14, 2016 at 10:07 AM, Gwen Shapira <g...@confluent.io> wrote: > Hi, > > 1. If you had problems with controlled shutdown, we need to know. Maybe > open a thread to discuss? > 2. Controlled shutdown is only used to reduce the downtime involved in > large number of leader elections. New leaders will get elected in any case. > 3. Controlled (or uncontrolled shutdown) does not change the preferred > leader. This happens only on re-assignment. > 4. #3 relies on the fact that if you are a brand new broker with absolutely > no data joining the cluster with id = "n", and the replica-map shows that > broker "n" has certain partitions (because we never assigned them away), > the new broker will immediately become follower for these partitions and > start replicating the missing data. > This makes automatic recover much easier. > > Gwen > > On Thu, Jan 14, 2016 at 10:49 AM, Luke Steensen < > luke.steen...@braintreepayments.com> wrote: > > > Hello, > > > > For #3, I assume this relies on controlled shutdown to transfer > leadership > > gracefully? Or is there some way to use partition reassignment to set the > > preferred leader of each partition? I ask because we've run into some > > problems relying on controlled shutdown and having a separate verifiable > > step would be nice. > > > > Thanks, > > Luke > > > > > > On Thu, Jan 14, 2016 at 9:36 AM, Gwen Shapira <g...@confluent.io> wrote: > > > > > Hi, > > > > > > There was a Jira to add "remove broker" option to the > > > partition-reassignment tool. I think it died in a long discussion > trying > > to > > > solve a harder problem... > > > > > > To your work-around - it is an acceptable work-around. > > > > > > Few improvements: > > > 1. Manually edit the resulting assignment json to avoid unnecessary > > moves. > > > Or even create your own assignment (either manually or using a small > > > script). > > > 2. We don't throttle the partition move automatically, so it can easily > > > take over the network if you are not careful. Therefore running the > > > reassignment tools multiple times to move partitions one-by-one is > often > > > safer. > > > 3. If you don't mean to permanently reduce the number of brokers but > > rather > > > to replace a broker, don't reassign. Just take down the existing broker > > and > > > give the new one the same ID. > > > > > > Hope this helps, > > > > > > Gwen > > > > > > On Wed, Jan 6, 2016 at 10:14 AM, Tom Crayford <tcrayf...@heroku.com> > > > wrote: > > > > > > > Hi there, > > > > > > > > Kafka's `kafka-reassign-partitions.sh` tool currently has no > mechanism > > > for > > > > removing brokers. However, it does have the ability to generate > > partition > > > > plans across arbitrary sets of brokers, by using `--generate`, > passing > > > all > > > > the topics in the cluster into it, then passing the generated plan to > > > > --execute. > > > > > > > > This isn't ideal, because it (from my understanding), potentially > moves > > > all > > > > the partitions in the entire cluster around, but it should work fine, > > and > > > > stop Kafka from having the partitions assigned to a broker that no > > longer > > > > exists. > > > > > > > > Am I missing something there? Or is this a reasonable workaround > until > > > > better partition reassignment tools turn up in the future? > > > > > > > > Thanks > > > > > > > > Tom > > > > > > > > > >
