Hi, 1. If you had problems with controlled shutdown, we need to know. Maybe open a thread to discuss? 2. Controlled shutdown is only used to reduce the downtime involved in large number of leader elections. New leaders will get elected in any case. 3. Controlled (or uncontrolled shutdown) does not change the preferred leader. This happens only on re-assignment. 4. #3 relies on the fact that if you are a brand new broker with absolutely no data joining the cluster with id = "n", and the replica-map shows that broker "n" has certain partitions (because we never assigned them away), the new broker will immediately become follower for these partitions and start replicating the missing data. This makes automatic recover much easier.
Gwen On Thu, Jan 14, 2016 at 10:49 AM, Luke Steensen < luke.steen...@braintreepayments.com> wrote: > Hello, > > For #3, I assume this relies on controlled shutdown to transfer leadership > gracefully? Or is there some way to use partition reassignment to set the > preferred leader of each partition? I ask because we've run into some > problems relying on controlled shutdown and having a separate verifiable > step would be nice. > > Thanks, > Luke > > > On Thu, Jan 14, 2016 at 9:36 AM, Gwen Shapira <g...@confluent.io> wrote: > > > Hi, > > > > There was a Jira to add "remove broker" option to the > > partition-reassignment tool. I think it died in a long discussion trying > to > > solve a harder problem... > > > > To your work-around - it is an acceptable work-around. > > > > Few improvements: > > 1. Manually edit the resulting assignment json to avoid unnecessary > moves. > > Or even create your own assignment (either manually or using a small > > script). > > 2. We don't throttle the partition move automatically, so it can easily > > take over the network if you are not careful. Therefore running the > > reassignment tools multiple times to move partitions one-by-one is often > > safer. > > 3. If you don't mean to permanently reduce the number of brokers but > rather > > to replace a broker, don't reassign. Just take down the existing broker > and > > give the new one the same ID. > > > > Hope this helps, > > > > Gwen > > > > On Wed, Jan 6, 2016 at 10:14 AM, Tom Crayford <tcrayf...@heroku.com> > > wrote: > > > > > Hi there, > > > > > > Kafka's `kafka-reassign-partitions.sh` tool currently has no mechanism > > for > > > removing brokers. However, it does have the ability to generate > partition > > > plans across arbitrary sets of brokers, by using `--generate`, passing > > all > > > the topics in the cluster into it, then passing the generated plan to > > > --execute. > > > > > > This isn't ideal, because it (from my understanding), potentially moves > > all > > > the partitions in the entire cluster around, but it should work fine, > and > > > stop Kafka from having the partitions assigned to a broker that no > longer > > > exists. > > > > > > Am I missing something there? Or is this a reasonable workaround until > > > better partition reassignment tools turn up in the future? > > > > > > Thanks > > > > > > Tom > > > > > >
