Hi Harsha, As far as I understand KIP-236 it's about enabling reassignment cancellation and as a future plan providing a queue of replica reassignment steps to allow manual reassignment chains. While I agree that the reassignment chain has a specific use case that allows fine grain control over reassignment process, My proposal on the other hand doesn't talk about cancellation but it only provides an automatic way to incrementalize an arbitrary reassignment which I think fits the general use case where users don't want that level of control but still would like a balanced way of reassignments. Therefore I think it's still relevant as an improvement of the current algorithm. Nevertheless I'm happy to add my ideas to KIP-236 as I think it would be a great improvement to Kafka.
Cheers, Viktor On Fri, Feb 22, 2019 at 5:05 PM Harsha <ka...@harsha.io> wrote: > Hi Viktor, > There is already KIP-236 for the same feature and George made > a PR for this as well. > Lets consolidate these two discussions. If you have any cases that are not > being solved by KIP-236 can you please mention them in that thread. We can > address as part of KIP-236. > > Thanks, > Harsha > > On Fri, Feb 22, 2019, at 5:44 AM, Viktor Somogyi-Vass wrote: > > Hi Folks, > > > > I've created a KIP about an improvement of the reassignment algorithm we > > have. It aims to enable partition-wise incremental reassignment. The > > motivation for this is to avoid excess load that the current replication > > algorithm implicitly carries as in that case there are points in the > > algorithm where both the new and old replica set could be online and > > replicating which puts double (or almost double) pressure on the brokers > > which could cause problems. > > Instead my proposal would slice this up into several steps where each > step > > is calculated based on the final target replicas and the current replica > > assignment taking into account scenarios where brokers could be offline > and > > when there are not enough replicas to fulfil the min.insync.replica > > requirement. > > > > The link to the KIP: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-435%3A+Incremental+Partition+Reassignment > > > > I'd be happy to receive any feedback. > > > > An important note is that this KIP and another one, KIP-236 that is > > about > > interruptible reassignment ( > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-236%3A+Interruptible+Partition+Reassignment > ) > > should be compatible. > > > > Thanks, > > Viktor > > >
