Re: [DISCUSS] KIP-46: Self Healing

[Aditya Auradkar]

Thanks Neha. I'll discard this for now. We can pick it up once replica
throttling and the default policies are available and tested.

On Thu, Feb 11, 2016 at 5:45 PM, Neha Narkhede  wrote:

> >
> > 1. Replica Throttling - I agree this is rather important to get done.
> > However, it may also be argued that this problem is orthogonal. We do not
> > have these protections currently yet we do run partition reassignment
> > fairly often. Having said that, I'm perfectly happy to tackle KIP-46
> after
> > this problem is solved. I understand it is actively being discussed in
> > KAFKA-1464.
>
>
> I think we are saying the same thing here. Replica throttling is required
> to be able to pull off any partition reassignment action. It removes the
> guesswork that comes from picking a batch size that is expressed in terms
> of partition count, which is an annoying hack.
>
>
> > 2. Pluggable policies - Can you elaborate on the need for pluggable
> > policies in the partition reassignment tool? Even if we make it pluggable
> > to begin with, this needs to ship with a default policy that makes sense
> > for most users. IMO, partition count is the most intuitive default and is
> > analogous to how we stripe partitions for new topics.
>
>
> Agree about the default. I was arguing for making it pluggable so we make
> it easy to test multiple policies. For instance, partition count is a
> decent one but I can imagine how one would want a policy that optimizes for
> balancing data sizes for instance.
>
>
> > 3. Even if the trigger were fully manual (as it is now), we could still
> > have the controller generate the assignment as per a configured policy
> i.e.
> > effectively the tool is built into Kafka itself. Following this approach
> to
> > begin with makes it easier to fully automate in the future since we will
> > only need to automate the trigger later.
>
>
> I would be much more comfortable adding the capability to move large
> amounts of data to the controller, after we are very sure that the default
> policy is well tested and the replica throttling works. If so, then it is
> just a matter of placing the trigger in the controller vs in the tool. But
> I'm skeptical of adding more things to the already messy controller,
> especially without being sure about how well it works.
>
> Thanks,
> Neha
>
> On Tue, Feb 9, 2016 at 12:53 PM, Aditya Auradkar <
> aaurad...@linkedin.com.invalid> wrote:
>
> > Hi Neha,
> >
> > Thanks for the detailed reply and apologies for my late response. I do
> have
> > a few comments.
> >
> > 1. Replica Throttling - I agree this is rather important to get done.
> > However, it may also be argued that this problem is orthogonal. We do not
> > have these protections currently yet we do run partition reassignment
> > fairly often. Having said that, I'm perfectly happy to tackle KIP-46
> after
> > this problem is solved. I understand it is actively being discussed in
> > KAFKA-1464.
> >
> > 2. Pluggable policies - Can you elaborate on the need for pluggable
> > policies in the partition reassignment tool? Even if we make it pluggable
> > to begin with, this needs to ship with a default policy that makes sense
> > for most users. IMO, partition count is the most intuitive default and is
> > analogous to how we stripe partitions for new topics.
> >
> > 3. Even if the trigger were fully manual (as it is now), we could still
> > have the controller generate the assignment as per a configured policy
> i.e.
> > effectively the tool is built into Kafka itself. Following this approach
> to
> > begin with makes it easier to fully automate in the future since we will
> > only need to automate the trigger later.
> >
> > Aditya
> >
> >
> >
> > On Wed, Feb 3, 2016 at 1:57 PM, Neha Narkhede  wrote:
> >
> > > Adi,
> > >
> > > Thanks for the write-up. Here are my thoughts:
> > >
> > > I think you are suggesting a way of automating resurrecting a topic’s
> > > replication factor in the presence of a specific scenario: in the event
> > of
> > > permanent broker failures. I agree that the partition reassignment
> > > mechanism should be used to add replicas when they are lost to
> permanent
> > > broker failures. But I think the KIP probably chews off more than we
> can
> > > digest.
> > >
> > > Before we automate detection of permanent broker failures and have the
> > > controller mitigate through automatic data balancing, I’d like to point
> > out
> > > that our current difficulty is not that but the ability to generate a
> > > workable partition assignment for rebalancing data in a cluster.
> > >
> > > There are 2 problems with partition rebalancing today:
> > >
> > >1. Lack of replica throttling for balancing data: In the absence of
> > >replica throttling, even if you come up with an assignment that
> might
> > be
> > >workable, it isn’t practical to kick it off without worrying about
> > > bringing
> > >the entire cluster down. I don’t think the hack of 

Re: [DISCUSS] KIP-46: Self Healing

[Neha Narkhede]

>
> 1. Replica Throttling - I agree this is rather important to get done.
> However, it may also be argued that this problem is orthogonal. We do not
> have these protections currently yet we do run partition reassignment
> fairly often. Having said that, I'm perfectly happy to tackle KIP-46 after
> this problem is solved. I understand it is actively being discussed in
> KAFKA-1464.


I think we are saying the same thing here. Replica throttling is required
to be able to pull off any partition reassignment action. It removes the
guesswork that comes from picking a batch size that is expressed in terms
of partition count, which is an annoying hack.


> 2. Pluggable policies - Can you elaborate on the need for pluggable
> policies in the partition reassignment tool? Even if we make it pluggable
> to begin with, this needs to ship with a default policy that makes sense
> for most users. IMO, partition count is the most intuitive default and is
> analogous to how we stripe partitions for new topics.


Agree about the default. I was arguing for making it pluggable so we make
it easy to test multiple policies. For instance, partition count is a
decent one but I can imagine how one would want a policy that optimizes for
balancing data sizes for instance.


> 3. Even if the trigger were fully manual (as it is now), we could still
> have the controller generate the assignment as per a configured policy i.e.
> effectively the tool is built into Kafka itself. Following this approach to
> begin with makes it easier to fully automate in the future since we will
> only need to automate the trigger later.


I would be much more comfortable adding the capability to move large
amounts of data to the controller, after we are very sure that the default
policy is well tested and the replica throttling works. If so, then it is
just a matter of placing the trigger in the controller vs in the tool. But
I'm skeptical of adding more things to the already messy controller,
especially without being sure about how well it works.

Thanks,
Neha

On Tue, Feb 9, 2016 at 12:53 PM, Aditya Auradkar <
aaurad...@linkedin.com.invalid> wrote:

> Hi Neha,
>
> Thanks for the detailed reply and apologies for my late response. I do have
> a few comments.
>
> 1. Replica Throttling - I agree this is rather important to get done.
> However, it may also be argued that this problem is orthogonal. We do not
> have these protections currently yet we do run partition reassignment
> fairly often. Having said that, I'm perfectly happy to tackle KIP-46 after
> this problem is solved. I understand it is actively being discussed in
> KAFKA-1464.
>
> 2. Pluggable policies - Can you elaborate on the need for pluggable
> policies in the partition reassignment tool? Even if we make it pluggable
> to begin with, this needs to ship with a default policy that makes sense
> for most users. IMO, partition count is the most intuitive default and is
> analogous to how we stripe partitions for new topics.
>
> 3. Even if the trigger were fully manual (as it is now), we could still
> have the controller generate the assignment as per a configured policy i.e.
> effectively the tool is built into Kafka itself. Following this approach to
> begin with makes it easier to fully automate in the future since we will
> only need to automate the trigger later.
>
> Aditya
>
>
>
> On Wed, Feb 3, 2016 at 1:57 PM, Neha Narkhede  wrote:
>
> > Adi,
> >
> > Thanks for the write-up. Here are my thoughts:
> >
> > I think you are suggesting a way of automating resurrecting a topic’s
> > replication factor in the presence of a specific scenario: in the event
> of
> > permanent broker failures. I agree that the partition reassignment
> > mechanism should be used to add replicas when they are lost to permanent
> > broker failures. But I think the KIP probably chews off more than we can
> > digest.
> >
> > Before we automate detection of permanent broker failures and have the
> > controller mitigate through automatic data balancing, I’d like to point
> out
> > that our current difficulty is not that but the ability to generate a
> > workable partition assignment for rebalancing data in a cluster.
> >
> > There are 2 problems with partition rebalancing today:
> >
> >1. Lack of replica throttling for balancing data: In the absence of
> >replica throttling, even if you come up with an assignment that might
> be
> >workable, it isn’t practical to kick it off without worrying about
> > bringing
> >the entire cluster down. I don’t think the hack of moving partitions
> in
> >batches is effective as it at-best a best guess.
> >2. Lack of support for policies in the rebalance tool that
> automatically
> >generate a workable partition assignment: There is no easy way to
> > generate
> >a partition reassignment JSON file. An example of a policy is “end up
> > with
> >an equal number of partitions on every broker while minimizing data
> >movement”. 

Re: [DISCUSS] KIP-46: Self Healing

[Aditya Auradkar]

Hi Neha,

Thanks for the detailed reply and apologies for my late response. I do have
a few comments.

1. Replica Throttling - I agree this is rather important to get done.
However, it may also be argued that this problem is orthogonal. We do not
have these protections currently yet we do run partition reassignment
fairly often. Having said that, I'm perfectly happy to tackle KIP-46 after
this problem is solved. I understand it is actively being discussed in
KAFKA-1464.

2. Pluggable policies - Can you elaborate on the need for pluggable
policies in the partition reassignment tool? Even if we make it pluggable
to begin with, this needs to ship with a default policy that makes sense
for most users. IMO, partition count is the most intuitive default and is
analogous to how we stripe partitions for new topics.

3. Even if the trigger were fully manual (as it is now), we could still
have the controller generate the assignment as per a configured policy i.e.
effectively the tool is built into Kafka itself. Following this approach to
begin with makes it easier to fully automate in the future since we will
only need to automate the trigger later.

Aditya



On Wed, Feb 3, 2016 at 1:57 PM, Neha Narkhede  wrote:

> Adi,
>
> Thanks for the write-up. Here are my thoughts:
>
> I think you are suggesting a way of automating resurrecting a topic’s
> replication factor in the presence of a specific scenario: in the event of
> permanent broker failures. I agree that the partition reassignment
> mechanism should be used to add replicas when they are lost to permanent
> broker failures. But I think the KIP probably chews off more than we can
> digest.
>
> Before we automate detection of permanent broker failures and have the
> controller mitigate through automatic data balancing, I’d like to point out
> that our current difficulty is not that but the ability to generate a
> workable partition assignment for rebalancing data in a cluster.
>
> There are 2 problems with partition rebalancing today:
>
>1. Lack of replica throttling for balancing data: In the absence of
>replica throttling, even if you come up with an assignment that might be
>workable, it isn’t practical to kick it off without worrying about
> bringing
>the entire cluster down. I don’t think the hack of moving partitions in
>batches is effective as it at-best a best guess.
>2. Lack of support for policies in the rebalance tool that automatically
>generate a workable partition assignment: There is no easy way to
> generate
>a partition reassignment JSON file. An example of a policy is “end up
> with
>an equal number of partitions on every broker while minimizing data
>movement”. There might be other policies that might make sense, we’d
> have
>to experiment.
>
> Broadly speaking, the data balancing problem is comprised of 3 parts:
>
>1. Trigger: An event that triggers data balancing to take place. KIP-46
>suggests a specific trigger and that is permanent broker failure. But
> there
>might be several other events that might make sense — Cluster expansion,
>decommissioning brokers, data imbalance
>2. Policy: Given a set of constraints, generate a target partition
>assignment that can be executed when triggered.
>3. Mechanism: Given a partition assignment, make the state changes and
>actually move the data until the target assignment is achieved.
>
> Currently, the trigger is manual through the rebalance tool, there is no
> support for any viable policy today and we have a built-in mechanism that,
> given a policy and upon a trigger, moves data in a cluster but does not
> support throttling.
>
> Given that both the policy and the throttling improvement to the mechanism
> are hard problems and given our past experience of operationalizing
> partition reassignment (required months of testing before we got it right),
> I strongly recommend attacking this problem in stages. I think a more
> practical approach would be to add the concept of pluggable policies in the
> rebalance tool, implement a practical policy that generates a workable
> partition assignment upon triggering the tool and improve the mechanism to
> support throttling so that a given policy can succeed without manual
> intervention. If we solved these problems first, the rebalance tool would
> be much more accessible to Kafka users and operators.
>
> Assuming that we do this, the problem that KIP-46 aims to solve becomes
> much easier. You can separate the detection of permanent broker failures
> (trigger) from the mitigation (above-mentioned improvements to data
> balancing). The latter will be a native capability in Kafka. Detecting
> permanent hardware failures is much easily done via an external script that
> uses a simple health check. (Part 1 of KIP-46).
>
> I agree that it will be great to *eventually* be able to fully automate
> both the trigger as well as the policies while also improving the
> 

[DISCUSS] KIP-46: Self Healing

[Aditya Auradkar]

Hey everyone,

I just created a kip to discuss automated replica reassignment when we lose
a broker in the cluster.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-46%3A+Self+Healing+Kafka

Any feedback is welcome.

Thanks,
Aditya