Re: [DISCUSS] KIP-879: Multi-level Rack Awareness
Hi Ziming, Thanks for the reply. It may not be a public interface, yes, but it is annotated with the @InterfaceStability.Unstable annotation from which I presumed it'll be public at some point. Therefore in my KIP I proposed to take it public as we've seen significant interest from our users towards stretch clusters and this kind of replica placement I described. In fact in our latest release we've already implemented something like this and we'd like to contribute it back if the community also feels the need for it. May I ask why don't you want to make this public? Thanks, Viktor On Thu, Dec 8, 2022 at 3:06 PM Andrew Otto wrote: > FWIW, the Wikimedia Foundation would find this change really helpful. We > are going to soon experiment with a stretched Kafka cluster, and it would > be nice to be able to target datacenter AND racks for replica placement. > > On Thu, Dec 8, 2022 at 3:37 AM ziming deng > wrote: > > > Hi Viktor, > > > > As far as I know, we haven't make ReplicaPlacer a public interface, and > we > > have no plan to make it public. I think you can submit a discussion or > > create a JIRA ticket directly without KIP if you have ideas on improving > > it, right? > > > > -- > > Best, > > Ziming > > > > > On Nov 29, 2022, at 21:52, Viktor Somogyi-Vass < > > viktor.somo...@cloudera.com.INVALID> wrote: > > > > > > Hi All, > > > > > > I'd like to bump this. I've also updated the KIP to incorporate the new > > > KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar > to > > > that, so mostly I've made some minor rewording, naming changes, etc. > > > > > > Again, the brief summary of the KIP: > > > - expose replica placement strategies with a new config > > > - create an admin API and protocol to expose replica placement > > > functionality (mainly for the reassignment tool) > > > - create a new multi-level rack awareness strategy which improves > > > availability on stretch clusters > > > > > > I'm happy for any feedback. > > > > > > Best, > > > Viktor > > > > > > On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass < > > > viktor.somo...@cloudera.com> wrote: > > > > > >> Hey all, > > >> > > >> I'd like to propose a new broker side replica assignment strategy and > an > > >> interface that generalizes replica assignment on brokers and makes > them > > >> pluggable. > > >> > > >> Briefly, the motivation for the new replica assignment strategy is > that > > >> more and more of our customers would want to run their clusters in a > > >> stretched environment, where for instance a cluster is running over > > >> multiple regions (and multiple racks inside a region). Since this > seems > > >> like a more common need, we'd like to contribute back our > implementation > > >> and also make a generalized interface, so that new strategies that > > people > > >> may come up with could be served better. > > >> > > >> I welcome any feedback on this KIP. > > >> > > >> The link: > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness > > >> > > >> Best to all, > > >> Viktor > > >> > > > > >
Re: [DISCUSS] KIP-879: Multi-level Rack Awareness
FWIW, the Wikimedia Foundation would find this change really helpful. We are going to soon experiment with a stretched Kafka cluster, and it would be nice to be able to target datacenter AND racks for replica placement. On Thu, Dec 8, 2022 at 3:37 AM ziming deng wrote: > Hi Viktor, > > As far as I know, we haven't make ReplicaPlacer a public interface, and we > have no plan to make it public. I think you can submit a discussion or > create a JIRA ticket directly without KIP if you have ideas on improving > it, right? > > -- > Best, > Ziming > > > On Nov 29, 2022, at 21:52, Viktor Somogyi-Vass < > viktor.somo...@cloudera.com.INVALID> wrote: > > > > Hi All, > > > > I'd like to bump this. I've also updated the KIP to incorporate the new > > KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar to > > that, so mostly I've made some minor rewording, naming changes, etc. > > > > Again, the brief summary of the KIP: > > - expose replica placement strategies with a new config > > - create an admin API and protocol to expose replica placement > > functionality (mainly for the reassignment tool) > > - create a new multi-level rack awareness strategy which improves > > availability on stretch clusters > > > > I'm happy for any feedback. > > > > Best, > > Viktor > > > > On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass < > > viktor.somo...@cloudera.com> wrote: > > > >> Hey all, > >> > >> I'd like to propose a new broker side replica assignment strategy and an > >> interface that generalizes replica assignment on brokers and makes them > >> pluggable. > >> > >> Briefly, the motivation for the new replica assignment strategy is that > >> more and more of our customers would want to run their clusters in a > >> stretched environment, where for instance a cluster is running over > >> multiple regions (and multiple racks inside a region). Since this seems > >> like a more common need, we'd like to contribute back our implementation > >> and also make a generalized interface, so that new strategies that > people > >> may come up with could be served better. > >> > >> I welcome any feedback on this KIP. > >> > >> The link: > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness > >> > >> Best to all, > >> Viktor > >> > >
Re: [DISCUSS] KIP-879: Multi-level Rack Awareness
Hi Viktor, As far as I know, we haven't make ReplicaPlacer a public interface, and we have no plan to make it public. I think you can submit a discussion or create a JIRA ticket directly without KIP if you have ideas on improving it, right? -- Best, Ziming > On Nov 29, 2022, at 21:52, Viktor Somogyi-Vass > wrote: > > Hi All, > > I'd like to bump this. I've also updated the KIP to incorporate the new > KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar to > that, so mostly I've made some minor rewording, naming changes, etc. > > Again, the brief summary of the KIP: > - expose replica placement strategies with a new config > - create an admin API and protocol to expose replica placement > functionality (mainly for the reassignment tool) > - create a new multi-level rack awareness strategy which improves > availability on stretch clusters > > I'm happy for any feedback. > > Best, > Viktor > > On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass < > viktor.somo...@cloudera.com> wrote: > >> Hey all, >> >> I'd like to propose a new broker side replica assignment strategy and an >> interface that generalizes replica assignment on brokers and makes them >> pluggable. >> >> Briefly, the motivation for the new replica assignment strategy is that >> more and more of our customers would want to run their clusters in a >> stretched environment, where for instance a cluster is running over >> multiple regions (and multiple racks inside a region). Since this seems >> like a more common need, we'd like to contribute back our implementation >> and also make a generalized interface, so that new strategies that people >> may come up with could be served better. >> >> I welcome any feedback on this KIP. >> >> The link: >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness >> >> Best to all, >> Viktor >>
Re: [DISCUSS] KIP-879: Multi-level Rack Awareness
Hi All, I'd like to bump this. I've also updated the KIP to incorporate the new KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar to that, so mostly I've made some minor rewording, naming changes, etc. Again, the brief summary of the KIP: - expose replica placement strategies with a new config - create an admin API and protocol to expose replica placement functionality (mainly for the reassignment tool) - create a new multi-level rack awareness strategy which improves availability on stretch clusters I'm happy for any feedback. Best, Viktor On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass < viktor.somo...@cloudera.com> wrote: > Hey all, > > I'd like to propose a new broker side replica assignment strategy and an > interface that generalizes replica assignment on brokers and makes them > pluggable. > > Briefly, the motivation for the new replica assignment strategy is that > more and more of our customers would want to run their clusters in a > stretched environment, where for instance a cluster is running over > multiple regions (and multiple racks inside a region). Since this seems > like a more common need, we'd like to contribute back our implementation > and also make a generalized interface, so that new strategies that people > may come up with could be served better. > > I welcome any feedback on this KIP. > > The link: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness > > Best to all, > Viktor >
[DISCUSS] KIP-879: Multi-level Rack Awareness
Hey all, I'd like to propose a new broker side replica assignment strategy and an interface that generalizes replica assignment on brokers and makes them pluggable. Briefly, the motivation for the new replica assignment strategy is that more and more of our customers would want to run their clusters in a stretched environment, where for instance a cluster is running over multiple regions (and multiple racks inside a region). Since this seems like a more common need, we'd like to contribute back our implementation and also make a generalized interface, so that new strategies that people may come up with could be served better. I welcome any feedback on this KIP. The link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness Best to all, Viktor
