If there are no further comments, I will start a vote on this next week. Thanks,
Tom On 20 October 2017 at 08:33, Tom Bentley <t.j.bent...@gmail.com> wrote: > Hi, > > I've made a fairly major update to KIP-179 to propose APIs for setting > throttled rates and throttled replicas with the ability to remove these > automatically at the end of reassignment. > > I'd be grateful for your feedback: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-179+-+Change+ > ReassignPartitionsCommand+to+use+AdminClient > > Thanks, > > Tom > > On 2 October 2017 at 13:15, Tom Bentley <t.j.bent...@gmail.com> wrote: > >> One question I have is about whether/how to scope throttling to a >> reassignment. Currently throttles are only loosely associated with >> reassignment: You can start a reassignment without any throttling, add >> throttling to an in-flight reassignment, and remember/forget to remove >> throttling after the reassignment is complete. There's is great flexibility >> in that, but also the risk that you forget the remove the throttle(s). >> >> Just adding an API for setting the throttled rate makes this situation >> worse: While it's nice to be able to auto-remove the throttles rate what >> about the config for the throttled replicas? Also you might add a throttle >> thinking a reassignment is in-flight, but it has in fact just finished: >> Those throttles will now hang around until reset or the end of the next >> reassignment. For these reasons it would be good if the throttle were more >> directly scoped to the reassignment. >> >> On the other hand, taking LinkedIn's Cruise Control as an example, there >> they seem to modify the reassignment znode directly and incrementally and >> so there is no notion of "the reassignment". Reassignments will be running >> continuously, with partitions added before all of the current partitions >> have completed. If there is no meaningful cluster-wide "reassignment" then >> it would be better to remove remove the throttle by changing the list of >> replicas as each replica catches up. >> >> I'm interested in any use cases people can share on this, as I'd like the >> throttle API to be useful for a broad range of use cases, rather than being >> too narrowly focussed on what's needed by the existing CLI tools. >> >> Thanks, >> >> Tom >> >> >> >> >> On 28 September 2017 at 17:22, Tom Bentley <t.j.bent...@gmail.com> wrote: >> >>> I'm starting to think about KIP-179 again. In order to have more >>> manageably-scoped KIPs and PRs I think it might be worth factoring-out the >>> throttling part into a separate KIP. Wdyt? >>> >>> Keeping the throttling discussion in this thread for the moment... >>> >>> The throttling behaviour is currently spread across the >>> `(leader|follower).replication.throttled.replicas` topic config and the >>> `(leader|follower).replication.throttled.rate` dynamic broker config. >>> It's not really clear to me exactly what "removing the throttle" is >>> supposed to mean. I mean we could reset the rate to Long.MAV_VALUE or we >>> could change the list of replicas to an empty list. The >>> ReassignPartitionsCommand does both, but there is some small utility in >>> leaving the rate, but clearing the list, if you've discovered the "right" >>> rate for your cluster/workload and to want it to be sticky for next time. >>> Does any one do this in practice? >>> >>> With regards to throttling, it would be >>>>> worth thinking about a way where the throttling configs can be >>>>> automatically removed without the user having to re-run the tool. >>>>> >>>> >>>> Isn't that just a matter of updating the topic configs for >>>> (leader|follower).replication.throttled.replicas at the same time we >>>> remove the reassignment znode? That leaves open the question about whether >>>> to reset the rates at the same time. >>>> >>> >>> Thinking some more about my "update the configs at the same time we >>> remove the reassignment znode" suggestion. The reassignment znode is >>> persistent, so the reassignment will survive a zookeeper restart. If there >>> was a flag for the auto-removal of the throttle it would likewise need to >>> be persistent. Otherwise a ZK restart would remember the reassignment, but >>> forget about the preference for auto removal of throttles. So, we would use >>> a persistent znode (a child of the reassignment path, perhaps) to store a >>> flag for throttle removal. >>> >>> Thoughts? >>> >>> Cheers, >>> >>> Tom >>> >> >> >
