[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17779769#comment-17779769 ] Sergio Troiano commented on KAFKA-4084: --- [~wushujames]  [~junrao] ,   I wanted to add some extra

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17772791#comment-17772791 ] Sergio Troiano commented on KAFKA-4084: --- [~junrao] ,   I dont want to keep adding examples of why

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145752#comment-17145752 ] Jun Rao commented on KAFKA-4084: [~hai_lin] : (1) If we just need a tool to disallow a broker from taking

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112508#comment-17112508 ] Hai Lin commented on KAFKA-4084: Thanks [~sql_consulting] point me to this ticket. [~junrao] Just want to

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103028#comment-17103028 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] hmm...That's weird. auto.leader.rebalance.enable

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102405#comment-17102405 ] Robin Palotai commented on KAFKA-4084: -- Hi guys! I'm trying to learn about the alternatives for

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099550#comment-17099550 ] Evan Williams commented on KAFKA-4084: -- Apologies for the delayed response [~sql_consulting]. Have

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086772#comment-17086772 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] I am not very familiar with 5.4 setup. Do you have

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085948#comment-17085948 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting]  Many thanks for that! I've just started

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077958#comment-17077958 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] Probably doing PLE with too many partitions at once is

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076167#comment-17076167 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] Many thanks for that details info. Sounds

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076124#comment-17076124 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] If this turns out to be positive in the testing. I

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076092#comment-17076092 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] Thanks. Yes, I already have that logic

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076089#comment-17076089 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] I think it's still ok to set

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076064#comment-17076064 ] Evan Williams commented on KAFKA-4084: -- Much appreciated [~sql_consulting].  I've requested access

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076006#comment-17076006 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] Have some free time this weekend to troubleshoot and

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073581#comment-17073581 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] Thanks! No stress at all. :) > automated

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072259#comment-17072259 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] Very busy with work and this COVID-19 Jira

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070032#comment-17070032 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] No pressure from my side, but how's

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051509#comment-17051509 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] sure. I am working on porting the patch from 1.1.x.

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049088#comment-17049088 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] No problem! Thanks for your help here.

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048116#comment-17048116 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] [~sriharsha] Hi Evan, I am so sorry I had been busy

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042554#comment-17042554 ] Evan Williams commented on KAFKA-4084: -- [~wushujames] Yes, I've now decided to set 

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042307#comment-17042307 ] Jun Rao commented on KAFKA-4084: https://issues.apache.org/jira/browse/KAFKA-9594 identifies an issue

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17042297#comment-17042297 ] James Cheng commented on KAFKA-4084: Hi   Just wanted to chime in with some of our experience on this

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039861#comment-17039861 ] Evan Williams commented on KAFKA-4084: -- [~sriharsha] That would be great. We are using Confluent

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039548#comment-17039548 ] Harsha commented on KAFKA-4084: --- [~blodsbror] if you are interested in trying out our patch we are happy to

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17035517#comment-17035517 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] [~junrao] FYI, I implemented separation

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032604#comment-17032604 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] That would be great! We are running the

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032598#comment-17032598 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] Which kafka version you are running? Maybe I can

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032285#comment-17032285 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] We are using min.insync.replicas=1.  And

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032267#comment-17032267 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] [~junrao][~sriharsha] With

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032200#comment-17032200 ] Evan Williams commented on KAFKA-4084: -- [~junrao] Yes, I can see that it's been throttled to

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17032006#comment-17032006 ] Jun Rao commented on KAFKA-4084: Do you see the incoming byte rate of that broker much more than

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031971#comment-17031971 ] Evan Williams commented on KAFKA-4084: -- [~junrao] My monitoring (datadog) isn't set up for that

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031955#comment-17031955 ] Jun Rao commented on KAFKA-4084: [~evanjpw]: There are metrics related to replication throttling

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17031916#comment-17031916 ] Evan Williams commented on KAFKA-4084: -- [~junrao] I've implemented throttling now. Even on quite a

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030864#comment-17030864 ] Jun Rao commented on KAFKA-4084: [~evanjpw]: I am not sure if you need the new config. The controller

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030825#comment-17030825 ] Evan Williams commented on KAFKA-4084: -- Thanks [~junrao] Totally agree.  # Set dynamic cluster

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030819#comment-17030819 ] Jun Rao commented on KAFKA-4084: [~evanjpw]: # For this case, you do have to script up things yourself.

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030379#comment-17030379 ] Evan Williams commented on KAFKA-4084: -- Thanks a lot guys. [~sql_consulting] I have actually

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030298#comment-17030298 ] Jun Rao commented on KAFKA-4084: [~sriharsha], hmm, if one sets follower.replication.throttled.rate to

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030290#comment-17030290 ] Harsha commented on KAFKA-4084: --- [~junrao] I am not sure if we can avoid a broker overload problem even

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17030183#comment-17030183 ] GEORGE LI commented on KAFKA-4084: -- [~evanjpw] One solution we have in-house is a tooling to do

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029638#comment-17029638 ] Evan Williams commented on KAFKA-4084: -- [~sql_consulting] Many thanks for the explanation. Is there

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029299#comment-17029299 ] GEORGE LI commented on KAFKA-4084: -- [~blodsbror] We have implemented KIP-491 internally. Using a

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029256#comment-17029256 ] Harsha commented on KAFKA-4084: --- [~junrao] any interest in KIP-491. We are using this in production and has

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029058#comment-17029058 ] Evan Williams commented on KAFKA-4084: --  We are seeing the same (or at least similar) issues. When

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16956143#comment-16956143 ] Marouane RAJI commented on KAFKA-4084: -- Hi [~jiaxinye],  Does 

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900461#comment-16900461 ] GEORGE LI commented on KAFKA-4084: -- [~jiaxinye] does the patch from

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846512#comment-16846512 ] Vivek Yadav commented on KAFKA-4084: [~jiaxinye] Is it works for you? > automated leader rebalance

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649839#comment-16649839 ] Jiaxin YE commented on KAFKA-4084: -- [~wushujames] : Wow, thanks! Let me try that out.  > automated

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648586#comment-16648586 ] James Cheng commented on KAFKA-4084: [~jiaxinye]: You are probably hitting

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648581#comment-16648581 ] Jiaxin YE commented on KAFKA-4084: -- Is this issue really resolved? I am using Kafka 1.1.0 and my cluster

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226013#comment-16226013 ] Jun Rao commented on KAFKA-4084: [~jeffwidman], the detailed perf results can be found in

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225870#comment-16225870 ] Jeff Widman commented on KAFKA-4084: [~junrao] any ballpark quantification to "much faster"? Are we

[jira] [Commented] (KAFKA-4084) automated leader rebalance causes replication downtime for clusters with too many partitions

[ https://issues.apache.org/jira/browse/KAFKA-4084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223612#comment-16223612 ] Ismael Juma commented on KAFKA-4084: [~tcrayford-heroku], can you please answer Jun's question? >