We don’t have GC problem. zookeeper runs at 0% GC Kafka broker runs at 1-3% GC
We don’t see many major GC's in our monitoring. We use Concurrent Mark and Sweep GC. -----Original Message----- From: Joe Stein [mailto:joe.st...@stealth.ly] Sent: Tuesday, September 30, 2014 6:41 PM To: users@kafka.apache.org Subject: Re: BadVersion state in Kafka Logs Also check for really long/bad GC pauses as another possibility. Not sure your JDK and JVM_OPTS and if you are setting like this https://kafka.apache.org/documentation.html#java or not. You need to find some "spike" somewhere right before that error happens to track down what is causing the timeouts. On Tue, Sep 30, 2014 at 6:33 PM, Joe Stein <joe.st...@stealth.ly> wrote: > It sounds like you have a much deeper rooted problem. Is zookeeper > swapping? Something has to be causing this. After you fix this > symptom you will probably start to see constant leader elections and > the isr shrinking/growing and constant consumer rebalancing (or at > least every > minute) and a herd affect up/down stream occuring. You need to figure > out what is causing the long session timeout and resolve that, IMHO. > Zookeeper health is the first place to look. Next would be the network. > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ********************************************/ > > On Tue, Sep 30, 2014 at 4:57 PM, Seshadri, Balaji < > balaji.sesha...@dish.com> wrote: > >> The zookeeper session timeout is 60 secs ,but that did not help. >> >> We are having broker crash and unresponsive, we got the "conditional >> update" failed error when broker crashed which confirmed that it is >> because of KAFKA-1382. >> >> server.log.2014-09-23:2014-09-23 13:54:48 ERROR utils.ZkUtils$ - >> Conditional update of path >> /brokers/topics/dish-promo-application-access/partitions/128/state >> with data { "controller_epoch":40, "isr":[ 6, 1 ], "leader":1, >> "leader_epoch":99, "version":1 } and expected version 150 failed due >> to >> org.apache.zookeeper.KeeperException$BadVersionException: >> KeeperErrorCode = BadVersion for >> /brokers/topics/dish-promo-application-access/partitions/128/state >> >> We are in very old version 0.8-beta so it's not just patch but >> switching to stable release version which also has the patch. >> >> -----Original Message----- >> From: Joe Stein [mailto:joe.st...@stealth.ly] >> Sent: Tuesday, September 30, 2014 2:01 PM >> To: users@kafka.apache.org >> Cc: Neha Narkhede >> Subject: Re: BadVersion state in Kafka Logs >> >> Have you tried increasing your broker's zookeeper session timeout as >> a work around for now to alleviate the issue? Is that an option for you? >> Assuming that is the culprit you are timing zk sessions out and >> bumping into >> KAFKA-1382 on the reconnect? Not knowing enough about what is going >> on with the cluster it is hard to say if anything negative will come >> from it but seems like it might be a an approach to try... if you can >> figure out what is causing the session to timeout and fix *that* it >> would be a solution also.... if it is happening every couple days (as >> another email thread >> states) something is going on that may not just be fixed by a single >> patch. >> >> /******************************************* >> Joe Stein >> Founder, Principal Consultant >> Big Data Open Source Security LLC >> http://www.stealth.ly >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> ********************************************/ >> >> On Tue, Sep 30, 2014 at 11:49 AM, Seshadri, Balaji < >> balaji.sesha...@dish.com >> > wrote: >> >> > Hi Joe, >> > >> > I did not try on 0.8.1 branch ,I can try and see if it goes through >> > when I get some breather. >> > >> > Thanks for initiating on 0.8.1.2. >> > >> > Thanks, >> > >> > Balaji >> > >> > -----Original Message----- >> > From: Joe Stein [mailto:joe.st...@stealth.ly] >> > Sent: Tuesday, September 30, 2014 9:34 AM >> > To: users@kafka.apache.org >> > Cc: Neha Narkhede >> > Subject: Re: BadVersion state in Kafka Logs >> > >> > Does the patch in KAFKA-1382 apply on the 0.8.1 branch? If not if >> > you could make a patch that does would be great. >> > >> > I will kick off a discussion for KAFKA-1382 and the scala 2.11 for >> > 0.8.1.2 release (and see what others may think we should do like >> > the gradle changes I think we should do too for src release issues >> > (and the jars in the repo)). I will send that on dev/user in a >> > little bit (please comment +1 community support please on that >> > thread for the >> release). >> > >> > /******************************************* >> > Joe Stein >> > Founder, Principal Consultant >> > Big Data Open Source Security LLC >> > http://www.stealth.ly >> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> > ********************************************/ >> > >> > On Tue, Sep 30, 2014 at 11:10 AM, Seshadri, Balaji < >> > balaji.sesha...@dish.com >> > > wrote: >> > >> > > I would love to help you guys to make Kafka best in Pub/Sub, will >> > > continue doing that whenever I can. >> > > >> > > Do we have 0.8.1.2 release tag or should we apply patch on top >> > > of >> > > 0.8.1.1 tag because we need this KAFKA-1382 JIRA ?. >> > > >> > > Balaji >> > > >> > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com] >> > > Sent: Monday, September 29, 2014 5:21 PM >> > > To: Seshadri, Balaji >> > > Cc: users@kafka.apache.org >> > > Subject: Re: BadVersion state in Kafka Logs >> > > >> > > It is difficult to predict an exact date. Though all the >> > > discussions of the progress and ETA are on the mailing list. You >> > > can follow the discussions to know the details and/or offer to >> > > help out on the outstanding issues. >> > > >> > > On Mon, Sep 29, 2014 at 3:48 PM, Seshadri, Balaji < >> > > balaji.sesha...@dish.com<mailto:balaji.sesha...@dish.com>> wrote: >> > > Neha, >> > > >> > > Do you know the date in Oct when 0.8.2 is going to be out ?. >> > > >> > > Thanks, >> > > >> > > Balaji >> > > >> > > From: Neha Narkhede [mailto:neha.narkh...@gmail.com<mailto: >> > > neha.narkh...@gmail.com>] >> > > Sent: Thursday, September 25, 2014 1:08 PM >> > > To: Seshadri, Balaji >> > > Cc: users@kafka.apache.org<mailto:users@kafka.apache.org> >> > > >> > > Subject: Re: BadVersion state in Kafka Logs >> > > >> > > We are close to the release. I'd probably expect 0.8.2 sometime >> > > in >> > October. >> > > >> > > On Thu, Sep 25, 2014 at 10:37 AM, Seshadri, Balaji < >> > > balaji.sesha...@dish.com<mailto:balaji.sesha...@dish.com>> wrote: >> > > Hi Neha, >> > > >> > > Do you know when are you guys releasing 0.8.2 ?. >> > > >> > > Thanks, >> > > >> > > Balaji >> > > >> > > -----Original Message----- >> > > From: Seshadri, Balaji [mailto:balaji.sesha...@dish.com<mailto: >> > > balaji.sesha...@dish.com>] >> > > Sent: Thursday, September 25, 2014 9:41 AM >> > > To: users@kafka.apache.org<mailto:users@kafka.apache.org> >> > > Subject: RE: BadVersion state in Kafka Logs >> > > >> > > Thanks for the replay. >> > > >> > > Please let me know if we can use trunk as 0.8.2 is not yet released. >> > > >> > > Balaji >> > > ________________________________________ >> > > From: Neha Narkhede [neha.narkh...@gmail.com<mailto: >> > > neha.narkh...@gmail.com>] >> > > Sent: Wednesday, September 24, 2014 6:32 PM >> > > To: users@kafka.apache.org<mailto:users@kafka.apache.org> >> > > Subject: Re: BadVersion state in Kafka Logs >> > > >> > > From the logs you've attached, my guess is it's most likely due >> > > to KAFKA-1382. >> > > >> > > Thanks, >> > > Neha >> > > >> > > On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji < >> > > balaji.sesha...@dish.com<mailto:balaji.sesha...@dish.com> >> > > > wrote: >> > > >> > > > Hi, >> > > > >> > > > >> > > > >> > > > We got the below error in our logs and our consumers stopped >> > > > consuming any data ?.It worked only after restart. >> > > > >> > > > >> > > > >> > > > We would like to confirm that it's because we are running with >> > > > 0.8-beta version and not 0.8 release version to convince "THE MGMT" >> > guys. >> > > > >> > > > >> > > > >> > > > Please let me know if it's this KAFKA-1382 causing the issue. >> > > > >> > > > >> > > > >> > > > Thanks, >> > > > >> > > > >> > > > >> > > > Balaji >> > > > >> > > > >> > > > >> > > > *From:* Gulia, Vikram >> > > > *Sent:* Wednesday, September 24, 2014 8:43 AM >> > > > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA >> > > > *Cc:* Alam, Mohammad Shah >> > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions. >> > > > >> > > > >> > > > >> > > > Adding full MAA distro. >> > > > >> > > > >> > > > >> > > > DES Offshore looked in to the logs on kafka servers and seems >> > > > like the issue we encountered yesterday may be described in >> > > > these threads, please have a look - >> > > > >> > > > >> > > > >> > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904 >> > > > >> > > > >> > > > >> > > > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes >> > > > the fix/patch which is available in >> > > > 0.8.1.2/0.8.2<http://0.8.1.2/0.8.2>) >> > > > >> > > > >> > > > >> > > > Thank You, >> > > > >> > > > Vikram Gulia >> > > > >> > > > >> > > > >> > > > *From:* Sharma, Navdeep >> > > > *Sent:* Wednesday, September 24, 2014 6:53 AM >> > > > *To:* Gulia, Vikram; #IT-MAD DES >> > > > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah >> > > > *Subject:* RE: 9/23 prod issue - offline kafka partitions. >> > > > >> > > > >> > > > >> > > > Hi Vikram, >> > > > >> > > > >> > > > >> > > > We analyzed below mentioned issue with MAA-Offshore (Abhishek) >> > > > and found that the error occurred only on 23 Sept. This is not >> > > > historical as we checked last 4 days logs. >> > > > >> > > > >> > > > >> > > > It looks like that consumer got stopped on September 22 2014 >> > > > for Linux patching activty.MAA started consumer September 23 >> > > > 2014 at >> > > > 1:00 >> > AM. >> > > > >> > > > >> > > > >> > > > *Issue *in server log *"* *BadVersion for >> > > > /brokers/topics/rain-burn-in/partitions/121/state"* but it is >> > > > not present in previous 4 days logs. >> > > > >> > > > More detail of this error can be found at- >> > > > >> > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904 >> > > > >> > > > >> > > > >> > > > We are not sure about data loss in this scenario and working on >> this. >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > Let us know if any concerns. >> > > > >> > > > >> > > > >> > > > [image: cid:image001.gif@01CF7B0A.03F21580] >> > > > >> > > > Navdeep Sharma >> > > > Developer - offshore, Middleware Applications & Development o >> India: >> > > > 0120-4532000 - 2234 >> > > > c: +91-9911698102<tel:%2B91-9911698102> >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > *From:* Gulia, Vikram >> > > > *Sent:* Tuesday, September 23, 2014 6:17 PM >> > > > *To:* #IT-MAD DES >> > > > *Subject:* FW: 9/23 prod issue - offline kafka partitions. >> > > > >> > > > >> > > > >> > > > DES Offshore dev, >> > > > >> > > > >> > > > >> > > > Please work with MAA offshore to monitor the kafka broker as we >> > > > had this incident where lot of partitions went offline around >> > > > 1.45 PM MST and MAA has to restart the kafka servers. We may >> > > > have lost messages and we need to see if there is a way to >> > > > figure out what was >> > the impact. >> > > > >> > > > >> > > > >> > > > Also, check the logs for kafka servers and see if we can figure >> > > > out why did partitions go offline or are un-available? Let us >> > > > know if you find anything relevant. >> > > > >> > > > >> > > > >> > > > Thank You, >> > > > >> > > > Vikram Gulia >> > > > >> > > > >> > > > >> > > > *From:* Nielsen, Andy >> > > > *Sent:* Tuesday, September 23, 2014 5:04 PM >> > > > *To:* #IT-MAD DES; Gulia, Vikram >> > > > *Cc:* #IT-MAA >> > > > *Subject:* 9/23 prod issue - offline kafka partitions. >> > > > >> > > > >> > > > >> > > > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic >> > > > --unavailable-partitions >> > > > >> > > > topic: account-access partition: 21 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: account-access partition: 51 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: account-access partition: 81 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: account-access partition: 111 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: account-activated partition: 13 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: account-activated partition: 43 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: account-activated partition: 73 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: account-activated partition: 103 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: account-adjustment-issued partition: 27 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: account-adjustment-issued partition: 57 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: account-adjustment-issued partition: 87 leader: -1 >> > > > replicas: 4,6,1 isr: 1 >> > > > >> > > > topic: account-adjustment-issued partition: 117 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: account-created partition: 11 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: account-created partition: 41 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: account-created partition: 71 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: account-created partition: 101 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: account-info-updated partition: 7 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: account-info-updated partition: 37 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: account-info-updated partition: 67 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: account-info-updated partition: 97 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: account-info-updated partition: 127 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: application-access partition: 21 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: application-access partition: 51 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: application-access partition: 81 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: application-access partition: 111 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: bill-generated partition: 3 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: bill-generated partition: 33 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: bill-generated partition: 63 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: bill-generated partition: 93 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: bill-generated partition: 123 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: collected-event partition: 29 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: collected-event partition: 59 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: collected-event partition: 89 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: collected-event partition: 119 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: customer-cues partition: 27 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: customer-cues partition: 57 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: customer-cues partition: 87 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: customer-cues partition: 117 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: dish-promo-application-access partition: 23 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: dish-promo-application-access partition: 53 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: dish-promo-application-access partition: 83 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: dish-promo-application-access partition: 113 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: event-response partition: 2 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: event-response partition: 32 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: event-response partition: 62 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: event-response partition: 92 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: event-response partition: 122 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: leads-service partition: 24 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: leads-service partition: 54 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: leads-service partition: 84 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: leads-service partition: 114 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: logprod_v3 partition: 3 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: logprod_v3 partition: 33 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: logprod_v3 partition: 63 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: logprod_v3 partition: 93 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: logprod_v3 partition: 123 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: online-account-registration-attempted partition: 21 >> leader: >> > > > -1 replicas: 4,6,1 isr: >> > > > >> > > > topic: online-account-registration-attempted partition: 51 >> leader: >> > > > -1 replicas: 4,6,1 isr: 1 >> > > > >> > > > topic: online-account-registration-attempted partition: 81 >> leader: >> > > > -1 replicas: 4,6,1 isr: >> > > > >> > > > topic: online-account-registration-attempted partition: 111 >> leader: >> > > > -1 replicas: 4,6,1 isr: >> > > > >> > > > topic: order-cancelled partition: 29 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-cancelled partition: 59 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-cancelled partition: 89 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-cancelled partition: 119 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: order-completed partition: 24 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-completed partition: 54 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-completed partition: 84 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: order-completed partition: 114 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-created partition: 25 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-created partition: 55 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-created partition: 85 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-created partition: 115 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-modified partition: 8 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: order-modified partition: 38 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-modified partition: 68 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-modified partition: 98 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-modified partition: 128 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: order-request partition: 24 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-request partition: 54 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-request partition: 84 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: order-request partition: 114 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-response partition: 27 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: order-response partition: 57 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-response partition: 87 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: order-response partition: 117 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: outbound-call-attempted partition: 13 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: outbound-call-attempted partition: 43 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: outbound-call-attempted partition: 73 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: outbound-call-attempted partition: 103 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: outbound-communications partition: 4 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: outbound-communications partition: 34 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: outbound-communications partition: 64 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: outbound-communications partition: 94 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: outbound-communications partition: 124 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > topic: postal-mail-undeliverable partition: 15 leader: -1 >> > > > replicas: 4,6,1 isr: 1 >> > > > >> > > > topic: postal-mail-undeliverable partition: 45 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: postal-mail-undeliverable partition: 75 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: postal-mail-undeliverable partition: 105 leader: -1 >> > > > replicas: 4,6,1 isr: >> > > > >> > > > topic: rain-burn-in partition: 4 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: rain-burn-in partition: 34 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-burn-in partition: 64 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-burn-in partition: 94 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: rain-burn-in partition: 124 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: rain-enhanced partition: 26 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-enhanced partition: 56 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-enhanced partition: 86 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: rain-enhanced partition: 116 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-listener partition: 23 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: rain-listener partition: 53 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: rain-listener partition: 83 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-listener partition: 113 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-load-test partition: 8 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: rain-load-test partition: 38 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-load-test partition: 68 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: rain-load-test partition: 98 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: rain-load-test partition: 128 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: submit-agreement partition: 2 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: submit-agreement partition: 32 leader: -1 replicas: >> 4,6,1 >> > > > isr: 1 >> > > > >> > > > topic: submit-agreement partition: 62 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: submit-agreement partition: 92 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: submit-agreement partition: 122 leader: -1 replicas: >> 4,6,1 >> > > > isr: >> > > > >> > > > topic: threshold-exceeded partition: 14 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: threshold-exceeded partition: 44 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: threshold-exceeded partition: 74 leader: -1 >> > replicas: >> > > > 4,6,1 isr: >> > > > >> > > > topic: threshold-exceeded partition: 104 leader: -1 >> > replicas: >> > > > 4,6,1 isr: 1 >> > > > >> > > > >> > > > >> > > > *Andy Nielsen* >> > > > >> > > > *Middleware Application Admin* >> > > > >> > > > *303-723-2347<tel:303-723-2347> >> > > > <303-723-2347<tel:303-723-2347>>* >> > > > >> > > > *cell:720-971-2856<tel:720-971-2856> >> > > > <720-971-2856<tel:720-971-2856>>* >> > > > >> > > > >> > > > >> > > >> > > >> > > >> > >> > >
