thanks Grant, i also found some more information (also the one your referring to). I think it is a kafka -> zookeeper connection issue on AWS. We are investigating further on this.
Cheers Fabian 2015-08-04 18:54 GMT+02:00 Grant Henke <ghe...@cloudera.com>: > The /brokers/ids nodes are ephemeral nodes that only exists while the > brokers maintain a session to zookeeper. There is more information on > Kafka's Zookeeper usage here: > - http://kafka.apache.org/documentation.html > - look for "Broker Node Registry" > - > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper > > Hopefully that helps debug your issue. > > Thank you, > Grant > > On Mon, Aug 3, 2015 at 5:20 AM, Wollert, Fabian <fabian.woll...@zalando.de > > > wrote: > > > hi everyone, > > > > we are trying to deploy Kafka 0.8.2.1 and Zookeeper on AWS using > > Cloudformation, ASG's and other Services. For Zookeeper we are using > > Netflix' Exhibitor (V 1.5.5) to ensure failover stability. > > > > What we are observing right now is that after some days our Brokers are > not > > registered anymore in the "/brokers/ids" path in Zookeeper. I was trying > to > > see when they get deleted to check the logs, but the ZK Transaction logs > > only shows the create stmt, no deletes or something (though deletes are > > written down there). Can someone explain me how the mechanism works with > > registering and deregistering in Zookeeper or point me to a doc or even > > source code, where this happens? Or some one has even some idea what > > happens there. > > > > Any experience on what to take care of deploying kafka on AWS (or > generally > > a cloud env) would be also helpful. > > > > Cheers > > > > -- > > *Fabian Wollert* > > Business Intelligence > > > > *POSTAL ADDRESS* > > Zalando SE > > 11501 Berlin > > > > *OFFICE* > > Zalando SE > > Mollstraße 1 > > 10178 Berlin > > Germany > > > > Phone: +49 30 20968 1819 > > Fax: +49 30 27594 693 > > E-Mail: fabian.woll...@zalando.de > > Web: www.zalando.de > > Jobs: jobs.zalando.de > > > > Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin > > Company registration: Amtsgericht Charlottenburg, HRB 158855 B > > Tax ID: 29/560/00596 * VAT registration number: DE 260543043 > > Management Board: Robert Gentz, David Schneider, Rubin Ritter > > Chairperson of the Supervisory Board: Cristina Stenbeck > > Registered office: Berlinn > > > > > > -- > Grant Henke > Software Engineer | Cloudera > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke > -- *Fabian Wollert* Business Intelligence *POSTAL ADDRESS* Zalando SE 11501 Berlin *OFFICE* Zalando SE Mollstraße 1 10178 Berlin Germany Phone: +49 30 20968 1819 Fax: +49 30 27594 693 E-Mail: fabian.woll...@zalando.de Web: www.zalando.de Jobs: jobs.zalando.de Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin Company registration: Amtsgericht Charlottenburg, HRB 158855 B Tax ID: 29/560/00596 * VAT registration number: DE 260543043 Management Board: Robert Gentz, David Schneider, Rubin Ritter Chairperson of the Supervisory Board: Cristina Stenbeck Registered office: Berlinn
