Thanks Tao. for reference, here is the corresponding listener https://github.com/apache/kafka/blob/9dbeb71ab258955e04b46991c1baf880b07633f4/core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala#L1000-L1002
On Sat, Sep 12, 2015 at 5:42 AM, tao xiao <xiaotao...@gmail.com> wrote: > It is handled by the implementation of the high level consumer > automatically. High level consumer internal registers an listener on ZK > topic znode. Any changes to it including topic creation high level consumer > will be notify and and then rebalanced topics if the new topic matches the > regex. You can check the source code high level for reference > On Sat, Sep 12, 2015 at 5:18 AM Alexis Midon > <alexis.mi...@airbnb.com.invalid> wrote: > > > When a new topic is created, I agree that the regex would remain > unchanged > > but how would an existing consumer be notified of the topic creation? > > afaik there's no such notification mechanism in the High level consumer. > > > > > > > > On Thu, Sep 10, 2015 at 8:43 AM, tao xiao <xiaotao...@gmail.com> wrote: > > > > > You can create message streams using regex that includes all topics. > The > > > beauty of regex is that any new topic created will be automatically > > > consumed as long as the name of the topic matches the regex > > > > > > You check the method createMessageStreamsByFilter in high level API > > > > > > On Thu, Sep 10, 2015 at 11:03 PM Gerrit Jansen van Vuuren < > > > gerrit...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > I'm not sure about the high level consumer but I maintain a kafka > > > consumer > > > > that can add and remove topics dynamically. > > > > > > > > https://github.com/gerritjvv/kafka-fast > > > > see > > > > > > > > > > > > > > https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java > > > > if you're using java/scala > > > > > > > > > > > > > > > > On Thu, Sep 10, 2015 at 3:48 PM, Helleren, Erik < > > > > erik.helle...@cmegroup.com> > > > > wrote: > > > > > > > > > So, there are several ways to do this. Lets assume the goal is to > > add > > > > > more topics to the application at runtime. And that this app is > > > > currently > > > > > written to be distributed via the magic of consumer groups. > Sadly, I > > > > > don¹t think the High level consumer is well designed for this > > > particular > > > > > use case. The app would have to poll using something like the > topic > > > list > > > > > script (bin/kafka-topics.sh --list Š), close the existing high > level > > > > > consumer on a change, and start a new one. And then do this on all > > the > > > > > nodes of your application (Should be easier than doing it on just > one > > > > > actually). This would result in a huge latency spike and a problem > > > when > > > > > it comes to migrating the state involved in your example > expectation. > > > > > > > > > > The next option still requires polling, but it needs a custom FT > and > > > > > distribution scheme. There might need a leader so some things only > > > > happen > > > > > once. Just use the simple consumer API, and have one thread per > > > > > partition. The leader would have to tell a follower something like > > > > ³Start > > > > > listening to topic X, partition Y², which is risky and difficult to > > do. > > > > > > > > > > The simplest option, assuming that each topic is independent when > it > > > > comes > > > > > to expectations, is don¹t go with a cluster. Just have a > > > script/watcher > > > > > app that does the polling and then, when it detects new topics, for > > > each > > > > > new topic: start a new instance of your app on a new box that > listens > > > to > > > > > that single topic. It might take a few seconds to startup, but its > > > easy > > > > > to code, easy to maintain, and easy to understand. Which makes > for a > > > > more > > > > > resilient application. > > > > > -Erik > > > > > > > > > > > > > > > > > > > > From: Joris Peeters <j.peet...@wintoncapital.com> > > > > > Reply-To: "users@kafka.apache.org" <users@kafka.apache.org> > > > > > Date: Thursday, September 10, 2015 at 6:09 AM > > > > > To: "users@kafka.apache.org" <users@kafka.apache.org> > > > > > Subject: automatically consume from all topics > > > > > > > > > > > > > > > Hello, > > > > > > > > > > Is there a simple way to set up a consumer that automatically picks > > up > > > > all > > > > > the topics for all the partitions, dynamically extending its range > as > > > new > > > > > topics get created? > > > > > > > > > > The underlying idea is that we want to have a few over-arching > > > consumers > > > > > (I¹m aware that¹s not great for the scalability, but that¹s not > such > > a > > > > > concern at present), to > > > > > - > > > > > Gather various statistics, metrics, system pressure, Š and dispatch > > to > > > > the > > > > > appropriate monitoring systems, > > > > > - > > > > > Apply some end-to-end business-logic testing, to continuously > assert > > > > > certain expectations (e.g. ³if this-sort-of message arrived, then > we > > > > > expect that-sort-of-message to be received within this time² etc). > > > > > > > > > > > > > > > I¹m sure I can piece something together that does this, but perhaps > > it > > > > > comes out of the box. (Couldn¹t find it, though). > > > > > We¹re using the Java client and Kafka 8.2.1. > > > > > > > > > > Joris Peeters > > > > > Developer > > > > > > > > > > Research and Data Technology > > > > > T: > > > > > +44 (0) 20 8576 5800 > > > > > > > > > > Winton > > > > > Grove House > > > > > 27 Hammersmith Grove > > > > > London W6 0NE > > > > > > > > > > wintoncapital.com <http://www.wintoncapital.com/> > > > > > > > > > > <http://www.wintoncapital.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Winton Capital Management Limited (³Winton²) is a limited company > > > > > registered in England and Wales with its registered offices at 16 > Old > > > > > Bailey, London, EC4M 7EG (Registered Company No. 3311531). > > > > > Winton is authorised and regulated by the Financial Conduct > > Authority > > > in > > > > > the United Kingdom, registered as an investment adviser with the US > > > > > Securities and Exchange Commission, registered with the US > Commodity > > > > > Futures Trading Commission and a member of the > > > > > National Futures Association in the United States. > > > > > This communication, including any attachments, is confidential and > > may > > > be > > > > > privileged. This email is for use by the intended recipient only. > If > > > you > > > > > receive it in error, please notify the sender and > > > > > delete it. You should not copy or disclose all or any part of this > > > > email. > > > > > This email does not constitute an offer or solicitation and nothing > > > > > contained in this email constitutes, and should not be construed > as, > > > > > investment advice. Prospective investors should request offering > > > > > materials and consult their own advisers with respect to > investment > > > > > decisions and inform themselves as to applicable legal > requirements, > > > > > exchange control regulations and taxes in the countries of their > > > > > citizenship, residence or domicile. Past performance is > > > > > not indicative of future results. > > > > > Winton takes reasonable steps to ensure the accuracy and integrity > of > > > its > > > > > communications, including emails. However Winton accepts no > liability > > > for > > > > > any materials transmitted. Emails are not secure > > > > > and cannot be guaranteed to be error free. > > > > > > > > > > > > > > > > > > > >
