Hi, I'm not sure about the high level consumer but I maintain a kafka consumer that can add and remove topics dynamically.
https://github.com/gerritjvv/kafka-fast see https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/java/kakfa_clj/core/Consumer.java if you're using java/scala On Thu, Sep 10, 2015 at 3:48 PM, Helleren, Erik <erik.helle...@cmegroup.com> wrote: > So, there are several ways to do this. Lets assume the goal is to add > more topics to the application at runtime. And that this app is currently > written to be distributed via the magic of consumer groups. Sadly, I > don¹t think the High level consumer is well designed for this particular > use case. The app would have to poll using something like the topic list > script (bin/kafka-topics.sh --list Š), close the existing high level > consumer on a change, and start a new one. And then do this on all the > nodes of your application (Should be easier than doing it on just one > actually). This would result in a huge latency spike and a problem when > it comes to migrating the state involved in your example expectation. > > The next option still requires polling, but it needs a custom FT and > distribution scheme. There might need a leader so some things only happen > once. Just use the simple consumer API, and have one thread per > partition. The leader would have to tell a follower something like ³Start > listening to topic X, partition Y², which is risky and difficult to do. > > The simplest option, assuming that each topic is independent when it comes > to expectations, is don¹t go with a cluster. Just have a script/watcher > app that does the polling and then, when it detects new topics, for each > new topic: start a new instance of your app on a new box that listens to > that single topic. It might take a few seconds to startup, but its easy > to code, easy to maintain, and easy to understand. Which makes for a more > resilient application. > -Erik > > > > From: Joris Peeters <j.peet...@wintoncapital.com> > Reply-To: "users@kafka.apache.org" <users@kafka.apache.org> > Date: Thursday, September 10, 2015 at 6:09 AM > To: "users@kafka.apache.org" <users@kafka.apache.org> > Subject: automatically consume from all topics > > > Hello, > > Is there a simple way to set up a consumer that automatically picks up all > the topics for all the partitions, dynamically extending its range as new > topics get created? > > The underlying idea is that we want to have a few over-arching consumers > (I¹m aware that¹s not great for the scalability, but that¹s not such a > concern at present), to > - > Gather various statistics, metrics, system pressure, Š and dispatch to the > appropriate monitoring systems, > - > Apply some end-to-end business-logic testing, to continuously assert > certain expectations (e.g. ³if this-sort-of message arrived, then we > expect that-sort-of-message to be received within this time² etc). > > > I¹m sure I can piece something together that does this, but perhaps it > comes out of the box. (Couldn¹t find it, though). > We¹re using the Java client and Kafka 8.2.1. > > Joris Peeters > Developer > > Research and Data Technology > T: > +44 (0) 20 8576 5800 > > Winton > Grove House > 27 Hammersmith Grove > London W6 0NE > > wintoncapital.com <http://www.wintoncapital.com/> > > <http://www.wintoncapital.com/> > > > > > > Winton Capital Management Limited (³Winton²) is a limited company > registered in England and Wales with its registered offices at 16 Old > Bailey, London, EC4M 7EG (Registered Company No. 3311531). > Winton is authorised and regulated by the Financial Conduct Authority in > the United Kingdom, registered as an investment adviser with the US > Securities and Exchange Commission, registered with the US Commodity > Futures Trading Commission and a member of the > National Futures Association in the United States. > This communication, including any attachments, is confidential and may be > privileged. This email is for use by the intended recipient only. If you > receive it in error, please notify the sender and > delete it. You should not copy or disclose all or any part of this email. > This email does not constitute an offer or solicitation and nothing > contained in this email constitutes, and should not be construed as, > investment advice. Prospective investors should request offering > materials and consult their own advisers with respect to investment > decisions and inform themselves as to applicable legal requirements, > exchange control regulations and taxes in the countries of their > citizenship, residence or domicile. Past performance is > not indicative of future results. > Winton takes reasonable steps to ensure the accuracy and integrity of its > communications, including emails. However Winton accepts no liability for > any materials transmitted. Emails are not secure > and cannot be guaranteed to be error free. > >
