Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Yes. You need to use Processor API to access the topic name via ProcessorContext that is given via Processor#init(). If you use DSL, you can mix-and-match with Processor API via KStream#process() - -Matthias On 11/14/16 12:42 AM, Timur Yusupov

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

How can I distinguish messages from different topics within one KStream if I subscribed to topics using regex ? On Thu, Nov 10, 2016 at 7:32 AM, Hans Jespersen wrote: > I believe that the new topics are picked up at the next metadata refresh > which is controlled by the

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

I believe that the new topics are picked up at the next metadata refresh which is controlled by the metadata.max.age.ms parameter. The default value is 30 (which is 5 minutes). -hans /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ On

RE: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

>From the document, the new topics are picked up after some period of time. My question is: how long is the duration before the new topics are detected and is the duration configurable? Much thanks! -Original Message- From: Matthias J. Sax [mailto:matth...@confluent.io] Sent:

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Yes. If new topics are created that match the regex, Streams will automatically subscribed to them. - -Mathias On 11/9/16 3:11 PM, Haopu Wang wrote: > Hi, do you mean that the new matched topics should be consumed > after the regex subscription

RE: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

Hi, do you mean that the new matched topics should be consumed after the regex subscription has been established? Thanks! -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: 2016年11月10日 3:41 To: users@kafka.apache.org Subject: Re: Is it possible to resubcribe

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

Timur, As Michael said one thing you can consider is the regex subscription if those topics have some common prefix / suffix / pattern. Another idea I can think of is to have two Streams app, one for reading the input topics and pipe them into the repartitioned intermediate topics that can be

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

I am not aware of any short-term plans to support that, but perhaps others in the community / mailing list are. On Wed, Nov 9, 2016 at 11:15 AM, Timur Yusupov wrote: > Are there any nearest plans to support that? > > On Wed, Nov 9, 2016 at 1:11 PM, Michael Noll

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

Are there any nearest plans to support that? On Wed, Nov 9, 2016 at 1:11 PM, Michael Noll wrote: > This is not possible at the moment. However, depending on your use case, > you might be able to leverage regex topic subscriptions (think: "b*" to > read from all topics

Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics?

This is not possible at the moment. However, depending on your use case, you might be able to leverage regex topic subscriptions (think: "b*" to read from all topics starting with letter `b`). On Wed, Nov 9, 2016 at 10:56 AM, Timur Yusupov wrote: > Hello, > > In our system