>From the document, the new topics are picked up after some period of time.
My question is: how long is the duration before the new topics are detected and is the duration configurable? Much thanks! -----Original Message----- From: Matthias J. Sax [mailto:matth...@confluent.io] Sent: 2016年11月10日 7:28 To: users@kafka.apache.org Subject: Re: Is it possible to resubcribe KafkaStreams in runtime to different set of topics? -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Yes. If new topics are created that match the regex, Streams will automatically subscribed to them. - -Mathias On 11/9/16 3:11 PM, Haopu Wang wrote: > Hi, do you mean that the new matched topics should be consumed > after the regex subscription has been established? Thanks! > > -----Original Message----- From: Guozhang Wang > [mailto:wangg...@gmail.com] Sent: 2016年11月10日 3:41 To: > users@kafka.apache.org Subject: Re: Is it possible to resubcribe > KafkaStreams in runtime to different set of topics? > > Timur, > > As Michael said one thing you can consider is the regex > subscription if those topics have some common prefix / suffix / > pattern. > > Another idea I can think of is to have two Streams app, one for > reading the input topics and pipe them into the repartitioned > intermediate topics that can be controlled by the streams app > itself, and the second app reading from the intermediate topics and > process the rest of the topology. When input topics change, you can > simple bounce the first app only, which is a very thin client and > hence should take much less time to restart. > > > Guozhang > > > On Wed, Nov 9, 2016 at 12:30 PM, Michael Noll > <mich...@confluent.io> wrote: > >> I am not aware of any short-term plans to support that, but >> perhaps others in the community / mailing list are. >> >> On Wed, Nov 9, 2016 at 11:15 AM, Timur Yusupov >> <ttyusu...@gmail.com> wrote: >> >>> Are there any nearest plans to support that? >>> >>> On Wed, Nov 9, 2016 at 1:11 PM, Michael Noll >>> <mich...@confluent.io> >> wrote: >>> >>>> This is not possible at the moment. However, depending on >>>> your use >> case, >>>> you might be able to leverage regex topic subscriptions >>>> (think: "b*" to read from all topics starting with letter >>>> `b`). >>>> >>>> On Wed, Nov 9, 2016 at 10:56 AM, Timur Yusupov >>>> <ttyusu...@gmail.com> wrote: >>>> >>>>> Hello, >>>>> >>>>> In our system it is possible to add/remove topics in >>>>> runtime and we >> are >>>>> trying to use KafkaStreams for incoming messages >>>>> processing. >>>>> >>>>> It is possible to resubscribe KafkaStreams instance to >>>>> updated set of topics? >>>>> >>>>> For now I see the only way is to shutdown exiting >>>>> KafkaStreams >> instance >>>> and >>>>> start a new one, but sometimes that takes up to 30-40 >>>>> seconds... >>>>> >>>>> Thanks, Timur. >>>>> >>>> >>> >> > > > -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJYI7DxAAoJECnhiMLycopPrOoP/2+6CXMFvE1jl1lvuR46fFeR XQoNB1m0waXOcSBfQVJnIy0sOCQakMHeW6rpc1Ji0w43K80H5Z7K0fC83X/awg1u 6LiEaAymTNaFuKyd49CzXSLiwKZKZiw4NpIXvJWPp5gw5EQazEfO84gA5uDBOm5F KdXATw8dREhv6PZtnxQ1+1vKsobNS1iGyI/ecjbDVjSFQtCY/97jY0xgoxe3W3H6 2+1NgxVlQ6o27Snkk7umcEPAmkPN1q+HhkZ9pBRUt9xr7MfAZSQqmlgoo28kNlAB M4CX+wV8XfL4JfOJqmvSv4FaLLuAKDvVyEHSfiNzxEayeIgnZTqlahRHkr7jW8T6 /TiJW+iwuAeERJwfH+0GoQgZDFodVh3/JfeRdTiS4KpnK6MywQP37H3Wp7y90sQt eYC9GOr0ZZF1xHjhYN974Wlb+RHmPY8c/FNcXkaZP6nBvUk5NoP4mvGJGImFZbXr 9/ENrJRW8HKc7XyXaL/vg8v0d7Hvpv3XTyHsfhUbR9NSufiML655cGRtDRF7ebDb MK2RyngdKf0YvXLv4uyQqWhIXSQvtPrqvJbzuUJhTIBOToNf0uupvXJ8CdVQPg80 ev7/eDFFTzUkPhYwinfP/Dn2lF6ob2BBmNeDI68mO/Z0vez/zda0lu3ofITo5mDv cTYKRiimCr471hNFmpPN =EYdQ -----END PGP SIGNATURE-----
