Hey Matthias, So I have a scenario where I need to batch a group of messages together.
I'm considering creating a new topic for each batch that arrives, i.e batch_<some_id>. Each batch_<id> topic will have a finite number of messages, and then it will remain empty. Essentially these will be throwaway topics. Is there any overhead to there being a lot of these topics, and having a listener for batch_.* , or is this effectively like having one listener for one topic? On Fri, Dec 2, 2016 at 11:09 PM, Matthias J. Sax <matth...@confluent.io> wrote: > 1) There will be once consumer per thread. The number of thread is > defined by the number of instances you start and how many threads you > configure for each instance via StreamConfig parameter > NUM_STREAM_THREADS_CONFIG. Thus, you control this completely by yourself. > > Depending on the number to partitions in your topics, each thread will > process one or multiple partitions. As a partition will be processed by > exactly one thread, the overall number of partitions over all you input > topics limits your max number of thread (if you have more threads, those > will just be idle) > > 2) Thus, there should be no performance issues. Furthermore, if you > create new topic while you application is running -- and if this might > overload you current application -- you can always start new instances > an scale-out you application dynamically -- Kafka Streams is fully elastic. > > Have a look here for more details: > http://docs.confluent.io/current/streams/architecture.html > > > -Matthias > > On 12/2/16 6:23 AM, Ali Akhtar wrote: > > That's pretty useful to know - thanks. > > > > 1) If I listened too foo-.*, and there were 5 foo topics created after > > kafka streaming was running: foo1, foo2, foo3, foo4, foo5, will this > create > > 5 consumers / threads / instances, or will it be just 1 instance that > > receives the messages for all of those topics? > > > > 2) Will this cause issues performance issues if i had a lot of throwaway > > foo topics being created, or will this scale? > > > > On Fri, Dec 2, 2016 at 7:17 PM, Damian Guy <damian....@gmail.com> wrote: > > > >> Hi Ali, > >> > >> The only way KafkaStreams will process new topics after start is if the > >> original stream was defined with a regular expression, i.e, > >> kafka.stream(Pattern.compile("foo-.*"); > >> > >> If any new topics are added after start that match the pattern, then > they > >> will also be consumed. > >> > >> Thanks, > >> Damian > >> > >> On Fri, 2 Dec 2016 at 13:13 Ali Akhtar <ali.rac...@gmail.com> wrote: > >> > >>> Heya, > >>> > >>> Normally, you add your topics and their callbacks to a StreamBuilder, > and > >>> then call KafkaStreams.start() to start ingesting those topics. > >>> > >>> Is it possible to add a new topic to the StreamBuilder, and start > >> ingesting > >>> that as well, after KafkaStreams.start() has been called? > >>> > >>> Thanks. > >>> > >> > > > >
