Thank you very much. Last q - Is it safe to do this from within a call back
processing that topic , once it reaches the last message? (It keeps a
count of how many messages processed vs how many remaining)
On 3 Dec 2016 11:36 a.m., "Matthias J. Sax" <matth...@confluent.io> wrote:
> You can use TopicCommand to delete a topic within Java:
>
> > final TopicCommand.TopicCommandOptions commandOptions = new
> TopicCommand.TopicCommandOptions(new String[]{
> > "--zookeeper", "zookeperHost:2181",
> > "--delete",
> > "--topic", "TOPIC-TO-BE-DELETED"});
> > TopicCommand.deleteTopic(zkUtils, commandOptions);
>
> So you can delete a topic within your Streams app.
>
> -Matthias
>
>
>
> On 12/2/16 9:25 PM, Ali Akhtar wrote:
> > Is there a way to delete the processed topics via streams or the java
> > driver? Or only thru the bash script?
> >
> > On 3 Dec 2016 5:27 a.m., "Matthias J. Sax" <matth...@confluent.io>
> wrote:
> >
> >> If you keep old topics that are completely processed, there would be
> >> increasing overhead, because Streams would try to read from those topics
> >> as long as they exist. Thus, more fetch request will be sent to those
> >> more topics over time, while most fetch request will return without any
> >> new data (as those old topic do not have new data)
> >>
> >> If you delete completely processed topics, there will be no overhead.
> >>
> >> -Matthias
> >>
> >> On 12/2/16 3:58 PM, Ali Akhtar wrote:
> >>> Hey Matthias,
> >>>
> >>> So I have a scenario where I need to batch a group of messages
> together.
> >>>
> >>> I'm considering creating a new topic for each batch that arrives, i.e
> >>> batch_<some_id>.
> >>>
> >>> Each batch_<id> topic will have a finite number of messages, and then
> it
> >>> will remain empty. Essentially these will be throwaway topics.
> >>>
> >>> Is there any overhead to there being a lot of these topics, and having
> a
> >>> listener for batch_.* , or is this effectively like having one listener
> >> for
> >>> one topic?
> >>>
> >>> On Fri, Dec 2, 2016 at 11:09 PM, Matthias J. Sax <
> matth...@confluent.io>
> >>> wrote:
> >>>
> >>>> 1) There will be once consumer per thread. The number of thread is
> >>>> defined by the number of instances you start and how many threads you
> >>>> configure for each instance via StreamConfig parameter
> >>>> NUM_STREAM_THREADS_CONFIG. Thus, you control this completely by
> >> yourself.
> >>>>
> >>>> Depending on the number to partitions in your topics, each thread will
> >>>> process one or multiple partitions. As a partition will be processed
> by
> >>>> exactly one thread, the overall number of partitions over all you
> input
> >>>> topics limits your max number of thread (if you have more threads,
> those
> >>>> will just be idle)
> >>>>
> >>>> 2) Thus, there should be no performance issues. Furthermore, if you
> >>>> create new topic while you application is running -- and if this might
> >>>> overload you current application -- you can always start new instances
> >>>> an scale-out you application dynamically -- Kafka Streams is fully
> >> elastic.
> >>>>
> >>>> Have a look here for more details:
> >>>> http://docs.confluent.io/current/streams/architecture.html
> >>>>
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 12/2/16 6:23 AM, Ali Akhtar wrote:
> >>>>> That's pretty useful to know - thanks.
> >>>>>
> >>>>> 1) If I listened too foo-.*, and there were 5 foo topics created
> after
> >>>>> kafka streaming was running: foo1, foo2, foo3, foo4, foo5, will this
> >>>> create
> >>>>> 5 consumers / threads / instances, or will it be just 1 instance that
> >>>>> receives the messages for all of those topics?
> >>>>>
> >>>>> 2) Will this cause issues performance issues if i had a lot of
> >> throwaway
> >>>>> foo topics being created, or will this scale?
> >>>>>
> >>>>> On Fri, Dec 2, 2016 at 7:17 PM, Damian Guy <damian....@gmail.com>
> >> wrote:
> >>>>>
> >>>>>> Hi Ali,
> >>>>>>
> >>>>>> The only way KafkaStreams will process new topics after start is if
> >> the
> >>>>>> original stream was defined with a regular expression, i.e,
> >>>>>> kafka.stream(Pattern.compile("foo-.*");
> >>>>>>
> >>>>>> If any new topics are added after start that match the pattern, then
> >>>> they
> >>>>>> will also be consumed.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Damian
> >>>>>>
> >>>>>> On Fri, 2 Dec 2016 at 13:13 Ali Akhtar <ali.rac...@gmail.com>
> wrote:
> >>>>>>
> >>>>>>> Heya,
> >>>>>>>
> >>>>>>> Normally, you add your topics and their callbacks to a
> StreamBuilder,
> >>>> and
> >>>>>>> then call KafkaStreams.start() to start ingesting those topics.
> >>>>>>>
> >>>>>>> Is it possible to add a new topic to the StreamBuilder, and start
> >>>>>> ingesting
> >>>>>>> that as well, after KafkaStreams.start() has been called?
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>
