A very good question from Joe. I have also the same question.
Hafsa
2016-05-24 18:00 GMT+02:00 Joe San <codeintheo...@gmail.com>:
> Interesting discussion!
>
> What do you mean here by a process? Is that a thread or the JVM process?
>
> On Tue, May 24, 2016 at 5:49 PM, Tom Crayford <tcrayf...@heroku.com>
> wrote:
>
> > Aha, yep that helped a lot.
> >
> > One producer per process. There's not really a per producer topic limit.
> > There's buffering and batching space, but assuming you have sufficient
> > memory (which is by the partition, not by topic), you'll be fine.
> >
> > Thanks
> >
> > Tom Crayford
> > Heroku Kafka
> >
> > On Tue, May 24, 2016 at 4:46 PM, Hafsa Asif <hafsa.a...@matchinguu.com>
> > wrote:
> >
> > > One more question:
> > > How many topics can be easily handled by one producer?
> > >
> > > Hafsa
> > >
> > > 2016-05-24 17:39 GMT+02:00 Hafsa Asif <hafsa.a...@matchinguu.com>:
> > >
> > > > Ok, let me rephrase (may be I am not using correct terms):
> > > > Simply consider I have 2 topics, and I have both Java and NodeJS
> client
> > > > for Kafka.
> > > >
> > > > *NodeJS:*
> > > > Is it good that I write two producers per each topic like that :
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > *var producer1 = new Producer(client);producer1.on('ready', function
> ()
> > > > {});producer1.on('error', function (err) {
> > });producer1.send(payloads,
> > > > cb);var producer2 = new Producer(client);producer2.on('ready',
> function
> > > ()
> > > > {});producer2.on('error', function (err) {
> > });producer2.send(payloads,
> > > > cb);*
> > > > Or, I should create one producer for all 2 topics.
> > > >
> > > >
> > > >
> > > > *Java:*Is it good that I write two producers per each topic like
> that :
> > > >
> > > >
> > > >
> > > >
> > > > *private static Producer<Integer, String> producer1;private static
> > > > Producer<Integer, String> producer2;producer1 = new Producer<>(new
> > > > ProducerConfig(properties));producer2 = new Producer<>(new
> > > > ProducerConfig(properties));*
> > > >
> > > > Or, I should create one producer for all 2 topics.
> > > >
> > > > Suggest your answer in the light of my estimations (10 topics and 1
> > > > million records per each topic in next week)
> > > >
> > > > Best Regards,
> > > > Hafsa
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2016-05-24 17:23 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
> > > >
> > > >> Hi,
> > > >>
> > > >> I think I'm a bit confused. When you say "one producer per topic",
> do
> > > you
> > > >> mean one instance of the JVM application that's producing per topic?
> > > >>
> > > >> Thanks
> > > >>
> > > >> Tom
> > > >>
> > > >> On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif <
> > hafsa.a...@matchinguu.com>
> > > >> wrote:
> > > >>
> > > >> > Tom,
> > > >> >
> > > >> > Thank you for your answer. No, I am talking about one PRODUCER for
> > > each
> > > >> > topic, not one instance of same producer class. I am asking for
> > > general
> > > >> > concept only.
> > > >> > Actually we are just growing and not so much far from the case
> of 1
> > > >> > million records per sec. Just considering our future case, I need
> > your
> > > >> > suggestion in more detail, that in general is it a good practice
> to:
> > > >> > 1. Prepare a single producer for multiple topics (consider 10
> > topics)
> > > .
> > > >> > 2. Prepare 10 producers for 10 topics respectively.
> > > >> >
> > > >> > Your answer is quite satisfying for me, but I need more details so
> > > that
> > > >> I
> > > >> > can convince my team in a good way.
> > > >> >
> > > >> > Best Regards,
> > > >> > Hafsa
> > > >> >
> > > >> > 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com>:
> > > >> >
> > > >> > > Is that "one instance of the producer class per topic"? I'd
> > > recommend
> > > >> > just
> > > >> > > having a single producer shared per process.
> > > >> > >
> > > >> > > 1 million records in a week is not very many records, it works
> > down
> > > to
> > > >> > ~1.6
> > > >> > > records a second on average, which is nothing (we typically see
> 1
> > > >> > million+
> > > >> > > messages per second on our clusters). Or maybe your load is
> > spikier
> > > >> than
> > > >> > > that?
> > > >> > >
> > > >> > > Generally if you have multiple producer instances they will fail
> > > >> slightly
> > > >> > > differently, but most failures that hit one (e.g. a broker going
> > > down
> > > >> and
> > > >> > > the controller not changing over the leader fast enough).
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > Tom Crayford
> > > >> > > Heroku Kafka
> > > >> > >
> > > >> > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <
> > > >> hafsa.a...@matchinguu.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hello Folks,
> > > >> > > >
> > > >> > > > I am using Kafka (0.9) in my company and it is expected that
> we
> > > are
> > > >> > going
> > > >> > > > to receive 1 million records in next week. I have many topics
> > for
> > > >> > solely
> > > >> > > > different purposes. Is it good that I define one producer per
> > > topic
> > > >> or
> > > >> > > > create one producer for every topic?
> > > >> > > >
> > > >> > > > Right now, I have only 4 topics and each one is expected to
> > > receive
> > > >> 1
> > > >> > > > million record in next week and after 4 months, we will
> receive
> > 10
> > > >> > > million
> > > >> > > > records.
> > > >> > > >
> > > >> > > >
> > > >> > > > Is it possible in Kafka that if one producer fails, then other
> > > >> producer
> > > >> > > > also does not work? Please also suggest the safe strategy to
> go.
> > > >> > > >
> > > >> > > > Best Regards,
> > > >> > > > Hafsa
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
