By process I mean a JVM process (if you're using the JVM clients and for
your app).
Thanks
Tom Crayford
Heroku Kafka
On Wednesday, 25 May 2016, Hafsa Asif <hafsa.a...@matchinguu.com> wrote:
> A very good question from Joe. I have also the same question.
>
> Hafsa
>
> 2016-05-24 18:00 GMT+02:00 Joe San <codeintheo...@gmail.com <javascript:;>
> >:
>
> > Interesting discussion!
> >
> > What do you mean here by a process? Is that a thread or the JVM process?
> >
> > On Tue, May 24, 2016 at 5:49 PM, Tom Crayford <tcrayf...@heroku.com
> <javascript:;>>
> > wrote:
> >
> > > Aha, yep that helped a lot.
> > >
> > > One producer per process. There's not really a per producer topic
> limit.
> > > There's buffering and batching space, but assuming you have sufficient
> > > memory (which is by the partition, not by topic), you'll be fine.
> > >
> > > Thanks
> > >
> > > Tom Crayford
> > > Heroku Kafka
> > >
> > > On Tue, May 24, 2016 at 4:46 PM, Hafsa Asif <hafsa.a...@matchinguu.com
> <javascript:;>>
> > > wrote:
> > >
> > > > One more question:
> > > > How many topics can be easily handled by one producer?
> > > >
> > > > Hafsa
> > > >
> > > > 2016-05-24 17:39 GMT+02:00 Hafsa Asif <hafsa.a...@matchinguu.com
> <javascript:;>>:
> > > >
> > > > > Ok, let me rephrase (may be I am not using correct terms):
> > > > > Simply consider I have 2 topics, and I have both Java and NodeJS
> > client
> > > > > for Kafka.
> > > > >
> > > > > *NodeJS:*
> > > > > Is it good that I write two producers per each topic like that :
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *var producer1 = new Producer(client);producer1.on('ready',
> function
> > ()
> > > > > {});producer1.on('error', function (err) {
> > > });producer1.send(payloads,
> > > > > cb);var producer2 = new Producer(client);producer2.on('ready',
> > function
> > > > ()
> > > > > {});producer2.on('error', function (err) {
> > > });producer2.send(payloads,
> > > > > cb);*
> > > > > Or, I should create one producer for all 2 topics.
> > > > >
> > > > >
> > > > >
> > > > > *Java:*Is it good that I write two producers per each topic like
> > that :
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > *private static Producer<Integer, String> producer1;private static
> > > > > Producer<Integer, String> producer2;producer1 = new Producer<>(new
> > > > > ProducerConfig(properties));producer2 = new Producer<>(new
> > > > > ProducerConfig(properties));*
> > > > >
> > > > > Or, I should create one producer for all 2 topics.
> > > > >
> > > > > Suggest your answer in the light of my estimations (10 topics and 1
> > > > > million records per each topic in next week)
> > > > >
> > > > > Best Regards,
> > > > > Hafsa
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2016-05-24 17:23 GMT+02:00 Tom Crayford <tcrayf...@heroku.com
> <javascript:;>>:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> I think I'm a bit confused. When you say "one producer per topic",
> > do
> > > > you
> > > > >> mean one instance of the JVM application that's producing per
> topic?
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >> Tom
> > > > >>
> > > > >> On Tue, May 24, 2016 at 4:19 PM, Hafsa Asif <
> > > hafsa.a...@matchinguu.com <javascript:;>>
> > > > >> wrote:
> > > > >>
> > > > >> > Tom,
> > > > >> >
> > > > >> > Thank you for your answer. No, I am talking about one PRODUCER
> for
> > > > each
> > > > >> > topic, not one instance of same producer class. I am asking for
> > > > general
> > > > >> > concept only.
> > > > >> > Actually we are just growing and not so much far from the case
> > of 1
> > > > >> > million records per sec. Just considering our future case, I
> need
> > > your
> > > > >> > suggestion in more detail, that in general is it a good practice
> > to:
> > > > >> > 1. Prepare a single producer for multiple topics (consider 10
> > > topics)
> > > > .
> > > > >> > 2. Prepare 10 producers for 10 topics respectively.
> > > > >> >
> > > > >> > Your answer is quite satisfying for me, but I need more details
> so
> > > > that
> > > > >> I
> > > > >> > can convince my team in a good way.
> > > > >> >
> > > > >> > Best Regards,
> > > > >> > Hafsa
> > > > >> >
> > > > >> > 2016-05-24 16:11 GMT+02:00 Tom Crayford <tcrayf...@heroku.com
> <javascript:;>>:
> > > > >> >
> > > > >> > > Is that "one instance of the producer class per topic"? I'd
> > > > recommend
> > > > >> > just
> > > > >> > > having a single producer shared per process.
> > > > >> > >
> > > > >> > > 1 million records in a week is not very many records, it works
> > > down
> > > > to
> > > > >> > ~1.6
> > > > >> > > records a second on average, which is nothing (we typically
> see
> > 1
> > > > >> > million+
> > > > >> > > messages per second on our clusters). Or maybe your load is
> > > spikier
> > > > >> than
> > > > >> > > that?
> > > > >> > >
> > > > >> > > Generally if you have multiple producer instances they will
> fail
> > > > >> slightly
> > > > >> > > differently, but most failures that hit one (e.g. a broker
> going
> > > > down
> > > > >> and
> > > > >> > > the controller not changing over the leader fast enough).
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > >
> > > > >> > > Tom Crayford
> > > > >> > > Heroku Kafka
> > > > >> > >
> > > > >> > > On Tue, May 24, 2016 at 3:03 PM, Hafsa Asif <
> > > > >> hafsa.a...@matchinguu.com <javascript:;>>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Hello Folks,
> > > > >> > > >
> > > > >> > > > I am using Kafka (0.9) in my company and it is expected that
> > we
> > > > are
> > > > >> > going
> > > > >> > > > to receive 1 million records in next week. I have many
> topics
> > > for
> > > > >> > solely
> > > > >> > > > different purposes. Is it good that I define one producer
> per
> > > > topic
> > > > >> or
> > > > >> > > > create one producer for every topic?
> > > > >> > > >
> > > > >> > > > Right now, I have only 4 topics and each one is expected to
> > > > receive
> > > > >> 1
> > > > >> > > > million record in next week and after 4 months, we will
> > receive
> > > 10
> > > > >> > > million
> > > > >> > > > records.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Is it possible in Kafka that if one producer fails, then
> other
> > > > >> producer
> > > > >> > > > also does not work? Please also suggest the safe strategy to
> > go.
> > > > >> > > >
> > > > >> > > > Best Regards,
> > > > >> > > > Hafsa
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
