Thanks for the info. I see there are tons of parameters but is there a place that lists some important performance specific parameters? On Fri, Oct 17, 2014 at 2:43 PM, Gwen Shapira <gshap...@cloudera.com> wrote:
> If I understand correctly (and I'll be happy if someone who knows more > will jump in and correct me): > > The Sync/Async part is not between the producer and the broker. Its > between you and the producer. The Sync producer takes your message and > immediately contacts the broker, sends the message, either wait for > acks or not and returns. The Async producer takes your message and > immediately returns. The producer will send the message to the broker > at some time later, batching multiple requests for efficiency and > throughput. > > So yeah, I think you got it mostly right. Just note that the producer > doesn't wait on .send, the producer executes the send - either > returning immediately (if async) or when it managed to contact the > broker (if sync). > > Gwen > > On Fri, Oct 17, 2014 at 4:38 PM, Mohit Anchlia <mohitanch...@gmail.com> > wrote: > > My understanding of sync is that producer waits on .send until Kafka > > receives the message. And async means it just dispatches the message > > without any gurantees that message is delivered. Did I get that part > right? > > On Fri, Oct 17, 2014 at 1:28 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > >> Sorry if I'm confusing you :) > >> > >> Kafka 0.8.1.1 has two producers sync and async. You are using the sync > >> producer without waiting for acks. I hope this helps? > >> > >> Regardless, did you check if the partition got created? are you able > >> to produce messages? are you able to consume them? > >> > >> Gwen > >> > >> On Fri, Oct 17, 2014 at 4:13 PM, Mohit Anchlia <mohitanch...@gmail.com> > >> wrote: > >> > Still don't understand the difference. If it's not waiting for the ack > >> then > >> > doesn't it make async? > >> > On Fri, Oct 17, 2014 at 12:55 PM, <gshap...@cloudera.com> wrote: > >> > > >> >> Its using the sync producer without waiting for any broker to > >> acknowledge > >> >> the write. This explains the lack of errors you are seeing. > >> >> > >> >> — > >> >> Sent from Mailbox > >> >> > >> >> On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia < > mohitanch...@gmail.com> > >> >> wrote: > >> >> > >> >> > Little confused :) From one of the examples I am using property > >> >> > request.required.acks=0, > >> >> > I thought this sets the producer to be async? > >> >> > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira < > gshap...@cloudera.com > >> > > >> >> > wrote: > >> >> >> 0.8.1.1 producer is Sync by default, and you can set > producer.type to > >> >> >> async if needed. > >> >> >> > >> >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia < > >> mohitanch...@gmail.com> > >> >> >> wrote: > >> >> >> > Thanks! How can I tell if I am using async producer? I thought > all > >> the > >> >> >> > sends are async in nature > >> >> >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira < > >> gshap...@cloudera.com > >> >> > > >> >> >> > wrote: > >> >> >> > > >> >> >> >> If you have "auto.create.topics.enable" set to "true" > (default), > >> >> >> >> producing to a topic creates it. > >> >> >> >> > >> >> >> >> Its a bit tricky because the "send" that creates the topic can > >> fail > >> >> >> >> with "leader not found" or similar issue. retrying few times > will > >> >> >> >> eventually succeed as the topic gets created and the leader > gets > >> >> >> >> elected. > >> >> >> >> > >> >> >> >> Is it possible that you are not getting errors because you are > >> using > >> >> >> >> async producer? > >> >> >> >> > >> >> >> >> Also "no messages are delivered" can have many causes. Check if > >> the > >> >> >> >> topic exists using: > >> >> >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 > >> >> >> >> > >> >> >> >> Perhaps the topic was created and the issue is elsewhere (the > >> >> consumer > >> >> >> >> is a usual suspect! perhaps look in the FAQ for tips with that > >> issue) > >> >> >> >> > >> >> >> >> Gwen > >> >> >> >> > >> >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < > >> >> mohitanch...@gmail.com > >> >> >> > > >> >> >> >> wrote: > >> >> >> >> > Is Kafka supposed to throw exception if topic doesn't exist? > It > >> >> >> appears > >> >> >> >> > that there is no exception thrown even though no messages are > >> >> >> delivered > >> >> >> >> and > >> >> >> >> > there are errors logged in Kafka logs. > >> >> >> >> > >> >> >> > >> >> > >> >
