Re: Topic doesn't exist exception
There's some good advice here: https://kafka.apache.org/081/ops.html And you may enjoy this post too: http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/ On Fri, Oct 17, 2014 at 5:52 PM, Mohit Anchlia wrote: > Thanks for the info. I see there are tons of parameters but is there a > place that lists some important performance specific parameters? > On Fri, Oct 17, 2014 at 2:43 PM, Gwen Shapira wrote: > >> If I understand correctly (and I'll be happy if someone who knows more >> will jump in and correct me): >> >> The Sync/Async part is not between the producer and the broker. Its >> between you and the producer. The Sync producer takes your message and >> immediately contacts the broker, sends the message, either wait for >> acks or not and returns. The Async producer takes your message and >> immediately returns. The producer will send the message to the broker >> at some time later, batching multiple requests for efficiency and >> throughput. >> >> So yeah, I think you got it mostly right. Just note that the producer >> doesn't wait on .send, the producer executes the send - either >> returning immediately (if async) or when it managed to contact the >> broker (if sync). >> >> Gwen >> >> On Fri, Oct 17, 2014 at 4:38 PM, Mohit Anchlia >> wrote: >> > My understanding of sync is that producer waits on .send until Kafka >> > receives the message. And async means it just dispatches the message >> > without any gurantees that message is delivered. Did I get that part >> right? >> > On Fri, Oct 17, 2014 at 1:28 PM, Gwen Shapira >> wrote: >> > >> >> Sorry if I'm confusing you :) >> >> >> >> Kafka 0.8.1.1 has two producers sync and async. You are using the sync >> >> producer without waiting for acks. I hope this helps? >> >> >> >> Regardless, did you check if the partition got created? are you able >> >> to produce messages? are you able to consume them? >> >> >> >> Gwen >> >> >> >> On Fri, Oct 17, 2014 at 4:13 PM, Mohit Anchlia >> >> wrote: >> >> > Still don't understand the difference. If it's not waiting for the ack >> >> then >> >> > doesn't it make async? >> >> > On Fri, Oct 17, 2014 at 12:55 PM, wrote: >> >> > >> >> >> Its using the sync producer without waiting for any broker to >> >> acknowledge >> >> >> the write. This explains the lack of errors you are seeing. >> >> >> >> >> >> — >> >> >> Sent from Mailbox >> >> >> >> >> >> On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia < >> mohitanch...@gmail.com> >> >> >> wrote: >> >> >> >> >> >> > Little confused :) From one of the examples I am using property >> >> >> > request.required.acks=0, >> >> >> > I thought this sets the producer to be async? >> >> >> > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira < >> gshap...@cloudera.com >> >> > >> >> >> > wrote: >> >> >> >> 0.8.1.1 producer is Sync by default, and you can set >> producer.type to >> >> >> >> async if needed. >> >> >> >> >> >> >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia < >> >> mohitanch...@gmail.com> >> >> >> >> wrote: >> >> >> >> > Thanks! How can I tell if I am using async producer? I thought >> all >> >> the >> >> >> >> > sends are async in nature >> >> >> >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira < >> >> gshap...@cloudera.com >> >> >> > >> >> >> >> > wrote: >> >> >> >> > >> >> >> >> >> If you have "auto.create.topics.enable" set to "true" >> (default), >> >> >> >> >> producing to a topic creates it. >> >> >> >> >> >> >> >> >> >> Its a bit tricky because the "send" that creates the topic can >> >> fail >> >> >> >> >> with "leader not found" or similar issue. retrying few times >> will >> >> >> >> >> eventually succeed as the topic gets created and the leader >> gets >> >> >> >> >> elected. >> >> >> >> >> >> >> >> >> >> Is it possible that you are not getting errors because you are >> >> using >> >> >> >> >> async producer? >> >> >> >> >> >> >> >> >> >> Also "no messages are delivered" can have many causes. Check if >> >> the >> >> >> >> >> topic exists using: >> >> >> >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 >> >> >> >> >> >> >> >> >> >> Perhaps the topic was created and the issue is elsewhere (the >> >> >> consumer >> >> >> >> >> is a usual suspect! perhaps look in the FAQ for tips with that >> >> issue) >> >> >> >> >> >> >> >> >> >> Gwen >> >> >> >> >> >> >> >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < >> >> >> mohitanch...@gmail.com >> >> >> >> > >> >> >> >> >> wrote: >> >> >> >> >> > Is Kafka supposed to throw exception if topic doesn't exist? >> It >> >> >> >> appears >> >> >> >> >> > that there is no exception thrown even though no messages are >> >> >> >> delivered >> >> >> >> >> and >> >> >> >> >> > there are errors logged in Kafka logs. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>
Re: Topic doesn't exist exception
Thanks for the info. I see there are tons of parameters but is there a place that lists some important performance specific parameters? On Fri, Oct 17, 2014 at 2:43 PM, Gwen Shapira wrote: > If I understand correctly (and I'll be happy if someone who knows more > will jump in and correct me): > > The Sync/Async part is not between the producer and the broker. Its > between you and the producer. The Sync producer takes your message and > immediately contacts the broker, sends the message, either wait for > acks or not and returns. The Async producer takes your message and > immediately returns. The producer will send the message to the broker > at some time later, batching multiple requests for efficiency and > throughput. > > So yeah, I think you got it mostly right. Just note that the producer > doesn't wait on .send, the producer executes the send - either > returning immediately (if async) or when it managed to contact the > broker (if sync). > > Gwen > > On Fri, Oct 17, 2014 at 4:38 PM, Mohit Anchlia > wrote: > > My understanding of sync is that producer waits on .send until Kafka > > receives the message. And async means it just dispatches the message > > without any gurantees that message is delivered. Did I get that part > right? > > On Fri, Oct 17, 2014 at 1:28 PM, Gwen Shapira > wrote: > > > >> Sorry if I'm confusing you :) > >> > >> Kafka 0.8.1.1 has two producers sync and async. You are using the sync > >> producer without waiting for acks. I hope this helps? > >> > >> Regardless, did you check if the partition got created? are you able > >> to produce messages? are you able to consume them? > >> > >> Gwen > >> > >> On Fri, Oct 17, 2014 at 4:13 PM, Mohit Anchlia > >> wrote: > >> > Still don't understand the difference. If it's not waiting for the ack > >> then > >> > doesn't it make async? > >> > On Fri, Oct 17, 2014 at 12:55 PM, wrote: > >> > > >> >> Its using the sync producer without waiting for any broker to > >> acknowledge > >> >> the write. This explains the lack of errors you are seeing. > >> >> > >> >> — > >> >> Sent from Mailbox > >> >> > >> >> On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia < > mohitanch...@gmail.com> > >> >> wrote: > >> >> > >> >> > Little confused :) From one of the examples I am using property > >> >> > request.required.acks=0, > >> >> > I thought this sets the producer to be async? > >> >> > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira < > gshap...@cloudera.com > >> > > >> >> > wrote: > >> >> >> 0.8.1.1 producer is Sync by default, and you can set > producer.type to > >> >> >> async if needed. > >> >> >> > >> >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia < > >> mohitanch...@gmail.com> > >> >> >> wrote: > >> >> >> > Thanks! How can I tell if I am using async producer? I thought > all > >> the > >> >> >> > sends are async in nature > >> >> >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira < > >> gshap...@cloudera.com > >> >> > > >> >> >> > wrote: > >> >> >> > > >> >> >> >> If you have "auto.create.topics.enable" set to "true" > (default), > >> >> >> >> producing to a topic creates it. > >> >> >> >> > >> >> >> >> Its a bit tricky because the "send" that creates the topic can > >> fail > >> >> >> >> with "leader not found" or similar issue. retrying few times > will > >> >> >> >> eventually succeed as the topic gets created and the leader > gets > >> >> >> >> elected. > >> >> >> >> > >> >> >> >> Is it possible that you are not getting errors because you are > >> using > >> >> >> >> async producer? > >> >> >> >> > >> >> >> >> Also "no messages are delivered" can have many causes. Check if > >> the > >> >> >> >> topic exists using: > >> >> >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 > >> >> >> >> > >> >> >> >> Perhaps the topic was created and the issue is elsewhere (the > >> >> consumer > >> >> >> >> is a usual suspect! perhaps look in the FAQ for tips with that > >> issue) > >> >> >> >> > >> >> >> >> Gwen > >> >> >> >> > >> >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < > >> >> mohitanch...@gmail.com > >> >> >> > > >> >> >> >> wrote: > >> >> >> >> > Is Kafka supposed to throw exception if topic doesn't exist? > It > >> >> >> appears > >> >> >> >> > that there is no exception thrown even though no messages are > >> >> >> delivered > >> >> >> >> and > >> >> >> >> > there are errors logged in Kafka logs. > >> >> >> >> > >> >> >> > >> >> > >> >
Re: Topic doesn't exist exception
If I understand correctly (and I'll be happy if someone who knows more will jump in and correct me): The Sync/Async part is not between the producer and the broker. Its between you and the producer. The Sync producer takes your message and immediately contacts the broker, sends the message, either wait for acks or not and returns. The Async producer takes your message and immediately returns. The producer will send the message to the broker at some time later, batching multiple requests for efficiency and throughput. So yeah, I think you got it mostly right. Just note that the producer doesn't wait on .send, the producer executes the send - either returning immediately (if async) or when it managed to contact the broker (if sync). Gwen On Fri, Oct 17, 2014 at 4:38 PM, Mohit Anchlia wrote: > My understanding of sync is that producer waits on .send until Kafka > receives the message. And async means it just dispatches the message > without any gurantees that message is delivered. Did I get that part right? > On Fri, Oct 17, 2014 at 1:28 PM, Gwen Shapira wrote: > >> Sorry if I'm confusing you :) >> >> Kafka 0.8.1.1 has two producers sync and async. You are using the sync >> producer without waiting for acks. I hope this helps? >> >> Regardless, did you check if the partition got created? are you able >> to produce messages? are you able to consume them? >> >> Gwen >> >> On Fri, Oct 17, 2014 at 4:13 PM, Mohit Anchlia >> wrote: >> > Still don't understand the difference. If it's not waiting for the ack >> then >> > doesn't it make async? >> > On Fri, Oct 17, 2014 at 12:55 PM, wrote: >> > >> >> Its using the sync producer without waiting for any broker to >> acknowledge >> >> the write. This explains the lack of errors you are seeing. >> >> >> >> — >> >> Sent from Mailbox >> >> >> >> On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia >> >> wrote: >> >> >> >> > Little confused :) From one of the examples I am using property >> >> > request.required.acks=0, >> >> > I thought this sets the producer to be async? >> >> > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira > > >> >> > wrote: >> >> >> 0.8.1.1 producer is Sync by default, and you can set producer.type to >> >> >> async if needed. >> >> >> >> >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia < >> mohitanch...@gmail.com> >> >> >> wrote: >> >> >> > Thanks! How can I tell if I am using async producer? I thought all >> the >> >> >> > sends are async in nature >> >> >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira < >> gshap...@cloudera.com >> >> > >> >> >> > wrote: >> >> >> > >> >> >> >> If you have "auto.create.topics.enable" set to "true" (default), >> >> >> >> producing to a topic creates it. >> >> >> >> >> >> >> >> Its a bit tricky because the "send" that creates the topic can >> fail >> >> >> >> with "leader not found" or similar issue. retrying few times will >> >> >> >> eventually succeed as the topic gets created and the leader gets >> >> >> >> elected. >> >> >> >> >> >> >> >> Is it possible that you are not getting errors because you are >> using >> >> >> >> async producer? >> >> >> >> >> >> >> >> Also "no messages are delivered" can have many causes. Check if >> the >> >> >> >> topic exists using: >> >> >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 >> >> >> >> >> >> >> >> Perhaps the topic was created and the issue is elsewhere (the >> >> consumer >> >> >> >> is a usual suspect! perhaps look in the FAQ for tips with that >> issue) >> >> >> >> >> >> >> >> Gwen >> >> >> >> >> >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < >> >> mohitanch...@gmail.com >> >> >> > >> >> >> >> wrote: >> >> >> >> > Is Kafka supposed to throw exception if topic doesn't exist? It >> >> >> appears >> >> >> >> > that there is no exception thrown even though no messages are >> >> >> delivered >> >> >> >> and >> >> >> >> > there are errors logged in Kafka logs. >> >> >> >> >> >> >> >> >> >>
Re: Topic doesn't exist exception
My understanding of sync is that producer waits on .send until Kafka receives the message. And async means it just dispatches the message without any gurantees that message is delivered. Did I get that part right? On Fri, Oct 17, 2014 at 1:28 PM, Gwen Shapira wrote: > Sorry if I'm confusing you :) > > Kafka 0.8.1.1 has two producers sync and async. You are using the sync > producer without waiting for acks. I hope this helps? > > Regardless, did you check if the partition got created? are you able > to produce messages? are you able to consume them? > > Gwen > > On Fri, Oct 17, 2014 at 4:13 PM, Mohit Anchlia > wrote: > > Still don't understand the difference. If it's not waiting for the ack > then > > doesn't it make async? > > On Fri, Oct 17, 2014 at 12:55 PM, wrote: > > > >> Its using the sync producer without waiting for any broker to > acknowledge > >> the write. This explains the lack of errors you are seeing. > >> > >> — > >> Sent from Mailbox > >> > >> On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia > >> wrote: > >> > >> > Little confused :) From one of the examples I am using property > >> > request.required.acks=0, > >> > I thought this sets the producer to be async? > >> > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira > > >> > wrote: > >> >> 0.8.1.1 producer is Sync by default, and you can set producer.type to > >> >> async if needed. > >> >> > >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia < > mohitanch...@gmail.com> > >> >> wrote: > >> >> > Thanks! How can I tell if I am using async producer? I thought all > the > >> >> > sends are async in nature > >> >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira < > gshap...@cloudera.com > >> > > >> >> > wrote: > >> >> > > >> >> >> If you have "auto.create.topics.enable" set to "true" (default), > >> >> >> producing to a topic creates it. > >> >> >> > >> >> >> Its a bit tricky because the "send" that creates the topic can > fail > >> >> >> with "leader not found" or similar issue. retrying few times will > >> >> >> eventually succeed as the topic gets created and the leader gets > >> >> >> elected. > >> >> >> > >> >> >> Is it possible that you are not getting errors because you are > using > >> >> >> async producer? > >> >> >> > >> >> >> Also "no messages are delivered" can have many causes. Check if > the > >> >> >> topic exists using: > >> >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 > >> >> >> > >> >> >> Perhaps the topic was created and the issue is elsewhere (the > >> consumer > >> >> >> is a usual suspect! perhaps look in the FAQ for tips with that > issue) > >> >> >> > >> >> >> Gwen > >> >> >> > >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < > >> mohitanch...@gmail.com > >> >> > > >> >> >> wrote: > >> >> >> > Is Kafka supposed to throw exception if topic doesn't exist? It > >> >> appears > >> >> >> > that there is no exception thrown even though no messages are > >> >> delivered > >> >> >> and > >> >> >> > there are errors logged in Kafka logs. > >> >> >> > >> >> > >> >
Re: Topic doesn't exist exception
Sorry if I'm confusing you :) Kafka 0.8.1.1 has two producers sync and async. You are using the sync producer without waiting for acks. I hope this helps? Regardless, did you check if the partition got created? are you able to produce messages? are you able to consume them? Gwen On Fri, Oct 17, 2014 at 4:13 PM, Mohit Anchlia wrote: > Still don't understand the difference. If it's not waiting for the ack then > doesn't it make async? > On Fri, Oct 17, 2014 at 12:55 PM, wrote: > >> Its using the sync producer without waiting for any broker to acknowledge >> the write. This explains the lack of errors you are seeing. >> >> — >> Sent from Mailbox >> >> On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia >> wrote: >> >> > Little confused :) From one of the examples I am using property >> > request.required.acks=0, >> > I thought this sets the producer to be async? >> > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira >> > wrote: >> >> 0.8.1.1 producer is Sync by default, and you can set producer.type to >> >> async if needed. >> >> >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia >> >> wrote: >> >> > Thanks! How can I tell if I am using async producer? I thought all the >> >> > sends are async in nature >> >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira > > >> >> > wrote: >> >> > >> >> >> If you have "auto.create.topics.enable" set to "true" (default), >> >> >> producing to a topic creates it. >> >> >> >> >> >> Its a bit tricky because the "send" that creates the topic can fail >> >> >> with "leader not found" or similar issue. retrying few times will >> >> >> eventually succeed as the topic gets created and the leader gets >> >> >> elected. >> >> >> >> >> >> Is it possible that you are not getting errors because you are using >> >> >> async producer? >> >> >> >> >> >> Also "no messages are delivered" can have many causes. Check if the >> >> >> topic exists using: >> >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 >> >> >> >> >> >> Perhaps the topic was created and the issue is elsewhere (the >> consumer >> >> >> is a usual suspect! perhaps look in the FAQ for tips with that issue) >> >> >> >> >> >> Gwen >> >> >> >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < >> mohitanch...@gmail.com >> >> > >> >> >> wrote: >> >> >> > Is Kafka supposed to throw exception if topic doesn't exist? It >> >> appears >> >> >> > that there is no exception thrown even though no messages are >> >> delivered >> >> >> and >> >> >> > there are errors logged in Kafka logs. >> >> >> >> >> >>
Re: Topic doesn't exist exception
Still don't understand the difference. If it's not waiting for the ack then doesn't it make async? On Fri, Oct 17, 2014 at 12:55 PM, wrote: > Its using the sync producer without waiting for any broker to acknowledge > the write. This explains the lack of errors you are seeing. > > — > Sent from Mailbox > > On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia > wrote: > > > Little confused :) From one of the examples I am using property > > request.required.acks=0, > > I thought this sets the producer to be async? > > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira > > wrote: > >> 0.8.1.1 producer is Sync by default, and you can set producer.type to > >> async if needed. > >> > >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia > >> wrote: > >> > Thanks! How can I tell if I am using async producer? I thought all the > >> > sends are async in nature > >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira > > >> > wrote: > >> > > >> >> If you have "auto.create.topics.enable" set to "true" (default), > >> >> producing to a topic creates it. > >> >> > >> >> Its a bit tricky because the "send" that creates the topic can fail > >> >> with "leader not found" or similar issue. retrying few times will > >> >> eventually succeed as the topic gets created and the leader gets > >> >> elected. > >> >> > >> >> Is it possible that you are not getting errors because you are using > >> >> async producer? > >> >> > >> >> Also "no messages are delivered" can have many causes. Check if the > >> >> topic exists using: > >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 > >> >> > >> >> Perhaps the topic was created and the issue is elsewhere (the > consumer > >> >> is a usual suspect! perhaps look in the FAQ for tips with that issue) > >> >> > >> >> Gwen > >> >> > >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia < > mohitanch...@gmail.com > >> > > >> >> wrote: > >> >> > Is Kafka supposed to throw exception if topic doesn't exist? It > >> appears > >> >> > that there is no exception thrown even though no messages are > >> delivered > >> >> and > >> >> > there are errors logged in Kafka logs. > >> >> > >> >
Re: Topic doesn't exist exception
Its using the sync producer without waiting for any broker to acknowledge the write. This explains the lack of errors you are seeing. — Sent from Mailbox On Fri, Oct 17, 2014 at 3:15 PM, Mohit Anchlia wrote: > Little confused :) From one of the examples I am using property > request.required.acks=0, > I thought this sets the producer to be async? > On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira > wrote: >> 0.8.1.1 producer is Sync by default, and you can set producer.type to >> async if needed. >> >> On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia >> wrote: >> > Thanks! How can I tell if I am using async producer? I thought all the >> > sends are async in nature >> > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira >> > wrote: >> > >> >> If you have "auto.create.topics.enable" set to "true" (default), >> >> producing to a topic creates it. >> >> >> >> Its a bit tricky because the "send" that creates the topic can fail >> >> with "leader not found" or similar issue. retrying few times will >> >> eventually succeed as the topic gets created and the leader gets >> >> elected. >> >> >> >> Is it possible that you are not getting errors because you are using >> >> async producer? >> >> >> >> Also "no messages are delivered" can have many causes. Check if the >> >> topic exists using: >> >> bin/kafka-topics.sh --list --zookeeper localhost:2181 >> >> >> >> Perhaps the topic was created and the issue is elsewhere (the consumer >> >> is a usual suspect! perhaps look in the FAQ for tips with that issue) >> >> >> >> Gwen >> >> >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia > > >> >> wrote: >> >> > Is Kafka supposed to throw exception if topic doesn't exist? It >> appears >> >> > that there is no exception thrown even though no messages are >> delivered >> >> and >> >> > there are errors logged in Kafka logs. >> >> >>
Re: Topic doesn't exist exception
Little confused :) From one of the examples I am using property request.required.acks=0, I thought this sets the producer to be async? On Fri, Oct 17, 2014 at 11:59 AM, Gwen Shapira wrote: > 0.8.1.1 producer is Sync by default, and you can set producer.type to > async if needed. > > On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia > wrote: > > Thanks! How can I tell if I am using async producer? I thought all the > > sends are async in nature > > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira > > wrote: > > > >> If you have "auto.create.topics.enable" set to "true" (default), > >> producing to a topic creates it. > >> > >> Its a bit tricky because the "send" that creates the topic can fail > >> with "leader not found" or similar issue. retrying few times will > >> eventually succeed as the topic gets created and the leader gets > >> elected. > >> > >> Is it possible that you are not getting errors because you are using > >> async producer? > >> > >> Also "no messages are delivered" can have many causes. Check if the > >> topic exists using: > >> bin/kafka-topics.sh --list --zookeeper localhost:2181 > >> > >> Perhaps the topic was created and the issue is elsewhere (the consumer > >> is a usual suspect! perhaps look in the FAQ for tips with that issue) > >> > >> Gwen > >> > >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia > > >> wrote: > >> > Is Kafka supposed to throw exception if topic doesn't exist? It > appears > >> > that there is no exception thrown even though no messages are > delivered > >> and > >> > there are errors logged in Kafka logs. > >> >
Re: Topic doesn't exist exception
0.8.1.1 producer is Sync by default, and you can set producer.type to async if needed. On Fri, Oct 17, 2014 at 2:57 PM, Mohit Anchlia wrote: > Thanks! How can I tell if I am using async producer? I thought all the > sends are async in nature > On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira > wrote: > >> If you have "auto.create.topics.enable" set to "true" (default), >> producing to a topic creates it. >> >> Its a bit tricky because the "send" that creates the topic can fail >> with "leader not found" or similar issue. retrying few times will >> eventually succeed as the topic gets created and the leader gets >> elected. >> >> Is it possible that you are not getting errors because you are using >> async producer? >> >> Also "no messages are delivered" can have many causes. Check if the >> topic exists using: >> bin/kafka-topics.sh --list --zookeeper localhost:2181 >> >> Perhaps the topic was created and the issue is elsewhere (the consumer >> is a usual suspect! perhaps look in the FAQ for tips with that issue) >> >> Gwen >> >> On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia >> wrote: >> > Is Kafka supposed to throw exception if topic doesn't exist? It appears >> > that there is no exception thrown even though no messages are delivered >> and >> > there are errors logged in Kafka logs. >>
Re: Topic doesn't exist exception
Thanks! How can I tell if I am using async producer? I thought all the sends are async in nature On Fri, Oct 17, 2014 at 11:44 AM, Gwen Shapira wrote: > If you have "auto.create.topics.enable" set to "true" (default), > producing to a topic creates it. > > Its a bit tricky because the "send" that creates the topic can fail > with "leader not found" or similar issue. retrying few times will > eventually succeed as the topic gets created and the leader gets > elected. > > Is it possible that you are not getting errors because you are using > async producer? > > Also "no messages are delivered" can have many causes. Check if the > topic exists using: > bin/kafka-topics.sh --list --zookeeper localhost:2181 > > Perhaps the topic was created and the issue is elsewhere (the consumer > is a usual suspect! perhaps look in the FAQ for tips with that issue) > > Gwen > > On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia > wrote: > > Is Kafka supposed to throw exception if topic doesn't exist? It appears > > that there is no exception thrown even though no messages are delivered > and > > there are errors logged in Kafka logs. >
Re: Topic doesn't exist exception
If you have "auto.create.topics.enable" set to "true" (default), producing to a topic creates it. Its a bit tricky because the "send" that creates the topic can fail with "leader not found" or similar issue. retrying few times will eventually succeed as the topic gets created and the leader gets elected. Is it possible that you are not getting errors because you are using async producer? Also "no messages are delivered" can have many causes. Check if the topic exists using: bin/kafka-topics.sh --list --zookeeper localhost:2181 Perhaps the topic was created and the issue is elsewhere (the consumer is a usual suspect! perhaps look in the FAQ for tips with that issue) Gwen On Fri, Oct 17, 2014 at 12:56 PM, Mohit Anchlia wrote: > Is Kafka supposed to throw exception if topic doesn't exist? It appears > that there is no exception thrown even though no messages are delivered and > there are errors logged in Kafka logs.
Topic doesn't exist exception
Is Kafka supposed to throw exception if topic doesn't exist? It appears that there is no exception thrown even though no messages are delivered and there are errors logged in Kafka logs.
