Hi Ewen, I did not see any "ERROR" messages in the connect logs. But, I checked the __consumer_offsets topic and it doesn't have anything in it. Should I provide write permissions to this topic for my Kafka client user? I am running my consumer using a different user than Kafka user.
Thanks, Sri On Tue, Jan 3, 2017 at 3:40 PM, Ewen Cheslack-Postava <[email protected]> wrote: > On Tue, Jan 3, 2017 at 12:58 PM, Srikrishna Alla < > [email protected]> > wrote: > > > Thanks for your response Ewen. I will try to make updates to the producer > > as suggested. Regd the Sink Connector consumer, Could it be that > > connect-offsets topic is not getting updated with the offset information > > per consumer? In that case, will the connector consume the same messages > > again and again? Also, if that is the case, how would I be able to > > troubleshoot? I am running a secured Kafka setup with SASL_PLAINTEXT > setup. > > Which users/groups should have access to write to the default topics? If > > not, please guide me in the right direction. > > > > > For sink connectors we actually don't use the connect-offsets topic. So if > you only have that one sink connector running, you shouldn't expect to see > any writes to it. Since sink connectors are just consumer groups, they use > the existing __consumer_offsets topic for storage and do the commits via > the normal consumer commit APIs. For ACLs, you'll want Read access to the > Group and the Topic. > > But I doubt it is ACL issues if you're only seeing this when there is heavy > load. You could use the consumer offset checker to see if any offsets are > committed for the group. Also, is there anything in the logs that might > indicate a problem with the consumer committing offsets? > > -Ewen > > > > Thanks, > > Sri > > > > On Tue, Jan 3, 2017 at 1:59 PM, Ewen Cheslack-Postava <[email protected] > > > > wrote: > > > > > On Tue, Jan 3, 2017 at 8:38 AM, Srikrishna Alla < > > [email protected] > > > > > > > wrote: > > > > > > > Hi, > > > > > > > > I am using Kafka/Kafka Connect to track certain events happening in > my > > > > application. This is how I have implemented it - > > > > 1. My application is opening a KafkaProducer every time this event > > > happens > > > > and writes to my topic. My application has several components running > > in > > > > Yarn and so I did not find a way to have just one producer and reuse > > it. > > > > Once the event has been published, producer is closed > > > > > > > > > > KafkaProducer is thread safe, so you can allocate a single producer per > > > process and use it every time the event occurs on any thread. Creating > > and > > > destroying a producer for every event will be very inefficient -- not > > only > > > are you opening new TCP connections every time, having to lookup > metadata > > > every time, etc, you also don't allow the producer to get any benefit > > from > > > batching so every message will require its own request/response. > > > > > > > > > > 2. I am using Kafka Connect Sink Connector to consume from my topic > and > > > > write to DB and do other processing. > > > > > > > > This setup is working great as long as we have a stable number of > > events > > > > published. The issue I am facing is when we have a huge number of > > > events(in > > > > thousands within minutes) hitting Kafka. In this case, my Sink > > Connector > > > is > > > > going into a loop and reading events from Kafka recursively and not > > > > stopping. What could have triggered this? Please provide your > valuable > > > > insights. > > > > > > > > > > What exactly do you mean by "reading events from Kafka recursively"? > > Unless > > > it's hitting some errors that are causing consumers to fall out of the > > > group uncleanly and then rejoin later, you shouldn't be seeing > > duplicates. > > > Is there anything from the logs that might help reveal the problem? > > > > > > -Ewen > > > > > > > > > > > > > > Thanks, > > > > Sri > > > > > > > > > >
