Hi Ewen, My assumption that this issue is happening when a huge number of events are getting published was wrong. That is when we discovered it. I looked closely at the log file and I seem to be having this issue for everything. So, my consumer is reading all the events from Kafka topic and restarting from the top again. This is happening continuously. I added read/write permissions for the Kafka client user to __consumer_offsets and restarted Kafka Connect. I am still facing this issue. Is there anything else I can check? I did not have this issue when we ran on an unsecured cluster. So, I keep thinking this has something to do with some permissions issue. Your help is greatly appreciated.
Thanks, Sri On Tue, Jan 3, 2017 at 5:21 PM, Ewen Cheslack-Postava <[email protected]> wrote: > It's a bit odd (and I just opened a JIRA about it), but you actually need > read permission for the group and read permission for the topic. > > There are some error responses which may only be logged at DEBUG level, but > I think they should all be throwing an exception and Kafka Connect would > log that at ERROR level. The only case I can find that doesn't do that is > if topic authorization failed. > > -Ewen > > On Tue, Jan 3, 2017 at 2:21 PM, Srikrishna Alla <[email protected] > > > wrote: > > > Hi Ewen, > > > > I did not see any "ERROR" messages in the connect logs. But, I checked > the > > __consumer_offsets topic and it doesn't have anything in it. Should I > > provide write permissions to this topic for my Kafka client user? I am > > running my consumer using a different user than Kafka user. > > > > Thanks, > > Sri > > > > On Tue, Jan 3, 2017 at 3:40 PM, Ewen Cheslack-Postava <[email protected] > > > > wrote: > > > > > On Tue, Jan 3, 2017 at 12:58 PM, Srikrishna Alla < > > > [email protected]> > > > wrote: > > > > > > > Thanks for your response Ewen. I will try to make updates to the > > producer > > > > as suggested. Regd the Sink Connector consumer, Could it be that > > > > connect-offsets topic is not getting updated with the offset > > information > > > > per consumer? In that case, will the connector consume the same > > messages > > > > again and again? Also, if that is the case, how would I be able to > > > > troubleshoot? I am running a secured Kafka setup with SASL_PLAINTEXT > > > setup. > > > > Which users/groups should have access to write to the default topics? > > If > > > > not, please guide me in the right direction. > > > > > > > > > > > For sink connectors we actually don't use the connect-offsets topic. So > > if > > > you only have that one sink connector running, you shouldn't expect to > > see > > > any writes to it. Since sink connectors are just consumer groups, they > > use > > > the existing __consumer_offsets topic for storage and do the commits > via > > > the normal consumer commit APIs. For ACLs, you'll want Read access to > the > > > Group and the Topic. > > > > > > But I doubt it is ACL issues if you're only seeing this when there is > > heavy > > > load. You could use the consumer offset checker to see if any offsets > are > > > committed for the group. Also, is there anything in the logs that might > > > indicate a problem with the consumer committing offsets? > > > > > > -Ewen > > > > > > > > > > Thanks, > > > > Sri > > > > > > > > On Tue, Jan 3, 2017 at 1:59 PM, Ewen Cheslack-Postava < > > [email protected] > > > > > > > > wrote: > > > > > > > > > On Tue, Jan 3, 2017 at 8:38 AM, Srikrishna Alla < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I am using Kafka/Kafka Connect to track certain events happening > in > > > my > > > > > > application. This is how I have implemented it - > > > > > > 1. My application is opening a KafkaProducer every time this > event > > > > > happens > > > > > > and writes to my topic. My application has several components > > running > > > > in > > > > > > Yarn and so I did not find a way to have just one producer and > > reuse > > > > it. > > > > > > Once the event has been published, producer is closed > > > > > > > > > > > > > > > > KafkaProducer is thread safe, so you can allocate a single producer > > per > > > > > process and use it every time the event occurs on any thread. > > Creating > > > > and > > > > > destroying a producer for every event will be very inefficient -- > not > > > > only > > > > > are you opening new TCP connections every time, having to lookup > > > metadata > > > > > every time, etc, you also don't allow the producer to get any > benefit > > > > from > > > > > batching so every message will require its own request/response. > > > > > > > > > > > > > > > > 2. I am using Kafka Connect Sink Connector to consume from my > topic > > > and > > > > > > write to DB and do other processing. > > > > > > > > > > > > This setup is working great as long as we have a stable number of > > > > events > > > > > > published. The issue I am facing is when we have a huge number of > > > > > events(in > > > > > > thousands within minutes) hitting Kafka. In this case, my Sink > > > > Connector > > > > > is > > > > > > going into a loop and reading events from Kafka recursively and > not > > > > > > stopping. What could have triggered this? Please provide your > > > valuable > > > > > > insights. > > > > > > > > > > > > > > > > What exactly do you mean by "reading events from Kafka > recursively"? > > > > Unless > > > > > it's hitting some errors that are causing consumers to fall out of > > the > > > > > group uncleanly and then rejoin later, you shouldn't be seeing > > > > duplicates. > > > > > Is there anything from the logs that might help reveal the problem? > > > > > > > > > > -Ewen > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > Sri > > > > > > > > > > > > > > > > > > > > >
