Re: In kafka streams consumer seems to hang while retrieving the offsets

Let me try to get the debug log when this error happens. Right now we have three instances each with 4 threads consuming from 12 partition topic. So one thread per partition. The application is running fine much better than before. Now it usually runs for a week even during peak load. Sometime o

Re: How to set offset for a consumer in Kafka 0.10.0.X

Hi Sudev, My version is pretty much the same as https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual. So, little point posting code which does the same thing, also it python, although with a different method of parsing the params to it. Sorry about the late reply, didn't

Re: offset.storage.filename configuration in kafka-connect-hdfs

The offset file is marked as required so we can fail early & fast, but if you only run sink connectors then offsets will be stored in Kafka's normal offsets topic and the file will never need to be created. (And the HDFS connector is even more unusual in that it doesn't even rely on Kafka offsets b

Re: Kafka Connect behaves weird in case of zombie Kafka brokers. Also, zombie brokers?

Is that node the only bootstrap broker provided? If the Connect worker was pinned to *only* that broker, it wouldn't have any chance of recovering correct cluster information from the healthy brokers. It sounds like there was a separate problem as well (the broker should have figured out it was in

Re: kafka connector for mongodb as a source

There is some log noise in there from Reflections, but it does look like your connector & task are being created: [2017-03-27 18:33:00,057] INFO Instantiated task mongodb-0 with version 0.10.0.1 of type org.apache.kafka.connect.mongodb.MongodbSourceTask (org.apache.kafka.connect.runtime.Worker:264

Re: Kafka producer and consumer within on sync execution

You posted the same question to Stack Overflow so I answered it there https://stackoverflow.com/questions/43302857/handling-sync-api-call-rest-spring-and-async-message-kafka-in-the-same-execu/43312070#43312070

RE: Topic deletion

Thank you Mayuresh, that's an excellent write-up. As you mention about reassignment blocking deletion; I think we are seeing an issue with our brokers and reassignment so that could well be the root cause of the deletion issue we experienced. The zKclient suggestion that I followed before resol

Re: In kafka streams consumer seems to hang while retrieving the offsets

Hi Sachin, It's not necessarily a deadlock. Do you have any debug traces from those nodes? Also would be useful to know the config (e.g., how many partitions do you have and how many app instances.) Thanks Eno > On 9 Apr 2017, at 04:45, Sachin Mittal wrote: > > Hi, > In my streams applicatio

In kafka streams consumer seems to hang while retrieving the offsets

Hi, In my streams applications cluster in one or more instances I see some threads always waiting with the following stack. Every time I check on jstack I see the following trace. Is this some kind of new deadlock that we have failed to identify. Thanks Sachin here is the stack trace: -

Re: Leader not available error after kafka node goes down

Hey Eno, So here's the sequence of events: - 48 hrs ago: Node went down and was brought back up - Yesterday: I tried to deploy a kafka streams app, but it gave that error for that topic name. I then changed the topic name, created that topic manually, and redeployed. This time it worked. Any id