kafka-consumer-groups.sh delete group with new-consumer

Hi all， When I use kafka-consumer-groups.sh to delete new-consumer's group, the shell note that "Option [delete] is not valid with [new-consumer]. Note that there's no need to delete group metadata for the new consumer as it is automatically deleted when the last member leaves." How to u

Kafka offset out of range using SimpleConsumer Example

Hello, I've implemented something quite similar to the SimpleConsumer example on https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example I'm using it to traverse a specific range of offsets. I find that sometimes, in the middle of this traversal, I end up hitting an "Offse

Co-locating brokers

Hi, We're considering Kafka to provide a queued transport mechanism for an ETL process with near-real-time capability. Kafka is looking pretty good but I'm wondering about a couple of things. It's not clear- my first inclination is to co-locate a broker on the client servers in order to pro

Re: Kafka ACLs CLI Auth Error

Just for the record. The Kafka/ZK clusters were in a bad state that caused this issue. I nuked the data dirs both ZK and Kafka and things work fine. Unfortunately, I couldn't reproduce the error. On Mon, Aug 8, 2016 at 5:10 PM, BigData dev wrote: > Hi, > I think jaas config file need to be chang

Re: Kafka consumer

You need to commit messages either manually or enable auto commit via "auto.commit.enable = true". -Matthias On 08/10/2016 07:34 PM, Auduru Suresh wrote: > Hi , > > We've wrote kafka consumer and for each restart it is reading messages from > starting of partition . Please help me how we can ma

Kafka consumer

Hi , We've wrote kafka consumer and for each restart it is reading messages from starting of partition . Please help me how we can make it read from last offset. Also, please subscribe me to the forum. Thanks, Suresh

Re: [kafka-clients] [ANNOUCE] Apache Kafka 0.10.0.1 Released

Ismael, Thanks for running the release. Jun On Wed, Aug 10, 2016 at 5:01 PM, Ismael Juma wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.1. > This is a bug fix release that fixes 53 issues in 0.10.0.0. > > All of the changes in this release can

Re: [ANNOUCE] Apache Kafka 0.10.0.1 Released

Thanks Ismael for managing the release! Guozhang On Thu, Aug 11, 2016 at 12:15 AM, Ismael Juma wrote: > Thank you Gwen. :) Also, thanks to Jun for copying the artifacts to the SVN > release repo (requires a PMC member) and to Gwen for answering my > questions. > > Ismael > > On Thu, Aug 11, 20

Re: Failed to start kafka cleaner thread

BTW, we are using Kafka 0.9.0.0. And I found the following thread describing the same issue: http://grokbase.com/t/kafka/users/159jbe18en/log-cleaner-thread-stops So is this still an issue in 0.9.0.0? Thanks, David On Thu, Aug 11, 2016 at 9:46 AM David Yu wrote: > Our Kafka cleaner has stoppe

Re: Kafka Connect questions

Thanks in advance. Jeyhun On Thu, Aug 11, 2016 at 7:09 AM Gwen Shapira wrote: > Maybe you need a different KafkaConnect source? I think this one may > fit your needs better: > https://github.com/jcustenborder/kafka-connect-spooldir > > It was built to copy data from files in directory into Kafk

Failed to start kafka cleaner thread

Our Kafka cleaner has stopped running log compactions. From the log, we found: 2016-08-03 20:57:19,240 ERROR kafka.log.LogCleaner: [kafka-log-cleaner-thread-0], Error due to java.lang.IllegalArgumentException: requirement failed: Last clean offset is 104326 but segment base offset is 0 for log se

kafka-list-topic showing leader: -1 when brokers 300 or 301 would be the leader

Our main concern is: how to nuke or repair things so that brokers 300 and 301 can rejoin the leader and isr lists properly. We have a 5-host cluster, each host has both zookeeper-3.3.6 and kafka-0.8 running on it. I sniffed around in zkCli.sh and didn't notice anyting particularly wrong or cor

Kafka consumer stuck in (Re-)joining group

Hi I have been using distributed Kafka Connect 0.10.0.0 very successfully for a while now. But then after a restart both machines get stuck as they try to join the group and marks the coordinator dead. I found no way of getting out of this state. Restarting the machines does not help. Here are so

Re: kafka broker is dropping the messages after acknowledging librdkafka

Are you running with unclean leader election on? Are you setting min in sync replicas at all? Can you attach controller and any other logs from the brokers you have? They would be crucial in debugging this kind of issue. Thanks Tom Crayford Heroku Kafka On Thursday, 11 August 2016, Mazhar Shaik

Re: How to Identify Consumers of a Topic?

For kafka version > 0.9, you can use kafka-consumer-groups.sh On Tue, Aug 9, 2016 at 5:21 AM, Jillian Cocklin < jillian.cock...@danalinc.com> wrote: > Thanks Derar, > > I'll check that out and see if it gives enough information about the > consumer to track it. > > Thanks! > Jillian > > -Or

Re: Deleting by writing null payload not working in compacted logs

Afaik, the current (most recent) segment is not touched by the cleaner. Not sure if this might be the problem in your case. Regards, Harald. On 11.08.2016 11:22, Christiane Lemke wrote: Hello all, @Tom - thank you for your answer :) Here's the link to a gist with a minimal example: https://g

Re: Deleting by writing null payload not working in compacted logs

Hello all, @Tom - thank you for your answer :) Here's the link to a gist with a minimal example: https://gist.github.com/anonymous/f78184eaeec3ee82b15182aec24a432a Here's the commands to create the compacted topic on a local kafka: kafka-topics.sh -zookeeper 127.0.0.1:2181 --create --topic compa

Re: Consumer skip/ignore uncommitted message

Hi, when starting the consumer you may want to call seekToEnd() https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#seekToEnd(org.apache.kafka.common.TopicPartition...) Harald. On 11.08.2016 02:26, sat wrote: Hi, We have come up with Kafka consumer and

Re: Verify log compaction

In case you are on Windows: compaction plainly does not work. See https://issues.apache.org/jira/browse/KAFKA-1194 Root cause seems to be Windows' file locking and Kafka trying to delete/rename files that are open in another thread/part of the broker. Harald. On 11.08.2016 07:38, David Yu wr

Kafka topic-partition won't serve if prefered election fails for a partition.

Hi, With *auto.leader.rebalance.enable=true* Kafka runs preferred election tool in regular intervals rebalancing topic-partition leaders. Now if preferred election fails to elect the preferred leader for a topic partition it falls back to any other leader in ISR. But for us in production when thi

Re: [ANNOUCE] Apache Kafka 0.10.0.1 Released

Thank you Gwen. :) Also, thanks to Jun for copying the artifacts to the SVN release repo (requires a PMC member) and to Gwen for answering my questions. Ismael On Thu, Aug 11, 2016 at 4:37 AM, Gwen Shapira wrote: > Woohoo! > > Thank you, Ismael! You make a great release manager :) > > On Wed, A