kafka 0.9.x : creating topic with delete.retention.ms is faling

with error below: UnrecognizedOptionException: 'delete.retention.ms' is not a recognized option I looked at the documentation for topics and seems like a valid option. I am trying to override this setting on the topic level. Appreciate some clarifications on this. Thanks, Nomar

Re: unknown (kafka) offsets after restart

Is the log compaction thread correctly working? The offsets are stored in a log compacted topic, and we have seen issues where the log cleaner thread dies and therefore the offsets topic just grows forever, which means it will take a long time to read in the topic. You can look in the

Re: KAFKA-3112

KAFKA-2657 is unresolved so you can safely assume it hasn't been fixed yet. On Fri, 6 May 2016 at 07:38 Raj Tanneru wrote: > Yeah it is a duplicate of KAFKA-2657. The question is how to check / know > if it is merged to 0.9.0.1 release. What are the options that I have if

Re: kafka-run-class.sh

Yes Dustin..that what i was looking for..thanks. > On May 6, 2016, at 11:55 PM, Dustin Cote wrote: > > Pulling down the kafka code and grepping through is one way to do it: > grep -r "def main(" . > > Mostly you'll be looking in these two directories: >

Re: kafka-run-class.sh

Pulling down the kafka code and grepping through is one way to do it: grep -r "def main(" . Mostly you'll be looking in these two directories: core/src/main/scala/kafka/tools/ core/src/main/scala/kafka/admin/ However, to echo what Christian said, it's probably worth deciding what you want to do

Re: Regarding Kafka Log compaction Features

hi! please have a look at this article. it help me touse the log compaction feature mechanism i hope thtat it helps. regards, florin http://www.shayne.me/blog/2015/2015-06-25-everything-about-kafka-part-2/ On Thursday, May 5, 2016, Behera, Himansu (Contractor) <

Re: [VOTE] 0.10.0.0 RC3

Thanks, Gwen. I was testing RC2 with Spark Streaming and found an issue that has a minor fix. I think this issue exists with RC3 as well: https://issues.apache.org/jira/browse/KAFKA-3669 On Thu, May 5, 2016 at 1:46 PM, Gwen Shapira wrote: > Hello Kafka users, developers and

Re: kafka-run-class.sh

Thanks chris but i am specifically looking for all Main Classes. > On May 6, 2016, at 11:07 PM, Christian Posta > wrote: > > Well, to find the ones that are used by the Kafka tools: > > find ./bin -type f | xargs grep kafka-run-class.sh | awk ' { print $4 }' > > >

Re: kafka-run-class.sh

Well, to find the ones that are used by the Kafka tools: find ./bin -type f | xargs grep kafka-run-class.sh | awk ' { print $4 }' kafka-run-class.sh basically just sets up the JVM classpath to call any of the Main classes inside kafka. What exactly are you looking to do? On Fri, May 6, 2016 at

Re: ktable implemented as compacted topic?

hi! thank you very much for your answer. what is unclear for me if can query the ktable using the keys and what is the relation between ktable and rocksdb? i look forward for your answers. regards, florin On Friday, May 6, 2016, Guozhang Wang wrote: > Hello, > > KTable's

kafka-run-class.sh

How can i get the list for all the class names i can run through ./kafka-run-class.sh [class-name] command? Thanks, Mudit

Re: ktable implemented as compacted topic?

Hello, KTable's changelog is using log compaction by default instead of log deletion. But we are adding mechanism to to let users configure their changelog topics. Guozhang On Thu, May 5, 2016 at 12:00 PM, Spico Florin wrote: > hello! > i would like to ask you if

RE: Client Protocols

If you're attempting to run a heterogeneous consumer group (different clients joining same group) then you must take care that the group protocols have been implemented the same. kafka-python attempts to mirror the Java client exactly. I haven't looked at pykafka in a while, but it is possible

Re: KAFKA-3112

Yeah it is a duplicate of KAFKA-2657. The question is how to check / know if it is merged to 0.9.0.1 release. What are the options that I have if I need this fix. How can I get patch for this on 0.8.2.1? Sent from my iPhone > On May 6, 2016, at 12:06 AM, tao xiao wrote:

Kafka - Networking considerations

Hello, I trust you are well. There's a wealth of great articles and presentations relating to the Kafka logic and design, thank you for these. But, I'm unable to find information relating to the network infrastructure that underpins Kafka. What I am trying to understand and put my Network

Fwd: unknown (kafka) offsets after restart

After a bit more looking into this we found out that the Offsetmanager is single threaded and due to our setup (few, powerful servers: rather bad for kafka I know..) it seems we are limiting ourselves severely by using kafka offsets. Any more insight is still welcome of course.

Re: Manual offset commit losing a message?

Sorry for the distraction. I think I had a typo in the property name: it is auto.commit.enable and not auto.commit.enabled (no 'd' at the end). It seems to be working as expected with the correction. Thanks hemanth On Fri, May 6, 2016 at 6:26 PM, Hemanth Yamijala wrote: >

Re: Tool to look at what's stored in internal topics on kafka 0.9

Hi Henry, You can also use any other consumer basically to read from the internal topics. The kafka-console-consumer would work for this purpose. If you're just looking to view the offsets for a consumer group though, might I suggest going with the built-in ConsumerGroupCommand? It has the

RE: Client Protocols

> To: users@kafka.apache.org > From: michael.to...@headforwards.com > Subject: Client Protocols > Date: Fri, 6 May 2016 09:26:59 +0100 > > > Hi, > > I'm doing some testing with two different kafka clients: pykafka and > kafkapython > > When switching between client library, I get an error:

Manual offset commit losing a message?

Hi all, I am using Kafka 0.9.0.1 and the old high level consumer API. I wanted to implement atleast once semantics in my app, and looking around it seemed like the right way to do it is to: - set "auto.commit.enabled=false" for the consumer properties. - Then, issue a

unknown (kafka) offsets after restart

We're using Kafka 0.8.2 and are puzzled by the offset behaviour when they are stored in kafka topics. Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can happen that the offsets are unknown and therefore stop consumers from consuming without knowing their offset.

Client Protocols

Hi, I'm doing some testing with two different kafka clients: pykafka and kafkapython When switching between client library, I get an error: InconsistentGroupProtocol I thought I was using latest protocol in both cases, but seems not. What different client protocols are there and is there

Re: KAFKA-3112

It said this is a duplication. This is the https://issues.apache.org/jira/browse/KAFKA-2657 that KAKFA-3112 duplicates to. On Thu, 5 May 2016 at 22:13 Raj Tanneru wrote: > > Hi All, > Does anyone know if KAFKA-3112 is merged to 0.9.0.1? Is there a place to > check which

Re: Tool to look at what's stored in internal topics on kafka 0.9

Hi! If you produce your messages with key type (optional) and value type of String then you can you Kafka tool: http://www.kafkatool.com/ I hope that it helps. Regards, Florin On Fri, May 6, 2016 at 12:10 AM, Henry Cai wrote: > When we are on kafka 0.8, all the