[VOTE] 0.9.0.0 Candiate 2

This is the second candidate for release of Apache Kafka 0.9.0.0. This a major release that includes (1) authentication (through SSL and SASL) and authorization, (2) a new java consumer, (3) a Kafka connect framework for data ingestion and egression, and (4) quotas. Since this is a major release, w

Re: [kafka-clients] Re: [VOTE] 0.9.0.0 Candiate 1

Thanks for identifying the issues. Will roll out RC2. Jun On Tue, Nov 10, 2015 at 12:42 PM, Gwen Shapira wrote: > BTW. I created a Jenkins job for the 0.9 branch: > https://builds.apache.org/job/kafka_0.9.0_jdk7/ > > Right now its pretty much identical to trunk, but since they may diverge, I >

Re: Kafka log retention questions

Thank you so much Gwen.This is really useful information :) On 11/13/15, 1:24 PM, "Gwen Shapira" wrote: >Mostly correct, yes. > >Just one additional point: >You configured the topic to retain data for 1s. However, there is also a >property for how often the broker will look for something to dele

Re: consumer offset tool and JMX metrics do not match

I also looked at this metric in JMX and it is also 0 *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash* On Fri, Nov 13, 2015 at 4:06 PM, allen chan wrote: > Hi All, > > I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX > (kafka.server:type=FetcherLag

consumer offset tool and JMX metrics do not match

Hi All, I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*) and they do not match. ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows 0 for all partitions.

Re: Kafka log retention questions

Mostly correct, yes. Just one additional point: You configured the topic to retain data for 1s. However, there is also a property for how often the broker will look for something to delete: log.retention.check.interval.ms. This defaults to 5 minute. So unless you tweak it, the old messages will on

Re: Apache Confluence link to inaccessible webpage

Hi Jordan, Thanks for pointing out. I have removed that link to the LI internal pages and we will provide a new apache confluence wiki for the client development guidance. Guozhang On Fri, Nov 13, 2015 at 9:07 AM, Pilat, Jordan wrote: > The page at the below link [1] contains the snippet, "Thi

Re: Log Compaction v. Retention

I think you can mark a tombstone (null value) for the keys explicitly, that will eventually delete those messages from the log. Thanks, Mayuresh On Wed, Nov 11, 2015 at 6:18 AM, Morgan Kenyon wrote: > I came across an interesting question on StackOverflow dealing with the > difference between

Re: Kafka log retention questions

Thanks a lot Gwen and Raju. This information was really helpful. I used the following example:- kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic --config retention.ms=1000 And it works as expected. By setting its limit to a particular value, it prunes messages after that limit

Apache Confluence link to inaccessible webpage

The page at the below link [1] contains the snippet, "This wiki provides a step by step guide for writing a non-java 0.9 client", where "This wiki" is a link to [2]. Is [2] a LinkedIn intranet page or something? Is it possible to copy/paste the content from the LinkedIn confluence instance to t

Re: Kafka Scalability & Partition

Hi, Having such a model will not scale. I think it it's mentioned in earlier posts as well as some wikis available out there Kafka works very well if you have more partitions, so you can massively parallel writes to Kafka. Also, your application need not send partition ids/numbers, unless you've

Nullpointer using 'old' producer with 0.9.0 when node fails

I don't think it's a big problem, but I just ran into an issue playing around with vagrant. I was using the 0.9.0 github branch to run kafka, and used vagrant to (by default) bring up one zookeeper and 3 broker instances. Then I created two topics like: ./bin/kafka-topics.sh --create --zookeeper 1

Re: Re: Re: Re: Re: Kafka lost data issue

Hi Gwen, If producer cant get a response but the message got committed, Because of retries, can this producer be committing messages more than once?? I'm trying to see this problem from the point of view of Idempotent Producer Thanks, Prabhjot On Nov 13, 2015 1:26 PM, "Hawin Jiang" wrote: > Tha

Kafka Scalability & Partition

All, I am looking for clarifications. I want to have Highly Scalable Kafka Environment, but I will have my topics without Partitions as my Application mayn't send the Partition Id. My question, will the Kafka Scale (Kafka Clusters) when there are Multiple Topics but they doesn't have any Partitio

Re: Kafka log retention questions

Adding some more info inline. On Fri, Nov 13, 2015 at 10:43 AM, Gwen Shapira wrote: > See answers inline > > On Thu, Nov 12, 2015 at 2:53 PM, Sandhu, Dilpreet > wrote: > > > Hi all, > >I am new to Kafka usage. Here are some questions that I have in > > mind. Kindly help me understand it