Re: Consumer Offsets Topic cleanup.policy

Thanks for the reply Jason. Our topics global retention is for 4 days and as we are planning to set the __consumer_offsets retention to the same interval, in the worst case, we won't loose any message offsets as the data will anyways be rotated. Reg. the problems with log cleaner: 1. I enabled lo

reporting the controller error in securing kafka cluster

HI, I'm building a safe Kafka cluster using SASL_PLAINTEXT。 It can create the topic, but not producing and consuming data。 The following is an related log : less controller.log [2016-03-08 11:43:44,164] INFO [Controller 0]: Controller starting up (kafka.controller.KafkaController) [2016-03-

Re: Consumer Offsets Topic cleanup.policy

This is actually a really good question. If you change the retention policy of the offsets topic, then in the worst case, consumer groups could lose their last committed positions and fall back to the auto reset behavior. However, if your consumers are not down for a long time and you set the reten

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Hi Jason, Yes, 0.9 clients should still work with 0.10 brokers. Thanks, Jiangjie (Becket) Qin On Mon, Mar 7, 2016 at 4:10 PM, Jason Gustafson wrote: > +users > > On Mon, Mar 7, 2016 at 4:09 PM, Jason Gustafson > wrote: > > > Hey Ismael, > > > > Thanks for bringing this up again. Just a quick

Re: First Kafka Meetup in Beijing

Hi, I checked the link, it’s OK. I guess Meetup was blocked by GFW, so you can not open the url in China. > On Mar 8, 2016, at 8:18 AM, Cheng Chen wrote: > > Hi Zhun, > > somehow I cant open your url, mind if I add you on wechat? (or feel free to > add me: 1999). I am excited to see th

Re: Seek to invalid offset, new consumer

Hi Giidox: Actually I just realized that position() call will trigger the underlying select() function when necessary, so your proposal above actually works. Guozhang On Mon, Mar 7, 2016 at 4:57 PM, Guozhang Wang wrote: > consumer.seekToBeginning / seekToEnd evaluates lazily, and hence you hav

Re: Larger Size Error Message

John, There is not a specific JIRA for this change as it is only implemented in the new Java producer: https://issues.apache.org/jira/browse/KAFKA-1239 Related classes are RecordAccumulator and MemoryRecords: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clie

Re: Seek to invalid offset, new consumer

consumer.seekToBeginning / seekToEnd evaluates lazily, and hence you have to call poll in order to really update the fetching position. The first record's offset would then be the starting offset. Guozhang On Mon, Mar 7, 2016 at 4:53 AM, Giidox wrote: > Excellent, many thanks! > > With “auto.o

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

+users On Mon, Mar 7, 2016 at 4:09 PM, Jason Gustafson wrote: > Hey Ismael, > > Thanks for bringing this up again. Just a quick question: if we do #1, > then there's no way that a user binary could work against both 0.9 and 0.10 > of kafka-clients, right? I'm not sure if that is much of a proble

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Coming back to this, see below. On Wed, Jan 27, 2016 at 9:01 PM, Jason Gustafson wrote: > > 1. For subscribe() and assign(), change the parameter type to collection as > planned in the KIP. This is at least source-compatible, so as long as users > compile against the updated release, there shoul

Re: migrating the main-page docs to gitbook format

All, Here's a first spike.. would love feedback.. happy to continue the convo here (https://issues.apache.org/jira/browse/KAFKA-2967) http://christianposta.com/kafka-docs/_book/index.html have the gitbook build integrated with the kafka gradle builds and rest of WIP here: https://github.com/chr

Re: test-log4j.properties purpose

Stevo, Turns out this is a leftover file from previous system tests. Filed KAFKA-3344 to remove it. On Sun, Mar 6, 2016 at 3:50 PM, Stevo Slavić wrote: > Hello Apache Kafka community, > > Is config/test-log4j.properties used for tests only? If so, would it make > sense to remove it from binary

Log cleaner error

Hello, We are using Kafka 0.8.2.2 and have modified most of our topics to use log compaction and a shorter retention.ms equivalent to 24 hours for those topics. We did not see any change in the disk usage, and on looking at the log cleaner log file, noticed the following message (repeated on e

Re: Larger Size Error Message

No, we don't have compression turned on the batch size is the default: 16384. But the message size is very small, even with that batch size, it is impossible to exceed the size limit. Thanks, Fang On Sun, Mar 6, 2016 at 6:09 PM, John Dennison wrote: > Guozhang, > > Do you know the ticket for fo

Re: 0.9.0.1 Kafka assign partition to new Consumer error

Hi Ken, It looks like what happened is this: 1. First thread joins the group and is assigned partitions 0 and 1 2. First thread races through a bunch of messages from these partitions. 3. Second thread joins and is assigned partition 1 (leaving partition 0 for the first thread) 4. Both threads re

Re: [kafka] decreasing Lag

So it looks like you need more logstash consumers, but you’ll want to look at the consumers you have and make sure they are working well and they’re not getting bogged down somewhere else, which is causing them to consume slower. Assuming they’re working fine, you can add 4 more. If that doesn’t d

[kafka] decreasing Lag

I am new to Kafka so please excuse me if this is a very basic question. I have a cluster set up with 3 zookeepers and 9 brokers. I have network security logs flowing into the kafka cluster. I am using logstash to read them from the cluster and ingest them into an elasticsearch cluster. My curre

Re: Uneven GC behavior between nodes

On Sat, Mar 5, 2016 at 4:35 AM, James Cheng wrote: > Your partitions are balanced, but is your data being evenly written across > all the partitions? How are you producing data? Are you producing them with > keys? Is it possible that the majority of the messages being written to > just a few part

First Kafka Meetup in Beijing

Hi there, First Kafka Meetup will be hold in Beijing on Saturday, March 12, 2016. If you are in Beijing and interested in Apache Kafka, come on join us! Check this link( http://www.meetup.com/Beijing-Kafka-Meetup/events/229408632/ f

Re: Seek to invalid offset, new consumer

Excellent, many thanks! With “auto.offset.reset” as “none”, I get OffsetOutOfRangeException, which is what I want. To find out how many messages the client I had missed, I called consumer.seekToBeginning and consumer.position (without calling poll). Is this a correct way to find out the first