Re: Kafka - deployment size and topologies

Thanks for the info, Todd. This is very useful. Please see my question inline: On Mon, Apr 6, 2015 at 10:24 AM, Todd Palino tpal...@gmail.com wrote: - Partition count (leader and follower combined) on each broker should stay under 4000 As far as topic volume goes, it varies widely. We

How can the only replica for a partition not be in the Isr list? (replication factor=1)

Hi all, I have faced a strange situation with a broker this week where some partitions of a topic with replication factor 1 became unavailable and stayed unavailable even after the broker node that had (network?) issues was back in the cluster. Even multiple restarts of the broker did not get

Re: offset-management-in-kafka

You are looking at the newer, and not-ready-yet consumer there. The configuration you are looking for is defined here: core/src/main/scala/kafka/consumer/ConsumerConfig.scala Gwen On Wed, Apr 8, 2015 at 9:25 AM, Yosi Botzer yosi.bot...@gmail.com wrote: Hi, I am looking at the java class

Re: offset-management-in-kafka

This is available from 0.8.2.0, and is enabled on server by default. The consumer needs to specify offsets.storage parameter - the default is still zookeeper, so the consumers should set it to 'kafka'. The documentation also explain how to migrate from zookeeper offsets to kafka offsets. Gwen

Re: offset-management-in-kafka

Thanks Gwan, Are there any plans to change the default from zookeeper to kafka in the future? On Wed, Apr 8, 2015 at 6:29 PM, Gwen Shapira gshap...@cloudera.com wrote: This is available from 0.8.2.0, and is enabled on server by default. The consumer needs to specify offsets.storage parameter

reassignment sequence

After initiating partition reassignment, does Kafka try to move all partitions at once or does it do a subset at a time? Thanks, Wes

expected behavior if a node undergoes unclean shutdown

Hello, I'm still trying to get to the bottom of an issue we had previously, with an unclean shutdown during an upgrade to 0.8.2.1 (from 0.8.1.1). In that case, the controlled shutdown was interrupted, and the node was shutdown abruptly. This resulted in about 5 minutes of unavailability for

Re: Kafka server relocation

thanks, but can you please tell which metrics could highlight the factor causing slow data migration by MirrorMaker? Regards, Nitin Kumar Sharma. On Tue, Apr 7, 2015 at 10:10 PM, tao xiao xiaotao...@gmail.com wrote: You may need to look into the consumer metrics and producer metrics to

Kafka ipv6

Hi, I am facing trouble in version 0.8.2.1. I saw this issue is closed. https://issues.apache.org/jira/browse/KAFKA-1123 The console-producer --broker-list option from command line tool does work with ipv6. But we are using the 'new Producer(config);', and

Re: Kafka ipv6

Resolved ... My cilent is compiled with a old version of kafka. Mingtao Best Regards, Mingtao On Wed, Apr 8, 2015 at 3:00 PM, Mingtao Zhang mail2ming...@gmail.com wrote: Hi, I am facing trouble in version 0.8.2.1. I saw this issue is closed. https://issues.apache.org/jira/browse/KAFKA-1123

Re: New broker ignoring retention

FWIW, we've had good luck changing the mtime. No problems found. On Mon, Apr 6, 2015 at 4:37 PM, Todd Palino tpal...@gmail.com wrote: I answered this in IRC, but the issue is that retention depends on the modification time of the log segments on disk. When you copy a partition from one broker

RE: Problem with node after restart no partitions?

Created issue https://issues.apache.org/jira/browse/KAFKA-2108 Cheers, Thunder -Original Message- From: Jason Rosenberg [mailto:j...@squareup.com] Sent: Tuesday, April 07, 2015 12:05 PM To: users@kafka.apache.org Subject: Re: Problem with node after restart no partitions? Thunder,

Re: Is there a complete Kafka 0.8.* replication design document

#2. /brokers/topics/[topic] stores the replica assignment for all partitions in the topic. /brokers/topics/[topic]/partitions/[partition_id]/state stores the leader/isr per partition. We did it this way since the leader/isr need to be updated on a per partition basis. #4. Yes, what you observed

Re: Kafka - deployment size and topologies

So as I noted, it really does depend on what you need. In the case of a small number of topics, I would say to make the number of partitions be a multiple of the number of brokers. That will balance them in the cluster, while still giving you some freedom to have larger partition counts for larger

Re: Kafka server relocation

Metrics like Bytepersec, FetchRequestRateAndTimeMs can help you to check if the consumer has problem processing messages On Thu, Apr 9, 2015 at 2:40 AM, nitin sharma kumarsharma.ni...@gmail.com wrote: thanks, but can you please tell which metrics could highlight the factor causing slow data

Upper-bound on number of consumers

Hello Kafka Friends, We are considering a use-case where we'd like to have a Kafka Cluster with potentially 1000's of partitions using a hashed key on customer userids. We have heard that Kafka can support 1000's of partitions in a single cluster and I wanted to find out if it's reasonable to

[DISCUSS] KIP-14 Tools Standardization

The Tool Standardization KIP that Jiangjie started has been updated to contain proposal details: https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization Any feedback is appreciated. Thanks, Matt

Re: expected behavior if a node undergoes unclean shutdown

I've confirmed that the same thing happens even if it's not the controller that's killed hard. Also, in several trials, it took between 10-30 seconds to recover. Jason On Wed, Apr 8, 2015 at 1:31 PM, Jason Rosenberg j...@squareup.com wrote: Hello, I'm still trying to get to the bottom of an

delete.retention.ms

I am new to Kafka that's the reason asking so many question KeyedMessageString, byte[] keyedMessage = new KeyedMessageString, byte[](request.getRequestTopicName(),SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently,I am sending message without any key maintained as

Re: Post on running Kafka at LinkedIn

Good questions. Here are the answers... - Yes, all brokers we run are hardware. We do not use virtual systems for Kafka or Zookeeper - There's a number of things we have done. I covered a lot of them last year at ApacheCon (