Re: Increasing partition count and preserving local order for a key

Hi You can’t move existing records between partitions, but one possibility would be to create a new topic with the required number of partitions, then copy the data from the original topic to the new one. The default partitioning algorithm would ensure that all records with the same key in the

Re: Scaling up kafka consumers

Hi If you have two consumers in your consumer group, but only one partition in the topic, then only one consumer will do any work. It’s not the case that “those two nodes start competing for messages” — one node will read from the partition, the other will have nothing to do. So to scale up by

Re: Scaling To Many Kafka Consumers For Particular Topic

> On Feb 22, 2017, at 1:02 PM, Sunil Patil > wrote: > > Hi Antony, > > > 1) Kafka only allows one consumer per partition, to guarantee order within > partition This is not true. Any number of Consumers can read data from a given Partition. It’s true that within a single Consumer Group, onl

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

+1 (non-binding) > On Jan 11, 2017, at 11:33 AM, Jay Kreps wrote: > > +1 > > On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford wrote: > >> Looks like there was a good consensus on the discuss thread for KIP-106 so >> lets move to a vote. >> >> Please chime in if you would like to change the def

Re: Kafka consumer cluster setup

list isn’t a list of the only Brokers that the client will talk to as it’s running. Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Oct 16, 2016, at 11:17 PM, sat wrote: > > Hi, > > We are planning to run Kafka servers along with Zookeeper in 3 remote

Re: rack aware consumer

Followers. Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Sep 30, 2016, at 12:32 PM, Ezra Stuetzel wrote: > > Hi, > Yeah I am aware of MirrorMaker. We tried to simplify our architecture so as > to avoid needing to use MirrorMaker and just rely on the rack re

Re: Questions about Apache Kafka

Actually, for 3 and 4 Kafka Streams is also a very good option — and comes as part of the standard Apache Kafka distribution. Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Aug 24, 2016, at 2:37 PM, David Garcia wrote: > > Regarding 3 and 4: https://calcite.a

Re: Offsets getting lost if no messages sent for a long time

looked at a topic. You can increase the value of offsets.retention.minutes to ensure that offset info isn’t cleaned out before more messages are written to a topic and read by the Consumer (and hence the Consumer updates its offset info in __consumer_offsets). Ian. --- Ian Wrigley Director

Re: compression ratio

I believe so, yes. --- Ian Wrigley Director, Education Services Confluent, Inc > On Jul 29, 2016, at 7:51 PM, Tauzell, Dave > wrote: > > A compression ratio of .5 means we are getting about 2x compression? > > Dave Tauzell | Senior Software Engineer | Surescripts

Re: compression ratio

Hi Dave The JMX metric compression-rate-avg should give you that info. Regards Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Jul 29, 2016, at 2:58 PM, Tauzell, Dave > wrote: > > Is there a good way to see what sort of compression ratio is being achieved

Re: automatically creating a new partition for topic

No — the number of partitions is specified when you create the topic, and you can subsequently modify it using the command-line tool, but if you try to write to a nonexistent partition you’ll get an exception. Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Jul 7, 2

Re: loading 15M messages to kafka

Depending on where your data is coming from, Kafka Connect may well be a good solution for you. Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Jul 5, 2016, at 10:18 AM, Nomar Morado wrote: > > Hi > > I am trying to load 15 million rows/records/messag

Re: My Use case is I want to delete the records instantly after consuming them

That’s really not what Kafka was designed to do. You can set a short log retention period, which will mean messages are deleted relatively soon after they were written to Kafka, but there’s no mechanism for deleting records on consumption. Ian. --- Ian Wrigley Director, Education Services

Re: About null keys

print.key and key.separator.) --- Ian Wrigley Director, Education Services Confluent, Inc > On Jun 16, 2016, at 5:58 AM, Adrienne Kole wrote: > > Thank you for your answers. I was trying to give input to topic from > console. Therefore, when I consume the message on kafka streams, I got &

Re: [COMMERCIAL] Re: download - 0.10.0.0 RC6

[Removing dev@ to avoid cross-posting] The second and third are binaries build against two different versions of Scala. Ian. > On May 18, 2016, at 1:28 PM, Ramanan, Buvana (Nokia - US) > wrote: > > Ian, > > Thanks a lot for the prompt response. > > What is the difference between the followi

Re: download - 0.10.0.0 RC6

Hi From Gwen’s announcement of RC6 last night: > * Release artifacts to be voted upon (source and binary): > http://home.apache.org/~gwenshap/0.10.0.0-rc6/ Regards Ian. --- Ian Wrigley Director, Education Services Confluent, Inc Cell: (323) 819-4075 > On May 18, 2016, at 1:04 PM

Re: How new consumer get to know about zookeeper URL?

Hi Ratha New Consumers don’t use ZooKeeper; all offsets are stored in a Kafka topic. Regards Ian. > On Apr 5, 2016, at 10:20 PM, Ratha v wrote: > > Hi all; > Im using kafka 0.9.0.1 V with new consume APIs. > > I would like to know how new consumers get to know about the zookeeper's > URL? >