These features are all nice, if one adds new brokers to support additional topics, or to move existing partitions or whole topics to new brokers. Referenced sentence is in paragraph named scalability. When I read "expanded" I was thinking of scaling out, extending parallelization capabilities, and parallelism in Kafka is achieved with partitions. So I understood that sentence that it is possible to add more partitions to existing topics at runtime, with no downtime.
I just found in source that there is API for adding new partitions to existing topics (see https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/admin/AdminUtils.scala#L101 ). Have to try it. I guess it should work during runtime, causing no downtime or data loss, or moving data from existing to new partition. Producers and consumers will eventually start writing to and reading from new partition, and consumers should be able to read previously published messages from old partitions, even messages which if they were sent again would end up assigned/written to new partition. Kind regards, Stevo Slavic. On Fri, Mar 13, 2015 at 8:27 PM, Joe Stein <joe.st...@stealth.ly> wrote: > https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion > > ~ Joe Stein > - - - - - - - - - - - - - - - - - > > http://www.stealth.ly > - - - - - - - - - - - - - - - - - > > On Fri, Mar 13, 2015 at 3:05 PM, sunil kalva <sambarc...@gmail.com> wrote: > > > Joe > > > > "Well, I know it is semantic but right now it "can" be elastically scaled > > without down time but you have to integrate into your environment for > what > > that means.... it has been that way since 0.8.0 imho" > > > > here what do you mean "you have to integrate into your environment", how > do > > i achieve elastically scaled cluster seamlessly ? > > > > SunilKalva > > > > On Fri, Mar 13, 2015 at 10:27 PM, Joe Stein <joe.st...@stealth.ly> > wrote: > > > > > Well, I know it is semantic but right now it "can" be elastically > scaled > > > without down time but you have to integrate into your environment for > > what > > > that means.... it has been that way since 0.8.0 imho. > > > > > > My point was just another way to-do that out of the box... folks do > this > > > elastic scailing today with AWS CloudFormation and internal systems > they > > > built too. > > > > > > So, it can be done... you just have todo it. > > > > > > ~ Joe Stein > > > - - - - - - - - - - - - - - - - - > > > > > > http://www.stealth.ly > > > - - - - - - - - - - - - - - - - - > > > > > > On Fri, Mar 13, 2015 at 12:39 PM, Stevo Slavić <ssla...@gmail.com> > > wrote: > > > > > > > OK, thanks for heads up. > > > > > > > > When reading Apache Kafka docs, and reading what Apache Kafka "can" I > > > > expect it to already be available in latest general availability > > release, > > > > not what's planned as part of some other project. > > > > > > > > Kind regards, > > > > Stevo Slavic. > > > > > > > > On Fri, Mar 13, 2015 at 2:32 PM, Joe Stein <joe.st...@stealth.ly> > > wrote: > > > > > > > > > Hey Stevo, "can be elastically and transparently expanded without > > > > > downtime." is > > > > > the goal of Kafka on Mesos https://github.com/mesos/kafka as Kafka > > as > > > > the > > > > > ability (knobs/levers) to-do this but has to be made to-do this out > > of > > > > the > > > > > box. > > > > > > > > > > e.g. in Kafka on Mesos when a broker fails, after the configurable > > max > > > > fail > > > > > over timeout (meaning it is truly deemed hard failure) then a > broker > > > > (with > > > > > the same id) will automatically be started on a another machine, > data > > > > > replicated and back in action once that is done, automatically. > Lots > > > more > > > > > features already in there... we are also in progress to auto > balance > > > > > partitions when increasing/decreasing the size of the cluster and > > some > > > > more > > > > > goodies too. > > > > > > > > > > ~ Joe Stein > > > > > - - - - - - - - - - - - - - - - - > > > > > > > > > > http://www.stealth.ly > > > > > - - - - - - - - - - - - - - - - - > > > > > > > > > > On Fri, Mar 13, 2015 at 8:43 AM, Stevo Slavić <ssla...@gmail.com> > > > wrote: > > > > > > > > > > > Hello Apache Kafka community, > > > > > > > > > > > > On Apache Kafka website home page http://kafka.apache.org/ it is > > > > stated > > > > > > that Kafka "can be elastically and transparently expanded without > > > > > > downtime." > > > > > > Is that really true? More specifically, can one just add one more > > > > broker, > > > > > > have another partition added for the topic, have new broker > > assigned > > > to > > > > > be > > > > > > the leader for new partition, have producers correctly write to > the > > > new > > > > > > partition, and consumers read from it, with no broker, consumer > or > > > > > producer > > > > > > downtime, no data loss, no manual action to move data from > existing > > > > > > partitions to new partition? > > > > > > > > > > > > Kind regards, > > > > > > Stevo Slavic. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > SunilKalva > > >
