Hey Jan, I am not sure if it is acceptable for producer to be stopped for a while, particularly for online application which requires low latency. I am also not sure how consumers can switch to a new topic. Does user application needs to explicitly specify a different topic for producer/consumer to subscribe to? It will be helpful for discussion if you can provide more detail on the interface change for this solution.
Thanks, Dong On Mon, Feb 26, 2018 at 12:48 AM, Jan Filipiak <jan.filip...@trivago.com> wrote: > Hi, > > just want to throw my though in. In general the functionality is very > usefull, we should though not try to find the architecture to hard while > implementing. > > The manual steps would be to > > create a new topic > the mirrormake from the new old topic to the new topic > wait for mirror making to catch up. > then put the consumers onto the new topic > (having mirrormaker spit out a mapping from old offsets to new offsets: > if topic is increased by factor X there is gonna be a clean > mapping from 1 offset in the old topic to X offsets in the new topic, > if there is no factor then there is no chance to generate a > mapping that can be reasonable used for continuing) > make consumers stop at appropriate points and continue consumption > with offsets from the mapping. > have the producers stop for a minimal time. > wait for mirrormaker to finish > let producer produce with the new metadata. > > > Instead of implementing the approach suggest in the KIP which will leave > log compacted topic completely crumbled and unusable. > I would much rather try to build infrastructure to support the mentioned > above operations more smoothly. > Especially having producers stop and use another topic is difficult and > it would be nice if one can trigger "invalid metadata" exceptions for them > and > if one could give topics aliases so that their produces with the old topic > will arrive in the new topic. > > The downsides are obvious I guess ( having the same data twice for the > transition period, but kafka tends to scale well with datasize). So its a > nicer fit into the architecture. > > I further want to argument that the functionality by the KIP can > completely be implementing in "userland" with a custom partitioner that > handles the transition as needed. I would appreciate if someone could point > out what a custom partitioner couldn't handle in this case? > > With the above approach, shrinking a topic becomes the same steps. Without > loosing keys in the discontinued partitions. > > Would love to hear what everyone thinks. > > Best Jan > > > > > > > > > > > > > > > > > > > On 11.02.2018 00:35, Dong Lin wrote: > >> Hi all, >> >> I have created KIP-253: Support in-order message delivery with partition >> expansion. See >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-253% >> 3A+Support+in-order+message+delivery+with+partition+expansion >> . >> >> This KIP provides a way to allow messages of the same key from the same >> producer to be consumed in the same order they are produced even if we >> expand partition of the topic. >> >> Thanks, >> Dong >> >> >
