Hello, Kind reminder about this KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint>
Regards, Levani > On Jul 9, 2019, at 11:38 AM, Levani Kokhreidze <levani.co...@gmail.com> wrote: > > Hello, > > In order to move this KIP forward, I’ve updated confluence page with the new > proposal > https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint > > <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint> > I’ve also filled “Rejected Alternatives” section. > > Looking forward to discuss this KIP :) > > King regards, > Levani > > >> On Jul 3, 2019, at 1:08 PM, Levani Kokhreidze <levani.co...@gmail.com >> <mailto:levani.co...@gmail.com>> wrote: >> >> Hello Matthias, >> >> Thanks for the feedback and ideas. >> I like the idea of introducing dedicated `Topic` class for topic >> configuration for internal operators like `groupedBy`. >> Would be great to hear others opinion about this as well. >> >> Kind regards, >> Levani >> >> >>> On Jul 3, 2019, at 7:00 AM, Matthias J. Sax <matth...@confluent.io >>> <mailto:matth...@confluent.io>> wrote: >>> >>> Levani, >>> >>> Thanks for picking up this KIP! And thanks for summarizing everything. >>> Even if some points may have been discussed already (can't really >>> remember), it's helpful to get a good summary to refresh the discussion. >>> >>> I think your reasoning makes sense. With regard to the distinction >>> between operators that manage topics and operators that use user-created >>> topics: Following this argument, it might indicate that leaving >>> `through()` as-is (as an operator that uses use-defined topics) and >>> introducing a new `repartition()` operator (an operator that manages >>> topics itself) might be good. Otherwise, there is one operator >>> `through()` that sometimes manages topics but sometimes not; a different >>> name, ie, new operator would make the distinction clearer. >>> >>> About adding `numOfPartitions` to `Grouped`. I am wondering if the same >>> argument as for `Produced` does apply and adding it is semantically >>> questionable? Might be good to get opinions of others on this, too. I am >>> not sure myself what solution I prefer atm. >>> >>> So far, KS uses configuration objects that allow to configure a certain >>> "entity" like a consumer, producer, store. If we assume that a topic is >>> a similar entity, I am wonder if we should have a >>> `Topic#withNumberOfPartitions()` class and method instead of a plain >>> integer? This would allow us to add other configs, like replication >>> factor, retention-time etc, easily, without the need to change the "main >>> API". >>> >>> Just want to give some ideas. Not sure if I like them myself. :) >>> >>> >>> -Matthias >>> >>> >>> >>> On 7/1/19 1:04 AM, Levani Kokhreidze wrote: >>>> Actually, giving it more though - maybe enhancing Produced with num of >>>> partitions configuration is not the best approach. Let me explain why: >>>> >>>> 1) If we enhance Produced class with this configuration, this will also >>>> affect KStream#to operation. Since KStream#to is the final sink of the >>>> topology, for me, it seems to be reasonable assumption that user needs to >>>> manually create sink topic in advance. And in that case, having num of >>>> partitions configuration doesn’t make much sense. >>>> >>>> 2) Looking at Produced class, based on API contract, seems like Produced >>>> is designed to be something that is explicitly for producer (key >>>> serializer, value serializer, partitioner those all are producer specific >>>> configurations) and num of partitions is topic level configuration. And I >>>> don’t think mixing topic and producer level configurations together in one >>>> class is the good approach. >>>> >>>> 3) Looking at KStream interface, seems like Produced parameter is for >>>> operations that work with non-internal (e.g topics created and managed >>>> internally by Kafka Streams) topics and I think we should leave it as it >>>> is in that case. >>>> >>>> Taking all this things into account, I think we should distinguish between >>>> DSL operations, where Kafka Streams should create and manage internal >>>> topics (KStream#groupBy) vs topics that should be created in advance (e.g >>>> KStream#to). >>>> >>>> To sum it up, I think adding numPartitions configuration in Produced will >>>> result in mixing topic and producer level configuration in one class and >>>> it’s gonna break existing API semantics. >>>> >>>> Regarding making topic name optional in KStream#through - I think >>>> underline idea is very useful and giving users possibility to specify num >>>> of partitions there is even more useful :) Considering arguments against >>>> adding num of partitions in Produced class, I see two options here: >>>> 1) Add following method overloads >>>> * through() - topic will be auto-generated and num of partitions will >>>> be taken from source topic >>>> * through(final int numOfPartitions) - topic will be auto generated >>>> with specified num of partitions >>>> * through(final int numOfPartitions, final Produced<K, V> produced) - >>>> topic will be with generated with specified num of partitions and >>>> configuration taken from produced parameter. >>>> 2) Leave KStream#through as it is and introduce new method - >>>> KStream#repartition (I think Matthias suggested this in one of the threads) >>>> >>>> Considering all mentioned above I propose the following plan: >>>> >>>> Option A: >>>> 1) Leave Produced as it is >>>> 2) Add num of partitions configuration to Grouped class (as mentioned in >>>> the KIP) >>>> 3) Add following method overloads to KStream#through >>>> * through() - topic will be auto-generated and num of partitions will >>>> be taken from source topic >>>> * through(final int numOfPartitions) - topic will be auto generated >>>> with specified num of partitions >>>> * through(final int numOfPartitions, final Produced<K, V> produced) - >>>> topic will be with generated with specified num of partitions and >>>> configuration taken from produced parameter. >>>> >>>> Option B: >>>> 1) Leave Produced as it is >>>> 2) Add num of partitions configuration to Grouped class (as mentioned in >>>> the KIP) >>>> 3) Add new operator KStream#repartition for creating and managing internal >>>> repartition topics >>>> >>>> P.S. I’m sorry if all of this was already discussed in the mailing list, >>>> but I kinda got with all the threads that were about this KIP :( >>>> >>>> Kind regards, >>>> Levani >>>> >>>>> On Jul 1, 2019, at 9:56 AM, Levani Kokhreidze <levani.co...@gmail.com >>>>> <mailto:levani.co...@gmail.com>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I would like to resurrect discussion around KIP-221. Going through the >>>>> discussion thread, there’s seems to agreement around usefulness of this >>>>> feature. >>>>> Regarding the implementation, as far as I understood, the most optimal >>>>> solution for me seems the following: >>>>> >>>>> 1) Add two method overloads to KStream#through method (essentially making >>>>> topic name optional) >>>>> 2) Enhance Produced class with numOfPartitions configuration field. >>>>> >>>>> Those two changes will allow DSL users to control parallelism and trigger >>>>> re-partition without doing stateful operations. >>>>> >>>>> I will update KIP with interface changes around KStream#through if this >>>>> changes sound sensible. >>>>> >>>>> Kind regards, >>>>> Levani >>>> >>>> >>> >> >
