Hi all, Here’s voting thread for this KIP: https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html <https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html>
Regards, Levani > On Jul 24, 2019, at 11:15 PM, Levani Kokhreidze <levani.co...@gmail.com> > wrote: > > Hi Matthias, > > Thanks for the suggestion. I Don’t have strong opinion on that one. > Agree that avoiding unnecessary method overloads is a good idea. > > Updated KIP > > Regards, > Levani > > >> On Jul 24, 2019, at 8:50 PM, Matthias J. Sax <matth...@confluent.io> wrote: >> >> One question: >> >> Why do we add >> >>> Repartitioned#with(final String name, final int numberOfPartitions) >> >> It seems that `#with(String name)`, `#numberOfPartitions(int)` in >> combination with `withName()` and `withNumberOfPartitions()` should be >> sufficient. Users can chain the method calls. >> >> (I think it's valuable to keep the number of overload small if possible.) >> >> Otherwise LGTM. >> >> >> -Matthias >> >> >> On 7/23/19 2:18 PM, Levani Kokhreidze wrote: >>> Hello, >>> >>> Thanks all for your feedback. >>> I started voting procedure for this KIP. If there’re any other concerns >>> about this KIP, please let me know. >>> >>> Regards, >>> Levani >>> >>>> On Jul 20, 2019, at 8:39 PM, Levani Kokhreidze <levani.co...@gmail.com> >>>> wrote: >>>> >>>> Hi Matthias, >>>> >>>> Thanks for the suggestion, makes sense. >>>> I’ve updated KIP >>>> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>> >>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint>). >>>> >>>> Regards, >>>> Levani >>>> >>>> >>>>> On Jul 20, 2019, at 3:53 AM, Matthias J. Sax <matth...@confluent.io >>>>> <mailto:matth...@confluent.io>> wrote: >>>>> >>>>> Thanks for driving the KIP. >>>>> >>>>> I agree that users need to be able to specify a partitioning strategy. >>>>> >>>>> Sophie raises a fair point about topic configs and producer configs. My >>>>> take is, that consider `Repartitioned` as an "extension" to `Produced`, >>>>> that adds topic configuration, is a good way to think about it and helps >>>>> to keep the API "clean". >>>>> >>>>> >>>>> With regard to method names. I would prefer to avoid abbreviations. Can >>>>> we rename: >>>>> >>>>> `withNumOfPartitions` -> `withNumberOfPartitions` >>>>> >>>>> Furthermore, it might be good to add some more `static` methods: >>>>> >>>>> - Repartitioned.with(Serde<K>, Serde<V>) >>>>> - Repartitioned.withNumberOfPartitions(int) >>>>> - Repartitioned.streamPartitioner(StreamPartitioner) >>>>> >>>>> >>>>> -Matthias >>>>> >>>>> On 7/19/19 3:33 PM, Levani Kokhreidze wrote: >>>>>> Totally agree. I think in KStream interface it makes sense to have some >>>>>> duplicate configurations between operators in order to keep API simple >>>>>> and usable. >>>>>> Also, as more surface API has, harder it is to have proper backward >>>>>> compatibility. >>>>>> While initial idea of keeping topic level configs separate was exciting, >>>>>> having Repartitioned class encapsulate some producer level configs makes >>>>>> API more readable. >>>>>> >>>>>> Regards, >>>>>> Levani >>>>>> >>>>>>> On Jul 20, 2019, at 1:15 AM, Sophie Blee-Goldman <sop...@confluent.io >>>>>>> <mailto:sop...@confluent.io>> wrote: >>>>>>> >>>>>>> I think that is a good point about trying to keep producer level >>>>>>> configurations and (repartition) topic level considerations separate. >>>>>>> Number of partitions is definitely purely a topic level configuration. >>>>>>> But >>>>>>> on some level, serdes and partitioners are just as much a topic >>>>>>> configuration as a producer one. You could have two producers configured >>>>>>> with different serdes and/or partitioners, but if they are writing to >>>>>>> the >>>>>>> same topic the result would be very difficult to part. So in a sense, >>>>>>> these >>>>>>> are configurations of topics in Streams, not just producers. >>>>>>> >>>>>>> Another way to think of it: while the Streams API is not always true to >>>>>>> this, ideally all the relevant configs for an operator are wrapped into >>>>>>> a >>>>>>> single object (in this case, Repartitioned). We could instead split out >>>>>>> the >>>>>>> fields in common with Produced into a separate parameter to keep topic >>>>>>> and >>>>>>> producer level configurations separate, but this increases the API >>>>>>> surface >>>>>>> area by a lot. It's much more straightforward to just say "this is >>>>>>> everything that this particular operator needs" without worrying about >>>>>>> what >>>>>>> exactly you're specifying. >>>>>>> >>>>>>> I suppose you could alternatively make Produced a field of >>>>>>> Repartitioned, >>>>>>> but I don't think we do this kind of composition elsewhere in Streams at >>>>>>> the moment >>>>>>> >>>>>>> On Fri, Jul 19, 2019 at 1:45 PM Levani Kokhreidze >>>>>>> <levani.co...@gmail.com <mailto:levani.co...@gmail.com>> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Bill, >>>>>>>> >>>>>>>> Thanks a lot for the feedback. >>>>>>>> Yes, that makes sense. I’ve updated KIP with >>>>>>>> `Repartitioned#partitioner` >>>>>>>> configuration. >>>>>>>> In the beginning, I wanted to introduce a class for topic level >>>>>>>> configuration and keep topic level and producer level configurations >>>>>>>> (such >>>>>>>> as Produced) separately (see my second email in this thread). >>>>>>>> But while looking at the semantics of KStream interface, I couldn’t >>>>>>>> really >>>>>>>> figure out good operation name for Topic level configuration class and >>>>>>>> just >>>>>>>> introducing `Topic` config class was kinda breaking the semantics. >>>>>>>> So I think having Repartitioned class which encapsulates topic and >>>>>>>> producer level configurations for internal topics is viable thing to >>>>>>>> do. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Levani >>>>>>>> >>>>>>>>> On Jul 19, 2019, at 7:47 PM, Bill Bejeck <bbej...@gmail.com >>>>>>>>> <mailto:bbej...@gmail.com>> wrote: >>>>>>>>> >>>>>>>>> Hi Lavani, >>>>>>>>> >>>>>>>>> Thanks for resurrecting this KIP. >>>>>>>>> >>>>>>>>> I'm also a +1 for adding a partition option. In addition to the >>>>>>>>> reason >>>>>>>>> provided by John, my reasoning is: >>>>>>>>> >>>>>>>>> 1. Users may want to use something other than hash-based partitioning >>>>>>>>> 2. Users may wish to partition on something different than the key >>>>>>>>> without having to change the key. For example: >>>>>>>>> 1. A combination of fields in the value in conjunction with the key >>>>>>>>> 2. Something other than the key >>>>>>>>> 3. We allow users to specify a partitioner on Produced hence in >>>>>>>>> KStream.to and KStream.through, so it makes sense for API consistency. >>>>>>>>> >>>>>>>>> Just my 2 cents. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Bill >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jul 19, 2019 at 5:46 AM Levani Kokhreidze < >>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi John, >>>>>>>>>> >>>>>>>>>> In my mind it makes sense. >>>>>>>>>> If we add partitioner configuration to Repartitioned class, with the >>>>>>>>>> combination of specifying number of partitions for internal topics, >>>>>>>>>> user >>>>>>>>>> will have opportunity to ensure co-partitioning before join >>>>>>>>>> operation. >>>>>>>>>> I think this can be quite powerful feature. >>>>>>>>>> Wondering what others think about this? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Levani >>>>>>>>>> >>>>>>>>>>> On Jul 18, 2019, at 1:20 AM, John Roesler <j...@confluent.io >>>>>>>>>>> <mailto:j...@confluent.io>> wrote: >>>>>>>>>>> >>>>>>>>>>> Yes, I believe that's what I had in mind. Again, not totally sure it >>>>>>>>>>> makes sense, but I believe something similar is the rationale for >>>>>>>>>>> having the partitioner option in Produced. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> -John >>>>>>>>>>> >>>>>>>>>>> On Wed, Jul 17, 2019 at 3:20 PM Levani Kokhreidze >>>>>>>>>>> <levani.co...@gmail.com <mailto:levani.co...@gmail.com>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hey John, >>>>>>>>>>>> >>>>>>>>>>>> Oh that’s interesting use-case. >>>>>>>>>>>> Do I understand this correctly, in your example I would first issue >>>>>>>>>> repartition(Repartitioned) with proper partitioner that essentially >>>>>>>> would >>>>>>>>>> be the same as the topic I want to join with and then do the >>>>>>>> KStream#join >>>>>>>>>> with DSL? >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Levani >>>>>>>>>>>> >>>>>>>>>>>>> On Jul 17, 2019, at 11:11 PM, John Roesler <j...@confluent.io >>>>>>>>>>>>> <mailto:j...@confluent.io>> >>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hey, all, just to chime in, >>>>>>>>>>>>> >>>>>>>>>>>>> I think it might be useful to have an option to specify the >>>>>>>>>>>>> partitioner. The case I have in mind is that some data may get >>>>>>>>>>>>> repartitioned and then joined with an input topic. If the >>>>>>>>>>>>> right-side >>>>>>>>>>>>> input topic uses a custom partitioning strategy, then the >>>>>>>>>>>>> repartitioned stream also needs to be partitioned with the same >>>>>>>>>>>>> strategy. >>>>>>>>>>>>> >>>>>>>>>>>>> Does that make sense, or did I maybe miss something important? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> -John >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jul 17, 2019 at 2:48 PM Levani Kokhreidze >>>>>>>>>>>>> <levani.co...@gmail.com <mailto:levani.co...@gmail.com>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Yes, I was thinking about it as well. To be honest I’m not sure >>>>>>>> about >>>>>>>>>> it yet. >>>>>>>>>>>>>> As Kafka Streams DSL user, I don’t really think I would need >>>>>>>>>>>>>> control >>>>>>>>>> over partitioner for internal topics. >>>>>>>>>>>>>> As a user, I would assume that Kafka Streams knows best how to >>>>>>>>>> partition data for internal topics. >>>>>>>>>>>>>> In this KIP I wrote that Produced should be used only for topics >>>>>>>> that >>>>>>>>>> are created by user In advance. >>>>>>>>>>>>>> In those cases maybe it make sense to have possibility to specify >>>>>>>> the >>>>>>>>>> partitioner. >>>>>>>>>>>>>> I don’t have clear answer on that yet, but I guess specifying the >>>>>>>>>> partitioner can be added as well if there’s agreement on this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Levani >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Jul 17, 2019, at 10:42 PM, Sophie Blee-Goldman < >>>>>>>>>> sop...@confluent.io <mailto:sop...@confluent.io>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for clearing that up. I agree that Repartitioned would >>>>>>>>>>>>>>> be a >>>>>>>>>> useful >>>>>>>>>>>>>>> addition. I'm wondering if it might also need to have >>>>>>>>>>>>>>> a withStreamPartitioner method/field, similar to Produced? I'm >>>>>>>>>>>>>>> not >>>>>>>>>> sure how >>>>>>>>>>>>>>> widely this feature is really used, but seems it should be >>>>>>>> available >>>>>>>>>> for >>>>>>>>>>>>>>> repartition topics. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 11:26 AM Levani Kokhreidze < >>>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hey Sophie, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> In both cases KStream#repartition and >>>>>>>>>> KStream#repartition(Repartitioned) >>>>>>>>>>>>>>>> topic will be created and managed by Kafka Streams. >>>>>>>>>>>>>>>> Idea of Repartitioned is to give user more control over the >>>>>>>>>>>>>>>> topic >>>>>>>>>> such as >>>>>>>>>>>>>>>> num of partitions. >>>>>>>>>>>>>>>> I feel like Repartitioned parameter is something that is >>>>>>>>>>>>>>>> missing >>>>>>>> in >>>>>>>>>>>>>>>> current DSL design. >>>>>>>>>>>>>>>> Essentially giving user control over parallelism by configuring >>>>>>>> num >>>>>>>>>> of >>>>>>>>>>>>>>>> partitions for internal topics. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hope this answers your question. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Jul 17, 2019, at 9:02 PM, Sophie Blee-Goldman < >>>>>>>>>> sop...@confluent.io <mailto:sop...@confluent.io>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hey Levani, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for the KIP! Can you clarify one thing for me -- for >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> KStream#repartition signature taking a Repartitioned, will the >>>>>>>>>> topic be >>>>>>>>>>>>>>>>> auto-created by Streams (which seems to be the case for the >>>>>>>>>> signature >>>>>>>>>>>>>>>>> without a Repartitioned) or does it have to be pre-created? >>>>>>>>>>>>>>>>> The >>>>>>>>>> wording >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>> the KIP makes it seem like one version of the method will >>>>>>>>>> auto-create >>>>>>>>>>>>>>>>> topics while the other will not. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>> Sophie >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 10:15 AM Levani Kokhreidze < >>>>>>>>>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> One more bump about KIP-221 ( >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>> >>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint> >>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>> so it doesn’t get lost in mailing list :) >>>>>>>>>>>>>>>>>> Would love to hear communities opinions/concerns about this >>>>>>>>>>>>>>>>>> KIP. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Jul 12, 2019, at 5:27 PM, Levani Kokhreidze < >>>>>>>>>> levani.co...@gmail.com >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Kind reminder about this KIP: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Jul 9, 2019, at 11:38 AM, Levani Kokhreidze < >>>>>>>>>>>>>>>> levani.co...@gmail.com >>>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> In order to move this KIP forward, I’ve updated confluence >>>>>>>> page >>>>>>>>>> with >>>>>>>>>>>>>>>>>> the new proposal >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I’ve also filled “Rejected Alternatives” section. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Looking forward to discuss this KIP :) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> King regards, >>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Jul 3, 2019, at 1:08 PM, Levani Kokhreidze < >>>>>>>>>>>>>>>> levani.co...@gmail.com >>>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hello Matthias, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks for the feedback and ideas. >>>>>>>>>>>>>>>>>>>>> I like the idea of introducing dedicated `Topic` class for >>>>>>>>>> topic >>>>>>>>>>>>>>>>>> configuration for internal operators like `groupedBy`. >>>>>>>>>>>>>>>>>>>>> Would be great to hear others opinion about this as well. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Jul 3, 2019, at 7:00 AM, Matthias J. Sax < >>>>>>>>>> matth...@confluent.io >>>>>>>>>>>>>>>>>> <mailto:matth...@confluent.io>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Levani, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for picking up this KIP! And thanks for >>>>>>>>>>>>>>>>>>>>>> summarizing >>>>>>>>>>>>>>>> everything. >>>>>>>>>>>>>>>>>>>>>> Even if some points may have been discussed already >>>>>>>>>>>>>>>>>>>>>> (can't >>>>>>>>>> really >>>>>>>>>>>>>>>>>>>>>> remember), it's helpful to get a good summary to refresh >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> discussion. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I think your reasoning makes sense. With regard to the >>>>>>>>>> distinction >>>>>>>>>>>>>>>>>>>>>> between operators that manage topics and operators that >>>>>>>>>>>>>>>>>>>>>> use >>>>>>>>>>>>>>>>>> user-created >>>>>>>>>>>>>>>>>>>>>> topics: Following this argument, it might indicate that >>>>>>>>>> leaving >>>>>>>>>>>>>>>>>>>>>> `through()` as-is (as an operator that uses use-defined >>>>>>>>>> topics) and >>>>>>>>>>>>>>>>>>>>>> introducing a new `repartition()` operator (an operator >>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>> manages >>>>>>>>>>>>>>>>>>>>>> topics itself) might be good. Otherwise, there is one >>>>>>>> operator >>>>>>>>>>>>>>>>>>>>>> `through()` that sometimes manages topics but sometimes >>>>>>>> not; a >>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>> name, ie, new operator would make the distinction >>>>>>>>>>>>>>>>>>>>>> clearer. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> About adding `numOfPartitions` to `Grouped`. I am >>>>>>>>>>>>>>>>>>>>>> wondering >>>>>>>>>> if the >>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>>> argument as for `Produced` does apply and adding it is >>>>>>>>>> semantically >>>>>>>>>>>>>>>>>>>>>> questionable? Might be good to get opinions of others on >>>>>>>>>> this, too. >>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>>>>>>> not sure myself what solution I prefer atm. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> So far, KS uses configuration objects that allow to >>>>>>>> configure >>>>>>>>>> a >>>>>>>>>>>>>>>>>> certain >>>>>>>>>>>>>>>>>>>>>> "entity" like a consumer, producer, store. If we assume >>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>> a topic >>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> a similar entity, I am wonder if we should have a >>>>>>>>>>>>>>>>>>>>>> `Topic#withNumberOfPartitions()` class and method >>>>>>>>>>>>>>>>>>>>>> instead of >>>>>>>>>> a plain >>>>>>>>>>>>>>>>>>>>>> integer? This would allow us to add other configs, like >>>>>>>>>> replication >>>>>>>>>>>>>>>>>>>>>> factor, retention-time etc, easily, without the need to >>>>>>>>>> change the >>>>>>>>>>>>>>>>>> "main >>>>>>>>>>>>>>>>>>>>>> API". >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Just want to give some ideas. Not sure if I like them >>>>>>>> myself. >>>>>>>>>> :) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 7/1/19 1:04 AM, Levani Kokhreidze wrote: >>>>>>>>>>>>>>>>>>>>>>> Actually, giving it more though - maybe enhancing >>>>>>>>>>>>>>>>>>>>>>> Produced >>>>>>>>>> with num >>>>>>>>>>>>>>>>>> of partitions configuration is not the best approach. Let me >>>>>>>>>> explain >>>>>>>>>>>>>>>> why: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 1) If we enhance Produced class with this configuration, >>>>>>>>>> this will >>>>>>>>>>>>>>>>>> also affect KStream#to operation. Since KStream#to is the >>>>>>>>>>>>>>>>>> final >>>>>>>>>> sink of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> topology, for me, it seems to be reasonable assumption that >>>>>>>>>>>>>>>>>> user >>>>>>>>>> needs >>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>> manually create sink topic in advance. And in that case, >>>>>>>>>>>>>>>>>> having >>>>>>>>>> num of >>>>>>>>>>>>>>>>>> partitions configuration doesn’t make much sense. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 2) Looking at Produced class, based on API contract, >>>>>>>>>>>>>>>>>>>>>>> seems >>>>>>>>>> like >>>>>>>>>>>>>>>>>> Produced is designed to be something that is explicitly for >>>>>>>>>> producer >>>>>>>>>>>>>>>> (key >>>>>>>>>>>>>>>>>> serializer, value serializer, partitioner those all are >>>>>>>>>>>>>>>>>> producer >>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>>> configurations) and num of partitions is topic level >>>>>>>>>> configuration. And >>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>> don’t think mixing topic and producer level configurations >>>>>>>>>> together in >>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>> class is the good approach. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 3) Looking at KStream interface, seems like Produced >>>>>>>>>> parameter is >>>>>>>>>>>>>>>>>> for operations that work with non-internal (e.g topics >>>>>>>>>>>>>>>>>> created >>>>>>>> and >>>>>>>>>>>>>>>> managed >>>>>>>>>>>>>>>>>> internally by Kafka Streams) topics and I think we should >>>>>>>>>>>>>>>>>> leave >>>>>>>>>> it as >>>>>>>>>>>>>>>> it is >>>>>>>>>>>>>>>>>> in that case. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Taking all this things into account, I think we should >>>>>>>>>> distinguish >>>>>>>>>>>>>>>>>> between DSL operations, where Kafka Streams should create and >>>>>>>>>> manage >>>>>>>>>>>>>>>>>> internal topics (KStream#groupBy) vs topics that should be >>>>>>>>>> created in >>>>>>>>>>>>>>>>>> advance (e.g KStream#to). >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> To sum it up, I think adding numPartitions >>>>>>>>>>>>>>>>>>>>>>> configuration in >>>>>>>>>>>>>>>> Produced >>>>>>>>>>>>>>>>>> will result in mixing topic and producer level configuration >>>>>>>>>>>>>>>>>> in >>>>>>>>>> one >>>>>>>>>>>>>>>> class >>>>>>>>>>>>>>>>>> and it’s gonna break existing API semantics. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Regarding making topic name optional in KStream#through >>>>>>>>>>>>>>>>>>>>>>> - I >>>>>>>>>> think >>>>>>>>>>>>>>>>>> underline idea is very useful and giving users possibility to >>>>>>>>>> specify >>>>>>>>>>>>>>>> num >>>>>>>>>>>>>>>>>> of partitions there is even more useful :) Considering >>>>>>>>>>>>>>>>>> arguments >>>>>>>>>> against >>>>>>>>>>>>>>>>>> adding num of partitions in Produced class, I see two options >>>>>>>>>> here: >>>>>>>>>>>>>>>>>>>>>>> 1) Add following method overloads >>>>>>>>>>>>>>>>>>>>>>> * through() - topic will be auto-generated and num of >>>>>>>>>> partitions >>>>>>>>>>>>>>>>>> will be taken from source topic >>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions) - topic will be >>>>>>>>>>>>>>>>>>>>>>> auto >>>>>>>>>>>>>>>>>> generated with specified num of partitions >>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions, final Produced<K, >>>>>>>>>>>>>>>>>>>>>>> V> >>>>>>>>>>>>>>>>>> produced) - topic will be with generated with specified num >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>> and configuration taken from produced parameter. >>>>>>>>>>>>>>>>>>>>>>> 2) Leave KStream#through as it is and introduce new >>>>>>>>>>>>>>>>>>>>>>> method >>>>>>>> - >>>>>>>>>>>>>>>>>> KStream#repartition (I think Matthias suggested this in one >>>>>>>>>>>>>>>>>> of >>>>>>>> the >>>>>>>>>>>>>>>> threads) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Considering all mentioned above I propose the following >>>>>>>> plan: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Option A: >>>>>>>>>>>>>>>>>>>>>>> 1) Leave Produced as it is >>>>>>>>>>>>>>>>>>>>>>> 2) Add num of partitions configuration to Grouped class >>>>>>>>>>>>>>>>>>>>>>> (as >>>>>>>>>>>>>>>>>> mentioned in the KIP) >>>>>>>>>>>>>>>>>>>>>>> 3) Add following method overloads to KStream#through >>>>>>>>>>>>>>>>>>>>>>> * through() - topic will be auto-generated and num of >>>>>>>>>> partitions >>>>>>>>>>>>>>>>>> will be taken from source topic >>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions) - topic will be >>>>>>>>>>>>>>>>>>>>>>> auto >>>>>>>>>>>>>>>>>> generated with specified num of partitions >>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions, final Produced<K, >>>>>>>>>>>>>>>>>>>>>>> V> >>>>>>>>>>>>>>>>>> produced) - topic will be with generated with specified num >>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>> and configuration taken from produced parameter. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Option B: >>>>>>>>>>>>>>>>>>>>>>> 1) Leave Produced as it is >>>>>>>>>>>>>>>>>>>>>>> 2) Add num of partitions configuration to Grouped class >>>>>>>>>>>>>>>>>>>>>>> (as >>>>>>>>>>>>>>>>>> mentioned in the KIP) >>>>>>>>>>>>>>>>>>>>>>> 3) Add new operator KStream#repartition for creating and >>>>>>>>>> managing >>>>>>>>>>>>>>>>>> internal repartition topics >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> P.S. I’m sorry if all of this was already discussed in >>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>> mailing >>>>>>>>>>>>>>>>>> list, but I kinda got with all the threads that were about >>>>>>>>>>>>>>>>>> this >>>>>>>>>> KIP :( >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Jul 1, 2019, at 9:56 AM, Levani Kokhreidze < >>>>>>>>>>>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I would like to resurrect discussion around KIP-221. >>>>>>>>>>>>>>>>>>>>>>>> Going >>>>>>>>>> through >>>>>>>>>>>>>>>>>> the discussion thread, there’s seems to agreement around >>>>>>>>>> usefulness of >>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>> feature. >>>>>>>>>>>>>>>>>>>>>>>> Regarding the implementation, as far as I understood, >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>> most >>>>>>>>>>>>>>>>>> optimal solution for me seems the following: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 1) Add two method overloads to KStream#through method >>>>>>>>>> (essentially >>>>>>>>>>>>>>>>>> making topic name optional) >>>>>>>>>>>>>>>>>>>>>>>> 2) Enhance Produced class with numOfPartitions >>>>>>>> configuration >>>>>>>>>>>>>>>> field. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Those two changes will allow DSL users to control >>>>>>>>>> parallelism and >>>>>>>>>>>>>>>>>> trigger re-partition without doing stateful operations. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I will update KIP with interface changes around >>>>>>>>>> KStream#through if >>>>>>>>>>>>>>>>>> this changes sound sensible. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >> >
