Hello all, Sorry for bringing this thread again, but I would like to get some attention on this PR: https://github.com/apache/kafka/pull/7170 <https://github.com/apache/kafka/pull/7170> It's been a while now and I would love to move on to other KIPs as well. Please let me know if you have any concerns.
Regards, Levani > On Jul 26, 2019, at 11:25 AM, Levani Kokhreidze <levani.co...@gmail.com> > wrote: > > Hi all, > > Here’s voting thread for this KIP: > https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html > <https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html> > > Regards, > Levani > >> On Jul 24, 2019, at 11:15 PM, Levani Kokhreidze <levani.co...@gmail.com >> <mailto:levani.co...@gmail.com>> wrote: >> >> Hi Matthias, >> >> Thanks for the suggestion. I Don’t have strong opinion on that one. >> Agree that avoiding unnecessary method overloads is a good idea. >> >> Updated KIP >> >> Regards, >> Levani >> >> >>> On Jul 24, 2019, at 8:50 PM, Matthias J. Sax <matth...@confluent.io >>> <mailto:matth...@confluent.io>> wrote: >>> >>> One question: >>> >>> Why do we add >>> >>>> Repartitioned#with(final String name, final int numberOfPartitions) >>> >>> It seems that `#with(String name)`, `#numberOfPartitions(int)` in >>> combination with `withName()` and `withNumberOfPartitions()` should be >>> sufficient. Users can chain the method calls. >>> >>> (I think it's valuable to keep the number of overload small if possible.) >>> >>> Otherwise LGTM. >>> >>> >>> -Matthias >>> >>> >>> On 7/23/19 2:18 PM, Levani Kokhreidze wrote: >>>> Hello, >>>> >>>> Thanks all for your feedback. >>>> I started voting procedure for this KIP. If there’re any other concerns >>>> about this KIP, please let me know. >>>> >>>> Regards, >>>> Levani >>>> >>>>> On Jul 20, 2019, at 8:39 PM, Levani Kokhreidze <levani.co...@gmail.com >>>>> <mailto:levani.co...@gmail.com>> wrote: >>>>> >>>>> Hi Matthias, >>>>> >>>>> Thanks for the suggestion, makes sense. >>>>> I’ve updated KIP >>>>> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>> >>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint> >>>>> >>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>> >>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint>>). >>>>> >>>>> Regards, >>>>> Levani >>>>> >>>>> >>>>>> On Jul 20, 2019, at 3:53 AM, Matthias J. Sax <matth...@confluent.io >>>>>> <mailto:matth...@confluent.io> <mailto:matth...@confluent.io >>>>>> <mailto:matth...@confluent.io>>> wrote: >>>>>> >>>>>> Thanks for driving the KIP. >>>>>> >>>>>> I agree that users need to be able to specify a partitioning strategy. >>>>>> >>>>>> Sophie raises a fair point about topic configs and producer configs. My >>>>>> take is, that consider `Repartitioned` as an "extension" to `Produced`, >>>>>> that adds topic configuration, is a good way to think about it and helps >>>>>> to keep the API "clean". >>>>>> >>>>>> >>>>>> With regard to method names. I would prefer to avoid abbreviations. Can >>>>>> we rename: >>>>>> >>>>>> `withNumOfPartitions` -> `withNumberOfPartitions` >>>>>> >>>>>> Furthermore, it might be good to add some more `static` methods: >>>>>> >>>>>> - Repartitioned.with(Serde<K>, Serde<V>) >>>>>> - Repartitioned.withNumberOfPartitions(int) >>>>>> - Repartitioned.streamPartitioner(StreamPartitioner) >>>>>> >>>>>> >>>>>> -Matthias >>>>>> >>>>>> On 7/19/19 3:33 PM, Levani Kokhreidze wrote: >>>>>>> Totally agree. I think in KStream interface it makes sense to have some >>>>>>> duplicate configurations between operators in order to keep API simple >>>>>>> and usable. >>>>>>> Also, as more surface API has, harder it is to have proper backward >>>>>>> compatibility. >>>>>>> While initial idea of keeping topic level configs separate was >>>>>>> exciting, having Repartitioned class encapsulate some producer level >>>>>>> configs makes API more readable. >>>>>>> >>>>>>> Regards, >>>>>>> Levani >>>>>>> >>>>>>>> On Jul 20, 2019, at 1:15 AM, Sophie Blee-Goldman <sop...@confluent.io >>>>>>>> <mailto:sop...@confluent.io> <mailto:sop...@confluent.io >>>>>>>> <mailto:sop...@confluent.io>>> wrote: >>>>>>>> >>>>>>>> I think that is a good point about trying to keep producer level >>>>>>>> configurations and (repartition) topic level considerations separate. >>>>>>>> Number of partitions is definitely purely a topic level configuration. >>>>>>>> But >>>>>>>> on some level, serdes and partitioners are just as much a topic >>>>>>>> configuration as a producer one. You could have two producers >>>>>>>> configured >>>>>>>> with different serdes and/or partitioners, but if they are writing to >>>>>>>> the >>>>>>>> same topic the result would be very difficult to part. So in a sense, >>>>>>>> these >>>>>>>> are configurations of topics in Streams, not just producers. >>>>>>>> >>>>>>>> Another way to think of it: while the Streams API is not always true to >>>>>>>> this, ideally all the relevant configs for an operator are wrapped >>>>>>>> into a >>>>>>>> single object (in this case, Repartitioned). We could instead split >>>>>>>> out the >>>>>>>> fields in common with Produced into a separate parameter to keep topic >>>>>>>> and >>>>>>>> producer level configurations separate, but this increases the API >>>>>>>> surface >>>>>>>> area by a lot. It's much more straightforward to just say "this is >>>>>>>> everything that this particular operator needs" without worrying about >>>>>>>> what >>>>>>>> exactly you're specifying. >>>>>>>> >>>>>>>> I suppose you could alternatively make Produced a field of >>>>>>>> Repartitioned, >>>>>>>> but I don't think we do this kind of composition elsewhere in Streams >>>>>>>> at >>>>>>>> the moment >>>>>>>> >>>>>>>> On Fri, Jul 19, 2019 at 1:45 PM Levani Kokhreidze >>>>>>>> <levani.co...@gmail.com <mailto:levani.co...@gmail.com> >>>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Bill, >>>>>>>>> >>>>>>>>> Thanks a lot for the feedback. >>>>>>>>> Yes, that makes sense. I’ve updated KIP with >>>>>>>>> `Repartitioned#partitioner` >>>>>>>>> configuration. >>>>>>>>> In the beginning, I wanted to introduce a class for topic level >>>>>>>>> configuration and keep topic level and producer level configurations >>>>>>>>> (such >>>>>>>>> as Produced) separately (see my second email in this thread). >>>>>>>>> But while looking at the semantics of KStream interface, I couldn’t >>>>>>>>> really >>>>>>>>> figure out good operation name for Topic level configuration class >>>>>>>>> and just >>>>>>>>> introducing `Topic` config class was kinda breaking the semantics. >>>>>>>>> So I think having Repartitioned class which encapsulates topic and >>>>>>>>> producer level configurations for internal topics is viable thing to >>>>>>>>> do. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Levani >>>>>>>>> >>>>>>>>>> On Jul 19, 2019, at 7:47 PM, Bill Bejeck <bbej...@gmail.com >>>>>>>>>> <mailto:bbej...@gmail.com> <mailto:bbej...@gmail.com >>>>>>>>>> <mailto:bbej...@gmail.com>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Lavani, >>>>>>>>>> >>>>>>>>>> Thanks for resurrecting this KIP. >>>>>>>>>> >>>>>>>>>> I'm also a +1 for adding a partition option. In addition to the >>>>>>>>>> reason >>>>>>>>>> provided by John, my reasoning is: >>>>>>>>>> >>>>>>>>>> 1. Users may want to use something other than hash-based partitioning >>>>>>>>>> 2. Users may wish to partition on something different than the key >>>>>>>>>> without having to change the key. For example: >>>>>>>>>> 1. A combination of fields in the value in conjunction with the key >>>>>>>>>> 2. Something other than the key >>>>>>>>>> 3. We allow users to specify a partitioner on Produced hence in >>>>>>>>>> KStream.to and KStream.through, so it makes sense for API >>>>>>>>>> consistency. >>>>>>>>>> >>>>>>>>>> Just my 2 cents. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Bill >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Fri, Jul 19, 2019 at 5:46 AM Levani Kokhreidze < >>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com> >>>>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>>> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi John, >>>>>>>>>>> >>>>>>>>>>> In my mind it makes sense. >>>>>>>>>>> If we add partitioner configuration to Repartitioned class, with the >>>>>>>>>>> combination of specifying number of partitions for internal topics, >>>>>>>>>>> user >>>>>>>>>>> will have opportunity to ensure co-partitioning before join >>>>>>>>>>> operation. >>>>>>>>>>> I think this can be quite powerful feature. >>>>>>>>>>> Wondering what others think about this? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Levani >>>>>>>>>>> >>>>>>>>>>>> On Jul 18, 2019, at 1:20 AM, John Roesler <j...@confluent.io >>>>>>>>>>>> <mailto:j...@confluent.io> <mailto:j...@confluent.io >>>>>>>>>>>> <mailto:j...@confluent.io>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Yes, I believe that's what I had in mind. Again, not totally sure >>>>>>>>>>>> it >>>>>>>>>>>> makes sense, but I believe something similar is the rationale for >>>>>>>>>>>> having the partitioner option in Produced. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> -John >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jul 17, 2019 at 3:20 PM Levani Kokhreidze >>>>>>>>>>>> <levani.co...@gmail.com <mailto:levani.co...@gmail.com> >>>>>>>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hey John, >>>>>>>>>>>>> >>>>>>>>>>>>> Oh that’s interesting use-case. >>>>>>>>>>>>> Do I understand this correctly, in your example I would first >>>>>>>>>>>>> issue >>>>>>>>>>> repartition(Repartitioned) with proper partitioner that essentially >>>>>>>>> would >>>>>>>>>>> be the same as the topic I want to join with and then do the >>>>>>>>> KStream#join >>>>>>>>>>> with DSL? >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Levani >>>>>>>>>>>>> >>>>>>>>>>>>>> On Jul 17, 2019, at 11:11 PM, John Roesler <j...@confluent.io >>>>>>>>>>>>>> <mailto:j...@confluent.io> <mailto:j...@confluent.io >>>>>>>>>>>>>> <mailto:j...@confluent.io>>> >>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hey, all, just to chime in, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think it might be useful to have an option to specify the >>>>>>>>>>>>>> partitioner. The case I have in mind is that some data may get >>>>>>>>>>>>>> repartitioned and then joined with an input topic. If the >>>>>>>>>>>>>> right-side >>>>>>>>>>>>>> input topic uses a custom partitioning strategy, then the >>>>>>>>>>>>>> repartitioned stream also needs to be partitioned with the same >>>>>>>>>>>>>> strategy. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Does that make sense, or did I maybe miss something important? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> -John >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 2:48 PM Levani Kokhreidze >>>>>>>>>>>>>> <levani.co...@gmail.com <mailto:levani.co...@gmail.com> >>>>>>>>>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes, I was thinking about it as well. To be honest I’m not sure >>>>>>>>> about >>>>>>>>>>> it yet. >>>>>>>>>>>>>>> As Kafka Streams DSL user, I don’t really think I would need >>>>>>>>>>>>>>> control >>>>>>>>>>> over partitioner for internal topics. >>>>>>>>>>>>>>> As a user, I would assume that Kafka Streams knows best how to >>>>>>>>>>> partition data for internal topics. >>>>>>>>>>>>>>> In this KIP I wrote that Produced should be used only for topics >>>>>>>>> that >>>>>>>>>>> are created by user In advance. >>>>>>>>>>>>>>> In those cases maybe it make sense to have possibility to >>>>>>>>>>>>>>> specify >>>>>>>>> the >>>>>>>>>>> partitioner. >>>>>>>>>>>>>>> I don’t have clear answer on that yet, but I guess specifying >>>>>>>>>>>>>>> the >>>>>>>>>>> partitioner can be added as well if there’s agreement on this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Jul 17, 2019, at 10:42 PM, Sophie Blee-Goldman < >>>>>>>>>>> sop...@confluent.io <mailto:sop...@confluent.io> >>>>>>>>>>> <mailto:sop...@confluent.io <mailto:sop...@confluent.io>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for clearing that up. I agree that Repartitioned would >>>>>>>>>>>>>>>> be a >>>>>>>>>>> useful >>>>>>>>>>>>>>>> addition. I'm wondering if it might also need to have >>>>>>>>>>>>>>>> a withStreamPartitioner method/field, similar to Produced? I'm >>>>>>>>>>>>>>>> not >>>>>>>>>>> sure how >>>>>>>>>>>>>>>> widely this feature is really used, but seems it should be >>>>>>>>> available >>>>>>>>>>> for >>>>>>>>>>>>>>>> repartition topics. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 11:26 AM Levani Kokhreidze < >>>>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com> >>>>>>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hey Sophie, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In both cases KStream#repartition and >>>>>>>>>>> KStream#repartition(Repartitioned) >>>>>>>>>>>>>>>>> topic will be created and managed by Kafka Streams. >>>>>>>>>>>>>>>>> Idea of Repartitioned is to give user more control over the >>>>>>>>>>>>>>>>> topic >>>>>>>>>>> such as >>>>>>>>>>>>>>>>> num of partitions. >>>>>>>>>>>>>>>>> I feel like Repartitioned parameter is something that is >>>>>>>>>>>>>>>>> missing >>>>>>>>> in >>>>>>>>>>>>>>>>> current DSL design. >>>>>>>>>>>>>>>>> Essentially giving user control over parallelism by >>>>>>>>>>>>>>>>> configuring >>>>>>>>> num >>>>>>>>>>> of >>>>>>>>>>>>>>>>> partitions for internal topics. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hope this answers your question. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Jul 17, 2019, at 9:02 PM, Sophie Blee-Goldman < >>>>>>>>>>> sop...@confluent.io <mailto:sop...@confluent.io> >>>>>>>>>>> <mailto:sop...@confluent.io <mailto:sop...@confluent.io>>> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hey Levani, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks for the KIP! Can you clarify one thing for me -- for >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> KStream#repartition signature taking a Repartitioned, will >>>>>>>>>>>>>>>>>> the >>>>>>>>>>> topic be >>>>>>>>>>>>>>>>>> auto-created by Streams (which seems to be the case for the >>>>>>>>>>> signature >>>>>>>>>>>>>>>>>> without a Repartitioned) or does it have to be pre-created? >>>>>>>>>>>>>>>>>> The >>>>>>>>>>> wording >>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>> the KIP makes it seem like one version of the method will >>>>>>>>>>> auto-create >>>>>>>>>>>>>>>>>> topics while the other will not. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>> Sophie >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 10:15 AM Levani Kokhreidze < >>>>>>>>>>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com> >>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com >>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com>>> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> One more bump about KIP-221 ( >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>> >>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint> >>>>>>>>> >>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>> >>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint>> >>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>> >>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint> >>>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>> so it doesn’t get lost in mailing list :) >>>>>>>>>>>>>>>>>>> Would love to hear communities opinions/concerns about this >>>>>>>>>>>>>>>>>>> KIP. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On Jul 12, 2019, at 5:27 PM, Levani Kokhreidze < >>>>>>>>>>> levani.co...@gmail.com >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Kind reminder about this KIP: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Jul 9, 2019, at 11:38 AM, Levani Kokhreidze < >>>>>>>>>>>>>>>>> levani.co...@gmail.com >>>>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com>> wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> In order to move this KIP forward, I’ve updated confluence >>>>>>>>> page >>>>>>>>>>> with >>>>>>>>>>>>>>>>>>> the new proposal >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I’ve also filled “Rejected Alternatives” section. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Looking forward to discuss this KIP :) >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> King regards, >>>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On Jul 3, 2019, at 1:08 PM, Levani Kokhreidze < >>>>>>>>>>>>>>>>> levani.co...@gmail.com >>>>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com>> wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hello Matthias, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback and ideas. >>>>>>>>>>>>>>>>>>>>>> I like the idea of introducing dedicated `Topic` class >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>> topic >>>>>>>>>>>>>>>>>>> configuration for internal operators like `groupedBy`. >>>>>>>>>>>>>>>>>>>>>> Would be great to hear others opinion about this as well. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On Jul 3, 2019, at 7:00 AM, Matthias J. Sax < >>>>>>>>>>> matth...@confluent.io >>>>>>>>>>>>>>>>>>> <mailto:matth...@confluent.io>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Levani, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks for picking up this KIP! And thanks for >>>>>>>>>>>>>>>>>>>>>>> summarizing >>>>>>>>>>>>>>>>> everything. >>>>>>>>>>>>>>>>>>>>>>> Even if some points may have been discussed already >>>>>>>>>>>>>>>>>>>>>>> (can't >>>>>>>>>>> really >>>>>>>>>>>>>>>>>>>>>>> remember), it's helpful to get a good summary to >>>>>>>>>>>>>>>>>>>>>>> refresh the >>>>>>>>>>>>>>>>>>> discussion. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I think your reasoning makes sense. With regard to the >>>>>>>>>>> distinction >>>>>>>>>>>>>>>>>>>>>>> between operators that manage topics and operators that >>>>>>>>>>>>>>>>>>>>>>> use >>>>>>>>>>>>>>>>>>> user-created >>>>>>>>>>>>>>>>>>>>>>> topics: Following this argument, it might indicate that >>>>>>>>>>> leaving >>>>>>>>>>>>>>>>>>>>>>> `through()` as-is (as an operator that uses use-defined >>>>>>>>>>> topics) and >>>>>>>>>>>>>>>>>>>>>>> introducing a new `repartition()` operator (an operator >>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>> manages >>>>>>>>>>>>>>>>>>>>>>> topics itself) might be good. Otherwise, there is one >>>>>>>>> operator >>>>>>>>>>>>>>>>>>>>>>> `through()` that sometimes manages topics but sometimes >>>>>>>>> not; a >>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>> name, ie, new operator would make the distinction >>>>>>>>>>>>>>>>>>>>>>> clearer. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> About adding `numOfPartitions` to `Grouped`. I am >>>>>>>>>>>>>>>>>>>>>>> wondering >>>>>>>>>>> if the >>>>>>>>>>>>>>>>>>> same >>>>>>>>>>>>>>>>>>>>>>> argument as for `Produced` does apply and adding it is >>>>>>>>>>> semantically >>>>>>>>>>>>>>>>>>>>>>> questionable? Might be good to get opinions of others on >>>>>>>>>>> this, too. >>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>>>>>>>> not sure myself what solution I prefer atm. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> So far, KS uses configuration objects that allow to >>>>>>>>> configure >>>>>>>>>>> a >>>>>>>>>>>>>>>>>>> certain >>>>>>>>>>>>>>>>>>>>>>> "entity" like a consumer, producer, store. If we assume >>>>>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>> a topic >>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>> a similar entity, I am wonder if we should have a >>>>>>>>>>>>>>>>>>>>>>> `Topic#withNumberOfPartitions()` class and method >>>>>>>>>>>>>>>>>>>>>>> instead of >>>>>>>>>>> a plain >>>>>>>>>>>>>>>>>>>>>>> integer? This would allow us to add other configs, like >>>>>>>>>>> replication >>>>>>>>>>>>>>>>>>>>>>> factor, retention-time etc, easily, without the need to >>>>>>>>>>> change the >>>>>>>>>>>>>>>>>>> "main >>>>>>>>>>>>>>>>>>>>>>> API". >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Just want to give some ideas. Not sure if I like them >>>>>>>>> myself. >>>>>>>>>>> :) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On 7/1/19 1:04 AM, Levani Kokhreidze wrote: >>>>>>>>>>>>>>>>>>>>>>>> Actually, giving it more though - maybe enhancing >>>>>>>>>>>>>>>>>>>>>>>> Produced >>>>>>>>>>> with num >>>>>>>>>>>>>>>>>>> of partitions configuration is not the best approach. Let me >>>>>>>>>>> explain >>>>>>>>>>>>>>>>> why: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 1) If we enhance Produced class with this >>>>>>>>>>>>>>>>>>>>>>>> configuration, >>>>>>>>>>> this will >>>>>>>>>>>>>>>>>>> also affect KStream#to operation. Since KStream#to is the >>>>>>>>>>>>>>>>>>> final >>>>>>>>>>> sink of >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>> topology, for me, it seems to be reasonable assumption that >>>>>>>>>>>>>>>>>>> user >>>>>>>>>>> needs >>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> manually create sink topic in advance. And in that case, >>>>>>>>>>>>>>>>>>> having >>>>>>>>>>> num of >>>>>>>>>>>>>>>>>>> partitions configuration doesn’t make much sense. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 2) Looking at Produced class, based on API contract, >>>>>>>>>>>>>>>>>>>>>>>> seems >>>>>>>>>>> like >>>>>>>>>>>>>>>>>>> Produced is designed to be something that is explicitly for >>>>>>>>>>> producer >>>>>>>>>>>>>>>>> (key >>>>>>>>>>>>>>>>>>> serializer, value serializer, partitioner those all are >>>>>>>>>>>>>>>>>>> producer >>>>>>>>>>>>>>>>> specific >>>>>>>>>>>>>>>>>>> configurations) and num of partitions is topic level >>>>>>>>>>> configuration. And >>>>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>> don’t think mixing topic and producer level configurations >>>>>>>>>>> together in >>>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>> class is the good approach. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> 3) Looking at KStream interface, seems like Produced >>>>>>>>>>> parameter is >>>>>>>>>>>>>>>>>>> for operations that work with non-internal (e.g topics >>>>>>>>>>>>>>>>>>> created >>>>>>>>> and >>>>>>>>>>>>>>>>> managed >>>>>>>>>>>>>>>>>>> internally by Kafka Streams) topics and I think we should >>>>>>>>>>>>>>>>>>> leave >>>>>>>>>>> it as >>>>>>>>>>>>>>>>> it is >>>>>>>>>>>>>>>>>>> in that case. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Taking all this things into account, I think we should >>>>>>>>>>> distinguish >>>>>>>>>>>>>>>>>>> between DSL operations, where Kafka Streams should create >>>>>>>>>>>>>>>>>>> and >>>>>>>>>>> manage >>>>>>>>>>>>>>>>>>> internal topics (KStream#groupBy) vs topics that should be >>>>>>>>>>> created in >>>>>>>>>>>>>>>>>>> advance (e.g KStream#to). >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> To sum it up, I think adding numPartitions >>>>>>>>>>>>>>>>>>>>>>>> configuration in >>>>>>>>>>>>>>>>> Produced >>>>>>>>>>>>>>>>>>> will result in mixing topic and producer level >>>>>>>>>>>>>>>>>>> configuration in >>>>>>>>>>> one >>>>>>>>>>>>>>>>> class >>>>>>>>>>>>>>>>>>> and it’s gonna break existing API semantics. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Regarding making topic name optional in >>>>>>>>>>>>>>>>>>>>>>>> KStream#through - I >>>>>>>>>>> think >>>>>>>>>>>>>>>>>>> underline idea is very useful and giving users possibility >>>>>>>>>>>>>>>>>>> to >>>>>>>>>>> specify >>>>>>>>>>>>>>>>> num >>>>>>>>>>>>>>>>>>> of partitions there is even more useful :) Considering >>>>>>>>>>>>>>>>>>> arguments >>>>>>>>>>> against >>>>>>>>>>>>>>>>>>> adding num of partitions in Produced class, I see two >>>>>>>>>>>>>>>>>>> options >>>>>>>>>>> here: >>>>>>>>>>>>>>>>>>>>>>>> 1) Add following method overloads >>>>>>>>>>>>>>>>>>>>>>>> * through() - topic will be auto-generated and num of >>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>> will be taken from source topic >>>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions) - topic will be >>>>>>>>>>>>>>>>>>>>>>>> auto >>>>>>>>>>>>>>>>>>> generated with specified num of partitions >>>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions, final Produced<K, >>>>>>>>>>>>>>>>>>>>>>>> V> >>>>>>>>>>>>>>>>>>> produced) - topic will be with generated with specified num >>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>> and configuration taken from produced parameter. >>>>>>>>>>>>>>>>>>>>>>>> 2) Leave KStream#through as it is and introduce new >>>>>>>>>>>>>>>>>>>>>>>> method >>>>>>>>> - >>>>>>>>>>>>>>>>>>> KStream#repartition (I think Matthias suggested this in one >>>>>>>>>>>>>>>>>>> of >>>>>>>>> the >>>>>>>>>>>>>>>>> threads) >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Considering all mentioned above I propose the following >>>>>>>>> plan: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Option A: >>>>>>>>>>>>>>>>>>>>>>>> 1) Leave Produced as it is >>>>>>>>>>>>>>>>>>>>>>>> 2) Add num of partitions configuration to Grouped >>>>>>>>>>>>>>>>>>>>>>>> class (as >>>>>>>>>>>>>>>>>>> mentioned in the KIP) >>>>>>>>>>>>>>>>>>>>>>>> 3) Add following method overloads to KStream#through >>>>>>>>>>>>>>>>>>>>>>>> * through() - topic will be auto-generated and num of >>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>> will be taken from source topic >>>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions) - topic will be >>>>>>>>>>>>>>>>>>>>>>>> auto >>>>>>>>>>>>>>>>>>> generated with specified num of partitions >>>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions, final Produced<K, >>>>>>>>>>>>>>>>>>>>>>>> V> >>>>>>>>>>>>>>>>>>> produced) - topic will be with generated with specified num >>>>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> partitions >>>>>>>>>>>>>>>>>>> and configuration taken from produced parameter. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Option B: >>>>>>>>>>>>>>>>>>>>>>>> 1) Leave Produced as it is >>>>>>>>>>>>>>>>>>>>>>>> 2) Add num of partitions configuration to Grouped >>>>>>>>>>>>>>>>>>>>>>>> class (as >>>>>>>>>>>>>>>>>>> mentioned in the KIP) >>>>>>>>>>>>>>>>>>>>>>>> 3) Add new operator KStream#repartition for creating >>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>> managing >>>>>>>>>>>>>>>>>>> internal repartition topics >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> P.S. I’m sorry if all of this was already discussed in >>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>> mailing >>>>>>>>>>>>>>>>>>> list, but I kinda got with all the threads that were about >>>>>>>>>>>>>>>>>>> this >>>>>>>>>>> KIP :( >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Jul 1, 2019, at 9:56 AM, Levani Kokhreidze < >>>>>>>>>>>>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>> >>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I would like to resurrect discussion around KIP-221. >>>>>>>>>>>>>>>>>>>>>>>>> Going >>>>>>>>>>> through >>>>>>>>>>>>>>>>>>> the discussion thread, there’s seems to agreement around >>>>>>>>>>> usefulness of >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>> feature. >>>>>>>>>>>>>>>>>>>>>>>>> Regarding the implementation, as far as I understood, >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>> most >>>>>>>>>>>>>>>>>>> optimal solution for me seems the following: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 1) Add two method overloads to KStream#through method >>>>>>>>>>> (essentially >>>>>>>>>>>>>>>>>>> making topic name optional) >>>>>>>>>>>>>>>>>>>>>>>>> 2) Enhance Produced class with numOfPartitions >>>>>>>>> configuration >>>>>>>>>>>>>>>>> field. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Those two changes will allow DSL users to control >>>>>>>>>>> parallelism and >>>>>>>>>>>>>>>>>>> trigger re-partition without doing stateful operations. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> I will update KIP with interface changes around >>>>>>>>>>> KStream#through if >>>>>>>>>>>>>>>>>>> this changes sound sensible. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>>>>>>>>>>>>>> Levani >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >> >
