Ah, thanks for the clarification. I missed your point.
I like the framework you've presented. It does seem simpler to assume
that they either care about the partition count and want to
repartition to realize it, or they don't care about the number.
Returning to this discussion, it does seem unlikely that they care
about the number and _don't_ care if it actually gets realized.
But then, it still seems like we can just keep the option as part of
Grouped. As in:
// user does not care
stream.groupByKey(Grouped /*not specifying partition count*/)
stream.groupBy(Grouped /*not specifying partition count*/)
// user does care
stream.repartition(Repartitioned)
stream.groupByKey(Grouped.numberOfPartitions(...))
stream.groupBy(Grouped.numberOfPartitions(...))
----
The above discussion got me thinking about algebra. Matthias is
absolutely right that `groupByKey(numPartitions)` is equivalent to
`repartition(numPartitions).groupByKey()`. I'm just not convinced that
we should force people to apply that expansion themselves vs. having a
more compact way to express it if they don't care where exactly the
repartition occurs. However, thinking about these operators
algebraically can really help *us* narrow down the number of different
expressions we have to consider.
Let's consider some identities:
A: groupBy(mapper) + agg = mapKey(mapper) + groupByKey + agg
B: src + ... + groupByKey + agg = src + ... + passthough + agg
C: mapKey(mapper) + ... + groupByKey + agg
= mapKey(mapper) + ... + repartition + groupByKey + agg
D: repartition = sink(managed) + src
In these identities, I used one special identifier (...), which means
any number (0+) of operations that are not src, mapKey, groupBy[Key],
repartition, or agg.
For mental clarity, I'm just going to make up a rule that groupBy
operations are not executable. In other words, you have to get to a
point where you can apply B to convert a groupByKey into a passthough
in order to execute the program. This is just a formal way of stating
what already happens in Kafka Streams.
By applying A, we can just completely leave `groupBy` out of our
analysis. It trivially decomposes into a mapKey followed by a
groupByKey.
Then, we can eliminate the "repartition required" case of `groupByKey`
by applying C followed by D to get to the "no repartition required"
version of groupByKey, which in turn sets us up to apply B to get an
executable topology.
Fundamentally, you can think about KIP-221 is as proposing a modified
D identity in which you can specify the partition count of the managed
sink topic:
D': repartition(pc) = sink(managed w/ pc) + src
Since users _could_ apply the identities above, we don't actually have
to add any partition count to groupBy[Key], but we decided early on in
the KIP discussion that it's more ergonomic to add it. In that case,
we also have to modify A and C:
A': groupBy(mapper, pc) + agg
= mapKey(mapper) + groupByKey(pc) + agg
C': mapKey(mapper) + ... + groupByKey(pc) + agg
= mapKey(mapper) + ... + repartition(pc) + groupByKey + agg
Which sets us up still to always be able to get back to a plain
`groupByKey` operation (with no `(pc)`) and then apply D' and
ultimately B to get an executable topology.
What about the optimizer?
The optimizer applies another set of graph-algebraic identities to
minimize the number of repartition topics in a topology.
(forgive my ascii art)
E: (merging repartition nodes)
(...) -> repartition -> X
\-> repartition -> Y
=
(... + repartition) -> X
\-> Y
F: (reordering around repartition)
Where SVO is any non-key-changing, stateless, operation:
repartition -> SVO = SVO -> repartition
In terms of these identities, what the optimizer does is apply F
repeatedly in either direction to a topology to factor out common in
branches so that it can apply E to merge repartition nodes. This was
especially necessary before KIP-221 because you couldn't directly
express `repartition` in the DSL, only indirectly via `groupBy[Key]`,
so there was no way to do the factoring manually.
We can now state very clearly that in KIP-221, explicit
`repartition()` operators should create a "reordering barrier". So, F
cannot be applied to an explicit `repartition()`. Also, I think we
decided earlier that explicit `repartition()` operations would also be
ineligible for merging, so E can't be applied to explicit
`repartition()` operations either. I think we feel we _could_ apply E
without harm, but we want to be conservative for now.
I think the salient point from the latter discussion has been that
when you use `Grouped.numberOfPartitions`, this does _not_ constitute
an explicit `repartition()` operator, and therefore, the resulting
repartition node remains eligible for optimization.
To be clear, I agree with Matthias that the provided partition count
_must_ be used in the resulting implicit repartition. This has some
implications for E. Namely, E could only be applied to two repartition
nodes that have the same partition count. This has always been
trivially true before KIP-221 because the partition count has always
been "unspecified", i.e., it would be determined at runtime by the
user-managed-topics' partition counts. Now, it could be specified or
unspecified. We can simply augment E to allow merging only repartition
nodes where the partition count is EITHER "specified and the same on
both sides", OR "unspecified on both sides".
Sorry for the long email, but I have a hope that it builds a solid
theoretical foundation for our decisions in KIP-221, so we can have
confidence that there are no edge cases for design flaws to hide.
Thanks,
-John
On Sat, Nov 9, 2019 at 10:37 PM Matthias J. Sax <matth...@confluent.io> wrote:
>
> > it seems like we do want to allow
> >> people to optionally specify a partition count as part of this
> >> operation, but we don't want that option to _force_ repartitioning
>
> Correct, ie, that is my suggestions.
>
> > "Use P partitions if repartitioning is necessary"
>
> I disagree here, because my reasoning is that:
>
> - if a user cares about the number of partition, the user wants to
> enforce a repartitioning
> - if a user does not case about the number of partitions, we don't need
> to provide them a way to pass in a "hint"
>
> Hence, it should be sufficient to support:
>
> // user does not care
>
> `stream.groupByKey(Grouped)`
> `stream.grouBy(..., Grouped)`
>
> // user does care
>
> `stream.repartition(Repartitioned).groupByKey()`
> `streams.groupBy(..., Repartitioned)`
>
>
>
> -Matthias
>
>
> On 11/9/19 8:10 PM, John Roesler wrote:
> > Thanks for those thoughts, Matthias,
> >
> > I find your reasoning about the optimization behavior compelling. The
> > `through` operation is very simple and clear to reason about. It just
> > passes the data exactly at the specified point in the topology exactly
> > through the specified topic. Likewise, if a user invokes a
> > `repartition` operator, the simplest behavior is if we just do what
> > they asked for.
> >
> > Stepping back to think about when optimizations are surprising and
> > when they aren't, it occurs to me that we should be free to move
> > around repartitions when users have asked to perform some operation
> > that implies a repartition, like "change keys, then filter, then
> > aggregate". This program requires a repartition, but it could be
> > anywhere between the key change and the aggregation. On the other
> > hand, if they say, "change keys, then filter, then repartition, then
> > aggregate", it seems like they were pretty clear about their desire,
> > and we should just take it at face value.
> >
> > So, I'm sold on just literally doing a repartition every time they
> > invoke the `repartition` operator.
> >
> >
> > The "partition count" modifier for `groupBy`/`groupByKey` is more nuanced.
> >
> > What you said about `groupByKey` makes sense. If they key hasn't
> > actually changed, then we don't need to repartition before
> > aggregating. On the other hand, `groupBy` is specifically changing the
> > key as part of the grouping operation, so (as you said) we definitely
> > have to do a repartition.
> >
> > If I'm reading the discussion right, it seems like we do want to allow
> > people to optionally specify a partition count as part of this
> > operation, but we don't want that option to _force_ repartitioning if
> > it's not needed. That last clause is the key. "Use P partitions if
> > repartitioning is necessary" is a directive that applies cleanly and
> > correctly to both `groupBy` and `groupByKey`. What if we call the
> > option `numberOfPartitionsHint`, which along with the "if necessary"
> > javadoc, should make it clear that the option won't force a
> > repartition, and also gives us enough latitude to still employ the
> > optimizer on those repartition topics?
> >
> > If we like the idea of expressing it as a "hint" for grouping and a
> > "command" for `repartition`, then it seems like it still makes sense
> > to keep Grouped and Repartitioned separate, as they would actually
> > offer different methods with distinct semantics.
> >
> > WDYT?
> >
> > Thanks,
> > -John
> >
> > On Sat, Nov 9, 2019 at 8:28 PM Matthias J. Sax <matth...@confluent.io>
> > wrote:
> >>
> >> Sorry for late reply.
> >>
> >> I guess, the question boils down to the intended semantics of
> >> `repartition()`. My understanding is as follows:
> >>
> >> - KS does auto-repartitioning for correctness reasons (using the
> >> upstream topic to determine the number of partitions)
> >> - KS does auto-repartitioning only for downstream DSL operators like
> >> `count()` (eg, a `transform()` does never trigger an auto-repartitioning
> >> even if the stream is marked as `repartitioningRequired`).
> >> - KS offers `through()` to enforce a repartitioning -- however, the user
> >> needs to create the topic manually (with the desired number of partitions).
> >>
> >> I see two main applications for `repartitioning()`:
> >>
> >> 1) repartition data before a `transform()` but user does not want to
> >> manage the topic
> >> 2) scale out a downstream subtopology
> >>
> >> Hence, I see `repartition()` similar to `through()`: if a users calls
> >> it, a repartitining is enforced, with the difference that KS manages the
> >> topic and the user does not need to create it.
> >>
> >> This behavior makes (1) and (2) possible.
> >>
> >>> I think many users would prefer to just say "if there *is* a repartition
> >>> required at this point in the topology, it should
> >>> have N partitions"
> >>
> >> Because of (2), I disagree. Either a user does not care about scaling
> >> out, for which case she would not specify the number of partitions. Or a
> >> user does care, and hence wants to enforce the scale out. I don't think
> >> that any user would say, "maybe scale out".
> >>
> >> Therefore, the optimizer should never ignore the repartition operation.
> >> As a "consequence" (because repartitioning is expensive) a user should
> >> make an explicit call to `repartition()` IMHO -- piggybacking an
> >> enforced repartitioning into `groupByKey()` seems to be "dangerous"
> >> because it might be too subtle and an "optional scaling out" as laid out
> >> above does not make sense IMHO.
> >>
> >> I am also not worried about "over repartitioning" because the result
> >> stream would never trigger auto-repartitioning. Only if multiple
> >> consecutive calls to `repartition()` are made it could be bad -- but
> >> that's the same with `through()`. In the end, there is always some
> >> responsibility on the user.
> >>
> >> Btw, for `.groupBy()` we know that repartitioning will be required,
> >> however, for `groupByKey()` it depends if the KStream is marked as
> >> `repartitioningRequired`.
> >>
> >> Hence, for `groupByKey()` it should not be possible for a user to set
> >> number of partitions IMHO. For `groupBy()` it's a different story,
> >> because calling
> >>
> >> `repartition().groupBy()`
> >>
> >> does not achieve what we want. Hence, allowing users to pass in the
> >> number of users partitions into `groupBy()` does actually makes sense,
> >> because repartitioning will happen anyway and thus we can piggyback a
> >> scaling decision.
> >>
> >> I think that John has a fair concern about the overloads, however, I am
> >> not convinced that using `Grouped` to specify the number of partitions
> >> is intuitive. I double checked `Grouped` and `Repartitioned` and both
> >> allow to specify a `name` and `keySerde/valueSerde`. Thus, I am
> >> wondering if we could bridge the gap between both, if we would make
> >> `Repartitioned extends Grouped`? For this case, we only need
> >> `groupBy(Grouped)` and a user can pass in both types what seems to make
> >> the API quite smooth:
> >>
> >> `stream.groupBy(..., Grouped...)`
> >>
> >> `stream.groupBy(..., Repartitioned...)`
> >>
> >>
> >> Thoughts?
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 11/7/19 10:59 AM, Levani Kokhreidze wrote:
> >>> Hi Sophie,
> >>>
> >>> Thank you for your reply, very insightful. Looking forward hearing others
> >>> opinion as well on this.
> >>>
> >>> Kind regards,
> >>> Levani
> >>>
> >>>
> >>>> On Nov 6, 2019, at 1:30 AM, Sophie Blee-Goldman <sop...@confluent.io>
> >>>> wrote:
> >>>>
> >>>>> Personally, I think Matthias’s concern is valid, but on the other hand
> >>>> Kafka Streams has already
> >>>>> optimizer in place which alters topology independently from user
> >>>>
> >>>> I agree (with you) and think this is a good way to put it -- we currently
> >>>> auto-repartition for the user so
> >>>> that they don't have to walk through their entire topology and reason
> >>>> about
> >>>> when and where to place a
> >>>> `.through` (or the new `.repartition`), so why suddenly force this onto
> >>>> the
> >>>> user? How certain are we that
> >>>> users will always get this right? It's easy to imagine that during
> >>>> development, you write your new app with
> >>>> correctly placed repartitions in order to use this new feature. During
> >>>> the
> >>>> course of development you end up
> >>>> tweaking the topology, but don't remember to review or move the
> >>>> repartitioning since you're used to Streams
> >>>> doing this for you. If you use only single-partition topics for testing,
> >>>> you might not even notice your app is
> >>>> spitting out incorrect results!
> >>>>
> >>>> Anyways, I feel pretty strongly that it would be weird to introduce a new
> >>>> feature and say that to use it, you can't take
> >>>> advantage of this other feature anymore. Also, is it possible our
> >>>> optimization framework could ever include an
> >>>> optimized repartitioning strategy that is better than what a user could
> >>>> achieve by manually inserting repartitions?
> >>>> Do we expect users to have a deep understanding of the best way to
> >>>> repartition their particular topology, or is it
> >>>> likely they will end up over-repartitioning either due to missed
> >>>> optimizations or unnecessary extra repartitions?
> >>>> I think many users would prefer to just say "if there *is* a repartition
> >>>> required at this point in the topology, it should
> >>>> have N partitions"
> >>>>
> >>>> As to the idea of adding `numberOfPartitions` to Grouped rather than
> >>>> adding a new parameter to groupBy, that does seem more in line with the
> >>>> current syntax so +1 from me
> >>>>
> >>>> On Tue, Nov 5, 2019 at 2:07 PM Levani Kokhreidze <levani.co...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hello all,
> >>>>>
> >>>>> While https://github.com/apache/kafka/pull/7170 <
> >>>>> https://github.com/apache/kafka/pull/7170> is under review and it’s
> >>>>> almost done, I want to resurrect discussion about this KIP to address
> >>>>> couple of concerns raised by Matthias and John.
> >>>>>
> >>>>> As a reminder, idea of the KIP-221 was to allow DSL users control over
> >>>>> repartitioning and parallelism of sub-topologies by:
> >>>>> 1) Introducing new KStream#repartition operation which is done in
> >>>>> https://github.com/apache/kafka/pull/7170 <
> >>>>> https://github.com/apache/kafka/pull/7170>
> >>>>> 2) Add new KStream#groupBy(Repartitioned) operation, which is planned to
> >>>>> be separate PR.
> >>>>>
> >>>>> While all agree about general implementation and idea behind
> >>>>> https://github.com/apache/kafka/pull/7170 <
> >>>>> https://github.com/apache/kafka/pull/7170> PR, introducing new
> >>>>> KStream#groupBy(Repartitioned) method overload raised some questions
> >>>>> during
> >>>>> the review.
> >>>>> Matthias raised concern that there can be cases when user uses
> >>>>> `KStream#groupBy(Repartitioned)` operation, but actual repartitioning
> >>>>> may
> >>>>> not required, thus configuration passed via `Repartitioned` would never
> >>>>> be
> >>>>> applied (Matthias, please correct me if I misinterpreted your comment).
> >>>>> So instead, if user wants to control parallelism of sub-topologies, he
> >>>>> or
> >>>>> she should always use `KStream#repartition` operation before groupBy.
> >>>>> Full
> >>>>> comment can be seen here:
> >>>>> https://github.com/apache/kafka/pull/7170#issuecomment-519303125 <
> >>>>> https://github.com/apache/kafka/pull/7170#issuecomment-519303125>
> >>>>>
> >>>>> On the same topic, John pointed out that, from API design perspective,
> >>>>> we
> >>>>> shouldn’t intertwine configuration classes of different operators
> >>>>> between
> >>>>> one another. So instead of introducing new
> >>>>> `KStream#groupBy(Repartitioned)`
> >>>>> for specifying number of partitions for internal topic, we should update
> >>>>> existing `Grouped` class with `numberOfPartitions` field.
> >>>>>
> >>>>> Personally, I think Matthias’s concern is valid, but on the other hand
> >>>>> Kafka Streams has already optimizer in place which alters topology
> >>>>> independently from user. So maybe it makes sense if Kafka Streams,
> >>>>> internally would optimize topology in the best way possible, even if in
> >>>>> some cases this means ignoring some operator configurations passed by
> >>>>> the
> >>>>> user. Also, I agree with John about API design semantics. If we go
> >>>>> through
> >>>>> with the changes for `KStream#groupBy` operation, it makes more sense to
> >>>>> add `numberOfPartitions` field to `Grouped` class instead of introducing
> >>>>> new `KStream#groupBy(Repartitioned)` method overload.
> >>>>>
> >>>>> I would really appreciate communities feedback on this.
> >>>>>
> >>>>> Kind regards,
> >>>>> Levani
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Oct 17, 2019, at 12:57 AM, Sophie Blee-Goldman <sop...@confluent.io>
> >>>>> wrote:
> >>>>>>
> >>>>>> Hey Levani,
> >>>>>>
> >>>>>> I think people are busy with the upcoming 2.4 release, and don't have
> >>>>> much
> >>>>>> spare time at the
> >>>>>> moment. It's kind of a difficult time to get attention on things, but
> >>>>> feel
> >>>>>> free to pick up something else
> >>>>>> to work on in the meantime until things have calmed down a bit!
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Sophie
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Oct 16, 2019 at 11:26 AM Levani Kokhreidze <
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hello all,
> >>>>>>>
> >>>>>>> Sorry for bringing this thread again, but I would like to get some
> >>>>>>> attention on this PR: https://github.com/apache/kafka/pull/7170 <
> >>>>> https://github.com/apache/kafka/pull/7170> <
> >>>>>>> https://github.com/apache/kafka/pull/7170 <
> >>>>> https://github.com/apache/kafka/pull/7170>>
> >>>>>>> It's been a while now and I would love to move on to other KIPs as
> >>>>>>> well.
> >>>>>>> Please let me know if you have any concerns.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Levani
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Jul 26, 2019, at 11:25 AM, Levani Kokhreidze <
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Here’s voting thread for this KIP:
> >>>>>>> https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html <
> >>>>> https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html> <
> >>>>>>> https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html <
> >>>>> https://www.mail-archive.com/dev@kafka.apache.org/msg99680.html>>
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Levani
> >>>>>>>>
> >>>>>>>>> On Jul 24, 2019, at 11:15 PM, Levani Kokhreidze <
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi Matthias,
> >>>>>>>>>
> >>>>>>>>> Thanks for the suggestion. I Don’t have strong opinion on that one.
> >>>>>>>>> Agree that avoiding unnecessary method overloads is a good idea.
> >>>>>>>>>
> >>>>>>>>> Updated KIP
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Levani
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Jul 24, 2019, at 8:50 PM, Matthias J. Sax <matth...@confluent.io
> >>>>> <mailto:matth...@confluent.io>
> >>>>>>> <mailto:matth...@confluent.io <mailto:matth...@confluent.io>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> One question:
> >>>>>>>>>>
> >>>>>>>>>> Why do we add
> >>>>>>>>>>
> >>>>>>>>>>> Repartitioned#with(final String name, final int
> >>>>>>>>>>> numberOfPartitions)
> >>>>>>>>>>
> >>>>>>>>>> It seems that `#with(String name)`, `#numberOfPartitions(int)` in
> >>>>>>>>>> combination with `withName()` and `withNumberOfPartitions()` should
> >>>>> be
> >>>>>>>>>> sufficient. Users can chain the method calls.
> >>>>>>>>>>
> >>>>>>>>>> (I think it's valuable to keep the number of overload small if
> >>>>>>> possible.)
> >>>>>>>>>>
> >>>>>>>>>> Otherwise LGTM.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> -Matthias
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 7/23/19 2:18 PM, Levani Kokhreidze wrote:
> >>>>>>>>>>> Hello,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks all for your feedback.
> >>>>>>>>>>> I started voting procedure for this KIP. If there’re any other
> >>>>>>> concerns about this KIP, please let me know.
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Levani
> >>>>>>>>>>>
> >>>>>>>>>>>> On Jul 20, 2019, at 8:39 PM, Levani Kokhreidze <
> >>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com> <mailto:
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Hi Matthias,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the suggestion, makes sense.
> >>>>>>>>>>>> I’ve updated KIP (
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>
> >>>>>>>>> ).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Regards,
> >>>>>>>>>>>> Levani
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On Jul 20, 2019, at 3:53 AM, Matthias J. Sax <
> >>>>> matth...@confluent.io <mailto:matth...@confluent.io>
> >>>>>>> <mailto:matth...@confluent.io <mailto:matth...@confluent.io>> <mailto:
> >>>>> matth...@confluent.io <mailto:matth...@confluent.io> <mailto:
> >>>>>>> matth...@confluent.io <mailto:matth...@confluent.io>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for driving the KIP.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I agree that users need to be able to specify a partitioning
> >>>>>>> strategy.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Sophie raises a fair point about topic configs and producer
> >>>>>>> configs. My
> >>>>>>>>>>>>> take is, that consider `Repartitioned` as an "extension" to
> >>>>>>> `Produced`,
> >>>>>>>>>>>>> that adds topic configuration, is a good way to think about it
> >>>>>>>>>>>>> and
> >>>>>>> helps
> >>>>>>>>>>>>> to keep the API "clean".
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> With regard to method names. I would prefer to avoid
> >>>>> abbreviations.
> >>>>>>> Can
> >>>>>>>>>>>>> we rename:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> `withNumOfPartitions` -> `withNumberOfPartitions`
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Furthermore, it might be good to add some more `static` methods:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - Repartitioned.with(Serde<K>, Serde<V>)
> >>>>>>>>>>>>> - Repartitioned.withNumberOfPartitions(int)
> >>>>>>>>>>>>> - Repartitioned.streamPartitioner(StreamPartitioner)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 7/19/19 3:33 PM, Levani Kokhreidze wrote:
> >>>>>>>>>>>>>> Totally agree. I think in KStream interface it makes sense to
> >>>>> have
> >>>>>>> some duplicate configurations between operators in order to keep API
> >>>>> simple
> >>>>>>> and usable.
> >>>>>>>>>>>>>> Also, as more surface API has, harder it is to have proper
> >>>>>>> backward compatibility.
> >>>>>>>>>>>>>> While initial idea of keeping topic level configs separate was
> >>>>>>> exciting, having Repartitioned class encapsulate some producer level
> >>>>>>> configs makes API more readable.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Jul 20, 2019, at 1:15 AM, Sophie Blee-Goldman <
> >>>>>>> sop...@confluent.io <mailto:sop...@confluent.io> <mailto:
> >>>>> sop...@confluent.io <mailto:sop...@confluent.io>> <mailto:
> >>>>>>> sop...@confluent.io <mailto:sop...@confluent.io> <mailto:
> >>>>> sop...@confluent.io <mailto:sop...@confluent.io>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think that is a good point about trying to keep producer
> >>>>>>>>>>>>>>> level
> >>>>>>>>>>>>>>> configurations and (repartition) topic level considerations
> >>>>>>> separate.
> >>>>>>>>>>>>>>> Number of partitions is definitely purely a topic level
> >>>>>>> configuration. But
> >>>>>>>>>>>>>>> on some level, serdes and partitioners are just as much a
> >>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>> configuration as a producer one. You could have two producers
> >>>>>>> configured
> >>>>>>>>>>>>>>> with different serdes and/or partitioners, but if they are
> >>>>>>> writing to the
> >>>>>>>>>>>>>>> same topic the result would be very difficult to part. So in a
> >>>>>>> sense, these
> >>>>>>>>>>>>>>> are configurations of topics in Streams, not just producers.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Another way to think of it: while the Streams API is not
> >>>>>>>>>>>>>>> always
> >>>>>>> true to
> >>>>>>>>>>>>>>> this, ideally all the relevant configs for an operator are
> >>>>>>> wrapped into a
> >>>>>>>>>>>>>>> single object (in this case, Repartitioned). We could instead
> >>>>>>> split out the
> >>>>>>>>>>>>>>> fields in common with Produced into a separate parameter to
> >>>>>>>>>>>>>>> keep
> >>>>>>> topic and
> >>>>>>>>>>>>>>> producer level configurations separate, but this increases the
> >>>>>>> API surface
> >>>>>>>>>>>>>>> area by a lot. It's much more straightforward to just say
> >>>>>>>>>>>>>>> "this
> >>>>> is
> >>>>>>>>>>>>>>> everything that this particular operator needs" without
> >>>>>>>>>>>>>>> worrying
> >>>>>>> about what
> >>>>>>>>>>>>>>> exactly you're specifying.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I suppose you could alternatively make Produced a field of
> >>>>>>> Repartitioned,
> >>>>>>>>>>>>>>> but I don't think we do this kind of composition elsewhere in
> >>>>>>> Streams at
> >>>>>>>>>>>>>>> the moment
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Fri, Jul 19, 2019 at 1:45 PM Levani Kokhreidze <
> >>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com> <mailto:
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>> <mailto:
> >>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com> <mailto:
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>>>
> >>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Hi Bill,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks a lot for the feedback.
> >>>>>>>>>>>>>>>> Yes, that makes sense. I’ve updated KIP with
> >>>>>>> `Repartitioned#partitioner`
> >>>>>>>>>>>>>>>> configuration.
> >>>>>>>>>>>>>>>> In the beginning, I wanted to introduce a class for topic
> >>>>>>>>>>>>>>>> level
> >>>>>>>>>>>>>>>> configuration and keep topic level and producer level
> >>>>>>> configurations (such
> >>>>>>>>>>>>>>>> as Produced) separately (see my second email in this thread).
> >>>>>>>>>>>>>>>> But while looking at the semantics of KStream interface, I
> >>>>>>> couldn’t really
> >>>>>>>>>>>>>>>> figure out good operation name for Topic level configuration
> >>>>>>> class and just
> >>>>>>>>>>>>>>>> introducing `Topic` config class was kinda breaking the
> >>>>>>> semantics.
> >>>>>>>>>>>>>>>> So I think having Repartitioned class which encapsulates
> >>>>>>>>>>>>>>>> topic
> >>>>>>> and
> >>>>>>>>>>>>>>>> producer level configurations for internal topics is viable
> >>>>>>> thing to do.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Jul 19, 2019, at 7:47 PM, Bill Bejeck <bbej...@gmail.com
> >>>>> <mailto:bbej...@gmail.com>
> >>>>>>> <mailto:bbej...@gmail.com <mailto:bbej...@gmail.com>> <mailto:
> >>>>> bbej...@gmail.com <mailto:bbej...@gmail.com> <mailto:
> >>>>>>> bbej...@gmail.com <mailto:bbej...@gmail.com>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Lavani,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks for resurrecting this KIP.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'm also a +1 for adding a partition option. In addition to
> >>>>>>> the reason
> >>>>>>>>>>>>>>>>> provided by John, my reasoning is:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> 1. Users may want to use something other than hash-based
> >>>>>>> partitioning
> >>>>>>>>>>>>>>>>> 2. Users may wish to partition on something different than
> >>>>>>>>>>>>>>>>> the
> >>>>>>> key
> >>>>>>>>>>>>>>>>> without having to change the key. For example:
> >>>>>>>>>>>>>>>>> 1. A combination of fields in the value in conjunction with
> >>>>>>> the key
> >>>>>>>>>>>>>>>>> 2. Something other than the key
> >>>>>>>>>>>>>>>>> 3. We allow users to specify a partitioner on Produced hence
> >>>>> in
> >>>>>>>>>>>>>>>>> KStream.to and KStream.through, so it makes sense for API
> >>>>>>> consistency.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Just my 2 cents.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>> Bill
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Fri, Jul 19, 2019 at 5:46 AM Levani Kokhreidze <
> >>>>>>>>>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>> <mailto:
> >>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com> <mailto:
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>>>
> >>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi John,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> In my mind it makes sense.
> >>>>>>>>>>>>>>>>>> If we add partitioner configuration to Repartitioned class,
> >>>>>>> with the
> >>>>>>>>>>>>>>>>>> combination of specifying number of partitions for internal
> >>>>>>> topics, user
> >>>>>>>>>>>>>>>>>> will have opportunity to ensure co-partitioning before join
> >>>>>>> operation.
> >>>>>>>>>>>>>>>>>> I think this can be quite powerful feature.
> >>>>>>>>>>>>>>>>>> Wondering what others think about this?
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Jul 18, 2019, at 1:20 AM, John Roesler <
> >>>>> j...@confluent.io <mailto:j...@confluent.io>
> >>>>>>> <mailto:j...@confluent.io <mailto:j...@confluent.io>> <mailto:
> >>>>> j...@confluent.io <mailto:j...@confluent.io> <mailto:
> >>>>>>> j...@confluent.io <mailto:j...@confluent.io>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Yes, I believe that's what I had in mind. Again, not
> >>>>>>>>>>>>>>>>>>> totally
> >>>>>>> sure it
> >>>>>>>>>>>>>>>>>>> makes sense, but I believe something similar is the
> >>>>> rationale
> >>>>>>> for
> >>>>>>>>>>>>>>>>>>> having the partitioner option in Produced.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 3:20 PM Levani Kokhreidze
> >>>>>>>>>>>>>>>>>>> <levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>>
> >>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>>>> <mailto:
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Hey John,
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Oh that’s interesting use-case.
> >>>>>>>>>>>>>>>>>>>> Do I understand this correctly, in your example I would
> >>>>>>> first issue
> >>>>>>>>>>>>>>>>>> repartition(Repartitioned) with proper partitioner that
> >>>>>>> essentially
> >>>>>>>>>>>>>>>> would
> >>>>>>>>>>>>>>>>>> be the same as the topic I want to join with and then do
> >>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>> KStream#join
> >>>>>>>>>>>>>>>>>> with DSL?
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Jul 17, 2019, at 11:11 PM, John Roesler <
> >>>>>>> j...@confluent.io <mailto:j...@confluent.io> <mailto:j...@confluent.io
> >>>>> <mailto:j...@confluent.io>> <mailto:j...@confluent.io <mailto:
> >>>>> j...@confluent.io>
> >>>>>>> <mailto:j...@confluent.io <mailto:j...@confluent.io>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Hey, all, just to chime in,
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> I think it might be useful to have an option to specify
> >>>>> the
> >>>>>>>>>>>>>>>>>>>>> partitioner. The case I have in mind is that some data
> >>>>>>>>>>>>>>>>>>>>> may
> >>>>>>> get
> >>>>>>>>>>>>>>>>>>>>> repartitioned and then joined with an input topic. If
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>> right-side
> >>>>>>>>>>>>>>>>>>>>> input topic uses a custom partitioning strategy, then
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>> repartitioned stream also needs to be partitioned with
> >>>>>>>>>>>>>>>>>>>>> the
> >>>>>>> same
> >>>>>>>>>>>>>>>>>>>>> strategy.
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Does that make sense, or did I maybe miss something
> >>>>>>> important?
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>>>>>> -John
> >>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 2:48 PM Levani Kokhreidze
> >>>>>>>>>>>>>>>>>>>>> <levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>>
> >>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>>>> <mailto:
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Yes, I was thinking about it as well. To be honest I’m
> >>>>> not
> >>>>>>> sure
> >>>>>>>>>>>>>>>> about
> >>>>>>>>>>>>>>>>>> it yet.
> >>>>>>>>>>>>>>>>>>>>>> As Kafka Streams DSL user, I don’t really think I would
> >>>>>>> need control
> >>>>>>>>>>>>>>>>>> over partitioner for internal topics.
> >>>>>>>>>>>>>>>>>>>>>> As a user, I would assume that Kafka Streams knows best
> >>>>>>> how to
> >>>>>>>>>>>>>>>>>> partition data for internal topics.
> >>>>>>>>>>>>>>>>>>>>>> In this KIP I wrote that Produced should be used only
> >>>>>>>>>>>>>>>>>>>>>> for
> >>>>>>> topics
> >>>>>>>>>>>>>>>> that
> >>>>>>>>>>>>>>>>>> are created by user In advance.
> >>>>>>>>>>>>>>>>>>>>>> In those cases maybe it make sense to have possibility
> >>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>> specify
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>> partitioner.
> >>>>>>>>>>>>>>>>>>>>>> I don’t have clear answer on that yet, but I guess
> >>>>>>> specifying the
> >>>>>>>>>>>>>>>>>> partitioner can be added as well if there’s agreement on
> >>>>> this.
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Jul 17, 2019, at 10:42 PM, Sophie Blee-Goldman <
> >>>>>>>>>>>>>>>>>> sop...@confluent.io <mailto:sop...@confluent.io> <mailto:
> >>>>> sop...@confluent.io <mailto:sop...@confluent.io>> <mailto:
> >>>>>>> sop...@confluent.io <mailto:sop...@confluent.io>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> Thanks for clearing that up. I agree that
> >>>>>>>>>>>>>>>>>>>>>>> Repartitioned
> >>>>>>> would be a
> >>>>>>>>>>>>>>>>>> useful
> >>>>>>>>>>>>>>>>>>>>>>> addition. I'm wondering if it might also need to have
> >>>>>>>>>>>>>>>>>>>>>>> a withStreamPartitioner method/field, similar to
> >>>>>>> Produced? I'm not
> >>>>>>>>>>>>>>>>>> sure how
> >>>>>>>>>>>>>>>>>>>>>>> widely this feature is really used, but seems it
> >>>>>>>>>>>>>>>>>>>>>>> should
> >>>>> be
> >>>>>>>>>>>>>>>> available
> >>>>>>>>>>>>>>>>>> for
> >>>>>>>>>>>>>>>>>>>>>>> repartition topics.
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 11:26 AM Levani Kokhreidze <
> >>>>>>>>>>>>>>>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>>>> <mailto:
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>>>
> >>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hey Sophie,
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> In both cases KStream#repartition and
> >>>>>>>>>>>>>>>>>> KStream#repartition(Repartitioned)
> >>>>>>>>>>>>>>>>>>>>>>>> topic will be created and managed by Kafka Streams.
> >>>>>>>>>>>>>>>>>>>>>>>> Idea of Repartitioned is to give user more control
> >>>>>>>>>>>>>>>>>>>>>>>> over
> >>>>>>> the topic
> >>>>>>>>>>>>>>>>>> such as
> >>>>>>>>>>>>>>>>>>>>>>>> num of partitions.
> >>>>>>>>>>>>>>>>>>>>>>>> I feel like Repartitioned parameter is something that
> >>>>> is
> >>>>>>> missing
> >>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>> current DSL design.
> >>>>>>>>>>>>>>>>>>>>>>>> Essentially giving user control over parallelism by
> >>>>>>> configuring
> >>>>>>>>>>>>>>>> num
> >>>>>>>>>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>>>>>>>> partitions for internal topics.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Hope this answers your question.
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Jul 17, 2019, at 9:02 PM, Sophie Blee-Goldman <
> >>>>>>>>>>>>>>>>>> sop...@confluent.io <mailto:sop...@confluent.io> <mailto:
> >>>>> sop...@confluent.io <mailto:sop...@confluent.io>> <mailto:
> >>>>>>> sop...@confluent.io <mailto:sop...@confluent.io>>>
> >>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Hey Levani,
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP! Can you clarify one thing for me
> >>>>> --
> >>>>>>> for the
> >>>>>>>>>>>>>>>>>>>>>>>>> KStream#repartition signature taking a
> >>>>>>>>>>>>>>>>>>>>>>>>> Repartitioned,
> >>>>>>> will the
> >>>>>>>>>>>>>>>>>> topic be
> >>>>>>>>>>>>>>>>>>>>>>>>> auto-created by Streams (which seems to be the case
> >>>>> for
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> signature
> >>>>>>>>>>>>>>>>>>>>>>>>> without a Repartitioned) or does it have to be
> >>>>>>> pre-created? The
> >>>>>>>>>>>>>>>>>> wording
> >>>>>>>>>>>>>>>>>>>>>>>> in
> >>>>>>>>>>>>>>>>>>>>>>>>> the KIP makes it seem like one version of the method
> >>>>>>> will
> >>>>>>>>>>>>>>>>>> auto-create
> >>>>>>>>>>>>>>>>>>>>>>>>> topics while the other will not.
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> Cheers,
> >>>>>>>>>>>>>>>>>>>>>>>>> Sophie
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Jul 17, 2019 at 10:15 AM Levani Kokhreidze <
> >>>>>>>>>>>>>>>>>>>>>>>> levani.co...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com>
> >>>>>>> <mailto:levani.co...@gmail.com <mailto:levani.co...@gmail.com>
> >>>>>>> <mailto:
> >>>>> levani.co...@gmail.com <mailto:levani.co...@gmail.com>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> One more bump about KIP-221 (
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>> <
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>
> >>>>>>> <
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>> )
> >>>>>>>>>>>>>>>>>>>>>>>>>> so it doesn’t get lost in mailing list :)
> >>>>>>>>>>>>>>>>>>>>>>>>>> Would love to hear communities opinions/concerns
> >>>>> about
> >>>>>>> this KIP.
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> On Jul 12, 2019, at 5:27 PM, Levani Kokhreidze <
> >>>>>>>>>>>>>>>>>> levani.co...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Kind reminder about this KIP:
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jul 9, 2019, at 11:38 AM, Levani Kokhreidze <
> >>>>>>>>>>>>>>>>>>>>>>>> levani.co...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> In order to move this KIP forward, I’ve updated
> >>>>>>> confluence
> >>>>>>>>>>>>>>>> page
> >>>>>>>>>>>>>>>>>> with
> >>>>>>>>>>>>>>>>>>>>>>>>>> the new proposal
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221%3A+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>>>>>>>>>>>>>>>>>>>>> <
> >>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-221:+Enhance+KStream+with+Connecting+Topic+Creation+and+Repartition+Hint
> >>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I’ve also filled “Rejected Alternatives” section.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Looking forward to discuss this KIP :)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> King regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jul 3, 2019, at 1:08 PM, Levani Kokhreidze <
> >>>>>>>>>>>>>>>>>>>>>>>> levani.co...@gmail.com
> >>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:levani.co...@gmail.com>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Matthias,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the feedback and ideas.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I like the idea of introducing dedicated `Topic`
> >>>>>>> class for
> >>>>>>>>>>>>>>>>>> topic
> >>>>>>>>>>>>>>>>>>>>>>>>>> configuration for internal operators like
> >>>>> `groupedBy`.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Would be great to hear others opinion about this
> >>>>> as
> >>>>>>> well.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jul 3, 2019, at 7:00 AM, Matthias J. Sax <
> >>>>>>>>>>>>>>>>>> matth...@confluent.io
> >>>>>>>>>>>>>>>>>>>>>>>>>> <mailto:matth...@confluent.io>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Levani,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for picking up this KIP! And thanks for
> >>>>>>> summarizing
> >>>>>>>>>>>>>>>>>>>>>>>> everything.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Even if some points may have been discussed
> >>>>>>> already (can't
> >>>>>>>>>>>>>>>>>> really
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> remember), it's helpful to get a good summary
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>> refresh the
> >>>>>>>>>>>>>>>>>>>>>>>>>> discussion.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your reasoning makes sense. With regard
> >>>>> to
> >>>>>>> the
> >>>>>>>>>>>>>>>>>> distinction
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> between operators that manage topics and
> >>>>> operators
> >>>>>>> that use
> >>>>>>>>>>>>>>>>>>>>>>>>>> user-created
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topics: Following this argument, it might
> >>>>> indicate
> >>>>>>> that
> >>>>>>>>>>>>>>>>>> leaving
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `through()` as-is (as an operator that uses
> >>>>>>> use-defined
> >>>>>>>>>>>>>>>>>> topics) and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introducing a new `repartition()` operator (an
> >>>>>>> operator that
> >>>>>>>>>>>>>>>>>> manages
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> topics itself) might be good. Otherwise, there
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>> one
> >>>>>>>>>>>>>>>> operator
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `through()` that sometimes manages topics but
> >>>>>>> sometimes
> >>>>>>>>>>>>>>>> not; a
> >>>>>>>>>>>>>>>>>>>>>>>>>> different
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name, ie, new operator would make the
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> distinction
> >>>>>>> clearer.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> About adding `numOfPartitions` to `Grouped`. I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> am
> >>>>>>> wondering
> >>>>>>>>>>>>>>>>>> if the
> >>>>>>>>>>>>>>>>>>>>>>>>>> same
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> argument as for `Produced` does apply and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> adding
> >>>>>>> it is
> >>>>>>>>>>>>>>>>>> semantically
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questionable? Might be good to get opinions of
> >>>>>>> others on
> >>>>>>>>>>>>>>>>>> this, too.
> >>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>> am
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not sure myself what solution I prefer atm.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So far, KS uses configuration objects that
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> allow
> >>>>> to
> >>>>>>>>>>>>>>>> configure
> >>>>>>>>>>>>>>>>>> a
> >>>>>>>>>>>>>>>>>>>>>>>>>> certain
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "entity" like a consumer, producer, store. If
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>> assume that
> >>>>>>>>>>>>>>>>>> a topic
> >>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a similar entity, I am wonder if we should
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> have a
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `Topic#withNumberOfPartitions()` class and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method
> >>>>>>> instead of
> >>>>>>>>>>>>>>>>>> a plain
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> integer? This would allow us to add other
> >>>>> configs,
> >>>>>>> like
> >>>>>>>>>>>>>>>>>> replication
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> factor, retention-time etc, easily, without the
> >>>>>>> need to
> >>>>>>>>>>>>>>>>>> change the
> >>>>>>>>>>>>>>>>>>>>>>>>>> "main
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> API".
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Just want to give some ideas. Not sure if I
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like
> >>>>>>> them
> >>>>>>>>>>>>>>>> myself.
> >>>>>>>>>>>>>>>>>> :)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 7/1/19 1:04 AM, Levani Kokhreidze wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Actually, giving it more though - maybe
> >>>>> enhancing
> >>>>>>> Produced
> >>>>>>>>>>>>>>>>>> with num
> >>>>>>>>>>>>>>>>>>>>>>>>>> of partitions configuration is not the best
> >>>>>>>>>>>>>>>>>>>>>>>>>> approach.
> >>>>>>> Let me
> >>>>>>>>>>>>>>>>>> explain
> >>>>>>>>>>>>>>>>>>>>>>>> why:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) If we enhance Produced class with this
> >>>>>>> configuration,
> >>>>>>>>>>>>>>>>>> this will
> >>>>>>>>>>>>>>>>>>>>>>>>>> also affect KStream#to operation. Since KStream#to
> >>>>>>>>>>>>>>>>>>>>>>>>>> is
> >>>>>>> the final
> >>>>>>>>>>>>>>>>>> sink of
> >>>>>>>>>>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>>>> topology, for me, it seems to be reasonable
> >>>>> assumption
> >>>>>>> that user
> >>>>>>>>>>>>>>>>>> needs
> >>>>>>>>>>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>>>>>>>>>> manually create sink topic in advance. And in that
> >>>>>>> case, having
> >>>>>>>>>>>>>>>>>> num of
> >>>>>>>>>>>>>>>>>>>>>>>>>> partitions configuration doesn’t make much sense.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Looking at Produced class, based on API
> >>>>>>> contract, seems
> >>>>>>>>>>>>>>>>>> like
> >>>>>>>>>>>>>>>>>>>>>>>>>> Produced is designed to be something that is
> >>>>>>> explicitly for
> >>>>>>>>>>>>>>>>>> producer
> >>>>>>>>>>>>>>>>>>>>>>>> (key
> >>>>>>>>>>>>>>>>>>>>>>>>>> serializer, value serializer, partitioner those all
> >>>>>>> are producer
> >>>>>>>>>>>>>>>>>>>>>>>> specific
> >>>>>>>>>>>>>>>>>>>>>>>>>> configurations) and num of partitions is topic
> >>>>>>>>>>>>>>>>>>>>>>>>>> level
> >>>>>>>>>>>>>>>>>> configuration. And
> >>>>>>>>>>>>>>>>>>>>>>>> I
> >>>>>>>>>>>>>>>>>>>>>>>>>> don’t think mixing topic and producer level
> >>>>>>> configurations
> >>>>>>>>>>>>>>>>>> together in
> >>>>>>>>>>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>>>>>>>>>>> class is the good approach.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Looking at KStream interface, seems like
> >>>>>>> Produced
> >>>>>>>>>>>>>>>>>> parameter is
> >>>>>>>>>>>>>>>>>>>>>>>>>> for operations that work with non-internal (e.g
> >>>>> topics
> >>>>>>> created
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>>>>>>>> managed
> >>>>>>>>>>>>>>>>>>>>>>>>>> internally by Kafka Streams) topics and I think we
> >>>>>>> should leave
> >>>>>>>>>>>>>>>>>> it as
> >>>>>>>>>>>>>>>>>>>>>>>> it is
> >>>>>>>>>>>>>>>>>>>>>>>>>> in that case.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Taking all this things into account, I think
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we
> >>>>>>> should
> >>>>>>>>>>>>>>>>>> distinguish
> >>>>>>>>>>>>>>>>>>>>>>>>>> between DSL operations, where Kafka Streams should
> >>>>>>> create and
> >>>>>>>>>>>>>>>>>> manage
> >>>>>>>>>>>>>>>>>>>>>>>>>> internal topics (KStream#groupBy) vs topics that
> >>>>>>> should be
> >>>>>>>>>>>>>>>>>> created in
> >>>>>>>>>>>>>>>>>>>>>>>>>> advance (e.g KStream#to).
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To sum it up, I think adding numPartitions
> >>>>>>> configuration in
> >>>>>>>>>>>>>>>>>>>>>>>> Produced
> >>>>>>>>>>>>>>>>>>>>>>>>>> will result in mixing topic and producer level
> >>>>>>> configuration in
> >>>>>>>>>>>>>>>>>> one
> >>>>>>>>>>>>>>>>>>>>>>>> class
> >>>>>>>>>>>>>>>>>>>>>>>>>> and it’s gonna break existing API semantics.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding making topic name optional in
> >>>>>>> KStream#through - I
> >>>>>>>>>>>>>>>>>> think
> >>>>>>>>>>>>>>>>>>>>>>>>>> underline idea is very useful and giving users
> >>>>>>> possibility to
> >>>>>>>>>>>>>>>>>> specify
> >>>>>>>>>>>>>>>>>>>>>>>> num
> >>>>>>>>>>>>>>>>>>>>>>>>>> of partitions there is even more useful :)
> >>>>> Considering
> >>>>>>> arguments
> >>>>>>>>>>>>>>>>>> against
> >>>>>>>>>>>>>>>>>>>>>>>>>> adding num of partitions in Produced class, I see
> >>>>>>>>>>>>>>>>>>>>>>>>>> two
> >>>>>>> options
> >>>>>>>>>>>>>>>>>> here:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add following method overloads
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * through() - topic will be auto-generated and
> >>>>>>> num of
> >>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>> will be taken from source topic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions) - topic
> >>>>> will
> >>>>>>> be auto
> >>>>>>>>>>>>>>>>>>>>>>>>>> generated with specified num of partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions, final
> >>>>>>> Produced<K, V>
> >>>>>>>>>>>>>>>>>>>>>>>>>> produced) - topic will be with generated with
> >>>>>>> specified num of
> >>>>>>>>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>> and configuration taken from produced parameter.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Leave KStream#through as it is and
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> introduce
> >>>>>>> new method
> >>>>>>>>>>>>>>>> -
> >>>>>>>>>>>>>>>>>>>>>>>>>> KStream#repartition (I think Matthias suggested
> >>>>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>> in one of
> >>>>>>>>>>>>>>>> the
> >>>>>>>>>>>>>>>>>>>>>>>> threads)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Considering all mentioned above I propose the
> >>>>>>> following
> >>>>>>>>>>>>>>>> plan:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Option A:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Leave Produced as it is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Add num of partitions configuration to
> >>>>> Grouped
> >>>>>>> class (as
> >>>>>>>>>>>>>>>>>>>>>>>>>> mentioned in the KIP)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Add following method overloads to
> >>>>>>> KStream#through
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * through() - topic will be auto-generated and
> >>>>>>> num of
> >>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>> will be taken from source topic
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions) - topic
> >>>>> will
> >>>>>>> be auto
> >>>>>>>>>>>>>>>>>>>>>>>>>> generated with specified num of partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> * through(final int numOfPartitions, final
> >>>>>>> Produced<K, V>
> >>>>>>>>>>>>>>>>>>>>>>>>>> produced) - topic will be with generated with
> >>>>>>> specified num of
> >>>>>>>>>>>>>>>>>>>>>>>> partitions
> >>>>>>>>>>>>>>>>>>>>>>>>>> and configuration taken from produced parameter.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Option B:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Leave Produced as it is
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Add num of partitions configuration to
> >>>>> Grouped
> >>>>>>> class (as
> >>>>>>>>>>>>>>>>>>>>>>>>>> mentioned in the KIP)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) Add new operator KStream#repartition for
> >>>>>>> creating and
> >>>>>>>>>>>>>>>>>> managing
> >>>>>>>>>>>>>>>>>>>>>>>>>> internal repartition topics
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> P.S. I’m sorry if all of this was already
> >>>>>>> discussed in the
> >>>>>>>>>>>>>>>>>> mailing
> >>>>>>>>>>>>>>>>>>>>>>>>>> list, but I kinda got with all the threads that
> >>>>>>>>>>>>>>>>>>>>>>>>>> were
> >>>>>>> about this
> >>>>>>>>>>>>>>>>>> KIP :(
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Levani
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Jul 1, 2019, at 9:56 AM, Levani
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kokhreidze <
> >>>>>>>>>>>>>>>>>>>>>>>>>> levani.co...@gmail.com <mailto:
> >>>>> levani.co...@gmail.com>>
> >>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would like to resurrect discussion around
> >>>>>>> KIP-221. Going
> >>>>>>>>>>>>>>>>>> through
> >>>>>>>>>>>>>>>>>>>>>>>>>> the discussion thread, there’s seems to agreement
> >>>>>>> around
> >>>>>>>>>>>>>>>>>> usefulness of
> >>>>>>>>>>>>>>>>>>>>>>>> this
> >>>>>>>>>>>>>>>>>>>>>>>>>> feature.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regarding the implementation, as far as I
> >>>>>>> understood, the
> >>>>>>>>>>>>>>>>>> most
> >>>>>>>>>>>>>>>>>>>>>>>>>> optimal solution for me seems the following:
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) Add two method overloads to
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#through
> >>>>>>> method
> >>>>>>>>>>>>>>>>>> (essentially
> >>>>>>>>>>>>>>>>>>>>>>>>>> making topic name optional)
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) Enhance Produced class with
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> numOfPartitions
> >>>>>>>>>>>>>>>> configuration
> >>>>>>>>>>>>>>>>>>>>>>>> field.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Those two changes will allow DSL users to
> >>>>> control
> >>>>>>>>>>>>>>>>>> parallelism and
> >>>>>>>>>>>>>>>>>>>>>>>>>> trigger re-partition without doing stateful
> >>>>> operations.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I will update KIP with interface changes
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> around
> >>>>>>>>>>>>>>>>>> KStream#through if
> >>>>>>>>>>>>>>>>>>>>>>>>>> this changes sound sensible.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Kind regards,
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Levani
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
