Regarding #6 above, I'm still not clear why we would need `commit()` in
both ProcessorContext and RecordContext, could you elaborate a bit more?
To me `commit()` is really a processor context not a record context
logically: when you call that function, it means we would commit the state
of the whole task up to this processed record, not only that single record
itself.
Guozhang
On Mon, Oct 16, 2017 at 9:19 AM, Jeyhun Karimov <je.kari...@gmail.com>
wrote:
> Hi,
>
> Thanks for the feedback.
>
>
> 0. RichInitializer definition seems missing.
>
>
>
> - Fixed.
>
>
> I'd suggest moving the key parameter in the RichValueXX and RichReducer
> > after the value parameters, as well as in the templates; e.g.
> > public interface RichValueJoiner<V1, V2, VR, K> {
> > VR apply(final V1 value1, final V2 value2, final K key, final
> > RecordContext
> > recordContext);
> > }
>
>
>
> - Fixed.
>
>
> 2. Some of the listed functions are not necessary since their pairing APIs
> > are being deprecated in 1.0 already:
> > <KR> KGroupedStream<KR, V> groupBy(final RichKeyValueMapper<? super K, ?
> > super V, KR> selector,
> > final Serde<KR> keySerde,
> > final Serde<V> valSerde);
> > <VT, VR> KStream<K, VR> leftJoin(final KTable<K, VT> table,
> > final RichValueJoiner<? super K, ? super
> > V,
> > ? super VT, ? extends VR> joiner,
> > final Serde<K> keySerde,
> > final Serde<V> valSerde);
>
>
> -Fixed
>
> 3. For a few functions where we are adding three APIs for a combo of both
> > mapper / joiner, or both initializer / aggregator, or adder / subtractor,
> > I'm wondering if we can just keep one that use "rich" functions for both;
> > so that we can have less overloads and let users who only want to access
> > one of them to just use dummy parameter declarations. For example:
> >
> > <GK, GV, RV> KStream<K, RV> join(final GlobalKTable<GK, GV> globalKTable,
> > final RichKeyValueMapper<? super K, ?
> > super
> > V, ? extends GK> keyValueMapper,
> > final RichValueJoiner<? super K, ? super
> > V,
> > ? super GV, ? extends RV> joiner);
>
>
>
> -Agreed. Fixed.
>
>
> 4. For TimeWindowedKStream, I'm wondering why we do not make its
> > Initializer also "rich" functions? I.e.
>
>
> - It was a typo. Fixed.
>
>
> 5. We need to move "RecordContext" from o.a.k.processor.internals to
> > o.a.k.processor.
> >
> > 6. I'm not clear why we want to move `commit()` from ProcessorContext to
> > RecordContext?
> >
>
> -
> Because it makes sense logically and to reduce code maintenance (both
> interfaces have offset() timestamp() topic() partition() methods), I
> inherit ProcessorContext from RecordContext.
> Since we need commit() method both in ProcessorContext and in RecordContext
> I move commit() method to parent class (RecordContext).
>
>
> Cheers,
> Jeyhun
>
>
>
> On Wed, Oct 11, 2017 at 12:59 AM, Guozhang Wang <wangg...@gmail.com>
> wrote:
>
> > Jeyhun,
> >
> > Thanks for the updated KIP, here are my comments.
> >
> > 0. RichInitializer definition seems missing.
> >
> > 1. I'd suggest moving the key parameter in the RichValueXX and
> RichReducer
> > after the value parameters, as well as in the templates; e.g.
> >
> > public interface RichValueJoiner<V1, V2, VR, K> {
> > VR apply(final V1 value1, final V2 value2, final K key, final
> > RecordContext
> > recordContext);
> > }
> >
> > My motivation is that for lambda expression in J8, users that would not
> > care about the key but only the context, or vice versa, is likely to
> write
> > it as (value1, value2, dummy, context) -> ... than putting the dummy at
> the
> > beginning of the parameter list. Generally speaking we'd like to make all
> > the "necessary" parameters prior to optional ones.
> >
> >
> > 2. Some of the listed functions are not necessary since their pairing
> APIs
> > are being deprecated in 1.0 already:
> >
> > <KR> KGroupedStream<KR, V> groupBy(final RichKeyValueMapper<? super K, ?
> > super V, KR> selector,
> > final Serde<KR> keySerde,
> > final Serde<V> valSerde);
> >
> > <VT, VR> KStream<K, VR> leftJoin(final KTable<K, VT> table,
> > final RichValueJoiner<? super K, ? super
> > V,
> > ? super VT, ? extends VR> joiner,
> > final Serde<K> keySerde,
> > final Serde<V> valSerde);
> >
> >
> >
> > 3. For a few functions where we are adding three APIs for a combo of both
> > mapper / joiner, or both initializer / aggregator, or adder / subtractor,
> > I'm wondering if we can just keep one that use "rich" functions for both;
> > so that we can have less overloads and let users who only want to access
> > one of them to just use dummy parameter declarations. For example:
> >
> >
> > <GK, GV, RV> KStream<K, RV> join(final GlobalKTable<GK, GV> globalKTable,
> > final RichKeyValueMapper<? super K, ?
> > super
> > V, ? extends GK> keyValueMapper,
> > final RichValueJoiner<? super K, ? super
> > V,
> > ? super GV, ? extends RV> joiner);
> >
> > <VR> KTable<K, VR> aggregate(final RichInitializer<K, VR> initializer,
> > final RichAggregator<? super K, ? super V,
> VR>
> > aggregator,
> > final Materialized<K, VR,
> KeyValueStore<Bytes,
> > byte[]>> materialized);
> >
> > Similarly for KGroupedTable, a bunch of aggregate() are deprecated so we
> do
> > not need to add its rich functions any more.
> >
> >
> > 4. For TimeWindowedKStream, I'm wondering why we do not make its
> > Initializer also "rich" functions? I.e.
> >
> > <VR> KTable<Windowed<K>, VR> aggregate(final RichInitializer<VR, K>
> > initializer,
> > final RichAggregator<? super K, ?
> > super V, VR> aggregator);
> > <VR> KTable<Windowed<K>, VR> aggregate(final RichInitializer<VR, K>
> > initializer,
> > final RichAggregator<? super K, ?
> > super V, VR> aggregator,
> > final Materialized<K, VR,
> > WindowStore<Bytes, byte[]>> materialized);
> >
> >
> > 5. We need to move "RecordContext" from o.a.k.processor.internals to
> > o.a.k.processor.
> >
> > 6. I'm not clear why we want to move `commit()` from ProcessorContext to
> > RecordContext? Conceptually I think it would better staying in the
> > ProcessorContext. Do you find this not doable in the internal
> > implementations?
> >
> >
> > Guozhang
> >
> >
> >
> > On Fri, Sep 22, 2017 at 1:09 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > recordContext = new RecordContext() { // recordContext
> > > initialization is added in this KIP
> > >
> > > This code snippet seems to be standard - would it make sense to pull it
> > > into a (sample) RecordContext implementation ?
> > >
> > > Cheers
> > >
> > > On Fri, Sep 22, 2017 at 12:14 PM, Jeyhun Karimov <je.kari...@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Ted,
> > > >
> > > > Thanks for your comments. I added a couple of comments in KIP to
> > clarify
> > > > some points.
> > > >
> > > >
> > > > bq. provides a hybrd solution
> > > > > Typo in hybrid.
> > > >
> > > >
> > > > - My bad. Thanks for the correction.
> > > >
> > > > It would be nice if you can name some Value operator as examples.
> > > >
> > > >
> > > > >
> > > > - I added the corresponding interface names to KIP.
> > > >
> > > >
> > > > <VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
> > > > > final Aggregator<? super K, ? super V,
> > VR>
> > > > > adder,
> > > > > The adder doesn't need to be RichAggregator ?
> > > >
> > > >
> > > >
> > > > - Exactly. However, there are 2 Aggregator-type arguments in the
> > related
> > > > method. So, I had to overload all possible their Rich counterparts:
> > > >
> > > > // adder with non-rich, subtrctor is rich
> > > > <VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
> > > > final Aggregator<? super K, ? super V,
> VR>
> > > > adder,
> > > > final RichAggregator<? super K, ? super
> V,
> > > VR>
> > > > subtractor,
> > > > final Materialized<K, VR,
> > > KeyValueStore<Bytes,
> > > > byte[]>> materialized);
> > > >
> > > > // adder withrich, subtrctor is non-rich
> > > > <VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
> > > > final RichAggregator<? super K, ? super
> V,
> > > VR>
> > > > adder,
> > > > final Aggregator<? super K, ? super V,
> VR>
> > > > subtractor,
> > > > final Materialized<K, VR,
> > > KeyValueStore<Bytes,
> > > > byte[]>> materialized);
> > > >
> > > > // both adder and subtractor are rich
> > > > <VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
> > > > final RichAggregator<? super K, ? super
> V,
> > > VR>
> > > > adder,
> > > > final RichAggregator<? super K, ? super
> V,
> > > VR>
> > > > subtractor,
> > > > final Materialized<K, VR,
> > > KeyValueStore<Bytes,
> > > > byte[]>> materialized);
> > > >
> > > >
> > > > Can you explain a bit about the above implementation ?
> > > > > void commit () {
> > > > > throw new UnsupportedOperationException("commit() is not
> > > supported
> > > > in
> > > > > this context");
> > > > > Is the exception going to be replaced with real code in the PR ?
> > > >
> > > >
> > > >
> > > > - I added some comments both inside and outside the code snippets in
> > KIP.
> > > > Specifically, for the code snippet above, we add *commit()* method to
> > > > *RecordContext* interface.
> > > > However, we want *commit()* method to be used only for
> *RecordContext*
> > > > instances (at least for now), so we add UnsupportedOperationException
> > in
> > > > all classes/interfaces that extend/implement *RecordContext.*
> > > > In general, 1) we make RecordContext publicly available within
> > > > ProcessorContext, 2) initialize its instance within all required
> > > > Processors and 3) pass it as an argument to the related Rich
> interfaces
> > > > inside Processors.
> > > >
> > > >
> > > >
> > > >
> > > > Cheers,
> > > > Jeyhun
> > > >
> > > > On Fri, Sep 22, 2017 at 6:44 PM Ted Yu <yuzhih...@gmail.com> wrote:
> > > >
> > > > > bq. provides a hybrd solution
> > > > >
> > > > > Typo in hybrid.
> > > > >
> > > > > bq. accessing read-only keys within XXXValues operators
> > > > >
> > > > > It would be nice if you can name some Value operator as examples.
> > > > >
> > > > > <VR> KTable<K, VR> aggregate(final Initializer<VR> initializer,
> > > > > final Aggregator<? super K, ? super V,
> > VR>
> > > > > adder,
> > > > >
> > > > > The adder doesn't need to be RichAggregator ?
> > > > >
> > > > > public RecordContext recordContext() {
> > > > > return this.recordContext();
> > > > >
> > > > > Can you explain a bit about the above implementation ?
> > > > >
> > > > > void commit () {
> > > > > throw new UnsupportedOperationException("commit() is not
> > > supported
> > > > in
> > > > > this context");
> > > > >
> > > > > Is the exception going to be replaced with real code in the PR ?
> > > > >
> > > > > Cheers
> > > > >
> > > > >
> > > > > On Fri, Sep 22, 2017 at 9:28 AM, Jeyhun Karimov <
> > je.kari...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Dear community,
> > > > > >
> > > > > > I updated the related KIP [1]. Please feel free to comment.
> > > > > >
> > > > > > Cheers,
> > > > > > Jeyhun
> > > > > >
> > > > > > [1]
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > 159%3A+Introducing+Rich+functions+to+Streams
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Fri, Sep 22, 2017 at 12:20 AM Jeyhun Karimov <
> > > je.kari...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Damian,
> > > > > > >
> > > > > > > Thanks for the update. I working on it and will provide an
> update
> > > > soon.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jeyhun
> > > > > > >
> > > > > > > On Thu, Sep 21, 2017 at 4:50 PM Damian Guy <
> damian....@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > >> Hi Jeyhun,
> > > > > > >>
> > > > > > >> All KIP-182 API PRs have now been merged. So you can consider
> it
> > > as
> > > > > > >> stable.
> > > > > > >> Thanks,
> > > > > > >> Damian
> > > > > > >>
> > > > > > >> On Thu, 21 Sep 2017 at 15:23 Jeyhun Karimov <
> > je.kari...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> > Hi all,
> > > > > > >> >
> > > > > > >> > Thanks a lot for your comments. For the single interface
> > > (RichXXX
> > > > > and
> > > > > > >> > XXXWithKey) solution, I have already submitted a PR but
> > probably
> > > > it
> > > > > is
> > > > > > >> > outdated (when the KIP first proposed), I need to revisit
> that
> > > > one.
> > > > > > >> >
> > > > > > >> > @Guozhang, from our (offline) discussion, I understood that
> we
> > > may
> > > > > not
> > > > > > >> make
> > > > > > >> > it merge this KIP into the upcoming release, as KIP-159 is
> not
> > > > voted
> > > > > > yet
> > > > > > >> > (because we want both KIP-149 and KIP-159 to be as an
> "atomic"
> > > > > merge).
> > > > > > >> So
> > > > > > >> > I decided to wait until KIP-182 gets stable (there are some
> > > minor
> > > > > > >> updates
> > > > > > >> > AFAIK) and update the KIP accordingly. Please correct me if
> I
> > am
> > > > > wrong
> > > > > > >> or I
> > > > > > >> > misunderstood.
> > > > > > >> >
> > > > > > >> > Cheers,
> > > > > > >> > Jeyhun
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Thu, Sep 21, 2017 at 4:11 PM Damian Guy <
> > > damian....@gmail.com>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > +1
> > > > > > >> > >
> > > > > > >> > > On Thu, 21 Sep 2017 at 13:46 Guozhang Wang <
> > > wangg...@gmail.com>
> > > > > > >> wrote:
> > > > > > >> > >
> > > > > > >> > > > +1 for me as well for collapsing.
> > > > > > >> > > >
> > > > > > >> > > > Jeyhun, could you update the wiki accordingly to show
> > what's
> > > > the
> > > > > > >> final
> > > > > > >> > > > updates post KIP-182 that needs to be done in KIP-159
> > > > including
> > > > > > >> > KIP-149?
> > > > > > >> > > > The child page I made is just a suggestion, but you
> would
> > > > still
> > > > > > >> need to
> > > > > > >> > > > update your proposal for people to comment and vote on.
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > Guozhang
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Thu, Sep 14, 2017 at 10:37 PM, Ted Yu <
> > > yuzhih...@gmail.com
> > > > >
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > > > +1
> > > > > > >> > > > >
> > > > > > >> > > > > One interface is cleaner.
> > > > > > >> > > > >
> > > > > > >> > > > > On Thu, Sep 14, 2017 at 7:26 AM, Bill Bejeck <
> > > > > bbej...@gmail.com
> > > > > > >
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > +1 for me on collapsing the RichXXXX and
> > > ValueXXXXWithKey
> > > > > > >> > interfaces
> > > > > > >> > > > > into 1
> > > > > > >> > > > > > interface.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks,
> > > > > > >> > > > > > Bill
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Wed, Sep 13, 2017 at 11:31 AM, Jeyhun Karimov <
> > > > > > >> > > je.kari...@gmail.com
> > > > > > >> > > > >
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi Damian,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks for your feedback. Actually, this (what you
> > > > > propose)
> > > > > > >> was
> > > > > > >> > the
> > > > > > >> > > > > first
> > > > > > >> > > > > > > idea of KIP-149. Then we decided to divide it into
> > two
> > > > > > KIPs. I
> > > > > > >> > also
> > > > > > >> > > > > > > expressed my opinion that keeping the two
> interfaces
> > > > (Rich
> > > > > > and
> > > > > > >> > > > withKey)
> > > > > > >> > > > > > > separate would add more overloads. So, email
> > > discussion
> > > > > > >> resulted
> > > > > > >> > > that
> > > > > > >> > > > > > this
> > > > > > >> > > > > > > would not be a problem.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Our initial idea was similar to :
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > public abstract class RichValueMapper<K, V, VR>
> > > > > implements
> > > > > > >> > > > > > > ValueMapperWithKey<K, V, VR>, RichFunction {
> > > > > > >> > > > > > > ......
> > > > > > >> > > > > > > }
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > So, we check the type of object, whether it is
> > RichXXX
> > > > or
> > > > > > >> > > XXXWithKey
> > > > > > >> > > > > > inside
> > > > > > >> > > > > > > the called method and continue accordingly.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > If this is ok with the community, I would like to
> > > revert
> > > > > the
> > > > > > >> > > current
> > > > > > >> > > > > > design
> > > > > > >> > > > > > > to this again.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Cheers,
> > > > > > >> > > > > > > Jeyhun
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Wed, Sep 13, 2017 at 3:02 PM Damian Guy <
> > > > > > >> damian....@gmail.com
> > > > > > >> > >
> > > > > > >> > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi Jeyhun,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thanks for sending out the update. I guess i was
> > > > > thinking
> > > > > > >> more
> > > > > > >> > > > along
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > lines of option 2 where we collapse the RichXXXX
> > and
> > > > > > >> > > > ValueXXXXWithKey
> > > > > > >> > > > > > etc
> > > > > > >> > > > > > > > interfaces into 1 interface that has all of the
> > > > > > arguments. I
> > > > > > >> > > think
> > > > > > >> > > > we
> > > > > > >> > > > > > > then
> > > > > > >> > > > > > > > only need to add one additional overload for
> each
> > > > > > operator?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thanks,
> > > > > > >> > > > > > > > Damian
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Wed, 13 Sep 2017 at 10:59 Jeyhun Karimov <
> > > > > > >> > > je.kari...@gmail.com>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Dear all,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > I would like to resume the discussion on
> > KIP-159.
> > > I
> > > > > (and
> > > > > > >> > > > Guozhang)
> > > > > > >> > > > > > > think
> > > > > > >> > > > > > > > > that releasing KIP-149 and KIP-159 in the same
> > > > release
> > > > > > >> would
> > > > > > >> > > make
> > > > > > >> > > > > > sense
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > avoid a release with "partial" public APIs.
> > There
> > > > is a
> > > > > > KIP
> > > > > > >> > [1]
> > > > > > >> > > > > > proposed
> > > > > > >> > > > > > > > by
> > > > > > >> > > > > > > > > Guozhang (and approved by me) to unify both
> > KIPs.
> > > > > > >> > > > > > > > > Please feel free to comment on this.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > [1]
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > https://cwiki.apache.org/
> > confluence/pages/viewpage.
> > > > > > >> > > > > > > action?pageId=73637757
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > Jeyhun
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Fri, Jul 21, 2017 at 2:00 AM Jeyhun
> Karimov <
> > > > > > >> > > > > je.kari...@gmail.com
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Hi Matthias, Damian, all,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks for your comments and sorry for
> > > super-late
> > > > > > >> update.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Sure, the DSL refactoring is not blocking
> for
> > > this
> > > > > > KIP.
> > > > > > >> > > > > > > > > > I made some changes to KIP document based on
> > my
> > > > > > >> prototype.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Please feel free to comment.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Cheers,
> > > > > > >> > > > > > > > > > Jeyhun
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Fri, Jul 7, 2017 at 9:35 PM Matthias J.
> > Sax <
> > > > > > >> > > > > > > matth...@confluent.io>
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >> I would not block this KIP with regard to
> DSL
> > > > > > >> refactoring.
> > > > > > >> > > > IMHO,
> > > > > > >> > > > > > we
> > > > > > >> > > > > > > > can
> > > > > > >> > > > > > > > > >> just finish this one and the DSL
> refactoring
> > > will
> > > > > > help
> > > > > > >> > later
> > > > > > >> > > > on
> > > > > > >> > > > > to
> > > > > > >> > > > > > > > > >> reduce the number of overloads.
> > > > > > >> > > > > > > > > >>
> > > > > > >> > > > > > > > > >> -Matthias
> > > > > > >> > > > > > > > > >>
> > > > > > >> > > > > > > > > >> On 7/7/17 5:28 AM, Jeyhun Karimov wrote:
> > > > > > >> > > > > > > > > >> > I am following the related thread in the
> > > > mailing
> > > > > > list
> > > > > > >> > and
> > > > > > >> > > > > > looking
> > > > > > >> > > > > > > > > >> forward
> > > > > > >> > > > > > > > > >> > for one-shot solution for overloads
> issue.
> > > > > > >> > > > > > > > > >> >
> > > > > > >> > > > > > > > > >> > Cheers,
> > > > > > >> > > > > > > > > >> > Jeyhun
> > > > > > >> > > > > > > > > >> >
> > > > > > >> > > > > > > > > >> > On Fri, Jul 7, 2017 at 10:32 AM Damian
> Guy
> > <
> > > > > > >> > > > > > damian....@gmail.com>
> > > > > > >> > > > > > > > > >> wrote:
> > > > > > >> > > > > > > > > >> >
> > > > > > >> > > > > > > > > >> >> Hi Jeyhun,
> > > > > > >> > > > > > > > > >> >>
> > > > > > >> > > > > > > > > >> >> About overrides, what other alternatives
> > do
> > > we
> > > > > > have?
> > > > > > >> > For
> > > > > > >> > > > > > > > > >> >>> backwards-compatibility we have to add
> > > extra
> > > > > > >> methods
> > > > > > >> > to
> > > > > > >> > > > the
> > > > > > >> > > > > > > > existing
> > > > > > >> > > > > > > > > >> >> ones.
> > > > > > >> > > > > > > > > >> >>>
> > > > > > >> > > > > > > > > >> >>>
> > > > > > >> > > > > > > > > >> >> It wasn't clear to me in the KIP if
> these
> > > are
> > > > > new
> > > > > > >> > methods
> > > > > > >> > > > or
> > > > > > >> > > > > > > > > replacing
> > > > > > >> > > > > > > > > >> >> existing ones.
> > > > > > >> > > > > > > > > >> >> Also, we are currently discussing
> options
> > > for
> > > > > > >> replacing
> > > > > > >> > > the
> > > > > > >> > > > > > > > > overrides.
> > > > > > >> > > > > > > > > >> >>
> > > > > > >> > > > > > > > > >> >> Thanks,
> > > > > > >> > > > > > > > > >> >> Damian
> > > > > > >> > > > > > > > > >> >>
> > > > > > >> > > > > > > > > >> >>
> > > > > > >> > > > > > > > > >> >>> About ProcessorContext vs
> RecordContext,
> > > you
> > > > > are
> > > > > > >> > right.
> > > > > > >> > > I
> > > > > > >> > > > > > think
> > > > > > >> > > > > > > I
> > > > > > >> > > > > > > > > >> need to
> > > > > > >> > > > > > > > > >> >>> implement a prototype to understand the
> > > full
> > > > > > >> picture
> > > > > > >> > as
> > > > > > >> > > > some
> > > > > > >> > > > > > > parts
> > > > > > >> > > > > > > > > of
> > > > > > >> > > > > > > > > >> the
> > > > > > >> > > > > > > > > >> >>> KIP might not be as straightforward as
> I
> > > > > thought.
> > > > > > >> > > > > > > > > >> >>>
> > > > > > >> > > > > > > > > >> >>>
> > > > > > >> > > > > > > > > >> >>> Cheers,
> > > > > > >> > > > > > > > > >> >>> Jeyhun
> > > > > > >> > > > > > > > > >> >>>
> > > > > > >> > > > > > > > > >> >>> On Wed, Jul 5, 2017 at 10:40 AM Damian
> > Guy
> > > <
> > > > > > >> > > > > > > damian....@gmail.com>
> > > > > > >> > > > > > > > > >> wrote:
> > > > > > >> > > > > > > > > >> >>>
> > > > > > >> > > > > > > > > >> >>>> HI Jeyhun,
> > > > > > >> > > > > > > > > >> >>>>
> > > > > > >> > > > > > > > > >> >>>> Is the intention that these methods
> are
> > > new
> > > > > > >> overloads
> > > > > > >> > > on
> > > > > > >> > > > > the
> > > > > > >> > > > > > > > > KStream,
> > > > > > >> > > > > > > > > >> >>>> KTable, etc?
> > > > > > >> > > > > > > > > >> >>>>
> > > > > > >> > > > > > > > > >> >>>> It is worth noting that a
> > ProcessorContext
> > > > is
> > > > > > not
> > > > > > >> a
> > > > > > >> > > > > > > > RecordContext.
> > > > > > >> > > > > > > > > A
> > > > > > >> > > > > > > > > >> >>>> RecordContext, as it stands, only
> exists
> > > > > during
> > > > > > >> the
> > > > > > >> > > > > > processing
> > > > > > >> > > > > > > > of a
> > > > > > >> > > > > > > > > >> >>> single
> > > > > > >> > > > > > > > > >> >>>> record. Whereas the ProcessorContext
> > > exists
> > > > > for
> > > > > > >> the
> > > > > > >> > > > > lifetime
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > > >> >>>> Processor. Sot it doesn't make sense
> to
> > > > cast a
> > > > > > >> > > > > > ProcessorContext
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > a
> > > > > > >> > > > > > > > > >> >>>> RecordContext.
> > > > > > >> > > > > > > > > >> >>>> You mentioned above passing the
> > > > > > >> > > InternalProcessorContext
> > > > > > >> > > > to
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > >> init()
> > > > > > >> > > > > > > > > >> >>>> calls. It is internal for a reason
> and i
> > > > think
> > > > > > it
> > > > > > >> > > should
> > > > > > >> > > > > > remain
> > > > > > >> > > > > > > > > that
> > > > > > >> > > > > > > > > >> >> way.
> > > > > > >> > > > > > > > > >> >>>> It might be better to move the
> > > > recordContext()
> > > > > > >> method
> > > > > > >> > > > from
> > > > > > >> > > > > > > > > >> >>>> InternalProcessorContext to
> > > > ProcessorContext.
> > > > > > >> > > > > > > > > >> >>>>
> > > > > > >> > > > > > > > > >> >>>> In the KIP you have an example
> showing:
> > > > > > >> > > > > > > > > >> >>>> richMapper.init((RecordContext)
> > > > > > processorContext);
> > > > > > >> > > > > > > > > >> >>>> But the interface is:
> > > > > > >> > > > > > > > > >> >>>> public interface RichValueMapper<V,
> VR>
> > {
> > > > > > >> > > > > > > > > >> >>>> VR apply(final V value, final
> > > > > RecordContext
> > > > > > >> > > > > > recordContext);
> > > > > > >> > > > > > > > > >> >>>> }
> > > > > > >> > > > > > > > > >> >>>> i.e., there is no init(...), besides
> as
> > > > above
> > > > > > this
> > > > > > >> > > > wouldn't
> > > > > > >> > > > > > > make
> > > > > > >> > > > > > > > > >> sense.
> > > > > > >> > > > > > > > > >> >>>>
> > > > > > >> > > > > > > > > >> >>>> Thanks,
> > > > > > >> > > > > > > > > >> >>>> Damian
> > > > > > >> > > > > > > > > >> >>>>
> > > > > > >> > > > > > > > > >> >>>> On Tue, 4 Jul 2017 at 23:30 Jeyhun
> > > Karimov <
> > > > > > >> > > > > > > je.kari...@gmail.com
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > >> >> wrote:
> > > > > > >> > > > > > > > > >> >>>>
> > > > > > >> > > > > > > > > >> >>>>> Hi Matthias,
> > > > > > >> > > > > > > > > >> >>>>>
> > > > > > >> > > > > > > > > >> >>>>> Actually my intend was to provide to
> > > > > > >> RichInitializer
> > > > > > >> > > and
> > > > > > >> > > > > > later
> > > > > > >> > > > > > > > on
> > > > > > >> > > > > > > > > we
> > > > > > >> > > > > > > > > >> >>>> could
> > > > > > >> > > > > > > > > >> >>>>> provide the context of the record as
> > you
> > > > also
> > > > > > >> > > mentioned.
> > > > > > >> > > > > > > > > >> >>>>> I remove that not to confuse the
> users.
> > > > > > >> > > > > > > > > >> >>>>> Regarding the RecordContext and
> > > > > > ProcessorContext
> > > > > > >> > > > > > interfaces, I
> > > > > > >> > > > > > > > > just
> > > > > > >> > > > > > > > > >> >>>>> realized the InternalProcessorContext
> > > > class.
> > > > > > >> Can't
> > > > > > >> > we
> > > > > > >> > > > pass
> > > > > > >> > > > > > > this
> > > > > > >> > > > > > > > > as a
> > > > > > >> > > > > > > > > >> >>>>> parameter to init() method of
> > processors?
> > > > > Then
> > > > > > we
> > > > > > >> > > would
> > > > > > >> > > > be
> > > > > > >> > > > > > > able
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > >> >> get
> > > > > > >> > > > > > > > > >> >>>>> RecordContext easily with just a
> method
> > > > call.
> > > > > > >> > > > > > > > > >> >>>>>
> > > > > > >> > > > > > > > > >> >>>>>
> > > > > > >> > > > > > > > > >> >>>>> Cheers,
> > > > > > >> > > > > > > > > >> >>>>> Jeyhun
> > > > > > >> > > > > > > > > >> >>>>>
> > > > > > >> > > > > > > > > >> >>>>> On Thu, Jun 29, 2017 at 10:14 PM
> > Matthias
> > > > J.
> > > > > > Sax
> > > > > > >> <
> > > > > > >> > > > > > > > > >> >>> matth...@confluent.io>
> > > > > > >> > > > > > > > > >> >>>>> wrote:
> > > > > > >> > > > > > > > > >> >>>>>
> > > > > > >> > > > > > > > > >> >>>>>> One more thing:
> > > > > > >> > > > > > > > > >> >>>>>>
> > > > > > >> > > > > > > > > >> >>>>>> I don't think `RichInitializer` does
> > > make
> > > > > > >> sense. As
> > > > > > >> > > we
> > > > > > >> > > > > > don't
> > > > > > >> > > > > > > > have
> > > > > > >> > > > > > > > > >> >> any
> > > > > > >> > > > > > > > > >> >>>>>> input record, there is also no
> > context.
> > > We
> > > > > > >> could of
> > > > > > >> > > > > course
> > > > > > >> > > > > > > > > provide
> > > > > > >> > > > > > > > > >> >>> the
> > > > > > >> > > > > > > > > >> >>>>>> context of the record that triggers
> > the
> > > > init
> > > > > > >> call,
> > > > > > >> > > but
> > > > > > >> > > > > this
> > > > > > >> > > > > > > > seems
> > > > > > >> > > > > > > > > >> >> to
> > > > > > >> > > > > > > > > >> >>> be
> > > > > > >> > > > > > > > > >> >>>>>> semantically questionable. Also, the
> > > > context
> > > > > > for
> > > > > > >> > this
> > > > > > >> > > > > first
> > > > > > >> > > > > > > > > record
> > > > > > >> > > > > > > > > >> >>> will
> > > > > > >> > > > > > > > > >> >>>>>> be provided by the consecutive call
> to
> > > > > > aggregate
> > > > > > >> > > > anyways.
> > > > > > >> > > > > > > > > >> >>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>
> > > > > > >> > > > > > > > > >> >>>>>> -Matthias
> > > > > > >> > > > > > > > > >> >>>>>>
> > > > > > >> > > > > > > > > >> >>>>>> On 6/29/17 1:11 PM, Matthias J. Sax
> > > wrote:
> > > > > > >> > > > > > > > > >> >>>>>>> Thanks for updating the KIP.
> > > > > > >> > > > > > > > > >> >>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>> I have one concern with regard to
> > > > backward
> > > > > > >> > > > > compatibility.
> > > > > > >> > > > > > > You
> > > > > > >> > > > > > > > > >> >>> suggest
> > > > > > >> > > > > > > > > >> >>>>> to
> > > > > > >> > > > > > > > > >> >>>>>>> use RecrodContext as base interface
> > for
> > > > > > >> > > > > ProcessorContext.
> > > > > > >> > > > > > > This
> > > > > > >> > > > > > > > > >> >> will
> > > > > > >> > > > > > > > > >> >>>>>>> break compatibility.
> > > > > > >> > > > > > > > > >> >>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>> I think, we should just have two
> > > > > independent
> > > > > > >> > > > interfaces.
> > > > > > >> > > > > > Our
> > > > > > >> > > > > > > > own
> > > > > > >> > > > > > > > > >> >>>>>>> ProcessorContextImpl class would
> > > > implement
> > > > > > >> both.
> > > > > > >> > > This
> > > > > > >> > > > > > allows
> > > > > > >> > > > > > > > us
> > > > > > >> > > > > > > > > >> >> to
> > > > > > >> > > > > > > > > >> >>>> cast
> > > > > > >> > > > > > > > > >> >>>>>>> it to `RecordContext` and thus
> limit
> > > the
> > > > > > >> visible
> > > > > > >> > > > scope.
> > > > > > >> > > > > > > > > >> >>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>> -Matthias
> > > > > > >> > > > > > > > > >> >>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>> On 6/27/17 1:35 PM, Jeyhun Karimov
> > > wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>> Hi all,
> > > > > > >> > > > > > > > > >> >>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>> I updated the KIP w.r.t.
> discussion
> > > and
> > > > > > >> comments.
> > > > > > >> > > > > > > > > >> >>>>>>>> Basically I eliminated overloads
> for
> > > > > > >> particular
> > > > > > >> > > > method
> > > > > > >> > > > > if
> > > > > > >> > > > > > > > they
> > > > > > >> > > > > > > > > >> >> are
> > > > > > >> > > > > > > > > >> >>>>> more
> > > > > > >> > > > > > > > > >> >>>>>>>> than 3.
> > > > > > >> > > > > > > > > >> >>>>>>>> As we can see there are a lot of
> > > > overloads
> > > > > > >> (and
> > > > > > >> > > more
> > > > > > >> > > > > will
> > > > > > >> > > > > > > > come
> > > > > > >> > > > > > > > > >> >>> with
> > > > > > >> > > > > > > > > >> >>>>>> KIP-149
> > > > > > >> > > > > > > > > >> >>>>>>>> :) )
> > > > > > >> > > > > > > > > >> >>>>>>>> So, is it wise to
> > > > > > >> > > > > > > > > >> >>>>>>>> wait the result of constructive
> DSL
> > > > thread
> > > > > > or
> > > > > > >> > > > > > > > > >> >>>>>>>> extend KIP to address this issue
> as
> > > well
> > > > > or
> > > > > > >> > > > > > > > > >> >>>>>>>> continue as it is?
> > > > > > >> > > > > > > > > >> >>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>> Cheers,
> > > > > > >> > > > > > > > > >> >>>>>>>> Jeyhun
> > > > > > >> > > > > > > > > >> >>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>> On Wed, Jun 14, 2017 at 11:29 PM
> > > > Guozhang
> > > > > > >> Wang <
> > > > > > >> > > > > > > > > >> >>> wangg...@gmail.com>
> > > > > > >> > > > > > > > > >> >>>>>> wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>> LGTM. Thanks!
> > > > > > >> > > > > > > > > >> >>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>> Guozhang
> > > > > > >> > > > > > > > > >> >>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>> On Tue, Jun 13, 2017 at 2:20 PM,
> > > Jeyhun
> > > > > > >> Karimov
> > > > > > >> > <
> > > > > > >> > > > > > > > > >> >>>>> je.kari...@gmail.com>
> > > > > > >> > > > > > > > > >> >>>>>>>>> wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>> Thanks for the comment Matthias.
> > > After
> > > > > all
> > > > > > >> the
> > > > > > >> > > > > > discussion
> > > > > > >> > > > > > > > > >> >>> (thanks
> > > > > > >> > > > > > > > > >> >>>> to
> > > > > > >> > > > > > > > > >> >>>>>> all
> > > > > > >> > > > > > > > > >> >>>>>>>>>> participants), I think this
> > (single
> > > > > method
> > > > > > >> that
> > > > > > >> > > > > passes
> > > > > > >> > > > > > > in a
> > > > > > >> > > > > > > > > >> >>>>>> RecordContext
> > > > > > >> > > > > > > > > >> >>>>>>>>>> object) is the best alternative.
> > > > > > >> > > > > > > > > >> >>>>>>>>>> Just a side note: I think
> > KAFKA-3907
> > > > [1]
> > > > > > can
> > > > > > >> > also
> > > > > > >> > > > be
> > > > > > >> > > > > > > > > >> >> integrated
> > > > > > >> > > > > > > > > >> >>>> into
> > > > > > >> > > > > > > > > >> >>>>>> the
> > > > > > >> > > > > > > > > >> >>>>>>>>>> KIP by adding related method
> > inside
> > > > > > >> > RecordContext
> > > > > > >> > > > > > > > interface.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>> [1]
> > > > > > >> > > > https://issues.apache.org/jira/browse/KAFKA-3907
> > > > > > >> > > > > > > > > >> >>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>> Cheers,
> > > > > > >> > > > > > > > > >> >>>>>>>>>> Jeyhun
> > > > > > >> > > > > > > > > >> >>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>> On Tue, Jun 13, 2017 at 7:50 PM
> > > > Matthias
> > > > > > J.
> > > > > > >> > Sax <
> > > > > > >> > > > > > > > > >> >>>>>> matth...@confluent.io>
> > > > > > >> > > > > > > > > >> >>>>>>>>>> wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> Hi,
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> I would like to push this
> > > discussion
> > > > > > >> further.
> > > > > > >> > It
> > > > > > >> > > > > seems
> > > > > > >> > > > > > > we
> > > > > > >> > > > > > > > > got
> > > > > > >> > > > > > > > > >> >>>> nice
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> alternatives (thanks for the
> > > summary
> > > > > > >> Jeyhun!).
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> With respect to RichFunctions
> and
> > > > > > allowing
> > > > > > >> > them
> > > > > > >> > > to
> > > > > > >> > > > > be
> > > > > > >> > > > > > > > > >> >>> stateful, I
> > > > > > >> > > > > > > > > >> >>>>>> have
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> my doubt as expressed already.
> > From
> > > > my
> > > > > > >> > > > > understanding,
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > >> >> idea
> > > > > > >> > > > > > > > > >> >>>> was
> > > > > > >> > > > > > > > > >> >>>>> to
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> give access to record metadata
> > > > > > information
> > > > > > >> > only.
> > > > > > >> > > > If
> > > > > > >> > > > > > you
> > > > > > >> > > > > > > > want
> > > > > > >> > > > > > > > > >> >> to
> > > > > > >> > > > > > > > > >> >>>> do
> > > > > > >> > > > > > > > > >> >>>>> a
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> stateful computation you should
> > > > rather
> > > > > > use
> > > > > > >> > > > > > #transform().
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> Furthermore, as pointed out, we
> > > would
> > > > > > need
> > > > > > >> to
> > > > > > >> > > > switch
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > a
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> supplier-pattern introducing
> many
> > > > more
> > > > > > >> > > overloads.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> For those reason, I advocate
> for
> > a
> > > > > simple
> > > > > > >> > > > interface
> > > > > > >> > > > > > > with a
> > > > > > >> > > > > > > > > >> >>> single
> > > > > > >> > > > > > > > > >> >>>>>>>>> method
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> that passes in a RecordContext
> > > > object.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> -Matthias
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> On 6/6/17 5:15 PM, Guozhang
> Wang
> > > > wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>> Thanks for the comprehensive
> > > > summary!
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>> Personally I'd prefer the
> option
> > > of
> > > > > > >> passing
> > > > > > >> > > > > > > RecordContext
> > > > > > >> > > > > > > > > as
> > > > > > >> > > > > > > > > >> >>> an
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> additional
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>> parameter into he overloaded
> > > > function.
> > > > > > But
> > > > > > >> > I'm
> > > > > > >> > > > also
> > > > > > >> > > > > > > open
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > >> >>>> other
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> arguments
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>> if there are sth. that I have
> > > > > > overlooked.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>> Guozhang
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>> On Mon, Jun 5, 2017 at 3:19
> PM,
> > > > Jeyhun
> > > > > > >> > Karimov
> > > > > > >> > > <
> > > > > > >> > > > > > > > > >> >>>>>> je.kari...@gmail.com
> > > > > > >> > > > > > > > > >> >>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Hi,
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Thanks for your comments
> > Matthias
> > > > and
> > > > > > >> > > Guozhang.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Below I mention the quick
> > summary
> > > > of
> > > > > > the
> > > > > > >> > main
> > > > > > >> > > > > > > > alternatives
> > > > > > >> > > > > > > > > >> >> we
> > > > > > >> > > > > > > > > >> >>>>>> looked
> > > > > > >> > > > > > > > > >> >>>>>>>>>> at
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> to
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> introduce the Rich functions
> (I
> > > > will
> > > > > > >> refer
> > > > > > >> > to
> > > > > > >> > > it
> > > > > > >> > > > > as
> > > > > > >> > > > > > > Rich
> > > > > > >> > > > > > > > > >> >>>>> functions
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> until we
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> find better/another name).
> > > > Initially
> > > > > > the
> > > > > > >> > > > proposed
> > > > > > >> > > > > > > > > >> >>> alternatives
> > > > > > >> > > > > > > > > >> >>>>> was
> > > > > > >> > > > > > > > > >> >>>>>>>>> not
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> backwards-compatible, so I
> will
> > > not
> > > > > > >> mention
> > > > > > >> > > > them.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> The related discussions are
> > > spread
> > > > in
> > > > > > >> > KIP-149
> > > > > > >> > > > and
> > > > > > >> > > > > in
> > > > > > >> > > > > > > > this
> > > > > > >> > > > > > > > > >> >> KIP
> > > > > > >> > > > > > > > > >> >>>>>>>>>> (KIP-159)
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> discussion threads.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> 1. The idea of rich functions
> > > came
> > > > > into
> > > > > > >> the
> > > > > > >> > > > stage
> > > > > > >> > > > > > with
> > > > > > >> > > > > > > > > >> >>> KIP-149,
> > > > > > >> > > > > > > > > >> >>>>> in
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> discussion thread. As a
> result
> > we
> > > > > > >> extended
> > > > > > >> > > > KIP-149
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > >> >> support
> > > > > > >> > > > > > > > > >> >>>>> Rich
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> functions as well.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> 2. To as part of the Rich
> > > > functions,
> > > > > > we
> > > > > > >> > > > provided
> > > > > > >> > > > > > init
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> (ProcessorContext)
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> method. Afterwards, Dammian
> > > > suggested
> > > > > > >> that
> > > > > > >> > we
> > > > > > >> > > > > should
> > > > > > >> > > > > > > not
> > > > > > >> > > > > > > > > >> >>>> provide
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> ProcessorContext to users.
> As a
> > > > > result,
> > > > > > >> we
> > > > > > >> > > > > separated
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > >> >> two
> > > > > > >> > > > > > > > > >> >>>>>>>>> problems
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> into
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> two separate KIPs, as it
> seems
> > > they
> > > > > can
> > > > > > >> be
> > > > > > >> > > > solved
> > > > > > >> > > > > in
> > > > > > >> > > > > > > > > >> >>> parallel.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> - One approach we considered
> > was
> > > :
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> public interface
> > > > > ValueMapperWithKey<K,
> > > > > > V,
> > > > > > >> > VR>
> > > > > > >> > > {
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> VR apply(final K key,
> > final V
> > > > > > value);
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> }
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> public interface
> > > RichValueMapper<K,
> > > > > V,
> > > > > > >> VR>
> > > > > > >> > > > extends
> > > > > > >> > > > > > > > > >> >>>> RichFunction{
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> }
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> public interface
> RichFunction {
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> void init(RecordContext
> > > > > > >> recordContext);
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> void close();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> }
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> public interface
> RecordContext
> > {
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> String applicationId();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> TaskId taskId();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> StreamsMetrics metrics();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> String topic();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> int partition();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> long offset();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> long timestamp();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Map<String, Object>
> > > > appConfigs();
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Map<String, Object>
> > > > > > >> > > > > appConfigsWithPrefix(String
> > > > > > >> > > > > > > > > >> >> prefix);
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> }
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> public interface
> > ProcessorContext
> > > > > > extends
> > > > > > >> > > > > > > RecordContext
> > > > > > >> > > > > > > > {
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> // all methods but the
> ones
> > in
> > > > > > >> > > RecordContext
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> }
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> As a result:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> * . All "withKey" and
> > > "withoutKey"
> > > > > > >> > interfaces
> > > > > > >> > > > can
> > > > > > >> > > > > be
> > > > > > >> > > > > > > > > >> >>> converted
> > > > > > >> > > > > > > > > >> >>>> to
> > > > > > >> > > > > > > > > >> >>>>>>>>>> their
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Rich counterparts (with empty
> > > > init()
> > > > > > and
> > > > > > >> > > close()
> > > > > > >> > > > > > > > methods)
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> *. All related Processors
> will
> > > > accept
> > > > > > >> Rich
> > > > > > >> > > > > > interfaces
> > > > > > >> > > > > > > in
> > > > > > >> > > > > > > > > >> >>> their
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> constructors.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> *. So, we convert the related
> > > > > "withKey"
> > > > > > >> or
> > > > > > >> > > > > > > "withoutKey"
> > > > > > >> > > > > > > > > >> >>>>> interfaces
> > > > > > >> > > > > > > > > >> >>>>>>>>> to
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> Rich
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> interface while building the
> > > > topology
> > > > > > and
> > > > > > >> > > > > initialize
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > >> >>>> related
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> processors
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> with Rich interfaces only.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> *. We will not need to
> > overloaded
> > > > > > methods
> > > > > > >> > for
> > > > > > >> > > > rich
> > > > > > >> > > > > > > > > >> >> functions
> > > > > > >> > > > > > > > > >> >>> as
> > > > > > >> > > > > > > > > >> >>>>>> Rich
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> interfaces extend withKey
> > > > interfaces.
> > > > > > We
> > > > > > >> > will
> > > > > > >> > > > just
> > > > > > >> > > > > > > check
> > > > > > >> > > > > > > > > >> >> the
> > > > > > >> > > > > > > > > >> >>>>> object
> > > > > > >> > > > > > > > > >> >>>>>>>>>> type
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> and act accordingly.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> 3. There was some thoughts
> that
> > > the
> > > > > > above
> > > > > > >> > > > approach
> > > > > > >> > > > > > > does
> > > > > > >> > > > > > > > > not
> > > > > > >> > > > > > > > > >> >>>>> support
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> lambdas
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> so we should support only one
> > > > method,
> > > > > > >> only
> > > > > > >> > > > > > > > > >> >>> init(RecordContext),
> > > > > > >> > > > > > > > > >> >>>>> as
> > > > > > >> > > > > > > > > >> >>>>>>>>>> part
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> of
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Rich interfaces.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> This is still in discussion.
> > > > > > Personally I
> > > > > > >> > > think
> > > > > > >> > > > > Rich
> > > > > > >> > > > > > > > > >> >>> interfaces
> > > > > > >> > > > > > > > > >> >>>>> are
> > > > > > >> > > > > > > > > >> >>>>>>>>> by
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> definition lambda-free and we
> > > > should
> > > > > > not
> > > > > > >> > care
> > > > > > >> > > > much
> > > > > > >> > > > > > > about
> > > > > > >> > > > > > > > > >> >> it.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> 4. Thanks to Matthias's
> > > discussion,
> > > > > an
> > > > > > >> > > > alternative
> > > > > > >> > > > > > we
> > > > > > >> > > > > > > > > >> >>>> considered
> > > > > > >> > > > > > > > > >> >>>>>> was
> > > > > > >> > > > > > > > > >> >>>>>>>>>> to
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> pass in the RecordContext as
> > > method
> > > > > > >> > parameter.
> > > > > > >> > > > > This
> > > > > > >> > > > > > > > might
> > > > > > >> > > > > > > > > >> >>> even
> > > > > > >> > > > > > > > > >> >>>>>>>>> allow
> > > > > > >> > > > > > > > > >> >>>>>>>>>> to
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> use Lambdas and we could keep
> > the
> > > > > name
> > > > > > >> > > > > RichFunction
> > > > > > >> > > > > > as
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > >> >>>>> preserve
> > > > > > >> > > > > > > > > >> >>>>>>>>> the
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> nature of being a function.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> "If you go with `init()` and
> > > > > `close()`
> > > > > > we
> > > > > > >> > > > > basically
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> allow users to have an
> > in-memory
> > > > > state
> > > > > > >> for a
> > > > > > >> > > > > > function.
> > > > > > >> > > > > > > > > >> >> Thus,
> > > > > > >> > > > > > > > > >> >>> we
> > > > > > >> > > > > > > > > >> >>>>>>>>> cannot
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> share a single instance of
> > > > > > >> RichValueMapper
> > > > > > >> > > (etc)
> > > > > > >> > > > > > over
> > > > > > >> > > > > > > > > >> >>> multiple
> > > > > > >> > > > > > > > > >> >>>>>> tasks
> > > > > > >> > > > > > > > > >> >>>>>>>>>> and
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> we would need a supplier
> > pattern
> > > > > > similar
> > > > > > >> to
> > > > > > >> > > > > > > > #transform().
> > > > > > >> > > > > > > > > >> >> And
> > > > > > >> > > > > > > > > >> >>>>> this
> > > > > > >> > > > > > > > > >> >>>>>>>>>> would
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> "break the flow" of the API,
> as
> > > > > > >> > > > > > > > (Rich)ValueMapperSupplier
> > > > > > >> > > > > > > > > >> >>> would
> > > > > > >> > > > > > > > > >> >>>>> not
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> inherit from ValueMapper and
> > thus
> > > > we
> > > > > > >> would
> > > > > > >> > > need
> > > > > > >> > > > > many
> > > > > > >> > > > > > > new
> > > > > > >> > > > > > > > > >> >>>> overload
> > > > > > >> > > > > > > > > >> >>>>>>>>> for
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> KStream/KTable classes".
> (Copy
> > > > paste
> > > > > > from
> > > > > > >> > > > > Matthias's
> > > > > > >> > > > > > > > > email)
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Cheers,
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> Jeyhun
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> On Mon, Jun 5, 2017 at 5:18
> AM
> > > > > Matthias
> > > > > > >> J.
> > > > > > >> > > Sax <
> > > > > > >> > > > > > > > > >> >>>>>>>>> matth...@confluent.io
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> Yes, we did consider this,
> and
> > > > there
> > > > > > is
> > > > > > >> no
> > > > > > >> > > > > > consensus
> > > > > > >> > > > > > > > yet
> > > > > > >> > > > > > > > > >> >>> what
> > > > > > >> > > > > > > > > >> >>>>> the
> > > > > > >> > > > > > > > > >> >>>>>>>>>> best
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> alternative is.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> @Jeyhun: the email thread
> got
> > > > pretty
> > > > > > >> long.
> > > > > > >> > > > Maybe
> > > > > > >> > > > > > you
> > > > > > >> > > > > > > > can
> > > > > > >> > > > > > > > > >> >>> give
> > > > > > >> > > > > > > > > >> >>>> a
> > > > > > >> > > > > > > > > >> >>>>>>>>> quick
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> summary of the current state
> > of
> > > > the
> > > > > > >> > > discussion?
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> -Matthias
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> On 6/4/17 6:04 PM, Guozhang
> > Wang
> > > > > > wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> Thanks for the explanation
> > > Jeyhun
> > > > > and
> > > > > > >> > > > Matthias.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> I have just read through
> both
> > > > > KIP-149
> > > > > > >> and
> > > > > > >> > > > > KIP-159
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > am
> > > > > > >> > > > > > > > > >> >>>>>> wondering
> > > > > > >> > > > > > > > > >> >>>>>>>>>> if
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> you
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> guys have considered a
> slight
> > > > > > different
> > > > > > >> > > > approach
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > > > rich
> > > > > > >> > > > > > > > > >> >>>>>>>>> function,
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> that
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> is
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> to add the `RecordContext`
> > into
> > > > the
> > > > > > >> apply
> > > > > > >> > > > > > functions
> > > > > > >> > > > > > > as
> > > > > > >> > > > > > > > > an
> > > > > > >> > > > > > > > > >> >>>>>>>>> additional
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> parameter. For example:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> ---------------------------
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> interface
> RichValueMapper<V,
> > > VR>
> > > > {
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> VR apply(final V value,
> final
> > > > > > >> > RecordContext
> > > > > > >> > > > > > > context);
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> }
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> ...
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> // then in KStreams
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> <VR> KStream<K, VR>
> > > > > > >> > mapValues(ValueMapper<?
> > > > > > >> > > > > super
> > > > > > >> > > > > > > V, ?
> > > > > > >> > > > > > > > > >> >>>> extends
> > > > > > >> > > > > > > > > >> >>>>>> VR>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> mapper);
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> <VR> KStream<K, VR>
> > > > > > >> > > > > > > > mapValueswithContext(RichValueMapper
> > > > > > >> > > > > > > > > >> >> <?
> > > > > > >> > > > > > > > > >> >>>>> super
> > > > > > >> > > > > > > > > >> >>>>>>>>>> V, ?
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> extends VR> mapper);
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > ------------------------------
> > > -
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> The caveat is that it will
> > > > > introduces
> > > > > > >> more
> > > > > > >> > > > > > > overloads;
> > > > > > >> > > > > > > > > >> >> but I
> > > > > > >> > > > > > > > > >> >>>>> think
> > > > > > >> > > > > > > > > >> >>>>>>>>>> the
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> #.overloads are mainly
> > > introduced
> > > > > by
> > > > > > 1)
> > > > > > >> > > serde
> > > > > > >> > > > > > > > overrides
> > > > > > >> > > > > > > > > >> >> and
> > > > > > >> > > > > > > > > >> >>>> 2)
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> state-store-supplier
> > overides,
> > > > both
> > > > > > of
> > > > > > >> > which
> > > > > > >> > > > can
> > > > > > >> > > > > > be
> > > > > > >> > > > > > > > > >> >> reduced
> > > > > > >> > > > > > > > > >> >>>> in
> > > > > > >> > > > > > > > > >> >>>>>> the
> > > > > > >> > > > > > > > > >> >>>>>>>>>>> near
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> future, and I felt this
> > > > overloading
> > > > > > is
> > > > > > >> > still
> > > > > > >> > > > > > > > worthwhile,
> > > > > > >> > > > > > > > > >> >> as
> > > > > > >> > > > > > > > > >> >>>> it
> > > > > > >> > > > > > > > > >> >>>>>> has
> > > > > > >> > > > > > > > > >> >>>>>>>>>> the
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> following benefits:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> 1) still allow lambda
> > > > expressions.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> 2) clearer code path (do
> not
> > > need
> > > > > to
> > > > > > >> > > "convert"
> > > > > > >> > > > > > from
> > > > > > >> > > > > > > > > >> >>> non-rich
> > > > > > >> > > > > > > > > >> >>>>>>>>>> functions
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> to
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> rich functions)
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> Maybe this approach has
> > already
> > > > > been
> > > > > > >> > > discussed
> > > > > > >> > > > > > and I
> > > > > > >> > > > > > > > may
> > > > > > >> > > > > > > > > >> >>> have
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> overlooked
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>> in
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> the email thread; anyways,
> > lmk.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> Guozhang
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> On Thu, Jun 1, 2017 at
> 10:18
> > > PM,
> > > > > > >> Matthias
> > > > > > >> > J.
> > > > > > >> > > > > Sax <
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> matth...@confluent.io>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>> wrote:
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> I agree with Jeyhun. As
> > > already
> > > > > > >> mention,
> > > > > > >> > > the
> > > > > > >> > > > > > > overall
> > > > > > >> > > > > > > > > API
> > > > > > >> > > > > > > > > >> >>>>>>>>>> improvement
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> ideas are overlapping
> and/or
> > > > > > >> > contradicting
> > > > > > >> > > > each
> > > > > > >> > > > > > > > other.
> > > > > > >> > > > > > > > > >> >> For
> > > > > > >> > > > > > > > > >> >>>>> this
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> reason,
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> not all ideas can be
> > > > accomplished
> > > > > > and
> > > > > > >> > some
> > > > > > >> > > > Jira
> > > > > > >> > > > > > > might
> > > > > > >> > > > > > > > > >> >> just
> > > > > > >> > > > > > > > > >> >>>> be
> > > > > > >> > > > > > > > > >> >>>>>>>>>> closed
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> as
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> "won't fix".
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> For this reason, we try to
> > do
> > > > > those
> > > > > > >> KIP
> > > > > > >> > > > > > discussion
> > > > > > >> > > > > > > > with
> > > > > > >> > > > > > > > > >> >>> are
> > > > > > >> > > > > > > > > >> >>>>>> large
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> scope
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> to get an overall picture
> to
> > > > > > converge
> > > > > > >> to
> > > > > > >> > an
> > > > > > >> > > > > > overall
> > > > > > >> > > > > > > > > >> >>>> consisted
> > > > > > >> > > > > > > > > >> >>>>>>>>> API.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> @Jeyhun: about the
> > overloads.
> > > > Yes,
> > > > > > we
> > > > > > >> > might
> > > > > > >> > > > get
> > > > > > >> > > > > > > more
> > > > > > >> > > > > > > > > >> >>>> overload.
> > > > > > >> > > > > > > > > >> >>>>>> It
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> might
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> be sufficient though, to
> do
> > a
> > > > > single
> > > > > > >> > > > > > > xxxWithContext()
> > > > > > >> > > > > > > > > >> >>>> overload
> > > > > > >> > > > > > > > > >> >>>>>>>>> that
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> will
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> provide key+value+context.
> > > > > > Otherwise,
> > > > > > >> if
> > > > > > >> > > > might
> > > > > > >> > > > > > get
> > > > > > >> > > > > > > > too
> > > > > > >> > > > > > > > > >> >>> messy
> > > > > > >> > > > > > > > > >> >>>>>>>>> having
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> ValueMapper,
> > > ValueMapperWithKey,
> > > > > > >> > > > > > > > > ValueMapperWithContext,
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > ValueMapperWithKeyWithContext.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> On the other hand, we also
> > > have
> > > > > the
> > > > > > >> > > "builder
> > > > > > >> > > > > > > pattern"
> > > > > > >> > > > > > > > > >> >> idea
> > > > > > >> > > > > > > > > >> >>>> as
> > > > > > >> > > > > > > > > >> >>>>> an
> > > > > > >> > > > > > > > > >> >>>>>>>>>> API
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> change and this might
> > mitigate
> > > > the
> > > > > > >> > overload
> > > > > > >> > > > > > > problem.
> > > > > > >> > > > > > > > > Not
> > > > > > >> > > > > > > > > >> >>> for
> > > > > > >> > > > > > > > > >> >>>>>>>>> simple
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> function like map/flatMap
> > etc
> > > > but
> > > > > > for
> > > > > > >> > joins
> > > > > > >> > > > and
> > > > > > >> > > > > > > > > >> >>>> aggregations.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> On the other hand, as I
> > > > mentioned
> > > > > in
> > > > > > >> an
> > > > > > >> > > older
> > > > > > >> > > > > > > email,
> > > > > > >> > > > > > > > I
> > > > > > >> > > > > > > > > >> >> am
> > > > > > >> > > > > > > > > >> >>>>>>>>>> personally
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> fine to break the pure
> > > > functional
> > > > > > >> > > interface,
> > > > > > >> > > > > and
> > > > > > >> > > > > > > add
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> - interface
> > > WithRecordContext
> > > > > with
> > > > > > >> > method
> > > > > > >> > > > > > > > > >> >>>>>> `open(RecordContext)`
> > > > > > >> > > > > > > > > >> >>>>>>>>>> (or
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> `init(...)`, or any better
> > > name)
> > > > > --
> > > > > > >> but
> > > > > > >> > not
> > > > > > >> > > > > > > > `close()`)
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> - interface
> > > > > > >> > ValueMapperWithRecordContext
> > > > > > >> > > > > > extends
> > > > > > >> > > > > > > > > >> >>>>> ValueMapper,
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> WithRecordContext
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> This would allow us to
> avoid
> > > any
> > > > > > >> > overload.
> > > > > > >> > > Of
> > > > > > >> > > > > > > course,
> > > > > > >> > > > > > > > > we
> > > > > > >> > > > > > > > > >> >>>> don't
> > > > > > >> > > > > > > > > >> >>>>>>>>> get
> > > > > > >> > > > > > > > > >> >>>>>>>>>> a
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> "pure function" interface
> > and
> > > > also
> > > > > > >> > > sacrifices
> > > > > > >> > > > > > > > Lambdas.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> I am personally a little
> bit
> > > > > > undecided
> > > > > > >> > what
> > > > > > >> > > > the
> > > > > > >> > > > > > > > better
> > > > > > >> > > > > > > > > >> >>>> option
> > > > > > >> > > > > > > > > >> >>>>>>>>> might
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>> be.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> Curious to hear what other
> > > think
> > > > > > about
> > > > > > >> > this
> > > > > > >> > > > > trade
> > > > > > >> > > > > > > > off.
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> -Matthias
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>>
> > > > > > >> > > > > > > > > >> >>>>>>>>>>>>>>>> On 6/1/17 6:13 PM, Jeyhun
> > > > Karimov
> > > > > > >> wrote:<
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
