Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Update: I want to give a quick update on what I found porting the 0.10 version towards 1.0. 1. It is difficult to provide a stock CombinedKey Serde. We effectively wrap 2 serdes for the key. We do not have good topic names to feed into the Avro Serde for K1 and K2 for the same topic.

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Hi Guozhang, thanks for the update. On 15.12.2017 22:54, Guozhang Wang wrote: Jan, Thanks for the updated KIP, and the raised questions. Here are my thoughts around the "back and forth mapper" approach on your wiki: 1) regarding the key-value types of KTableValueGetter, we do not

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Jan, Thanks for the updated KIP, and the raised questions. Here are my thoughts around the "back and forth mapper" approach on your wiki: 1) regarding the key-value types of KTableValueGetter, we do not necessarily enforce its template K, V to be the same as its calling Processor, although

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Hi, I updated the KIP I would be open for this: We mark the "less intrusive" and the "back and forth mapper" approach as rejected alternatives. and implement the two remaining methods. any thoughts? Best jan On 07.12.2017 12:58, Jan Filipiak wrote: On 05.12.2017 00:42, Matthias J. Sax

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

On 05.12.2017 00:42, Matthias J. Sax wrote: Jan, The KTableValueGetter thing is a valid point. I think we would need a backwards mapper (or merge both into one and sacrifices lambdas?). Another alternative would be, to drop the optimization and materialize the KTable.operator() result... (not

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Jan, The KTableValueGetter thing is a valid point. I think we would need a backwards mapper (or merge both into one and sacrifices lambdas?). Another alternative would be, to drop the optimization and materialize the KTable.operator() result... (not a great solution either). I am personally fine

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Hi Matthias, 2 things that pop into my mind sunday morning. Can we provide an KTableValueGetter when key in the store is different from the key forwarded? 1. we would need a backwards mapper 2. I am not sure if we can pull it of w/o said forth generic type in KTable (that I am in favour of

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Just to clarify, though "CombinedKey" will still be exposed as public APIs it would not be used in any of the returned key types, so users do not need to worry providing a serde for it at all; it will only be used in the Mapper parameter, and internally Streams library would know how to serde it

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Jan, Thanks for explaining the Serde issue! This makes a lot of sense. I discussed with Guozhang about this issue and came up with the following idea that bridges both APIs: We still introduce CombinedKey as a public interface and exploit it to manage the key in the store and the changelog

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Just list what each thing is: K0: key type of first/this table K1: key type of second/other table KO: key type of result table (concatenation of both input keys ) something like this (not sure it the example above is correct---it's just for illustration) -Matthias On 11/18/17 2:30 PM, Jan

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Hello Jan, I think I get your point about the cumbersome that CombinedKey would introduce for serialization and tooling based on serdes. What I'm still wondering is the underlying of joinPrefixFakers mapper: from your latest comment it seems this mapper will be a one-time mapper: we use this to

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

-> it think the relationships between the different used types, K0,K1,KO should be explains explicitly (all information is there implicitly, but one need to think hard to figure it out) I'm probably blind for this. can you help me here? how would you formulate this? Thanks, Jan On

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

On 17.11.2017 06:59, Guozhang Wang wrote: Thanks for the explanation Jan. On top of my head I'm leaning towards the "more intrusive" approach to resolve the race condition issue we discussed above. Matthias has some arguments for this approach already, so I would not re-iterate them here. To me

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Hi Matthias answers to the questions inline. On 16.11.2017 23:18, Matthias J. Sax wrote: Hi, I am just catching up on this discussion and did re-read the KIP and discussion thread. In contrast to you, I prefer the second approach with CombinedKey as return type for the following reasons:

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Thanks for the explanation Jan. On top of my head I'm leaning towards the "more intrusive" approach to resolve the race condition issue we discussed above. Matthias has some arguments for this approach already, so I would not re-iterate them here. To me I find the "ValueMapper

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Hi, I am just catching up on this discussion and did re-read the KIP and discussion thread. In contrast to you, I prefer the second approach with CombinedKey as return type for the following reasons: 1) the oneToManyJoin() method had less parameter 2) those parameters are easy to understand

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

1. Going over KIP-213, I am leaning toward the "less intrusive" approach. In my use case, I am planning on performing a sequence of several oneToMany joins, From my understanding, the more intrusive approach would result in several nested levels of CombinedKey's. For example, consider Tables A, B,

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

We are running this perfectly fine. for us the smaller table changes rather infrequent say. only a few times per day. The performance of the flush is way lower than the computing power you need to bring to the table to account for all the records beeing emmited after the one single update.

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Hi Trevor, thank you very much for your interested. Too keep discussion mailing list focused and not Jira or Confluence I decided to reply here. 1. its tricky activity is indeed very low. In the KIP-213 there are 2 proposals about the return type of the join. I would like to settle on one.

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

;> In both alternatives the API will need to use the `Joined` class >>>>>>>>> rather >>>>>>>>> than than passing in `Serde`s. Also, as with all other joins etc, >>>>>>>>> there >>>>>>>>> probabl

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

, Oct 26, 2017 at 11:04 PM, Jan Filipiak <jan.filip...@trivago.com wrote: Thanks for the remarks. hope I didn't miss any. Not even sure if it makes sense to introduce A and B or just stick with "this ktable", "other ktable" Thank you Jan On 27.10.2017 06:58, Ted Yu wrote:

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

gt;>>>>> argument >>>>>> with this IQ overloads things. >>>>>> Didn't expect something to have happend already so I just copied from >>>>>> the >>>>>> PR. Will update. >>>>>> Will also add the o

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

ilipiak <jan.filip...@trivago.com wrote: Thanks for the remarks. hope I didn't miss any. Not even sure if it makes sense to introduce A and B or just stick with "this ktable", "other ktable" Thank you Jan On 27.10.2017 06:58, Ted Yu wrote: Do you mind addressing my

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

makes sense to introduce A and B or just stick with "this ktable", "other ktable" Thank you Jan On 27.10.2017 06:58, Ted Yu wrote: Do you mind addressing my previous comments ? http://search-hadoop.com/m/Kafka/uyzND1hzF8SRzUqb?subj=Re+ DISCUSS+KIP+213+Support+non+key+joining+in

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

+ Protobuf +  custom Serdes and Hadoop >>> Writables. Its a nightmare for JSON serdes. >>> >>> >>> >>>> Thanks, >>>> Damian >>>> >>>> >>>> On Fri, 27 Oct 2017 at 10:27 Ted Yu <yuzh

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

B or just stick with "this ktable", "other ktable" Thank you Jan On 27.10.2017 06:58, Ted Yu wrote: Do you mind addressing my previous comments ? http://search-hadoop.com/m/Kafka/uyzND1hzF8SRzUqb?subj=Re+ DISCUSS+KIP+213+Support+non+key+joining+in+KTable On Thu, Oct 26, 2017 at

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Cheers On Thu, Oct 26, 2017 at 11:04 PM, Jan Filipiak <jan.filip...@trivago.com wrote: Thanks for the remarks. hope I didn't miss any. Not even sure if it makes sense to introduce A and B or just stick with "this ktable", "other ktable" Thank you Jan On 27.10.2017 06:58, T

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

lt;jan.filip...@trivago.com >>> > >>> wrote: >>> >>> >>>> Thanks for the remarks. hope I didn't miss any. >>>> Not even sure if it makes sense to introduce A and B or just stick with >>>> "this ktable", "other

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

s ? http://search-hadoop.com/m/Kafka/uyzND1hzF8SRzUqb?subj=Re+ DISCUSS+KIP+213+Support+non+key+joining+in+KTable On Thu, Oct 26, 2017 at 9:38 PM, Jan Filipiak <jan.filip...@trivago.com wrote: Hello everyone, this is the new discussion thread after the ID-clash. Best Jan __ Hello Ka

Re: Re: [DISCUSS] KIP-213 Support non-key joining in KTable

didn't miss any. > > Not even sure if it makes sense to introduce A and B or just stick with > > "this ktable", "other ktable" > > > > Thank you > > Jan > > > > > > On 27.10.2017 06:58, Ted Yu wrote: > > > >> Do

Re: Re: [DISCUSS] KIP-213 Support non-key joining in KTable

> Not even sure if it makes sense to introduce A and B or just stick with > "this ktable", "other ktable" > > Thank you > Jan > > > On 27.10.2017 06:58, Ted Yu wrote: > >> Do you mind addressing my previous comments ? >> >> http://search

Fwd: Re: [DISCUSS] KIP-213 Support non-key joining in KTable

my previous comments ? http://search-hadoop.com/m/Kafka/uyzND1hzF8SRzUqb?subj=Re+DISCUSS+KIP+213+Support+non+key+joining+in+KTable On Thu, Oct 26, 2017 at 9:38 PM, Jan Filipiak <jan.filip...@trivago.com> wrote: Hello everyone, this is the new discussion thread after the ID-clash. Best Jan

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Do you mind addressing my previous comments ? http://search-hadoop.com/m/Kafka/uyzND1hzF8SRzUqb?subj=Re+DISCUSS+KIP+213+Support+non+key+joining+in+KTable On Thu, Oct 26, 2017 at 9:38 PM, Jan Filipiak <jan.filip...@trivago.com> wrote: > Hello everyone, > > this is the new discussi

[DISCUSS] KIP-213 Support non-key joining in KTable

Hello everyone, this is the new discussion thread after the ID-clash. Best Jan __ Hello Kafka-users, I want to continue with the development of KAFKA-3705, which allows the Streams DSL to perform KTableKTable-Joins when the KTables have a one-to-many relationship. To make sure we

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Done. I changed my KIP to KIP-214. So this KIP doesn't need to change. On Wed, Oct 25, 2017 at 10:33 PM, Onur Karaman wrote: > Looks like Jan technically made his KIP wiki page first so I'll just > change my KIP number. > > On Wed, Oct 25, 2017 at 4:59 PM,

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Looks like Jan technically made his KIP wiki page first so I'll just change my KIP number. On Wed, Oct 25, 2017 at 4:59 PM, Matthias J. Sax wrote: > Thanks a lot for the KIP. Can we please move the discussion to the dev > list? > > Thus, after fixing the KIP collision,

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Thanks a lot for the KIP. Can we please move the discussion to the dev list? Thus, after fixing the KIP collision, just start a new DISCUSS thread. Thx. -Matthias On 10/25/17 4:20 PM, Ted Yu wrote: > Have you seen the email a moment ago from Onur which uses the same KIP > number ? > > Looks