Matthias Thanks for the links. I have seen those before but I will dig deeper into them, especially around the CombinedKey and the flush + cache + rangescan functionality. I believe Jan had a PR with many of the changes in there, perhaps I can use some of the work that was done there to help leverage a similar (or identical) design.
I will certainly be able to make a PoC before going to vote on this one. It is a larger change and I suspect that we will need to review some of the finer points to ensure that the design is still suitable and sufficiently performant. I'll post back when I have something more concrete, but in the meantime I welcome all other concerns and comments. Thanks On Mon, Jun 18, 2018 at 10:05 PM, Matthias J. Sax <matth...@confluent.io> wrote: > Adam, > > thanks a lot for the KIP. I agree that this would be a valuable feature > to add. It's a very complex one though. You correctly pointed out, that > the GlobalKTable (or global stores in general) cannot be the "driver" > atm and are passively updated only. This is by design. Are you familiar > with the KIP discussion of KIP-99? > (https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=67633649 > ) > Would be worth to refresh to understand the tradeoffs and design decisions. > > It's unclear to me, what the impact will be if we want to change the > current design. Even if no GlobalKTable is used, it might have impact on > performance and for sure on code complexity. Overall, it seems that a > POC might be required before we can consider adding this (with the > danger, that it does not get accepted in the end). > > Are you aware of KIP-213: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 213+Support+non-key+joining+in+KTable > > It suggest to add non-key joins and a lot of issues how to implement > this were discussed already. As a KTable-GloblKTable join is a non-key > join, too, it seems that those discussion apply to your KIP too. > > Hope this helps to make the next steps. > > > -Matthias > > > On 6/18/18 1:15 PM, Adam Bellemare wrote: > > Hi All > > > > I created KIP-314 and I would like to initiate a discussion on it. > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 314%3A+KTable+to+GlobalKTable+Bi-directional+Join > > > > The primary goal of this KIP is to improve the way that Kafka can deal > with > > relational data at scale. This KIP would alter the way that GlobalKTables > > can be used in relation to KTables. I believe that this would be a very > > useful change but I need some eyes on the technical aspects to validate > or > > refute the strategy. > > > > Thanks > > > > Adam Bellemare > > > >
