A comment on the PR pointed out the advantages of a relatively minor adjustment to the method org.apache.kafka.streams.processor.addGlobalStore. It should take a StateStoreSupplier<KeyValueStore>" type as the first argument, not a StateStore. Note that one can obtain a StateStore by calling the .get() method on a StateStoreSupplier.
Adjusted the KIP accordingly. Thanks Eno > On 26 Apr 2017, at 09:21, Eno Thereska <eno.there...@gmail.com> wrote: > > Made a slight change to the wiki since I had forgotten to state the fact that > some APIs don’t take a store name, but a StateStoreSupplier (that contains > the name). That’s already the case in the code but thought of making that > explicit. FYI, no action needed. > > Thanks > Eno >> On Apr 21, 2017, at 7:47 PM, Eno Thereska <eno.there...@gmail.com> wrote: >> >> Added the .tostream() discussion in the Wiki. >> >> Thanks >> Eno >>> On 21 Apr 2017, at 18:48, Matthias J. Sax <matth...@confluent.io> wrote: >>> >>> I agree with Eno about the renaming. >>> >>> @Eno: can you add this to the Wiki Discussion page? >>> >>> >>> -Matthias >>> >>> >>> On 4/21/17 1:11 AM, Eno Thereska wrote: >>>> Hi Guozhang, >>>> >>>> Thanks for the feedback. Comments inline: >>>> >>>>> 1. Regarding the user-facing semantics, I thought we will claim that >>>>> "KTables generated from functions that do NOT specify a table name will >>>>> NOT >>>>> be queryable"; but it seems you're proposing to claim it "may not possible >>>>> to be queryable", i.e. if users happen to know the internal name if it is >>>>> materialized, she can still query it. I feel its potential benefits are >>>>> well overwhelmed by the confusion it may introduce. So I'd suggest we just >>>>> be strict and say "no store name, not queryable". >>>> >>>> Sure. "No store name, not queryable" sounds fine. If the user is brave and >>>> digs deep they will be able to query these stores that are always created >>>> (like when we do aggregates), but I agree that there is no reason we need >>>> to make a promise to them if they don't provide a name. I'll change the >>>> wording. >>>> >>>>> >>>>> 2. Is there a difference between "calling the overloaded function with >>>>> store name, but specify the value as null" and "calling the overloaded >>>>> function without store name"? I thought they will be implemented the same >>>>> way. But after reading through the wiki I'm not sure. So just clarifying. >>>>> >>>> >>>> There is no difference. I'll clarify. >>>> >>>> >>>>> 3. Personally I'm still a bit preferring renaming "KTable#toStream" to >>>>> sth. >>>>> like "KTable#getChangelog()" or "#toChangelog", since to me it feels more >>>>> understandable from user's point of view. WDPT? >>>>> >>>> >>>> So I left this out of this KIP, since it's not directly related to the >>>> scope. Perhaps we can do it in a cleanup KIP? >>>> >>>> Thanks >>>> Eno >>>> >>>> >>>>> >>>>> Guozhang >>>>> >>>>> >>>>> On Tue, Apr 11, 2017 at 11:53 AM, Matthias J. Sax <matth...@confluent.io> >>>>> wrote: >>>>> >>>>>> +1 >>>>>> >>>>>> On 4/11/17 10:34 AM, Eno Thereska wrote: >>>>>>> Hi Matthias, >>>>>>> >>>>>>> >>>>>>>> On 11 Apr 2017, at 09:41, Matthias J. Sax <matth...@confluent.io> >>>>>> wrote: >>>>>>>> >>>>>>>> Not sure, if we are on the same page already? >>>>>>>> >>>>>>>>> "A __store__ can be queryable whether is't materialized or not" >>>>>>>> >>>>>>>> This does not make sense -- there is nothing like a non-materialized >>>>>>>> store -- only non-materialized KTables. >>>>>>> >>>>>>> Yes, there are stores that are simple views, i.e., non-materialized. >>>>>> Damian has such a prototype for Global Tables (it didn't go into trunk). >>>>>>> It's still a store, e.g., a KeyValueStore, but when you do a get() it >>>>>> recomputes the result on the fly (e.g., it applies a filter). >>>>>>> >>>>>>> Eno >>>>>>> >>>>>>>> >>>>>>>>> "Yes, there is nothing that will prevent users from querying >>>>>>>> internally generated stores, but they cannot assume a store will >>>>>>>> necessarily be queryable." >>>>>>>> >>>>>>>> That is what I disagree on. Stores should be queryable all the time. >>>>>>>> >>>>>>>> Furthermore, we should have all non-materialized KTables to be >>>>>>>> queryable, too. >>>>>>>> >>>>>>>> >>>>>>>> Or maybe there is just some missunderstand going as, and there is some >>>>>>>> mix-up between "store" and "KTable" >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -Matthias >>>>>>>> >>>>>>>> >>>>>>>> On 4/11/17 9:34 AM, Eno Thereska wrote: >>>>>>>>> Hi Matthias, >>>>>>>>> >>>>>>>>> See my note: "A store can be queryable whether it's materialized or >>>>>> not". I think we're on the same page. Stores with an internal name are >>>>>> also >>>>>> queryable. >>>>>>>>> >>>>>>>>> I'm just pointing out that. although that is the case today and with >>>>>> this KIP, I don't think we have an obligation to make stores with >>>>>> internal >>>>>> names queryable in the future. However, that is a discussion for a future >>>>>> point. >>>>>>>>> >>>>>>>>> Eno >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 11 Apr 2017, at 08:56, Matthias J. Sax <matth...@confluent.io> >>>>>> wrote: >>>>>>>>>> >>>>>>>>>> +1 on including GlobalKTable >>>>>>>>>> >>>>>>>>>> But I am not sure about the materialization / queryable question. For >>>>>>>>>> full consistency, all KTables should be queryable nevertheless if >>>>>>>>>> they >>>>>>>>>> are materialized or not. -- Maybe this is a second step though (even >>>>>> if >>>>>>>>>> I would like to get this done right away) >>>>>>>>>> >>>>>>>>>> If we don't want all KTables to be queryable, ie, only those KTables >>>>>>>>>> that are materialized, then we should have a clear definition about >>>>>>>>>> this, and only allow to query stores, the user did specify a name >>>>>>>>>> for. >>>>>>>>>> This will simply the reasoning for users, what stores are queryable >>>>>> and >>>>>>>>>> what not. Otherwise, we still end up confusing user. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -Matthias >>>>>>>>>> >>>>>>>>>> On 4/11/17 8:23 AM, Damian Guy wrote: >>>>>>>>>>> Eno, re: GlobalKTable - yeah that seems fine. >>>>>>>>>>> >>>>>>>>>>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com> >>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> About GlobalKTables, I suppose there is no reason why they cannot >>>>>> also use >>>>>>>>>>>> this KIP for consistency, e.g., today you have: >>>>>>>>>>>> >>>>>>>>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> >>>>>> keySerde, >>>>>>>>>>>> final Serde<V> valSerde, >>>>>>>>>>>> final String topic, >>>>>>>>>>>> final String storeName) >>>>>>>>>>>> >>>>>>>>>>>> For consistency with the KIP you could also have an overload >>>>>> without the >>>>>>>>>>>> store name, for people who want to construct a global ktable, but >>>>>> don't >>>>>>>>>>>> care about querying it directly: >>>>>>>>>>>> >>>>>>>>>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> >>>>>> keySerde, >>>>>>>>>>>> final Serde<V> valSerde, >>>>>>>>>>>> final String topic) >>>>>>>>>>>> >>>>>>>>>>>> Damian, what do you think? I'm thinking of adding this to KIP. >>>>>> Thanks to >>>>>>>>>>>> Michael for bringing it up. >>>>>>>>>>>> >>>>>>>>>>>> Eno >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com> >>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Michael, comments inline: >>>>>>>>>>>>> >>>>>>>>>>>>>> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io> >>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for the updates, Eno! >>>>>>>>>>>>>> >>>>>>>>>>>>>> In addition to what has already been said: We should also >>>>>> explicitly >>>>>>>>>>>>>> mention that this KIP is not touching GlobalKTable. I'm sure >>>>>> that some >>>>>>>>>>>>>> users will throw KTable and GlobalKTable into one conceptual >>>>>> "it's all >>>>>>>>>>>>>> tables!" bucket and then wonder how the KIP might affect global >>>>>> tables. >>>>>>>>>>>>> >>>>>>>>>>>>> Good point, I'll add. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Damian wrote: >>>>>>>>>>>>>>> I think if no store name is provided users would still be able >>>>>> to query >>>>>>>>>>>>>> the >>>>>>>>>>>>>>> store, just the store name would be some internally generated >>>>>> name. >>>>>>>>>>>> They >>>>>>>>>>>>>>> would be able to discover those names via the IQ API. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I, too, think that users should be able to query a store even if >>>>>> its >>>>>>>>>>>> name >>>>>>>>>>>>>> was internally generated. After all, the data is already there / >>>>>>>>>>>>>> materialized. >>>>>>>>>>>>> >>>>>>>>>>>>> Yes, there is nothing that will prevent users from querying >>>>>> internally >>>>>>>>>>>> generated stores, but they cannot >>>>>>>>>>>>> assume a store will necessarily be queryable. So if it's there, >>>>>> they can >>>>>>>>>>>> query it. If it's not there, and they didn't >>>>>>>>>>>>> provide a queryable name, they cannot complain and say "hey, where >>>>>> is my >>>>>>>>>>>> store". If they must absolutely be certain that >>>>>>>>>>>>> a store is queryable, then they must provide a queryable name. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Damian wrote: >>>>>>>>>>>>>>> I think for some stores it will make sense to not create a >>>>>> physical >>>>>>>>>>>>>> store, i.e., >>>>>>>>>>>>>>> for thinks like `filter`, as this will save the rocksdb >>>>>> overhead. But i >>>>>>>>>>>>>> guess that >>>>>>>>>>>>>>> is more of an implementation detail. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think it would help if the KIP would clarify what we'd do in >>>>>> such a >>>>>>>>>>>>>> case. For example, if the user did not specify a store name for >>>>>>>>>>>>>> `KTable#filter` -- would it be queryable? If so, would this >>>>>> imply we'd >>>>>>>>>>>>>> always materialize the state store, or...? >>>>>>>>>>>>> >>>>>>>>>>>>> I'll clarify in the KIP with some more examples. Materialization >>>>>> will be >>>>>>>>>>>> an internal concept. A store can be queryable whether it's >>>>>> materialized or >>>>>>>>>>>> not >>>>>>>>>>>>> (e.g., through advanced implementations that compute the value of >>>>>>>>>>>>> a >>>>>>>>>>>> filter on a fly, rather than materialize the answer). >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Eno >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Michael >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy <damian....@gmail.com >>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for the update. I agree with what Matthias said. I wonder >>>>>> if >>>>>>>>>>>> the KIP >>>>>>>>>>>>>>> should talk less about materialization and more about querying? >>>>>> After >>>>>>>>>>>> all, >>>>>>>>>>>>>>> that is what is being provided from an end-users perspective. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think if no store name is provided users would still be able >>>>>>>>>>>>>>> to >>>>>>>>>>>> query the >>>>>>>>>>>>>>> store, just the store name would be some internally generated >>>>>> name. >>>>>>>>>>>> They >>>>>>>>>>>>>>> would be able to discover those names via the IQ API >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think for some stores it will make sense to not create a >>>>>> physical >>>>>>>>>>>> store, >>>>>>>>>>>>>>> i.e., for thinks like `filter`, as this will save the rocksdb >>>>>>>>>>>> overhead. But >>>>>>>>>>>>>>> i guess that is more of an implementation detail. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Damian >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska < >>>>>> eno.there...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Matthias, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> However, this still forces users, to provide a name for store >>>>>> that we >>>>>>>>>>>>>>>>> must materialize, even if users are not interested in querying >>>>>> the >>>>>>>>>>>>>>>>> stores. Thus, I would like to have overloads for all currently >>>>>>>>>>>> existing >>>>>>>>>>>>>>>>> methods having mandatory storeName paremeter, with overloads, >>>>>> that do >>>>>>>>>>>>>>>>> not require the storeName parameter. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't >>>>>> make it >>>>>>>>>>>>>>>> clear, I'll clarify. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax < >>>>>> matth...@confluent.io> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks for pushing this KIP Eno. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The update give a very clear description about the scope, that >>>>>> is >>>>>>>>>>>> super >>>>>>>>>>>>>>>>> helpful for the discussion! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - To put it into my own words, the KIP focus is on enable to >>>>>> query >>>>>>>>>>>> all >>>>>>>>>>>>>>>>> KTables. >>>>>>>>>>>>>>>>> ** The ability to query a store is determined by providing a >>>>>> name for >>>>>>>>>>>>>>>>> the store. >>>>>>>>>>>>>>>>> ** At the same time, providing a name -- and thus making a >>>>>> store >>>>>>>>>>>>>>>>> queryable -- does not say anything about an actual >>>>>> materialization >>>>>>>>>>>> (ie, >>>>>>>>>>>>>>>>> being queryable and being materialized are orthogonal). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I like this overall a lot. However, I would go one step >>>>>> further. >>>>>>>>>>>> Right >>>>>>>>>>>>>>>>> now, you suggest to add new overload methods that allow users >>>>>> to >>>>>>>>>>>>>>> specify >>>>>>>>>>>>>>>>> a storeName -- if `null` is provided and the store is not >>>>>>>>>>>> materialized, >>>>>>>>>>>>>>>>> we ignore it completely -- if `null` is provided but the store >>>>>> must >>>>>>>>>>>> be >>>>>>>>>>>>>>>>> materialized we generate a internal name. So far so good. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> However, this still forces users, to provide a name for store >>>>>> that we >>>>>>>>>>>>>>>>> must materialize, even if users are not interested in querying >>>>>> the >>>>>>>>>>>>>>>>> stores. Thus, I would like to have overloads for all currently >>>>>>>>>>>> existing >>>>>>>>>>>>>>>>> methods having mandatory storeName paremeter, with overloads, >>>>>> that do >>>>>>>>>>>>>>>>> not require the storeName parameter. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Otherwise, we would still have some methods which optional >>>>>> storeName >>>>>>>>>>>>>>>>> parameter and other method with mandatory storeName parameter >>>>>> -- >>>>>>>>>>>> thus, >>>>>>>>>>>>>>>>> still some inconsistency. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 4/9/17 8:35 AM, Eno Thereska wrote: >>>>>>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I've now done a V2 of the KIP, that hopefully addresses the >>>>>> feedback >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> this discussion thread: >>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>>>>>>>>> 114%3A+KTable+materialization+and+improved+semantics >>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>>>>>>>>> 114:+KTable+materialization+and+improved+semantics>. >>>>>>>>>>>>>>>> Notable changes: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> - clearly outline what is in the scope of the KIP and what is >>>>>> not. >>>>>>>>>>>> We >>>>>>>>>>>>>>>> ran into the issue where lots of useful, but somewhat >>>>>>>>>>>>>>>> tangential >>>>>>>>>>>>>>>> discussions came up on interactive queries, declarative DSL >>>>>> etc. The >>>>>>>>>>>>>>> exact >>>>>>>>>>>>>>>> scope of this KIP is spelled out. >>>>>>>>>>>>>>>>>> - decided to go with overloaded methods, not .materialize(), >>>>>> to stay >>>>>>>>>>>>>>>> within the spirit of the current declarative DSL. >>>>>>>>>>>>>>>>>> - clarified the depreciation plan >>>>>>>>>>>>>>>>>> - listed part of the discussion we had under rejected >>>>>> alternatives >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> If you have any further feedback on this, let's continue on >>>>>> this >>>>>>>>>>>>>>> thread. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thank you >>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska < >>>>>> eno.there...@gmail.com> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP so >>>>>> I'll do >>>>>>>>>>>>>>>> that and we can see how it looks and continue the discussion >>>>>> from >>>>>>>>>>>> there. >>>>>>>>>>>>>>>> Stay tuned. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax < >>>>>> matth...@confluent.io> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I think Eno's separation is very clear and helpful. In >>>>>> order to >>>>>>>>>>>>>>>>>>>> streamline this discussion, I would suggest we focus back >>>>>> on point >>>>>>>>>>>>>>> (1) >>>>>>>>>>>>>>>>>>>> only, as this is the original KIP question. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Even if I started to DSL design discussion somehow, because >>>>>> I >>>>>>>>>>>>>>> thought >>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>> might be helpful to resolve both in a single shot, I feel >>>>>> that we >>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>> too many options about DSL design and we should split it up >>>>>> in two >>>>>>>>>>>>>>>>>>>> steps. This will have the disadvantage that we will change >>>>>> the API >>>>>>>>>>>>>>>>>>>> twice, but still, I think it will be a more focused >>>>>> discussion. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I just had another look at the KIP, an it proposes 3 >>>>>> changes: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 1. add .materialized() -> IIRC it was suggested to name >>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>> .materialize() though (can you maybe update the KIP Eno?) >>>>>>>>>>>>>>>>>>>> 2. remove print(), writeAsText(), and foreach() >>>>>>>>>>>>>>>>>>>> 3. rename toStream() to toKStream() >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I completely agree with (2) -- not sure about (3) though >>>>>> because >>>>>>>>>>>>>>>>>>>> KStreamBuilder also hast .stream() and .table() as methods. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> However, we might want to introduce a KStream#toTable() -- >>>>>> this >>>>>>>>>>>> was >>>>>>>>>>>>>>>>>>>> requested multiple times -- might also be part of a >>>>>> different KIP. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thus, we end up with (1). I would suggest to do a step >>>>>> backward >>>>>>>>>>>> here >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>> instead of a discussion how to express the changes in the >>>>>> DSL (new >>>>>>>>>>>>>>>>>>>> overload, new methods...) we should discuss what the actual >>>>>> change >>>>>>>>>>>>>>>>>>>> should be. Like (1) materialize all KTable all the time (2) >>>>>> all >>>>>>>>>>>> the >>>>>>>>>>>>>>>> user >>>>>>>>>>>>>>>>>>>> to force a materialization to enable querying the KTable >>>>>> (3) allow >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>> queryable non-materialized KTable. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On more question is, if we want to allow a user-forced >>>>>>>>>>>>>>> materialization >>>>>>>>>>>>>>>>>>>> only as as local store without changelog, or both (together >>>>>> / >>>>>>>>>>>>>>>>>>>> independently)? We got some request like this already. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote: >>>>>>>>>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> thanks for putting into different points. I want to put a >>>>>> few >>>>>>>>>>>>>>> remarks >>>>>>>>>>>>>>>>>>>>> inline. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>>> So I think there are several important discussion threads >>>>>> that >>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>> emerging here. Let me try to tease them apart: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 1. inconsistency in what is materialized and what is not, >>>>>> what >>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> queryable and what is not. I think we all agree there is >>>>>> some >>>>>>>>>>>>>>>>>>>>>> inconsistency there and this will be addressed with any >>>>>> of the >>>>>>>>>>>>>>>>>>>>>> proposed approaches. Addressing the inconsistency is the >>>>>> point >>>>>>>>>>>> of >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> original KIP. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 2. the exact API for materializing a KTable. We can >>>>>> specify 1) a >>>>>>>>>>>>>>>>>>>>>> "store name" (as we do today) or 2) have a >>>>>> ".materialize[d]" >>>>>>>>>>>> call >>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4) >>>>>> have a >>>>>>>>>>>>>>> builder >>>>>>>>>>>>>>>>>>>>>> construct. So we have discussed 4 options. It is >>>>>> important to >>>>>>>>>>>>>>>> remember >>>>>>>>>>>>>>>>>>>>>> in this discussion that IQ is not designed for just local >>>>>>>>>>>> queries, >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>> also for distributed queries. In all cases an identifying >>>>>>>>>>>> name/id >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> needed for the store that the user is interested in >>>>>> querying. So >>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>> end up with a discussion on who provides the name, the >>>>>> user (as >>>>>>>>>>>>>>> done >>>>>>>>>>>>>>>>>>>>>> today) or if it is generated automatically (as Jan >>>>>> suggests, as >>>>>>>>>>>> I >>>>>>>>>>>>>>>>>>>>>> understand it). If it is generated automatically we need >>>>>> a way >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>> expose these auto-generated names to the users and link >>>>>> them to >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> KTables they care to query. >>>>>>>>>>>>>>>>>>>>> Hi, the last sentence is what I currently arguing against. >>>>>> The >>>>>>>>>>>> user >>>>>>>>>>>>>>>>>>>>> would never see a stringtype indentifier name or anything. >>>>>> All he >>>>>>>>>>>>>>>> gets >>>>>>>>>>>>>>>>>>>>> is the queryHandle if he executes a get(K) that will be an >>>>>>>>>>>>>>>> interactive >>>>>>>>>>>>>>>>>>>>> query get. with all the finding the right servers that >>>>>> currently >>>>>>>>>>>>>>>> have a >>>>>>>>>>>>>>>>>>>>> copy of this underlying store stuff going on. The nice >>>>>> part is >>>>>>>>>>>> that >>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>>>>>> someone retrieves a queryHandle, you know that you have to >>>>>>>>>>>>>>>> materialized >>>>>>>>>>>>>>>>>>>>> (if you are not already) as queries will be coming. Taking >>>>>> away >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>> confusion mentioned in point 1 IMO. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 3. The exact boundary between the DSL, that is the >>>>>> processing >>>>>>>>>>>>>>>>>>>>>> language, and the storage/IQ queries, and how we jump >>>>>> from one >>>>>>>>>>>> to >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>> other. This is mostly for how we get a handle on a store >>>>>> (so >>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>>>>> related to point 2), rather than for how we query the >>>>>> store. I >>>>>>>>>>>>>>> think >>>>>>>>>>>>>>>>>>>>>> we all agree that we don't want to limit ways one can >>>>>> query a >>>>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>>>> (e.g., using gets or range queries etc) and the query >>>>>> APIs are >>>>>>>>>>>> not >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>> the scope of the DSL. >>>>>>>>>>>>>>>>>>>>> Does the IQ work with range currently? The range would >>>>>> have to be >>>>>>>>>>>>>>>>>>>>> started on all stores and then merged by maybe the client. >>>>>> Range >>>>>>>>>>>>>>>> force a >>>>>>>>>>>>>>>>>>>>> flush to RocksDB currently so I am sure you would get a >>>>>>>>>>>> performance >>>>>>>>>>>>>>>> hit >>>>>>>>>>>>>>>>>>>>> right there. Time-windows might be okay, but I am not sure >>>>>> if the >>>>>>>>>>>>>>>> first >>>>>>>>>>>>>>>>>>>>> version should offer the user range access. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> 4. The nature of the DSL and whether its declarative >>>>>> enough, or >>>>>>>>>>>>>>>>>>>>>> flexible enough. Damian made the point that he likes the >>>>>> builder >>>>>>>>>>>>>>>>>>>>>> pattern since users can specify, per KTable, things like >>>>>> caching >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>> logging needs. His observation (as I understand it) is >>>>>> that the >>>>>>>>>>>>>>>>>>>>>> processor API (PAPI) is flexible but doesn't provide any >>>>>> help at >>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>> to users. The current DSL provides declarative >>>>>> abstractions, but >>>>>>>>>>>>>>>> it's >>>>>>>>>>>>>>>>>>>>>> not fine-grained enough. This point is much broader than >>>>>> the >>>>>>>>>>>> KIP, >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>> discussing it in this KIPs context is ok, since we don't >>>>>> want to >>>>>>>>>>>>>>>> make >>>>>>>>>>>>>>>>>>>>>> small piecemeal changes and then realise we're not in the >>>>>> spot >>>>>>>>>>>> we >>>>>>>>>>>>>>>> want >>>>>>>>>>>>>>>>>>>>>> to be. >>>>>>>>>>>>>>>>>>>>> This is indeed much broader. My guess here is that's why >>>>>> both >>>>>>>>>>>> API's >>>>>>>>>>>>>>>>>>>>> exists and helping the users to switch back and forth >>>>>> might be a >>>>>>>>>>>>>>>> thing. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Feel free to pitch in if I have misinterpreted something. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak < >>>>>>>>>>>> jan.filip...@trivago.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> I have a really hard time understanding why we can't. >>>>>> From my >>>>>>>>>>>>>>> point >>>>>>>>>>>>>>>>>>>>>>> of view everything could be super elegant DSL only + >>>>>> public api >>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>> the PAPI-people as already exist. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> The above aproach implementing a .get(K) on KTable is >>>>>> foolisch >>>>>>>>>>>> in >>>>>>>>>>>>>>>> my >>>>>>>>>>>>>>>>>>>>>>> opinion as it would be to late to know that >>>>>> materialisation >>>>>>>>>>>> would >>>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>> required. >>>>>>>>>>>>>>>>>>>>>>> But having an API that allows to indicate I want to >>>>>> query this >>>>>>>>>>>>>>>> table >>>>>>>>>>>>>>>>>>>>>>> and then wrapping the say table's processorname can work >>>>>> out >>>>>>>>>>>>>>> really >>>>>>>>>>>>>>>>>>>>>>> really nice. The only obstacle I see is people not >>>>>> willing to >>>>>>>>>>>>>>> spend >>>>>>>>>>>>>>>>>>>>>>> the additional time in implementation and just want a >>>>>> quick >>>>>>>>>>>> shot >>>>>>>>>>>>>>>>>>>>>>> option to make it work. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> For me it would look like this: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> table = builder.table() >>>>>>>>>>>>>>>>>>>>>>> filteredTable = table.filter() >>>>>>>>>>>>>>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the >>>>>> materialisation, >>>>>>>>>>>>>>>>>>>>>>> really all names possible but id rather hide the >>>>>> implication of >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>> materializes >>>>>>>>>>>>>>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() // >>>>>> this >>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>> _not_ materialize again of course, the source or the >>>>>> aggregator >>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>> stay the only materialized processors >>>>>>>>>>>>>>>>>>>>>>> streams = new streams(builder) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> This middle part is highly flexible I could imagin to >>>>>> force the >>>>>>>>>>>>>>>> user >>>>>>>>>>>>>>>>>>>>>>> todo something like this. This implies to the user that >>>>>> his >>>>>>>>>>>>>>> streams >>>>>>>>>>>>>>>>>>>>>>> need to be running >>>>>>>>>>>>>>>>>>>>>>> instead of propagating the missing initialisation back >>>>>>>>>>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>> exceptions. >>>>>>>>>>>>>>>>>>>>>>> Also if the users is forced to pass the appropriate >>>>>> streams >>>>>>>>>>>>>>>> instance >>>>>>>>>>>>>>>>>>>>>>> back can change. >>>>>>>>>>>>>>>>>>>>>>> I think its possible to build multiple streams out of >>>>>> one >>>>>>>>>>>>>>> topology >>>>>>>>>>>>>>>>>>>>>>> so it would be easiest to implement aswell. This is just >>>>>> what I >>>>>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>>>>> had liked the most >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> streams.start(); >>>>>>>>>>>>>>>>>>>>>>> rawHandle.prepare(streams) >>>>>>>>>>>>>>>>>>>>>>> filteredHandle.prepare(streams) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> later the users can do >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> V value = rawHandle.get(K) >>>>>>>>>>>>>>>>>>>>>>> V value = filteredHandle.get(K) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> This could free DSL users from anything like storenames >>>>>> and how >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>> what to materialize. Can someone indicate what the >>>>>> problem >>>>>>>>>>>> would >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>> implementing it like this. >>>>>>>>>>>>>>>>>>>>>>> Yes I am aware that the current IQ API will not support >>>>>>>>>>>> querying >>>>>>>>>>>>>>> by >>>>>>>>>>>>>>>>>>>>>>> KTableProcessorName instread of statestoreName. But I >>>>>> think >>>>>>>>>>>> that >>>>>>>>>>>>>>>> had >>>>>>>>>>>>>>>>>>>>>>> to change if you want it to be intuitive >>>>>>>>>>>>>>>>>>>>>>> IMO you gotta apply the filter read time >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Looking forward to your opinions >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> The inconsistency will be resolved, whether with >>>>>> materialize >>>>>>>>>>>> or >>>>>>>>>>>>>>>>>>>>>>>> overloaded methods. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> With the discussion on the DSL & stores I feel we've >>>>>> gone in a >>>>>>>>>>>>>>>>>>>>>>>> slightly different tangent, which is worth discussing >>>>>>>>>>>>>>> nonetheless. >>>>>>>>>>>>>>>>>>>>>>>> We have entered into an argument around the scope of >>>>>> the DSL. >>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>> DSL has been designed primarily for processing. The DSL >>>>>> does >>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>> dictate ways to access state stores or what hind of >>>>>> queries to >>>>>>>>>>>>>>>>>>>>>>>> perform on them. Hence, I see the mechanism for >>>>>> accessing >>>>>>>>>>>>>>> storage >>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>> decoupled from the DSL. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> We could think of ways to get store handles from part >>>>>> of the >>>>>>>>>>>>>>> DSL, >>>>>>>>>>>>>>>>>>>>>>>> like the KTable abstraction. However, subsequent >>>>>> queries will >>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm not >>>>>> sure we >>>>>>>>>>>>>>> get >>>>>>>>>>>>>>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing >>>>>> that the >>>>>>>>>>>>>>>>>>>>>>>> current way of getting a handle on state stores is >>>>>>>>>>>>>>>>>>>>>>>> fine. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang < >>>>>> wangg...@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thinking loud here about the API options (materialize >>>>>> v.s. >>>>>>>>>>>>>>>> overloaded >>>>>>>>>>>>>>>>>>>>>>>>> functions) and its impact on IQ: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 1. The first issue of the current DSL is that, there >>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>> inconsistency upon >>>>>>>>>>>>>>>>>>>>>>>>> whether / how KTables should be materialized: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> a) in many cases the library HAS TO materialize >>>>>> KTables no >>>>>>>>>>>>>>>>>>>>>>>>> matter what, >>>>>>>>>>>>>>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables, >>>>>> and hence >>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>> enforce >>>>>>>>>>>>>>>>>>>>>>>>> users to provide store names and throw RTE if it is >>>>>> null; >>>>>>>>>>>>>>>>>>>>>>>>> b) in some other cases, the KTable can be materialized >>>>>> or >>>>>>>>>>>> not; >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>> example in KStreamBuilder.table(), store names can be >>>>>>>>>>>> nullable >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>> in which >>>>>>>>>>>>>>>>>>>>>>>>> case the KTable would not be materialized; >>>>>>>>>>>>>>>>>>>>>>>>> c) in some other cases, the KTable will never be >>>>>>>>>>>> materialized, >>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>> example KTable.filter() resulted KTables, and users >>>>>> have no >>>>>>>>>>>>>>>> options to >>>>>>>>>>>>>>>>>>>>>>>>> enforce them to be materialized; >>>>>>>>>>>>>>>>>>>>>>>>> d) this is related to a), where some KTables are >>>>>> required to >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>> materialized, but we do not enforce users to provide a >>>>>> state >>>>>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>>>>>>> name, >>>>>>>>>>>>>>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown >>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>> immediately but >>>>>>>>>>>>>>>>>>>>>>>>> later in this case. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 2. The second issue is related to IQ, where state >>>>>> stores are >>>>>>>>>>>>>>>>>>>>>>>>> accessed by >>>>>>>>>>>>>>>>>>>>>>>>> their state stores; so only those KTable's that have >>>>>>>>>>>>>>>> user-specified >>>>>>>>>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>>>>>>>>> stores will be queryable. But because of 1) above, >>>>>>>>>>>>>>>>>>>>>>>>> many >>>>>>>>>>>> stores >>>>>>>>>>>>>>>> may >>>>>>>>>>>>>>>>>>>>>>>>> not be >>>>>>>>>>>>>>>>>>>>>>>>> interested to users for IQ but they still need to >>>>>> provide a >>>>>>>>>>>>>>>>>>>>>>>>> (dummy?) state >>>>>>>>>>>>>>>>>>>>>>>>> store name for them; while on the other hand users >>>>>> cannot >>>>>>>>>>>> query >>>>>>>>>>>>>>>>>>>>>>>>> some state >>>>>>>>>>>>>>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() as >>>>>> there >>>>>>>>>>>> is >>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>>>> APIs for >>>>>>>>>>>>>>>>>>>>>>>>> them to specify a state store name. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> 3. We are aware from user feedbacks that such backend >>>>>> details >>>>>>>>>>>>>>>> would be >>>>>>>>>>>>>>>>>>>>>>>>> better be abstracted away from the DSL layer, where >>>>>>>>>>>>>>>>>>>>>>>>> app >>>>>>>>>>>>>>>> developers >>>>>>>>>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>> just focus on processing logic, while state stores >>>>>> along with >>>>>>>>>>>>>>>> their >>>>>>>>>>>>>>>>>>>>>>>>> changelogs etc would better be in a different >>>>>> mechanism; same >>>>>>>>>>>>>>>>>>>>>>>>> arguments >>>>>>>>>>>>>>>>>>>>>>>>> have been discussed for serdes / windowing triggers as >>>>>> well. >>>>>>>>>>>>>>> For >>>>>>>>>>>>>>>>>>>>>>>>> serdes >>>>>>>>>>>>>>>>>>>>>>>>> specifically, we had a very long discussion about it >>>>>> and >>>>>>>>>>>>>>>> concluded >>>>>>>>>>>>>>>>>>>>>>>>> that, at >>>>>>>>>>>>>>>>>>>>>>>>> least in Java7, we cannot completely abstract serde >>>>>> away in >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> DSL, so we >>>>>>>>>>>>>>>>>>>>>>>>> choose the other extreme to enforce users to be >>>>>> completely >>>>>>>>>>>>>>> aware >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> serde requirements when some KTables may need to be >>>>>>>>>>>>>>> materialized >>>>>>>>>>>>>>>> vis >>>>>>>>>>>>>>>>>>>>>>>>> overloaded API functions. While for the state store >>>>>> names, I >>>>>>>>>>>>>>> feel >>>>>>>>>>>>>>>>>>>>>>>>> it is a >>>>>>>>>>>>>>>>>>>>>>>>> different argument than serdes (details below). >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded >>>>>> functions >>>>>>>>>>>>>>>>>>>>>>>>> directions, >>>>>>>>>>>>>>>>>>>>>>>>> the first thing I'd like to resolve is the >>>>>> inconsistency >>>>>>>>>>>> issue >>>>>>>>>>>>>>>>>>>>>>>>> mentioned >>>>>>>>>>>>>>>>>>>>>>>>> above. So in either case: KTable materialization will >>>>>> not be >>>>>>>>>>>>>>>> affect >>>>>>>>>>>>>>>>>>>>>>>>> by user >>>>>>>>>>>>>>>>>>>>>>>>> providing state store name or not, but will only be >>>>>> decided >>>>>>>>>>>> by >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> library >>>>>>>>>>>>>>>>>>>>>>>>> when it is necessary. More specifically, only join >>>>>> operator >>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>> builder.table() resulted KTables are not always >>>>>> materialized, >>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>> are still >>>>>>>>>>>>>>>>>>>>>>>>> likely to be materialized lazily (e.g. when >>>>>> participated in a >>>>>>>>>>>>>>>> join >>>>>>>>>>>>>>>>>>>>>>>>> operator). >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> For overloaded functions that would mean: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> a) we have an overloaded function for ALL operators >>>>>> that >>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>>>>>>>> result >>>>>>>>>>>>>>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the >>>>>> function >>>>>>>>>>>>>>>> without >>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>> param it is null by default); >>>>>>>>>>>>>>>>>>>>>>>>> b) null-state-store-name do not indicate that a KTable >>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>> not be >>>>>>>>>>>>>>>>>>>>>>>>> materialized, but that it will not be used for IQ at >>>>>> all >>>>>>>>>>>>>>>> (internal >>>>>>>>>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>>>>>>>>> store names will be generated when necessary). >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> For materialize() that would mean: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> a) we will remove state store names from ALL operators >>>>>> that >>>>>>>>>>>>>>>> could >>>>>>>>>>>>>>>>>>>>>>>>> result in a KTable. >>>>>>>>>>>>>>>>>>>>>>>>> b) KTables that not calling materialized do not >>>>>> indicate that >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>> would not be materialized, but that it will not be >>>>>> used for >>>>>>>>>>>> IQ >>>>>>>>>>>>>>>> at all >>>>>>>>>>>>>>>>>>>>>>>>> (internal state store names will be generated when >>>>>>>>>>>> necessary). >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Again, in either ways the API itself does not "hint" >>>>>> about >>>>>>>>>>>>>>>> anything >>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>> materializing a KTable or not at all; it is still >>>>>> purely >>>>>>>>>>>>>>>> determined >>>>>>>>>>>>>>>>>>>>>>>>> by the >>>>>>>>>>>>>>>>>>>>>>>>> library when parsing the DSL for now. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Following these thoughts, I feel that 1) we should >>>>>> probably >>>>>>>>>>>>>>>> change >>>>>>>>>>>>>>>>>>>>>>>>> the name >>>>>>>>>>>>>>>>>>>>>>>>> "materialize" since it may be misleading to users as >>>>>> what >>>>>>>>>>>>>>>> actually >>>>>>>>>>>>>>>>>>>>>>>>> happened >>>>>>>>>>>>>>>>>>>>>>>>> behind the scene, to e.g. Damian suggested >>>>>>>>>>>>>>> "queryableStore(String >>>>>>>>>>>>>>>>>>>>>>>>> storeName)", >>>>>>>>>>>>>>>>>>>>>>>>> which returns a QueryableStateStore, and can replace >>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those two >>>>>> options >>>>>>>>>>>>>>>>>>>>>>>>> assuming we >>>>>>>>>>>>>>>>>>>>>>>>> get rid of the misleading function name, I personally >>>>>> favor >>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>> adding more >>>>>>>>>>>>>>>>>>>>>>>>> overloading functions as it keeps the API simpler. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Guozhang >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak >>>>>>>>>>>>>>>>>>>>>>>>> <jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> thanks for your mail, felt like this can clarify some >>>>>>>>>>>> things! >>>>>>>>>>>>>>>> The >>>>>>>>>>>>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>>>>>>>>>>>> unfortunately split but as all branches close in on >>>>>> what my >>>>>>>>>>>>>>>>>>>>>>>>>> suggestion was >>>>>>>>>>>>>>>>>>>>>>>>>> about Ill pick this to continue >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Of course only the table the user wants to query >>>>>> would be >>>>>>>>>>>>>>>>>>>>>>>>>> materialized. >>>>>>>>>>>>>>>>>>>>>>>>>> (retrieving the queryhandle implies materialisation). >>>>>> So In >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> example of >>>>>>>>>>>>>>>>>>>>>>>>>> KTable::filter if you call >>>>>>>>>>>>>>>>>>>>>>>>>> getIQHandle on both tables only the one source that >>>>>> is there >>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>> materialize and the QueryHandleabstraction would make >>>>>> sure >>>>>>>>>>>> it >>>>>>>>>>>>>>>> gets >>>>>>>>>>>>>>>>>>>>>>>>>> mapped >>>>>>>>>>>>>>>>>>>>>>>>>> and filtered and what not uppon read as usual. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe >>>>>> only >>>>>>>>>>>> wrap >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> storeName / table unique identifier and a way to >>>>>> access the >>>>>>>>>>>>>>>> streams >>>>>>>>>>>>>>>>>>>>>>>>>> instance and then basically uses the same mechanism >>>>>> that is >>>>>>>>>>>>>>>>>>>>>>>>>> currently used. >>>>>>>>>>>>>>>>>>>>>>>>>> From my point of view this is the least confusing way >>>>>> for >>>>>>>>>>>> DSL >>>>>>>>>>>>>>>>>>>>>>>>>> users. If >>>>>>>>>>>>>>>>>>>>>>>>>> its to tricky to get a hand on the streams instance >>>>>> one >>>>>>>>>>>> could >>>>>>>>>>>>>>>> ask >>>>>>>>>>>>>>>>>>>>>>>>>> the user >>>>>>>>>>>>>>>>>>>>>>>>>> to pass it in before executing queries, therefore >>>>>> making >>>>>>>>>>>> sure >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> streams >>>>>>>>>>>>>>>>>>>>>>>>>> instance has been build. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> The effort to implement this is indeed some orders of >>>>>>>>>>>>>>> magnitude >>>>>>>>>>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>>>>>>>>> than the overloaded materialized call. As long as I >>>>>> could >>>>>>>>>>>> help >>>>>>>>>>>>>>>>>>>>>>>>>> getting a >>>>>>>>>>>>>>>>>>>>>>>>>> different view I am happy. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jan, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your concern. One implication of not >>>>>> passing >>>>>>>>>>>> any >>>>>>>>>>>>>>>>>>>>>>>>>>> store name >>>>>>>>>>>>>>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables >>>>>> would >>>>>>>>>>>> need >>>>>>>>>>>>>>>> to be >>>>>>>>>>>>>>>>>>>>>>>>>>> materialised. Currently the store name (or proposed >>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() call) >>>>>>>>>>>>>>>>>>>>>>>>>>> act as hints on whether to materialise the KTable or >>>>>> not. >>>>>>>>>>>>>>>>>>>>>>>>>>> Materialising >>>>>>>>>>>>>>>>>>>>>>>>>>> every KTable can be expensive, although there are >>>>>> some >>>>>>>>>>>> tricks >>>>>>>>>>>>>>>> one >>>>>>>>>>>>>>>>>>>>>>>>>>> can play, >>>>>>>>>>>>>>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed by >>>>>> a >>>>>>>>>>>> Kafka >>>>>>>>>>>>>>>> topic. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> However, even with the above, after getting an IQ >>>>>> handle, >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> user would >>>>>>>>>>>>>>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As >>>>>> such, we >>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>>>>>>>>> continue to be outside the original DSL so this >>>>>> wouldn't >>>>>>>>>>>>>>>> address >>>>>>>>>>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>>>>>>>>> original concern. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs by >>>>>>>>>>>> removing >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>>>>>>>>> name, at the cost of having to materialise every >>>>>> KTable. >>>>>>>>>>>> It's >>>>>>>>>>>>>>>>>>>>>>>>>>> definitely an >>>>>>>>>>>>>>>>>>>>>>>>>>> option we'll consider as part of this KIP. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak < >>>>>>>>>>>>>>>> jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Exactly >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I know it works from the Processor API, but my >>>>>> suggestion >>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>> prevent >>>>>>>>>>>>>>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> In general I am pro switching between DSL and >>>>>> Processor >>>>>>>>>>>> API >>>>>>>>>>>>>>>>>>>>>>>>>>>> easily. (In >>>>>>>>>>>>>>>>>>>>>>>>>>>> my Stream applications I do this a lot with >>>>>> reflection and >>>>>>>>>>>>>>>>>>>>>>>>>>>> instanciating >>>>>>>>>>>>>>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that >>>>>> there >>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>>>>>>>>>>>> be a DSL >>>>>>>>>>>>>>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. This >>>>>> can be >>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>> Method like >>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable::retrieveIQHandle():InteractiveQueryHandle, >>>>>> the >>>>>>>>>>>>>>> table >>>>>>>>>>>>>>>>>>>>>>>>>>>> would know >>>>>>>>>>>>>>>>>>>>>>>>>>>> to materialize, and the user had a reference to the >>>>>> "store >>>>>>>>>>>>>>>> and the >>>>>>>>>>>>>>>>>>>>>>>>>>>> distributed query mechanism by the Interactive >>>>>>>>>>>>>>>>>>>>>>>>>>>> Query >>>>>>>>>>>> Handle" >>>>>>>>>>>>>>>>>>>>>>>>>>>> under the hood >>>>>>>>>>>>>>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people >>>>>> again. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I hope you see my point J >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jan, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but >>>>>> can also >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>> used for >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism >>>>>> that does >>>>>>>>>>>>>>> work >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for PAPI >>>>>>>>>>>>>>>>>>>>>>>>>>>>> and DSL. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we could >>>>>>>>>>>> provide >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> better API >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for KTable stores including the discovery of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> remote >>>>>>>>>>>> shards >>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the same >>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of >>>>>> overloads >>>>>>>>>>>> and >>>>>>>>>>>>>>> I >>>>>>>>>>>>>>>> am >>>>>>>>>>>>>>>>>>>>>>>>>>>>> not a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a >>>>>>>>>>>>>>>>>>>>>>>>>>>>> builder >>>>>>>>>>>>>>> pattern. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> But that >>>>>>>>>>>>>>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, if >>>>>> we >>>>>>>>>>>> would >>>>>>>>>>>>>>>> aim >>>>>>>>>>>>>>>>>>>>>>>>>>>>> for a API >>>>>>>>>>>>>>>>>>>>>>>>>>>>> rework, we should get the changes with regard to >>>>>> stores >>>>>>>>>>>>>>> right >>>>>>>>>>>>>>>>>>>>>>>>>>>>> from the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later >>>>>> on.) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> something like: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> stream.groupyByKey() >>>>>>>>>>>>>>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000)) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> .aggregate(...) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde()) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> .withStoreName("storeName); >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy -- >>>>>> maybe a >>>>>>>>>>>>>>>> personal >>>>>>>>>>>>>>>>>>>>>>>>>>>>> pain >>>>>>>>>>>>>>>>>>>>>>>>>>>>> point right now :)) >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i >>>>>> don't >>>>>>>>>>>> find >>>>>>>>>>>>>>>> them >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> anywhere >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is >>>>>> that >>>>>>>>>>>> people >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to know >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the Store name), so we are working on different >>>>>> levels >>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> achieve a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> single goal. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What is your peoples opinion on having a method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>> KTABLE >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> returns >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is of >>>>>> course >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> problems like >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are >>>>>> going and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> groupmembership >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is established..." but the benefit would be that >>>>>> for the >>>>>>>>>>>>>>>> user >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> there is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it >>>>>> materialized as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> querries gonna >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can >>>>>> execute >>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> querries on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1 step. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to >>>>>> say >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialize on this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find >>>>>> its store >>>>>>>>>>>>>>>> name >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and then >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for >>>>>> the >>>>>>>>>>>> store >>>>>>>>>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> therefore >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> confuse him less. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always >>>>>> need to be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e., >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it >>>>>> doesn't >>>>>>>>>>>>>>>> need yet >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> another >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> physical state store. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll < >>>>>>>>>>>>>>>> mich...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am more >>>>>> in >>>>>>>>>>>> favor >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloading >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods rather than introducing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `materialize()`. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for >>>>>> e.g. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A related but slightly different question is >>>>>> what e.g. >>>>>>>>>>>>>>> Jan >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipiak >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why >>>>>> KIP-114 >>>>>>>>>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> propose >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always >>>>>> materializing >>>>>>>>>>>>>>>> tables/state >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> stores. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be >>>>>> querable by >>>>>>>>>>>> IQ? >>>>>>>>>>>>>>> If >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you uses >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the filter >>>>>> and >>>>>>>>>>>>>>>> should be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> completely >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or IQ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>> accessing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it? How >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new method help? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional >>>>>>>>>>>> materialize >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method being >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have >>>>>> strong >>>>>>>>>>>>>>> opinions >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> seems to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> be unrelated. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too >>>>>> in case >>>>>>>>>>>>>>>> people >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" < >>>>>> matth...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable >>>>>>>>>>>>>>> materialization >>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> improved >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> semantics >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact". >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to >>>>>> materialize a >>>>>>>>>>>>>>> KTable: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> creating a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- users >>>>>> should >>>>>>>>>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> aware about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better >>>>>> expressed by >>>>>>>>>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by >>>>>> using a >>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overload of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a method. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact and >>>>>> mine >>>>>>>>>>>> are >>>>>>>>>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ;-P >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not really a >>>>>> huge >>>>>>>>>>>>>>> impact >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMO. It is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability, >>>>>> usability of >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> API. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J. >>>>>> Sax < >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not >>>>>> agree with >>>>>>>>>>>>>>> it. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" is >>>>>> not as >>>>>>>>>>>>>>>> nice) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> than the second version. Adding a >>>>>> stateStoreName >>>>>>>>>>>>>>>> parameter >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is quite >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- thus, I >>>>>> prefer >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verbose >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but explicit version. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think it >>>>>>>>>>>> interrupts >>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> table.mapValue(..).materialize().join(..).materialize() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide >>>>>> overloaded >>>>>>>>>>>>>>> methods >>>>>>>>>>>>>>>> where >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> people can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want, >>>>>> otherwise >>>>>>>>>>>> we >>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2 >>>>>> cents: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about >>>>>> removing >>>>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names >>>>>> (however, I >>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not >>>>>> force >>>>>>>>>>>> users >>>>>>>>>>>>>>>> to call >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but add >>>>>> one more >>>>>>>>>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the >>>>>> KTable is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thus, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a >>>>>> parameter >>>>>>>>>>>>>>>> storeName >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (ie, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a >>>>>> null store >>>>>>>>>>>>>>>> name (to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but >>>>>> throw an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see >>>>>> below). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call >>>>>> materialize >>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another >>>>>>>>>>>> StateStore >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but >>>>>> see >>>>>>>>>>>> below. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from >>>>>> Guozhang, there >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worry >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no >>>>>> exception >>>>>>>>>>>>>>>> must be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a >>>>>> "materialized >>>>>>>>>>>>>>>> flag" (ie, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for >>>>>>>>>>>> consistency. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really >>>>>> required. We >>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>> use >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream` >>>>>> would make >>>>>>>>>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the >>>>>>>>>>>>>>>> KStreamBuilder >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this KIP). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods but >>>>>> only >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A general note: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments >>>>>> "Rejected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" was >>>>>>>>>>>>>>> rejected. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But your >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog >>>>>> abstraction >>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> secondary >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> after those changes and the "view" abstraction >>>>>> is >>>>>>>>>>>> what >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view >>>>>>>>>>>> abstraction" >>>>>>>>>>>>>>>> (as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> optional). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few >>>>>> meta >>>>>>>>>>>>>>> comments >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a few >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() function in >>>>>>>>>>>> general, >>>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be >>>>>> updated >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> -- Guozhang >>>> >>> >> >
