Hi Matthias,
> On 11 Apr 2017, at 09:41, Matthias J. Sax <matth...@confluent.io> wrote: > > Not sure, if we are on the same page already? > >> "A __store__ can be queryable whether is't materialized or not" > > This does not make sense -- there is nothing like a non-materialized > store -- only non-materialized KTables. Yes, there are stores that are simple views, i.e., non-materialized. Damian has such a prototype for Global Tables (it didn't go into trunk). It's still a store, e.g., a KeyValueStore, but when you do a get() it recomputes the result on the fly (e.g., it applies a filter). Eno > >> "Yes, there is nothing that will prevent users from querying > internally generated stores, but they cannot assume a store will > necessarily be queryable." > > That is what I disagree on. Stores should be queryable all the time. > > Furthermore, we should have all non-materialized KTables to be > queryable, too. > > > Or maybe there is just some missunderstand going as, and there is some > mix-up between "store" and "KTable" > > > > -Matthias > > > On 4/11/17 9:34 AM, Eno Thereska wrote: >> Hi Matthias, >> >> See my note: "A store can be queryable whether it's materialized or not". I >> think we're on the same page. Stores with an internal name are also >> queryable. >> >> I'm just pointing out that. although that is the case today and with this >> KIP, I don't think we have an obligation to make stores with internal names >> queryable in the future. However, that is a discussion for a future point. >> >> Eno >> >> >> >> >>> On 11 Apr 2017, at 08:56, Matthias J. Sax <matth...@confluent.io> wrote: >>> >>> +1 on including GlobalKTable >>> >>> But I am not sure about the materialization / queryable question. For >>> full consistency, all KTables should be queryable nevertheless if they >>> are materialized or not. -- Maybe this is a second step though (even if >>> I would like to get this done right away) >>> >>> If we don't want all KTables to be queryable, ie, only those KTables >>> that are materialized, then we should have a clear definition about >>> this, and only allow to query stores, the user did specify a name for. >>> This will simply the reasoning for users, what stores are queryable and >>> what not. Otherwise, we still end up confusing user. >>> >>> >>> -Matthias >>> >>> On 4/11/17 8:23 AM, Damian Guy wrote: >>>> Eno, re: GlobalKTable - yeah that seems fine. >>>> >>>> On Tue, 11 Apr 2017 at 14:18 Eno Thereska <eno.there...@gmail.com> wrote: >>>> >>>>> About GlobalKTables, I suppose there is no reason why they cannot also use >>>>> this KIP for consistency, e.g., today you have: >>>>> >>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> keySerde, >>>>> final Serde<V> valSerde, >>>>> final String topic, >>>>> final String storeName) >>>>> >>>>> For consistency with the KIP you could also have an overload without the >>>>> store name, for people who want to construct a global ktable, but don't >>>>> care about querying it directly: >>>>> >>>>> public <K, V> GlobalKTable<K, V> globalTable(final Serde<K> keySerde, >>>>> final Serde<V> valSerde, >>>>> final String topic) >>>>> >>>>> Damian, what do you think? I'm thinking of adding this to KIP. Thanks to >>>>> Michael for bringing it up. >>>>> >>>>> Eno >>>>> >>>>> >>>>> >>>>>> On 11 Apr 2017, at 06:13, Eno Thereska <eno.there...@gmail.com> wrote: >>>>>> >>>>>> Hi Michael, comments inline: >>>>>> >>>>>>> On 11 Apr 2017, at 03:25, Michael Noll <mich...@confluent.io> wrote: >>>>>>> >>>>>>> Thanks for the updates, Eno! >>>>>>> >>>>>>> In addition to what has already been said: We should also explicitly >>>>>>> mention that this KIP is not touching GlobalKTable. I'm sure that some >>>>>>> users will throw KTable and GlobalKTable into one conceptual "it's all >>>>>>> tables!" bucket and then wonder how the KIP might affect global tables. >>>>>> >>>>>> Good point, I'll add. >>>>>> >>>>>> >>>>>>> >>>>>>> Damian wrote: >>>>>>>> I think if no store name is provided users would still be able to query >>>>>>> the >>>>>>>> store, just the store name would be some internally generated name. >>>>> They >>>>>>>> would be able to discover those names via the IQ API. >>>>>>> >>>>>>> I, too, think that users should be able to query a store even if its >>>>> name >>>>>>> was internally generated. After all, the data is already there / >>>>>>> materialized. >>>>>> >>>>>> Yes, there is nothing that will prevent users from querying internally >>>>> generated stores, but they cannot >>>>>> assume a store will necessarily be queryable. So if it's there, they can >>>>> query it. If it's not there, and they didn't >>>>>> provide a queryable name, they cannot complain and say "hey, where is my >>>>> store". If they must absolutely be certain that >>>>>> a store is queryable, then they must provide a queryable name. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>> Damian wrote: >>>>>>>> I think for some stores it will make sense to not create a physical >>>>>>> store, i.e., >>>>>>>> for thinks like `filter`, as this will save the rocksdb overhead. But i >>>>>>> guess that >>>>>>>> is more of an implementation detail. >>>>>>> >>>>>>> I think it would help if the KIP would clarify what we'd do in such a >>>>>>> case. For example, if the user did not specify a store name for >>>>>>> `KTable#filter` -- would it be queryable? If so, would this imply we'd >>>>>>> always materialize the state store, or...? >>>>>> >>>>>> I'll clarify in the KIP with some more examples. Materialization will be >>>>> an internal concept. A store can be queryable whether it's materialized or >>>>> not >>>>>> (e.g., through advanced implementations that compute the value of a >>>>> filter on a fly, rather than materialize the answer). >>>>>> >>>>>> Thanks, >>>>>> Eno >>>>>> >>>>>> >>>>>>> >>>>>>> -Michael >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Apr 11, 2017 at 9:14 AM, Damian Guy <damian....@gmail.com> >>>>> wrote: >>>>>>> >>>>>>>> Hi Eno, >>>>>>>> >>>>>>>> Thanks for the update. I agree with what Matthias said. I wonder if >>>>> the KIP >>>>>>>> should talk less about materialization and more about querying? After >>>>> all, >>>>>>>> that is what is being provided from an end-users perspective. >>>>>>>> >>>>>>>> I think if no store name is provided users would still be able to >>>>> query the >>>>>>>> store, just the store name would be some internally generated name. >>>>> They >>>>>>>> would be able to discover those names via the IQ API >>>>>>>> >>>>>>>> I think for some stores it will make sense to not create a physical >>>>> store, >>>>>>>> i.e., for thinks like `filter`, as this will save the rocksdb >>>>> overhead. But >>>>>>>> i guess that is more of an implementation detail. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Damian >>>>>>>> >>>>>>>> On Tue, 11 Apr 2017 at 00:36 Eno Thereska <eno.there...@gmail.com> >>>>> wrote: >>>>>>>> >>>>>>>>> Hi Matthias, >>>>>>>>> >>>>>>>>>> However, this still forces users, to provide a name for store that we >>>>>>>>>> must materialize, even if users are not interested in querying the >>>>>>>>>> stores. Thus, I would like to have overloads for all currently >>>>> existing >>>>>>>>>> methods having mandatory storeName paremeter, with overloads, that do >>>>>>>>>> not require the storeName parameter. >>>>>>>>> >>>>>>>>> >>>>>>>>> Oh yeah, absolutely, this is part of the KIP. I guess I didn't make it >>>>>>>>> clear, I'll clarify. >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Eno >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 10 Apr 2017, at 16:00, Matthias J. Sax <matth...@confluent.io> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Thanks for pushing this KIP Eno. >>>>>>>>>> >>>>>>>>>> The update give a very clear description about the scope, that is >>>>> super >>>>>>>>>> helpful for the discussion! >>>>>>>>>> >>>>>>>>>> - To put it into my own words, the KIP focus is on enable to query >>>>> all >>>>>>>>>> KTables. >>>>>>>>>> ** The ability to query a store is determined by providing a name for >>>>>>>>>> the store. >>>>>>>>>> ** At the same time, providing a name -- and thus making a store >>>>>>>>>> queryable -- does not say anything about an actual materialization >>>>> (ie, >>>>>>>>>> being queryable and being materialized are orthogonal). >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I like this overall a lot. However, I would go one step further. >>>>> Right >>>>>>>>>> now, you suggest to add new overload methods that allow users to >>>>>>>> specify >>>>>>>>>> a storeName -- if `null` is provided and the store is not >>>>> materialized, >>>>>>>>>> we ignore it completely -- if `null` is provided but the store must >>>>> be >>>>>>>>>> materialized we generate a internal name. So far so good. >>>>>>>>>> >>>>>>>>>> However, this still forces users, to provide a name for store that we >>>>>>>>>> must materialize, even if users are not interested in querying the >>>>>>>>>> stores. Thus, I would like to have overloads for all currently >>>>> existing >>>>>>>>>> methods having mandatory storeName paremeter, with overloads, that do >>>>>>>>>> not require the storeName parameter. >>>>>>>>>> >>>>>>>>>> Otherwise, we would still have some methods which optional storeName >>>>>>>>>> parameter and other method with mandatory storeName parameter -- >>>>> thus, >>>>>>>>>> still some inconsistency. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -Matthias >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 4/9/17 8:35 AM, Eno Thereska wrote: >>>>>>>>>>> Hi there, >>>>>>>>>>> >>>>>>>>>>> I've now done a V2 of the KIP, that hopefully addresses the feedback >>>>>>>> in >>>>>>>>> this discussion thread: >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>> 114%3A+KTable+materialization+and+improved+semantics >>>>>>>>> < >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >>>>>>>> 114:+KTable+materialization+and+improved+semantics>. >>>>>>>>> Notable changes: >>>>>>>>>>> >>>>>>>>>>> - clearly outline what is in the scope of the KIP and what is not. >>>>> We >>>>>>>>> ran into the issue where lots of useful, but somewhat tangential >>>>>>>>> discussions came up on interactive queries, declarative DSL etc. The >>>>>>>> exact >>>>>>>>> scope of this KIP is spelled out. >>>>>>>>>>> - decided to go with overloaded methods, not .materialize(), to stay >>>>>>>>> within the spirit of the current declarative DSL. >>>>>>>>>>> - clarified the depreciation plan >>>>>>>>>>> - listed part of the discussion we had under rejected alternatives >>>>>>>>>>> >>>>>>>>>>> If you have any further feedback on this, let's continue on this >>>>>>>> thread. >>>>>>>>>>> >>>>>>>>>>> Thank you >>>>>>>>>>> Eno >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On 1 Feb 2017, at 09:04, Eno Thereska <eno.there...@gmail.com> >>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Thanks everyone! I think it's time to do a V2 on the KIP so I'll do >>>>>>>>> that and we can see how it looks and continue the discussion from >>>>> there. >>>>>>>>> Stay tuned. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Eno >>>>>>>>>>>> >>>>>>>>>>>>> On 30 Jan 2017, at 17:23, Matthias J. Sax <matth...@confluent.io> >>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I think Eno's separation is very clear and helpful. In order to >>>>>>>>>>>>> streamline this discussion, I would suggest we focus back on point >>>>>>>> (1) >>>>>>>>>>>>> only, as this is the original KIP question. >>>>>>>>>>>>> >>>>>>>>>>>>> Even if I started to DSL design discussion somehow, because I >>>>>>>> thought >>>>>>>>> it >>>>>>>>>>>>> might be helpful to resolve both in a single shot, I feel that we >>>>>>>> have >>>>>>>>>>>>> too many options about DSL design and we should split it up in two >>>>>>>>>>>>> steps. This will have the disadvantage that we will change the API >>>>>>>>>>>>> twice, but still, I think it will be a more focused discussion. >>>>>>>>>>>>> >>>>>>>>>>>>> I just had another look at the KIP, an it proposes 3 changes: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. add .materialized() -> IIRC it was suggested to name this >>>>>>>>>>>>> .materialize() though (can you maybe update the KIP Eno?) >>>>>>>>>>>>> 2. remove print(), writeAsText(), and foreach() >>>>>>>>>>>>> 3. rename toStream() to toKStream() >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I completely agree with (2) -- not sure about (3) though because >>>>>>>>>>>>> KStreamBuilder also hast .stream() and .table() as methods. >>>>>>>>>>>>> >>>>>>>>>>>>> However, we might want to introduce a KStream#toTable() -- this >>>>> was >>>>>>>>>>>>> requested multiple times -- might also be part of a different KIP. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Thus, we end up with (1). I would suggest to do a step backward >>>>> here >>>>>>>>> and >>>>>>>>>>>>> instead of a discussion how to express the changes in the DSL (new >>>>>>>>>>>>> overload, new methods...) we should discuss what the actual change >>>>>>>>>>>>> should be. Like (1) materialize all KTable all the time (2) all >>>>> the >>>>>>>>> user >>>>>>>>>>>>> to force a materialization to enable querying the KTable (3) allow >>>>>>>> for >>>>>>>>>>>>> queryable non-materialized KTable. >>>>>>>>>>>>> >>>>>>>>>>>>> On more question is, if we want to allow a user-forced >>>>>>>> materialization >>>>>>>>>>>>> only as as local store without changelog, or both (together / >>>>>>>>>>>>> independently)? We got some request like this already. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -Matthias >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 1/30/17 3:50 AM, Jan Filipiak wrote: >>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>> >>>>>>>>>>>>>> thanks for putting into different points. I want to put a few >>>>>>>> remarks >>>>>>>>>>>>>> inline. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 30.01.2017 12:19, Eno Thereska wrote: >>>>>>>>>>>>>>> So I think there are several important discussion threads that >>>>> are >>>>>>>>>>>>>>> emerging here. Let me try to tease them apart: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. inconsistency in what is materialized and what is not, what >>>>> is >>>>>>>>>>>>>>> queryable and what is not. I think we all agree there is some >>>>>>>>>>>>>>> inconsistency there and this will be addressed with any of the >>>>>>>>>>>>>>> proposed approaches. Addressing the inconsistency is the point >>>>> of >>>>>>>>> the >>>>>>>>>>>>>>> original KIP. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. the exact API for materializing a KTable. We can specify 1) a >>>>>>>>>>>>>>> "store name" (as we do today) or 2) have a ".materialize[d]" >>>>> call >>>>>>>> or >>>>>>>>>>>>>>> 3) get a handle from a KTable ".getQueryHandle" or 4) have a >>>>>>>> builder >>>>>>>>>>>>>>> construct. So we have discussed 4 options. It is important to >>>>>>>>> remember >>>>>>>>>>>>>>> in this discussion that IQ is not designed for just local >>>>> queries, >>>>>>>>> but >>>>>>>>>>>>>>> also for distributed queries. In all cases an identifying >>>>> name/id >>>>>>>> is >>>>>>>>>>>>>>> needed for the store that the user is interested in querying. So >>>>>>>> we >>>>>>>>>>>>>>> end up with a discussion on who provides the name, the user (as >>>>>>>> done >>>>>>>>>>>>>>> today) or if it is generated automatically (as Jan suggests, as >>>>> I >>>>>>>>>>>>>>> understand it). If it is generated automatically we need a way >>>>> to >>>>>>>>>>>>>>> expose these auto-generated names to the users and link them to >>>>>>>> the >>>>>>>>>>>>>>> KTables they care to query. >>>>>>>>>>>>>> Hi, the last sentence is what I currently arguing against. The >>>>> user >>>>>>>>>>>>>> would never see a stringtype indentifier name or anything. All he >>>>>>>>> gets >>>>>>>>>>>>>> is the queryHandle if he executes a get(K) that will be an >>>>>>>>> interactive >>>>>>>>>>>>>> query get. with all the finding the right servers that currently >>>>>>>>> have a >>>>>>>>>>>>>> copy of this underlying store stuff going on. The nice part is >>>>> that >>>>>>>>> if >>>>>>>>>>>>>> someone retrieves a queryHandle, you know that you have to >>>>>>>>> materialized >>>>>>>>>>>>>> (if you are not already) as queries will be coming. Taking away >>>>> the >>>>>>>>>>>>>> confusion mentioned in point 1 IMO. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3. The exact boundary between the DSL, that is the processing >>>>>>>>>>>>>>> language, and the storage/IQ queries, and how we jump from one >>>>> to >>>>>>>>> the >>>>>>>>>>>>>>> other. This is mostly for how we get a handle on a store (so >>>>> it's >>>>>>>>>>>>>>> related to point 2), rather than for how we query the store. I >>>>>>>> think >>>>>>>>>>>>>>> we all agree that we don't want to limit ways one can query a >>>>>>>> store >>>>>>>>>>>>>>> (e.g., using gets or range queries etc) and the query APIs are >>>>> not >>>>>>>>> in >>>>>>>>>>>>>>> the scope of the DSL. >>>>>>>>>>>>>> Does the IQ work with range currently? The range would have to be >>>>>>>>>>>>>> started on all stores and then merged by maybe the client. Range >>>>>>>>> force a >>>>>>>>>>>>>> flush to RocksDB currently so I am sure you would get a >>>>> performance >>>>>>>>> hit >>>>>>>>>>>>>> right there. Time-windows might be okay, but I am not sure if the >>>>>>>>> first >>>>>>>>>>>>>> version should offer the user range access. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 4. The nature of the DSL and whether its declarative enough, or >>>>>>>>>>>>>>> flexible enough. Damian made the point that he likes the builder >>>>>>>>>>>>>>> pattern since users can specify, per KTable, things like caching >>>>>>>> and >>>>>>>>>>>>>>> logging needs. His observation (as I understand it) is that the >>>>>>>>>>>>>>> processor API (PAPI) is flexible but doesn't provide any help at >>>>>>>> all >>>>>>>>>>>>>>> to users. The current DSL provides declarative abstractions, but >>>>>>>>> it's >>>>>>>>>>>>>>> not fine-grained enough. This point is much broader than the >>>>> KIP, >>>>>>>>> but >>>>>>>>>>>>>>> discussing it in this KIPs context is ok, since we don't want to >>>>>>>>> make >>>>>>>>>>>>>>> small piecemeal changes and then realise we're not in the spot >>>>> we >>>>>>>>> want >>>>>>>>>>>>>>> to be. >>>>>>>>>>>>>> This is indeed much broader. My guess here is that's why both >>>>> API's >>>>>>>>>>>>>> exists and helping the users to switch back and forth might be a >>>>>>>>> thing. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Feel free to pitch in if I have misinterpreted something. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 30 Jan 2017, at 10:22, Jan Filipiak < >>>>> jan.filip...@trivago.com >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Eno, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have a really hard time understanding why we can't. From my >>>>>>>> point >>>>>>>>>>>>>>>> of view everything could be super elegant DSL only + public api >>>>>>>> for >>>>>>>>>>>>>>>> the PAPI-people as already exist. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The above aproach implementing a .get(K) on KTable is foolisch >>>>> in >>>>>>>>> my >>>>>>>>>>>>>>>> opinion as it would be to late to know that materialisation >>>>> would >>>>>>>>> be >>>>>>>>>>>>>>>> required. >>>>>>>>>>>>>>>> But having an API that allows to indicate I want to query this >>>>>>>>> table >>>>>>>>>>>>>>>> and then wrapping the say table's processorname can work out >>>>>>>> really >>>>>>>>>>>>>>>> really nice. The only obstacle I see is people not willing to >>>>>>>> spend >>>>>>>>>>>>>>>> the additional time in implementation and just want a quick >>>>> shot >>>>>>>>>>>>>>>> option to make it work. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For me it would look like this: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> table = builder.table() >>>>>>>>>>>>>>>> filteredTable = table.filter() >>>>>>>>>>>>>>>> rawHandle = table.getQueryHandle() // Does the materialisation, >>>>>>>>>>>>>>>> really all names possible but id rather hide the implication of >>>>>>>> it >>>>>>>>>>>>>>>> materializes >>>>>>>>>>>>>>>> filteredTableHandle = filteredTable.getQueryHandle() // this >>>>>>>> would >>>>>>>>>>>>>>>> _not_ materialize again of course, the source or the aggregator >>>>>>>>> would >>>>>>>>>>>>>>>> stay the only materialized processors >>>>>>>>>>>>>>>> streams = new streams(builder) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This middle part is highly flexible I could imagin to force the >>>>>>>>> user >>>>>>>>>>>>>>>> todo something like this. This implies to the user that his >>>>>>>> streams >>>>>>>>>>>>>>>> need to be running >>>>>>>>>>>>>>>> instead of propagating the missing initialisation back by >>>>>>>>> exceptions. >>>>>>>>>>>>>>>> Also if the users is forced to pass the appropriate streams >>>>>>>>> instance >>>>>>>>>>>>>>>> back can change. >>>>>>>>>>>>>>>> I think its possible to build multiple streams out of one >>>>>>>> topology >>>>>>>>>>>>>>>> so it would be easiest to implement aswell. This is just what I >>>>>>>>> maybe >>>>>>>>>>>>>>>> had liked the most >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> streams.start(); >>>>>>>>>>>>>>>> rawHandle.prepare(streams) >>>>>>>>>>>>>>>> filteredHandle.prepare(streams) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> later the users can do >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> V value = rawHandle.get(K) >>>>>>>>>>>>>>>> V value = filteredHandle.get(K) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This could free DSL users from anything like storenames and how >>>>>>>> and >>>>>>>>>>>>>>>> what to materialize. Can someone indicate what the problem >>>>> would >>>>>>>> be >>>>>>>>>>>>>>>> implementing it like this. >>>>>>>>>>>>>>>> Yes I am aware that the current IQ API will not support >>>>> querying >>>>>>>> by >>>>>>>>>>>>>>>> KTableProcessorName instread of statestoreName. But I think >>>>> that >>>>>>>>> had >>>>>>>>>>>>>>>> to change if you want it to be intuitive >>>>>>>>>>>>>>>> IMO you gotta apply the filter read time >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Looking forward to your opinions >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 30.01.2017 10:42, Eno Thereska wrote: >>>>>>>>>>>>>>>>> Hi there, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The inconsistency will be resolved, whether with materialize >>>>> or >>>>>>>>>>>>>>>>> overloaded methods. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> With the discussion on the DSL & stores I feel we've gone in a >>>>>>>>>>>>>>>>> slightly different tangent, which is worth discussing >>>>>>>> nonetheless. >>>>>>>>>>>>>>>>> We have entered into an argument around the scope of the DSL. >>>>>>>> The >>>>>>>>>>>>>>>>> DSL has been designed primarily for processing. The DSL does >>>>> not >>>>>>>>>>>>>>>>> dictate ways to access state stores or what hind of queries to >>>>>>>>>>>>>>>>> perform on them. Hence, I see the mechanism for accessing >>>>>>>> storage >>>>>>>>> as >>>>>>>>>>>>>>>>> decoupled from the DSL. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We could think of ways to get store handles from part of the >>>>>>>> DSL, >>>>>>>>>>>>>>>>> like the KTable abstraction. However, subsequent queries will >>>>> be >>>>>>>>>>>>>>>>> store-dependent and not rely on the DSL, hence I'm not sure we >>>>>>>> get >>>>>>>>>>>>>>>>> any grand-convergence DSL-Store here. So I am arguing that the >>>>>>>>>>>>>>>>> current way of getting a handle on state stores is fine. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 30 Jan 2017, at 03:56, Guozhang Wang <wangg...@gmail.com> >>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thinking loud here about the API options (materialize v.s. >>>>>>>>> overloaded >>>>>>>>>>>>>>>>>> functions) and its impact on IQ: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 1. The first issue of the current DSL is that, there is >>>>>>>>>>>>>>>>>> inconsistency upon >>>>>>>>>>>>>>>>>> whether / how KTables should be materialized: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> a) in many cases the library HAS TO materialize KTables no >>>>>>>>>>>>>>>>>> matter what, >>>>>>>>>>>>>>>>>> e.g. KStream / KTable aggregation resulted KTables, and hence >>>>>>>> we >>>>>>>>>>>>>>>>>> enforce >>>>>>>>>>>>>>>>>> users to provide store names and throw RTE if it is null; >>>>>>>>>>>>>>>>>> b) in some other cases, the KTable can be materialized or >>>>> not; >>>>>>>>> for >>>>>>>>>>>>>>>>>> example in KStreamBuilder.table(), store names can be >>>>> nullable >>>>>>>>> and >>>>>>>>>>>>>>>>>> in which >>>>>>>>>>>>>>>>>> case the KTable would not be materialized; >>>>>>>>>>>>>>>>>> c) in some other cases, the KTable will never be >>>>> materialized, >>>>>>>>> for >>>>>>>>>>>>>>>>>> example KTable.filter() resulted KTables, and users have no >>>>>>>>> options to >>>>>>>>>>>>>>>>>> enforce them to be materialized; >>>>>>>>>>>>>>>>>> d) this is related to a), where some KTables are required to >>>>>>>> be >>>>>>>>>>>>>>>>>> materialized, but we do not enforce users to provide a state >>>>>>>>> store >>>>>>>>>>>>>>>>>> name, >>>>>>>>>>>>>>>>>> e.g. KTables involved in joins; a RTE will be thrown not >>>>>>>>>>>>>>>>>> immediately but >>>>>>>>>>>>>>>>>> later in this case. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2. The second issue is related to IQ, where state stores are >>>>>>>>>>>>>>>>>> accessed by >>>>>>>>>>>>>>>>>> their state stores; so only those KTable's that have >>>>>>>>> user-specified >>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>> stores will be queryable. But because of 1) above, many >>>>> stores >>>>>>>>> may >>>>>>>>>>>>>>>>>> not be >>>>>>>>>>>>>>>>>> interested to users for IQ but they still need to provide a >>>>>>>>>>>>>>>>>> (dummy?) state >>>>>>>>>>>>>>>>>> store name for them; while on the other hand users cannot >>>>> query >>>>>>>>>>>>>>>>>> some state >>>>>>>>>>>>>>>>>> stores, e.g. the ones generated by KTable.filter() as there >>>>> is >>>>>>>> no >>>>>>>>>>>>>>>>>> APIs for >>>>>>>>>>>>>>>>>> them to specify a state store name. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 3. We are aware from user feedbacks that such backend details >>>>>>>>> would be >>>>>>>>>>>>>>>>>> better be abstracted away from the DSL layer, where app >>>>>>>>> developers >>>>>>>>>>>>>>>>>> should >>>>>>>>>>>>>>>>>> just focus on processing logic, while state stores along with >>>>>>>>> their >>>>>>>>>>>>>>>>>> changelogs etc would better be in a different mechanism; same >>>>>>>>>>>>>>>>>> arguments >>>>>>>>>>>>>>>>>> have been discussed for serdes / windowing triggers as well. >>>>>>>> For >>>>>>>>>>>>>>>>>> serdes >>>>>>>>>>>>>>>>>> specifically, we had a very long discussion about it and >>>>>>>>> concluded >>>>>>>>>>>>>>>>>> that, at >>>>>>>>>>>>>>>>>> least in Java7, we cannot completely abstract serde away in >>>>> the >>>>>>>>>>>>>>>>>> DSL, so we >>>>>>>>>>>>>>>>>> choose the other extreme to enforce users to be completely >>>>>>>> aware >>>>>>>>> of >>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> serde requirements when some KTables may need to be >>>>>>>> materialized >>>>>>>>> vis >>>>>>>>>>>>>>>>>> overloaded API functions. While for the state store names, I >>>>>>>> feel >>>>>>>>>>>>>>>>>> it is a >>>>>>>>>>>>>>>>>> different argument than serdes (details below). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> So to me, for either materialize() v.s. overloaded functions >>>>>>>>>>>>>>>>>> directions, >>>>>>>>>>>>>>>>>> the first thing I'd like to resolve is the inconsistency >>>>> issue >>>>>>>>>>>>>>>>>> mentioned >>>>>>>>>>>>>>>>>> above. So in either case: KTable materialization will not be >>>>>>>>> affect >>>>>>>>>>>>>>>>>> by user >>>>>>>>>>>>>>>>>> providing state store name or not, but will only be decided >>>>> by >>>>>>>>> the >>>>>>>>>>>>>>>>>> library >>>>>>>>>>>>>>>>>> when it is necessary. More specifically, only join operator >>>>> and >>>>>>>>>>>>>>>>>> builder.table() resulted KTables are not always materialized, >>>>>>>> but >>>>>>>>>>>>>>>>>> are still >>>>>>>>>>>>>>>>>> likely to be materialized lazily (e.g. when participated in a >>>>>>>>> join >>>>>>>>>>>>>>>>>> operator). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> For overloaded functions that would mean: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> a) we have an overloaded function for ALL operators that >>>>> could >>>>>>>>>>>>>>>>>> result >>>>>>>>>>>>>>>>>> in a KTable, and allow it to be null (i.e. for the function >>>>>>>>> without >>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>> param it is null by default); >>>>>>>>>>>>>>>>>> b) null-state-store-name do not indicate that a KTable would >>>>>>>>>>>>>>>>>> not be >>>>>>>>>>>>>>>>>> materialized, but that it will not be used for IQ at all >>>>>>>>> (internal >>>>>>>>>>>>>>>>>> state >>>>>>>>>>>>>>>>>> store names will be generated when necessary). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> For materialize() that would mean: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> a) we will remove state store names from ALL operators that >>>>>>>>> could >>>>>>>>>>>>>>>>>> result in a KTable. >>>>>>>>>>>>>>>>>> b) KTables that not calling materialized do not indicate that >>>>>>>> a >>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>> would not be materialized, but that it will not be used for >>>>> IQ >>>>>>>>> at all >>>>>>>>>>>>>>>>>> (internal state store names will be generated when >>>>> necessary). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Again, in either ways the API itself does not "hint" about >>>>>>>>> anything >>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>> materializing a KTable or not at all; it is still purely >>>>>>>>> determined >>>>>>>>>>>>>>>>>> by the >>>>>>>>>>>>>>>>>> library when parsing the DSL for now. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Following these thoughts, I feel that 1) we should probably >>>>>>>>> change >>>>>>>>>>>>>>>>>> the name >>>>>>>>>>>>>>>>>> "materialize" since it may be misleading to users as what >>>>>>>>> actually >>>>>>>>>>>>>>>>>> happened >>>>>>>>>>>>>>>>>> behind the scene, to e.g. Damian suggested >>>>>>>> "queryableStore(String >>>>>>>>>>>>>>>>>> storeName)", >>>>>>>>>>>>>>>>>> which returns a QueryableStateStore, and can replace the >>>>>>>>>>>>>>>>>> `KafkaStreams.store` function; 2) comparing those two options >>>>>>>>>>>>>>>>>> assuming we >>>>>>>>>>>>>>>>>> get rid of the misleading function name, I personally favor >>>>> not >>>>>>>>>>>>>>>>>> adding more >>>>>>>>>>>>>>>>>> overloading functions as it keeps the API simpler. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Guozhang >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Sat, Jan 28, 2017 at 2:32 PM, Jan Filipiak >>>>>>>>>>>>>>>>>> <jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> thanks for your mail, felt like this can clarify some >>>>> things! >>>>>>>>> The >>>>>>>>>>>>>>>>>>> thread >>>>>>>>>>>>>>>>>>> unfortunately split but as all branches close in on what my >>>>>>>>>>>>>>>>>>> suggestion was >>>>>>>>>>>>>>>>>>> about Ill pick this to continue >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Of course only the table the user wants to query would be >>>>>>>>>>>>>>>>>>> materialized. >>>>>>>>>>>>>>>>>>> (retrieving the queryhandle implies materialisation). So In >>>>>>>> the >>>>>>>>>>>>>>>>>>> example of >>>>>>>>>>>>>>>>>>> KTable::filter if you call >>>>>>>>>>>>>>>>>>> getIQHandle on both tables only the one source that is there >>>>>>>>> would >>>>>>>>>>>>>>>>>>> materialize and the QueryHandleabstraction would make sure >>>>> it >>>>>>>>> gets >>>>>>>>>>>>>>>>>>> mapped >>>>>>>>>>>>>>>>>>> and filtered and what not uppon read as usual. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Of Course the Object you would retrieve would maybe only >>>>> wrap >>>>>>>>> the >>>>>>>>>>>>>>>>>>> storeName / table unique identifier and a way to access the >>>>>>>>> streams >>>>>>>>>>>>>>>>>>> instance and then basically uses the same mechanism that is >>>>>>>>>>>>>>>>>>> currently used. >>>>>>>>>>>>>>>>>>> From my point of view this is the least confusing way for >>>>> DSL >>>>>>>>>>>>>>>>>>> users. If >>>>>>>>>>>>>>>>>>> its to tricky to get a hand on the streams instance one >>>>> could >>>>>>>>> ask >>>>>>>>>>>>>>>>>>> the user >>>>>>>>>>>>>>>>>>> to pass it in before executing queries, therefore making >>>>> sure >>>>>>>>> the >>>>>>>>>>>>>>>>>>> streams >>>>>>>>>>>>>>>>>>> instance has been build. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> The effort to implement this is indeed some orders of >>>>>>>> magnitude >>>>>>>>>>>>>>>>>>> higher >>>>>>>>>>>>>>>>>>> than the overloaded materialized call. As long as I could >>>>> help >>>>>>>>>>>>>>>>>>> getting a >>>>>>>>>>>>>>>>>>> different view I am happy. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On 28.01.2017 09:36, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi Jan, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I understand your concern. One implication of not passing >>>>> any >>>>>>>>>>>>>>>>>>>> store name >>>>>>>>>>>>>>>>>>>> and just getting an IQ handle is that all KTables would >>>>> need >>>>>>>>> to be >>>>>>>>>>>>>>>>>>>> materialised. Currently the store name (or proposed >>>>>>>>>>>>>>>>>>>> .materialize() call) >>>>>>>>>>>>>>>>>>>> act as hints on whether to materialise the KTable or not. >>>>>>>>>>>>>>>>>>>> Materialising >>>>>>>>>>>>>>>>>>>> every KTable can be expensive, although there are some >>>>> tricks >>>>>>>>> one >>>>>>>>>>>>>>>>>>>> can play, >>>>>>>>>>>>>>>>>>>> e.g., have a virtual store rather than one backed by a >>>>> Kafka >>>>>>>>> topic. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> However, even with the above, after getting an IQ handle, >>>>> the >>>>>>>>>>>>>>>>>>>> user would >>>>>>>>>>>>>>>>>>>> still need to use IQ APIs to query the state. As such, we >>>>>>>> would >>>>>>>>>>>>>>>>>>>> still >>>>>>>>>>>>>>>>>>>> continue to be outside the original DSL so this wouldn't >>>>>>>>> address >>>>>>>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>>>> original concern. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> So I read this suggestion as simplifying the APIs by >>>>> removing >>>>>>>>> the >>>>>>>>>>>>>>>>>>>> store >>>>>>>>>>>>>>>>>>>> name, at the cost of having to materialise every KTable. >>>>> It's >>>>>>>>>>>>>>>>>>>> definitely an >>>>>>>>>>>>>>>>>>>> option we'll consider as part of this KIP. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> On 28 Jan 2017, at 06:49, Jan Filipiak < >>>>>>>>> jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>> Hi Exactly >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I know it works from the Processor API, but my suggestion >>>>>>>>> would >>>>>>>>>>>>>>>>>>>>> prevent >>>>>>>>>>>>>>>>>>>>> DSL users dealing with storenames what so ever. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> In general I am pro switching between DSL and Processor >>>>> API >>>>>>>>>>>>>>>>>>>>> easily. (In >>>>>>>>>>>>>>>>>>>>> my Stream applications I do this a lot with reflection and >>>>>>>>>>>>>>>>>>>>> instanciating >>>>>>>>>>>>>>>>>>>>> KTableImpl) Concerning this KIP all I say is that there >>>>>>>> should >>>>>>>>>>>>>>>>>>>>> be a DSL >>>>>>>>>>>>>>>>>>>>> concept of "I want to expose this __KTable__. This can be >>>>> a >>>>>>>>>>>>>>>>>>>>> Method like >>>>>>>>>>>>>>>>>>>>> KTable::retrieveIQHandle():InteractiveQueryHandle, the >>>>>>>> table >>>>>>>>>>>>>>>>>>>>> would know >>>>>>>>>>>>>>>>>>>>> to materialize, and the user had a reference to the "store >>>>>>>>> and the >>>>>>>>>>>>>>>>>>>>> distributed query mechanism by the Interactive Query >>>>> Handle" >>>>>>>>>>>>>>>>>>>>> under the hood >>>>>>>>>>>>>>>>>>>>> it can use the same mechanism as the PIP people again. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> I hope you see my point J >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On 27.01.2017 21:59, Matthias J. Sax wrote: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Jan, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> the IQ feature is not limited to Streams DSL but can also >>>>>>>> be >>>>>>>>>>>>>>>>>>>>>> used for >>>>>>>>>>>>>>>>>>>>>> Stores used in PAPI. Thus, we need a mechanism that does >>>>>>>> work >>>>>>>>>>>>>>>>>>>>>> for PAPI >>>>>>>>>>>>>>>>>>>>>> and DSL. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Nevertheless I see your point and I think we could >>>>> provide >>>>>>>> a >>>>>>>>>>>>>>>>>>>>>> better API >>>>>>>>>>>>>>>>>>>>>> for KTable stores including the discovery of remote >>>>> shards >>>>>>>> of >>>>>>>>>>>>>>>>>>>>>> the same >>>>>>>>>>>>>>>>>>>>>> KTable. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> @Michael: Yes, right now we do have a lot of overloads >>>>> and >>>>>>>> I >>>>>>>>> am >>>>>>>>>>>>>>>>>>>>>> not a >>>>>>>>>>>>>>>>>>>>>> big fan of those -- I would rather prefer a builder >>>>>>>> pattern. >>>>>>>>>>>>>>>>>>>>>> But that >>>>>>>>>>>>>>>>>>>>>> might be a different discussion (nevertheless, if we >>>>> would >>>>>>>>> aim >>>>>>>>>>>>>>>>>>>>>> for a API >>>>>>>>>>>>>>>>>>>>>> rework, we should get the changes with regard to stores >>>>>>>> right >>>>>>>>>>>>>>>>>>>>>> from the >>>>>>>>>>>>>>>>>>>>>> beginning on, in order to avoid a redesign later on.) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> something like: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> stream.groupyByKey() >>>>>>>>>>>>>>>>>>>>>> .window(TimeWindow.of(5000)) >>>>>>>>>>>>>>>>>>>>>> .aggregate(...) >>>>>>>>>>>>>>>>>>>>>> .withAggValueSerde(new CustomTypeSerde()) >>>>>>>>>>>>>>>>>>>>>> .withStoreName("storeName); >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> (This would also reduce JavaDoc redundancy -- maybe a >>>>>>>>> personal >>>>>>>>>>>>>>>>>>>>>> pain >>>>>>>>>>>>>>>>>>>>>> point right now :)) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> On 1/27/17 11:10 AM, Jan Filipiak wrote: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Yeah, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Maybe my bad that I refuse to look into IQ as i don't >>>>> find >>>>>>>>> them >>>>>>>>>>>>>>>>>>>>>>> anywhere >>>>>>>>>>>>>>>>>>>>>>> close to being interesting. The Problem IMO is that >>>>> people >>>>>>>>>>>>>>>>>>>>>>> need to know >>>>>>>>>>>>>>>>>>>>>>> the Store name), so we are working on different levels >>>>> to >>>>>>>>>>>>>>>>>>>>>>> achieve a >>>>>>>>>>>>>>>>>>>>>>> single goal. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> What is your peoples opinion on having a method on >>>>> KTABLE >>>>>>>>> that >>>>>>>>>>>>>>>>>>>>>>> returns >>>>>>>>>>>>>>>>>>>>>>> them something like a Keyvalue store. There is of course >>>>>>>>>>>>>>>>>>>>>>> problems like >>>>>>>>>>>>>>>>>>>>>>> "it cant be used before the streamthreads are going and >>>>>>>>>>>>>>>>>>>>>>> groupmembership >>>>>>>>>>>>>>>>>>>>>>> is established..." but the benefit would be that for the >>>>>>>>> user >>>>>>>>>>>>>>>>>>>>>>> there is >>>>>>>>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>> consistent way of saying "Hey I need it materialized as >>>>>>>>>>>>>>>>>>>>>>> querries gonna >>>>>>>>>>>>>>>>>>>>>>> be comming" + already get a Thing that he can execute >>>>> the >>>>>>>>>>>>>>>>>>>>>>> querries on >>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>> 1 step. >>>>>>>>>>>>>>>>>>>>>>> What I think is unintuitive here is you need to say >>>>>>>>>>>>>>>>>>>>>>> materialize on this >>>>>>>>>>>>>>>>>>>>>>> Ktable and then you go somewhere else and find its store >>>>>>>>> name >>>>>>>>>>>>>>>>>>>>>>> and then >>>>>>>>>>>>>>>>>>>>>>> you go to the kafkastreams instance and ask for the >>>>> store >>>>>>>>> with >>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>> name. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> So one could the user help to stay in DSL land and >>>>>>>> therefore >>>>>>>>>>>>>>>>>>>>>>> maybe >>>>>>>>>>>>>>>>>>>>>>> confuse him less. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> #DeathToIQMoreAndBetterConnectors :) >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> On 27.01.2017 16:51, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I think Jan is saying that they don't always need to be >>>>>>>>>>>>>>>>>>>>>>>> materialized, >>>>>>>>>>>>>>>>>>>>>>>> i.e., >>>>>>>>>>>>>>>>>>>>>>>> filter just needs to apply the ValueGetter, it doesn't >>>>>>>>> need yet >>>>>>>>>>>>>>>>>>>>>>>> another >>>>>>>>>>>>>>>>>>>>>>>> physical state store. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> On Fri, 27 Jan 2017 at 15:49 Michael Noll < >>>>>>>>> mich...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Like Damian, and for the same reasons, I am more in >>>>> favor >>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>> overloading >>>>>>>>>>>>>>>>>>>>>>>>> methods rather than introducing `materialize()`. >>>>>>>>>>>>>>>>>>>>>>>>> FWIW, we already have a similar API setup for e.g. >>>>>>>>>>>>>>>>>>>>>>>>> `KTable#through(topicName, stateStoreName)`. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> A related but slightly different question is what e.g. >>>>>>>> Jan >>>>>>>>>>>>>>>>>>>>>>>>> Filipiak >>>>>>>>>>>>>>>>>>>>>>>>> mentioned earlier in this thread: >>>>>>>>>>>>>>>>>>>>>>>>> I think we need to explain more clearly why KIP-114 >>>>>>>>> doesn't >>>>>>>>>>>>>>>>>>>>>>>>> propose >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> seemingly simpler solution of always materializing >>>>>>>>> tables/state >>>>>>>>>>>>>>>>>>>>>>>>> stores. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> On Fri, Jan 27, 2017 at 4:38 PM, Jan Filipiak < >>>>>>>>>>>>>>>>>>>>>>>>> jan.filip...@trivago.com> >>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>> Yeah its confusing, Why shoudn't it be querable by >>>>> IQ? >>>>>>>> If >>>>>>>>>>>>>>>>>>>>>>>>>> you uses >>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> ValueGetter of Filter it will apply the filter and >>>>>>>>> should be >>>>>>>>>>>>>>>>>>>>>>>>>> completely >>>>>>>>>>>>>>>>>>>>>>>>>> transparent as to if another processor or IQ is >>>>>>>> accessing >>>>>>>>>>>>>>>>>>>>>>>>>> it? How >>>>>>>>>>>>>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> new method help? >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> I cannot see the reason for the additional >>>>> materialize >>>>>>>>>>>>>>>>>>>>>>>>>> method being >>>>>>>>>>>>>>>>>>>>>>>>>> required! Hence I suggest leave it alone. >>>>>>>>>>>>>>>>>>>>>>>>>> regarding removing the others I dont have strong >>>>>>>> opinions >>>>>>>>>>>>>>>>>>>>>>>>>> and it >>>>>>>>>>>>>>>>>>>>>>>>>> seems to >>>>>>>>>>>>>>>>>>>>>>>>>> be unrelated. >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Best Jan >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On 26.01.2017 20:48, Eno Thereska wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Forwarding this thread to the users list too in case >>>>>>>>> people >>>>>>>>>>>>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>> comment. It is also on the dev list. >>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>>>>>>>>>>> Eno >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> From: "Matthias J. Sax" <matth...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: [DISCUSS] KIP-114: KTable >>>>>>>> materialization >>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>> improved >>>>>>>>>>>>>>>>>>>>>>>>>>>> semantics >>>>>>>>>>>>>>>>>>>>>>>>>>>> Date: 24 January 2017 at 19:30:10 GMT >>>>>>>>>>>>>>>>>>>>>>>>>>>> To: dev@kafka.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>> Reply-To: dev@kafka.apache.org >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> That not what I meant by "huge impact". >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I refer to the actions related to materialize a >>>>>>>> KTable: >>>>>>>>>>>>>>>>>>>>>>>>>>>> creating a >>>>>>>>>>>>>>>>>>>>>>>>>>>> RocksDB store and a changelog topic -- users should >>>>>>>> be >>>>>>>>>>>>>>>>>>>>>>>>>>>> aware about >>>>>>>>>>>>>>>>>>>>>>>>>>>> runtime implication and this is better expressed by >>>>>>>> an >>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>>>>>>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>> call, rather than implicitly triggered by using a >>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>> overload of >>>>>>>>>>>>>>>>>>>>>>>>>>>> a method. >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/24/17 1:35 AM, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> I think your definition of a huge impact and mine >>>>> are >>>>>>>>> rather >>>>>>>>>>>>>>>>>>>>>>>>>>>>> different >>>>>>>>>>>>>>>>>>>>>>>>>>>>> ;-P >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Overloading a few methods is not really a huge >>>>>>>> impact >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IMO. It is >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>> also a >>>>>>>>>>>>>>>>>>>>>>>>>> sacrifice worth making for readability, usability of >>>>>>>> the >>>>>>>>> API. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 17:55 Matthias J. Sax < >>>>>>>>>>>>>>>>>>>>>>>>>>>>> matth...@confluent.io> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> I understand your argument, but do not agree with >>>>>>>> it. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Your first version (even if the "flow" is not as >>>>>>>>> nice) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> explicit >>>>>>>>>>>>>>>>>>>>>>>>>> than the second version. Adding a stateStoreName >>>>>>>>> parameter >>>>>>>>>>>>>>>>>>>>>>>>>> is quite >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> implicit but has a huge impact -- thus, I prefer >>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rather more >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> verbose >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> but explicit version. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/23/17 1:39 AM, Damian Guy wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm not a fan of materialize. I think it >>>>> interrupts >>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> flow, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> i.e, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>> table.mapValue(..).materialize().join(..).materialize() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> compared to: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> table.mapValues(..).join(..) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I know which one i prefer. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> My preference is stil to provide overloaded >>>>>>>> methods >>>>>>>>> where >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> people can >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> specify the store names if they want, otherwise >>>>> we >>>>>>>>> just >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> generate >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> them. >>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, 23 Jan 2017 at 05:30 Matthias J. Sax >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <matth...@confluent.io >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> thanks for the KIP Eno! Here are my 2 cents: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1) I like Guozhang's proposal about removing >>>>>>>> store >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name from >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> all >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods and generate internal names (however, I >>>>>>>>> would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do this >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> overloads). Furthermore, I would not force >>>>> users >>>>>>>>> to call >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if they want to query a store, but add one more >>>>>>>>> method >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .stateStoreName() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that returns the store name if the KTable is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thus, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> also >>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() must not necessarily have a parameter >>>>>>>>> storeName >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (ie, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should have some overloads here). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would also not allow to provide a null store >>>>>>>>> name (to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> indicate no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization if not necessary) but throw an >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> exception. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This yields some simplification (see below). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2) I also like Guozhang's proposal about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KStream#toTable() >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3. What will happen when you call materialize >>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> that is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> already >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialized? Will it create another >>>>> StateStore >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (providing >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> name >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> different), throw an Exception? >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Currently an exception is thrown, but see >>>>> below. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> If we follow approach (1) from Guozhang, there >>>>>>>> is >>>>>>>>> no >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> need to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> worry >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> a second materialization and also no exception >>>>>>>>> must be >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> throws. A >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> call to >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> .materialize() basically sets a "materialized >>>>>>>>> flag" (ie, >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> idempotent >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> operation) and sets a new name. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 4) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Rename toStream() to toKStream() for >>>>> consistency. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Not sure whether that is really required. We >>>>>>>> also >>>>>>>>> use >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#stream()` and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> `KStreamBuilder#table()`, for >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> don't care about the "K" prefix. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Eno's reply: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I think changing it to `toKStream` would make >>>>> it >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> absolutely >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> clear >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> what >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> we are converting it to. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'd say we should probably change the >>>>>>>>> KStreamBuilder >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> methods >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this KIP). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would keep #toStream(). (see below) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 5) We should not remove any methods but only >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> deprecate them. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> A general note: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I do not understand your comments "Rejected >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Alternatives". You >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> say >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "Have >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the KTable be the materialized view" was >>>>>>>> rejected. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But your >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KIP >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> actually >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> does exactly this -- the changelog abstraction >>>>> of >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> secondary >>>>>>>>>>>>>>>>>>>>>>>>>> after those changes and the "view" abstraction is >>>>> what >>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> KTable is. >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> just to be clear, I like this a lot: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with the name KTable >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - is aligns with stream-table-duality >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> - it aligns with IQ >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would say that a KTable is a "view >>>>> abstraction" >>>>>>>>> (as >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> materialization is >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> optional). >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -Matthias >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 1/22/17 5:05 PM, Guozhang Wang wrote: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for the KIP Eno, I have a few meta >>>>>>>> comments >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and a few >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> detailed >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comments: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1. I like the materialize() function in >>>>> general, >>>>>>>>> but >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I would >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> like >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>>>>>>> see >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> how other KTable functions should be updated >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> accordingly. For >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> example, >>>>>>>>>>>>>>>>>>>>> >>>> >>> >> >
