> It seems like it must be a ReadOnlyKeyValueStore. Does that sound right?
Yes, it is. Would it be better to add a detailed description of how this feature effects interactive query, with examples? Best, Dongjin On Tue, Sep 29, 2020 at 10:31 AM John Roesler <vvcep...@apache.org> wrote: > Hi Dongjin, > > Thanks! Sorry, I missed your prior message. The proposed API looks good to > me. > > I’m wondering if we should specify what kind of store view would be > returned when querying the operation result. It seems like it must be a > ReadOnlyKeyValueStore. Does that sound right? > > Thanks! > John > > On Mon, Sep 28, 2020, at 10:06, Dongjin Lee wrote: > > Hi John, > > > > I updated the KIP with the discussion above. The 'Public Interfaces' > > section describes the new API, and the 'Rejected Alternatives' section > > describes the reasoning about why we selected this API design and > rejected > > the other alternatives. > > > > Please have a look when you are free. And please note that the KIP freeze > > for 2.7.0 is imminent. > > > > Thanks, > > Dongjin > > > > On Mon, Sep 21, 2020 at 11:35 PM Dongjin Lee <dong...@apache.org> wrote: > > > > > Hi John, > > > > > > I updated the PR applying the API changes we discussed above. I am now > > > updating the KIP document. > > > > > > Thanks, > > > Dongjin > > > > > > On Sat, Sep 19, 2020 at 10:42 AM John Roesler <vvcep...@apache.org> > wrote: > > > > > >> Hi Dongjin, > > >> > > >> Yes, that’s right. My the time of KIP-307, we had no choice but to > add a > > >> second name. But we do have a choice with Suppress. > > >> > > >> Thanks! > > >> -John > > >> > > >> On Thu, Sep 17, 2020, at 13:14, Dongjin Lee wrote: > > >> > Hi John, > > >> > > > >> > I just reviewed KIP-307. As far as I understood, ... > > >> > > > >> > 1. There was Materialized name initially. > > >> > 2. With KIP-307, Named Operations were added. > > >> > 3. Now we have two options for materializing suppression. If we take > > >> > Materialized name here, we have two names for the same operation, > which > > >> is > > >> > not feasible. > > >> > > > >> > Do I understand correctly? > > >> > > > >> > > Do you have a use case in mind for having two separate names for > the > > >> > operation and the view? > > >> > > > >> > No. I am now entirely convinced with your suggestion. > > >> > > > >> > I just started to update the draft implementation. If I understand > > >> > correctly, please notify me; I will update the KIP by adding the > > >> discussion > > >> > above. > > >> > > > >> > Best, > > >> > Dongjin > > >> > > > >> > On Thu, Sep 17, 2020 at 11:06 AM John Roesler <vvcep...@apache.org> > > >> wrote: > > >> > > > >> > > Hi Dongjin, > > >> > > > > >> > > Thanks for the reply. Yes, that’s correct, we added that method to > > >> name > > >> > > the operation. But the operation seems synonymous with the view > > >> produced > > >> > > the operation, right? > > >> > > > > >> > > During KIP-307, I remember thinking that it’s unfortunate the we > had > > >> to > > >> > > have two different “name” concepts for the same thing just because > > >> setting > > >> > > the name on Materialized is equivalent both to making it > queriable and > > >> > > actually materializing it. > > >> > > > > >> > > If we were to reconsider the API, it would be nice to treat these > > >> three as > > >> > > orthogonal: > > >> > > * specify a name > > >> > > * flag to make the view queriable > > >> > > * flag to materialize the view > > >> > > > > >> > > That was the context behind my suggestion. Do you have a use case > in > > >> mind > > >> > > for having two separate names for the operation and the view? > > >> > > > > >> > > Thanks, > > >> > > John > > >> > > > > >> > > On Wed, Sep 16, 2020, at 11:43, Dongjin Lee wrote: > > >> > > > Hi John, > > >> > > > > > >> > > > It seems like the available alternatives in this point is clear: > > >> > > > > > >> > > > 1. Pass queriable name as a separate parameter (i.e., > > >> > > > `KTable#suppress(Suppressed, String)`) > > >> > > > 2. Make use of the Suppression processor name as a queryable > name by > > >> > > adding > > >> > > > `enableQuery` optional flag to `Suppressed`. > > >> > > > > > >> > > > However, I doubt the second approach a little bit; As far as I > > >> know, the > > >> > > > processor name is introduced in KIP-307[^1] to make debugging > > >> topology > > >> > > easy > > >> > > > and understandable. Since the processor name is an independent > > >> concept > > >> > > with > > >> > > > the materialization, I feel the first approach is more natural > and > > >> > > > consistent. Is there any specific reason that you prefer the > second > > >> > > > approach? > > >> > > > > > >> > > > Thanks, > > >> > > > Dongjin > > >> > > > > > >> > > > [^1]: > > >> > > > > > >> > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Wed, Sep 16, 2020 at 11:48 PM John Roesler < > vvcep...@apache.org> > > >> > > wrote: > > >> > > > > > >> > > > > Hi Dongjin, > > >> > > > > > > >> > > > > Yes, that's where I was leaning. Although, I'd prefer adding > > >> > > > > the option to Suppressed instead of adding a new argument to > > >> > > > > the method call. > > >> > > > > > > >> > > > > What do you think about: > > >> > > > > > > >> > > > > class Suppressed<K> { > > >> > > > > + public Suppressed<K> enableQuery(); > > >> > > > > } > > >> > > > > > > >> > > > > Since Suppressed already has `withName(String)`, it seems > > >> > > > > like all we need to add is a boolean flag. > > >> > > > > > > >> > > > > Does that seem sensible to you? > > >> > > > > > > >> > > > > Thanks, > > >> > > > > -John > > >> > > > > > > >> > > > > On Wed, 2020-09-16 at 21:50 +0900, Dongjin Lee wrote: > > >> > > > > > Hi John, > > >> > > > > > > > >> > > > > > > Although it's not great to have "special snowflakes" in > the > > >> API, > > >> > > > > Choice B > > >> > > > > > does seem safer in the short term. We would basically be > > >> proposing a > > >> > > > > > temporary API to make the suppressed view queriable without > a > > >> > > > > Materialized > > >> > > > > > argument. > > >> > > > > > > > >> > > > > > Then, it seems like you prefer `KTable#suppress(Suppressed, > > >> String)` > > >> > > > > (i.e., > > >> > > > > > queriable name only as a parameter) for this time, and > refine > > >> API > > >> > > with > > >> > > > > the > > >> > > > > > other related KIPs later. > > >> > > > > > > > >> > > > > > Do I understand correctly? > > >> > > > > > > > >> > > > > > Thanks, > > >> > > > > > Dongjin > > >> > > > > > > > >> > > > > > On Wed, Sep 16, 2020 at 2:17 AM John Roesler < > > >> vvcep...@apache.org> > > >> > > > > wrote: > > >> > > > > > > > >> > > > > > > Hi Dongjin, > > >> > > > > > > > > >> > > > > > > Thanks for presenting these options. The concern that > > >> > > > > > > Matthias brought up is a very deep problem that afflicts > all > > >> > > > > > > operations downstream of windowing operations. It's the > same > > >> > > > > > > thing that derailed KIP-300. For the larger context, I > have > > >> > > > > > > developed a couple of approaches to resolve this > situation, > > >> > > > > > > but I think it makes sense to finish up KIP-478 before > > >> > > > > > > presenting them. > > >> > > > > > > > > >> > > > > > > However, I don't think that we need in particular to block > > >> > > > > > > the current proposal on solving that long-running and deep > > >> > > > > > > issue with the DSL. Instead, we should make a top-level > > >> > > > > > > decision whether to: > > >> > > > > > > > > >> > > > > > > A: Make Suppress just like all the other KTable > operations. > > >> > > > > > > It will have the same pathological behavior that the > keyset > > >> > > > > > > is unbounded while the store implementation is only a > > >> > > > > > > KeyValueStore. Again, this exact pathology currently > affects > > >> > > > > > > all KTable operations that follow from windowing > operations. > > >> > > > > > > For example, it applies to the current workaround that > > >> > > > > > > Dongjin documented in the KIP: > > >> > > > > > > suppress().filter(Materialized<KeyValueStore>). This is > > >> > > > > > > Option 2 that Dongjin presented. > > >> > > > > > > > > >> > > > > > > B: Do something different with Suppress in order to side- > > >> > > > > > > step the problem. For example, Suppress does not _need_ to > > >> > > > > > > have a separate state store at all. If we just give > people a > > >> > > > > > > switch to make the operation queriable, we can implement a > > >> > > > > > > ReadOnlyKeyValueStore interface by querying the > "priorValue" > > >> > > > > > > of the buffer first and then querying the upstream > > >> > > > > > > ValueGetter. This broad category of "do something > different > > >> > > > > > > with Suppress" encompases Option 1 and Option 3 that > Dongjin > > >> > > > > > > presented. > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > Speaking personally, I think Choice A would be the most > > >> > > > > > > obvious and least weird choice, but it presents a serious > > >> > > > > > > risk of escalating the severity of the problem of > unbounded > > >> > > > > > > state. This is currently a risk that we're aware of, but > has > > >> > > > > > > not yet become a big problem in practice. As Matthias > > >> > > > > > > pointed out, Suppress is far more likely to be used > > >> > > > > > > downstream of windowed tables than other operations, so > > >> > > > > > > having a Materialized<KVStore> overload has the > significant > > >> > > > > > > risk of getting people into a bad state. Note, broadly > > >> > > > > > > advertising the workaround from the KIP would have the > exact > > >> > > > > > > same impact, so we should be careful about recommending > it. > > >> > > > > > > > > >> > > > > > > Although it's not great to have "special snowflakes" in > the > > >> > > > > > > API, Choice B does seem safer in the short term. We would > > >> > > > > > > basically be proposing a temporary API to make the > > >> > > > > > > suppressed view queriable without a Materialized argument. > > >> > > > > > > Then, once we fix the main KIP-300 problem, we would look > at > > >> > > > > > > converging Suppress with the rest of the KTable > > >> > > > > > > materialization APIs. > > >> > > > > > > > > >> > > > > > > WDYT? > > >> > > > > > > Thanks, > > >> > > > > > > -John > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > On Wed, 2020-09-16 at 00:01 +0900, Dongjin Lee wrote: > > >> > > > > > > > Hi Matthias, > > >> > > > > > > > > > >> > > > > > > > Thank you very much for the detailed feedback. Here are > my > > >> > > opinions: > > >> > > > > > > > > > >> > > > > > > > > Because there is no final result for non-windowed > > >> KTables, it > > >> > > seems > > >> > > > > > > that > > >> > > > > > > > this new feature only make sense for the > > >> windowed-aggregation > > >> > > case? > > >> > > > > > > > > > >> > > > > > > > I think a little bit different. Of course, for windowed > > >> KTable, > > >> > > this > > >> > > > > > > > feature provides the final state; for non-windowed > KTables, > > >> it > > >> > > > > provides a > > >> > > > > > > > view to the records received more than the predefined > > >> waiting > > >> > > time > > >> > > > > ago - > > >> > > > > > > > excluding the records that are waiting for more events. > > >> > > > > > > > > > >> > > > > > > > > Thus, the signature of `Materialized` should take a > > >> > > `WindowStore` > > >> > > > > > > instead > > >> > > > > > > > of a `KeyValueStore`? > > >> > > > > > > > > > >> > > > > > > > I reviewed the implementation following your comments > and > > >> found > > >> > > the > > >> > > > > > > > following: > > >> > > > > > > > > > >> > > > > > > > 1. `Materialized` instance includes the following: > KeySerde, > > >> > > > > ValueSerde, > > >> > > > > > > > StoreSupplier, and Queriable Name. > > >> > > > > > > > 2. The other `Materialized` method variants in `KTable` > are > > >> > > making > > >> > > > > use of > > >> > > > > > > > KeySerde, ValueSerde, and Queriable Name only. (That is, > > >> > > > > StoreSupplier is > > >> > > > > > > > ignored.) > > >> > > > > > > > 3. `KTable#suppress(Suppressed, Materialized)` uses the > > >> Queriable > > >> > > > > Name > > >> > > > > > > > only. StoreSupplier is also ignored. > > >> > > > > > > > > > >> > > > > > > > So, we have three choices for the method signature: > > >> > > > > > > > > > >> > > > > > > > 1. `KTable#suppress(Suppressed, String)` (i.e., passing > the > > >> > > Queriable > > >> > > > > > > Name > > >> > > > > > > > only): > > >> > > > > > > > > > >> > > > > > > > This is the simplest; however, it is inconsistent > with the > > >> > > other > > >> > > > > > > > Materialized variant methods. > > >> > > > > > > > > > >> > > > > > > > 2. `KTable#suppress(Suppressed, Materialized<K, V, > > >> > > KeyValueStore>)` > > >> > > > > > > (i.e., > > >> > > > > > > > current proposal) > > >> > > > > > > > > > >> > > > > > > > This approach is harmless at this point, for > > >> StoreSupplier is > > >> > > > > ignored; > > >> > > > > > > > However, since suppression can be used to both of > > >> > > `KeyValueStore` and > > >> > > > > > > > `WindowStore`, this approach is not only weird but also > > >> leaving > > >> > > some > > >> > > > > > > > potential risk to the future. (On second thoughts, I > agree, > > >> this > > >> > > API > > >> > > > > > > design > > >> > > > > > > > is bad and dangerous.) > > >> > > > > > > > > > >> > > > > > > > 3. `KTable#suppress(Suppressed, Materialized<K, V, > > >> StateStore>)` > > >> > > > > > > > > > >> > > > > > > > This approach embraces both of `KeyValueStore` and > > >> > > `WindowStore` > > >> > > > > cases. > > >> > > > > > > > Since the concrete class type of `Suppressed` instance > > >> differs > > >> > > for > > >> > > > > the > > >> > > > > > > > `StateStore`[^1], it seems like we can validate the > > >> arguments on > > >> > > the > > >> > > > > > > method > > >> > > > > > > > call. (e.g., throw `IllegalArgumentException` if when > > >> > > `Suppressed` > > >> > > > > > > instance > > >> > > > > > > > on `KeyValueStore` is given with `Materialized` > instance of > > >> > > > > > > `WindowStore`.) > > >> > > > > > > > This approach not only breaks the API consistency but > also > > >> guards > > >> > > > > from a > > >> > > > > > > > miss-usage of the API. > > >> > > > > > > > > > >> > > > > > > > How do you think? I am now making a try on the third > > >> approach. > > >> > > > > > > > > > >> > > > > > > > Thanks, > > >> > > > > > > > Dongjin > > >> > > > > > > > > > >> > > > > > > > [^1]: `SuppressedInternal` for `KeyValueStore` and > > >> > > > > > > > `FinalResultsSuppressionBuilder` for `WindowStore`. > > >> > > > > > > > > > >> > > > > > > > On Sat, Sep 12, 2020 at 3:29 AM Matthias J. Sax < > > >> > > mj...@apache.org> > > >> > > > > > > wrote: > > >> > > > > > > > > Thanks for updating the KIP. > > >> > > > > > > > > > > >> > > > > > > > > I think there is still one open question. `suppress()` > > >> can be > > >> > > used > > >> > > > > on > > >> > > > > > > > > non-windowed KTable for rate control, as well as on a > > >> > > > > windowed-KTable > > >> > > > > > > > > (also for rate control, but actually mainly) for only > > >> emitting > > >> > > the > > >> > > > > > > final > > >> > > > > > > > > result of a windowed aggregation. For the non-windowed > > >> case, we > > >> > > > > use a > > >> > > > > > > > > KeyValueStore while for the windowed cases, we use a > > >> > > WindowStore. > > >> > > > > > > > > > > >> > > > > > > > > Because there is no final result for non-windowed > > >> KTables, it > > >> > > seems > > >> > > > > > > that > > >> > > > > > > > > this new feature only make sense for the > > >> windowed-aggregation > > >> > > case? > > >> > > > > > > > > Thus, the signature of `Materialized` should take a > > >> > > `WindowStore` > > >> > > > > > > > > instead of a `KeyValueStore`? > > >> > > > > > > > > > > >> > > > > > > > > If that's correct, I am wondering: > > >> > > > > > > > > > > >> > > > > > > > > - Can we guard from a miss-usage of the API if the > > >> upstream > > >> > > > > KTable is > > >> > > > > > > > > not windowed (or maybe it's not necessary to guard)? > > >> > > > > > > > > - Can we actually implement it? We had issues with > > >> regard to > > >> > > > > KIP-300 > > >> > > > > > > to > > >> > > > > > > > > materialize windowed-KTables? > > >> > > > > > > > > > > >> > > > > > > > > Would be worth to clarify upfront. Maybe, we even > need a > > >> POC > > >> > > > > > > > > implementation to verify that it works? > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > -Matthias > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > On 9/11/20 12:26 AM, Dongjin Lee wrote: > > >> > > > > > > > > > Hi All, > > >> > > > > > > > > > > > >> > > > > > > > > > Here is the voting thread: > > >> > > > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > https://lists.apache.org/thread.html/r5653bf2dafbb27b247bf20dbe6f070c151b3823d96c9c9ca94183e20%40%3Cdev.kafka.apache.org%3E > > >> > > > > > > > > > Thanks, > > >> > > > > > > > > > Dongjin > > >> > > > > > > > > > > > >> > > > > > > > > > On Fri, Sep 11, 2020 at 4:23 PM Dongjin Lee < > > >> > > dong...@apache.org> > > >> > > > > > > wrote: > > >> > > > > > > > > > > Hi John, > > >> > > > > > > > > > > > > >> > > > > > > > > > > Thanks for the feedback. I will open the Vote > thread > > >> now. > > >> > > > > > > > > > > > > >> > > > > > > > > > > Best, > > >> > > > > > > > > > > Dongjin > > >> > > > > > > > > > > > > >> > > > > > > > > > > On Fri, Sep 11, 2020 at 2:00 AM John Roesler < > > >> > > > > vvcep...@apache.org> > > >> > > > > > > > > wrote: > > >> > > > > > > > > > > > Hi Dongjin, > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Sorry for the delay. I'm glad you're still > pushing > > >> this > > >> > > > > > > > > > > > forward. It would be nice to get this in to the > 2.7 > > >> > > release. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > I just took another look at the KIP, and it > looks > > >> good to > > >> > > > > > > > > > > > me! > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > I think this is ready for a vote. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks, > > >> > > > > > > > > > > > -John > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > On Wed, 2020-08-05 at 22:04 +0900, Dongjin Lee > > >> wrote: > > >> > > > > > > > > > > > > Hi All, > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > I updated the KIP > > >> > > > > > > > > > > > > < > > >> > > > > > > > > >> > > > > > > >> > > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-508%3A+Make+Suppression+State+Queriable > > >> > > > > > > > > > > > > and the implementation, following the > discussion > > >> here. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > You must be working hard preparing the > release of > > >> > > 2.6.0, so > > >> > > > > > > please > > >> > > > > > > > > have > > >> > > > > > > > > > > > a > > >> > > > > > > > > > > > > look after your work is done. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks, > > >> > > > > > > > > > > > > Dongjin > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Sun, Mar 8, 2020 at 12:20 PM John Roesler < > > >> > > > > > > vvcep...@apache.org> > > >> > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > Thanks Matthias, > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Good idea. I've changed the ticket name and > > >> added a > > >> > > note > > >> > > > > > > > > > > > > > clarifying that this ticket is not the same > as > > >> > > > > > > > > > > > > > > > >> https://issues.apache.org/jira/browse/KAFKA-7224 > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Incidentally, I learned that I never > documented > > >> my > > >> > > > > reasons > > >> > > > > > > > > > > > > > for abandoning my work on KAFKA-7224 ! I've > now > > >> > > updated > > >> > > > > > > > > > > > > > that ticket, too, so your question had an > > >> unexpected > > >> > > > > > > side-benefit. > > >> > > > > > > > > > > > > > Thanks, > > >> > > > > > > > > > > > > > -John > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Sat, Mar 7, 2020, at 18:01, Matthias J. > Sax > > >> wrote: > > >> > > > > > > > > > Thanks for clarification. > > >> > > > > > > > > > > > >> > > > > > > > > > Can you maybe update the Jira ticket? Do we have a > > >> ticket for > > >> > > > > > > > > > spill-to-disk? Maybe link to it and explain that > it's > > >> two > > >> > > > > different > > >> > > > > > > > > > things? Maybe even rename the ticket to something > more > > >> > > clear, ie, > > >> > > > > > > > > > "make suppress result queryable" or simliar? > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > -Matthias > > >> > > > > > > > > > > > >> > > > > > > > > > On 3/7/20 1:58 PM, John Roesler wrote: > > >> > > > > > > > > > > > > > > > > Hey Matthias, > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > I’m sorry if the ticket was poorly > > >> stated. The > > >> > > > > ticket > > >> > > > > > > is to add a > > >> > > > > > > > > > DSL overload to pass a Materialized argument to > > >> suppress. As > > >> > > a > > >> > > > > > > > > > > > > result, > > >> > > > > > > > > > the result of the suppression would be queriable. > > >> > > > > > > > > > > > > > > > > This is unrelated to “persistent > buffer” > > >> aka > > >> > > > > > > “spill-to-disk”. > > >> > > > > > > > > > > > > > > > > There was some confusion before about > > >> whether > > >> > > this > > >> > > > > > > ticket could be > > >> > > > > > > > > > implemented as “query the buffer”. Maybe it can, > but not > > >> > > > > trivially. > > >> > > > > > > > > > The obvious way is just to add a new state store > which > > >> we > > >> > > write > > >> > > > > the > > >> > > > > > > > > > results into just before we forward. I.e., it’s > exactly > > >> like > > >> > > the > > >> > > > > > > > > > materialized variant of any stateless KTable > operation. > > >> > > > > > > > > > > > > > > > > Thanks, John > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Sat, Mar 7, 2020, at 15:32, > Matthias > > >> J. Sax > > >> > > > > wrote: > > >> > > > > > > Thanks for > > >> > > > > > > > > > > > > > > > > the KIP Dongjin, > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > I am still not sure if I can follow, > what > > >> might > > >> > > > > also > > >> > > > > > > be caused by > > >> > > > > > > > > > > > > > > > > the backing JIRA ticket (maybe John > can > > >> > > clarify the > > >> > > > > > > intent of the > > >> > > > > > > > > > > > > > > > > ticket as he created it): > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Currently, suppress() only uses an > > >> in-memory > > >> > > buffer > > >> > > > > > > and my > > >> > > > > > > > > > > > > > > > > understanding of the Jira is, to add > the > > >> > > ability to > > >> > > > > > > use a > > >> > > > > > > > > > > > > > > > > persistent buffer (ie, spill to disk > > >> backed by > > >> > > > > > > RocksDB). > > >> > > > > > > > > > > > > > > > > Adding a persistent buffer is > completely > > >> > > unrelated > > >> > > > > to > > >> > > > > > > allow > > >> > > > > > > > > > > > > > > > > querying the buffer. In fact, one > could > > >> query > > >> > > an > > >> > > > > > > in-memory buffer, > > >> > > > > > > > > > > > > > > > > too. However, querying the buffer > does not > > >> > > really > > >> > > > > seem > > >> > > > > > > to be > > >> > > > > > > > > > > > > useful > > >> > > > > > > > > > > > > > > > > as pointed out by John, as you can > always > > >> > > query the > > >> > > > > > > upstream > > >> > > > > > > > > > > > > KTable > > >> > > > > > > > > > > > > > > > > store. > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Also note that for the > > >> emit-on-window-close > > >> > > case > > >> > > > > the > > >> > > > > > > result is > > >> > > > > > > > > > > > > > > > > deleted from the buffer when it is > > >> emitted, and > > >> > > > > thus > > >> > > > > > > cannot be > > >> > > > > > > > > > > > > > > > > queried any longe r. > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Can you please clarify if you intend > to > > >> allow > > >> > > > > spilling > > >> > > > > > > to disk or > > >> > > > > > > > > > > > > > > > > if you intent to enable IQ (even if I > > >> don't > > >> > > see why > > >> > > > > > > querying make > > >> > > > > > > > > > > > > > > > > sense, as the data is either upstream > or > > >> > > deleted). > > >> > > > > > > Also, if you > > >> > > > > > > > > > > > > > > > > want to enable IQ, why do we need all > > >> those new > > >> > > > > > > interfaces? The > > >> > > > > > > > > > > > > > > > > result of a suppress() is a KTable > that > > >> is the > > >> > > > > same as > > >> > > > > > > any other > > >> > > > > > > > > > > > > > > > > key-value/windowed/sessions store? > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > We should also have corresponding Jira > > >> tickets > > >> > > for > > >> > > > > > > different cases > > >> > > > > > > > > > > > > > > > > to avoid the confusion I am in atm :) > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > -Matthias > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On 2/27/20 8:21 AM, John Roesler > wrote: > > >> > > > > > > > > > > > > > > > > > > > Hi Dongjin, > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > No problem; glad we got it > sorted > > >> out. > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks again for picking this > up! > > >> -John > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Wed, Feb 26, 2020, at 09:24, > > >> Dongjin > > >> > > Lee > > >> > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > > > > > I was under the impression > that > > >> you > > >> > > > > wanted > > >> > > > > > > to expand the > > >> > > > > > > > > > > > > > > > > > > > > > scope of the KIP > > >> > > > > > > > > > > > > > > > > > > > > to additionally allow > querying the > > >> > > internal > > >> > > > > > > buffer, not > > >> > > > > > > > > > > > > > > > > > > > > just the result. Can you > clarify > > >> > > whether > > >> > > > > you > > >> > > > > > > are proposing > > >> > > > > > > > > > > > > > > > > > > > > to allow querying the state > of the > > >> > > internal > > >> > > > > > > buffer, the > > >> > > > > > > > > > > > > > > > > > > > > result, or both? > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Sorry for the confusion. As we > > >> already > > >> > > > > talked > > >> > > > > > > with, we > > >> > > > > > > > > > > > > only > > >> > > > > > > > > > > > > > > > > > > > > need to query the suppressed > > >> output, > > >> > > not > > >> > > > > the > > >> > > > > > > internal > > >> > > > > > > > > > > > > > > > > > > > > buffer. The current > > >> implementation is > > >> > > > > wrong. > > >> > > > > > > After > > >> > > > > > > > > > > > > refining > > >> > > > > > > > > > > > > > > > > > > > > the KIP and implementation > > >> accordingly > > >> > > I > > >> > > > > will > > >> > > > > > > notify you - > > >> > > > > > > > > > > > > > > > > > > > > I must be confused, also. > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > On Tue, Feb 25, 2020 at 12:17 > AM > > >> John > > >> > > > > Roesler > > >> > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> wrote: > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Hi Dongjin, > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Ah, I think I may have been > > >> > > confused. I > > >> > > > > 100% > > >> > > > > > > agree that > > >> > > > > > > > > > > > > > > > > > > > > > we need a materialized > variant > > >> for > > >> > > > > > > suppress(). Then, you > > >> > > > > > > > > > > > > > > > > > > > > > could do: ...suppress(..., > > >> > > > > > > > > > > > > > > > > > > > > > > Materialized.as(“final-count”)) > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > If that’s your proposal, > then > > >> we are > > >> > > on > > >> > > > > the > > >> > > > > > > same page. > > >> > > > > > > > > > > > > > > > > > > > > > I was under the impression > that > > >> you > > >> > > > > wanted > > >> > > > > > > to expand the > > >> > > > > > > > > > > > > > > > > > > > > > scope of the KIP to > additionally > > >> > > allow > > >> > > > > > > querying the > > >> > > > > > > > > > > > > > > > > > > > > > internal buffer, not just > the > > >> > > result. Can > > >> > > > > > > you clarify > > >> > > > > > > > > > > > > > > > > > > > > > whether you are proposing to > > >> allow > > >> > > > > querying > > >> > > > > > > the state of > > >> > > > > > > > > > > > > > > > > > > > > > the internal buffer, the > > >> result, or > > >> > > both? > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Thanks, John > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 20, 2020, at > 08:41, > > >> > > Dongjin > > >> > > > > Lee > > >> > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > > > > > > Hi John, Thanks for your > kind > > >> > > > > explanation > > >> > > > > > > with an > > >> > > > > > > > > > > > > > > > > > > > > > > example. > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > But it feels like you're > > >> saying > > >> > > > > you're > > >> > > > > > > trying to do > > >> > > > > > > > > > > > > > > > > > > > > > > > something different > > >> > > > > > > > > > > > > > > > > > > > > > > than just query the > windowed > > >> key > > >> > > and > > >> > > > > get > > >> > > > > > > back the > > >> > > > > > > > > > > > > > > > > > > > > > > current count? > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Yes, for example, what if > we > > >> need > > >> > > to > > >> > > > > > > retrieve the (all > > >> > > > > > > > > > > > > > > > > > > > > > > or range) keys > > >> > > > > > > > > > > > > > > > > > > > > > with > > >> > > > > > > > > > > > > > > > > > > > > > > a closed window? In this > > >> example, > > >> > > let's > > >> > > > > > > imagine we > > >> > > > > > > > > > > > > need > > >> > > > > > > > > > > > > > > > > > > > > > > to retrieve only (key=A, > > >> > > window=10), > > >> > > > > not > > >> > > > > > > (key=A, > > >> > > > > > > > > > > > > > > > > > > > > > > window=20). > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Of course, the value > > >> accompanied > > >> > > by a > > >> > > > > > > flushed key is > > >> > > > > > > > > > > > > > > > > > > > > > > exactly the same to the > one > > >> in the > > >> > > > > > > upstream KTable; > > >> > > > > > > > > > > > > > > > > > > > > > > However, if our intention > is > > >> not > > >> > > > > pointing > > >> > > > > > > out a > > >> > > > > > > > > > > > > > > > > > > > > > > specific key but > retrieving a > > >> > > group of > > >> > > > > > > unspecified > > >> > > > > > > > > > > > > > > > > > > > > > > keys, we stuck > > >> > > > > > > > > > > > > > > > > > > > > > in > > >> > > > > > > > > > > > > > > > > > > > > > > trouble - since we can't > be > > >> sure > > >> > > which > > >> > > > > key > > >> > > > > > > is flushed > > >> > > > > > > > > > > > > > > > > > > > > > > out beforehand. > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > One workaround would be > > >> > > materializing > > >> > > > > it > > >> > > > > > > with > > >> > > > > > > > > > > > > > > > > > > > > > > `suppressed.filter(e -> > > >> > > > > > > > > > > > > > > > > > > > > > true, > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> Materialized.as("final-count"))`. > > >> > > But I > > >> > > > > > > think > > >> > > > > > > > > > > > > providing > > >> > > > > > > > > > > > > > > > > > > > > > > a materialized variant for > > >> suppress > > >> > > > > method > > >> > > > > > > is better > > >> > > > > > > > > > > > > > > > > > > > > > > than this workaround. > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 20, 2020 at > 1:26 > > >> AM > > >> > > John > > >> > > > > > > Roesler > > >> > > > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> > > >> > > > > > > > > > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > > > > > > > Thanks for the response, > > >> Dongjin, > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > I'm sorry, but I'm > still not > > >> > > > > following. > > >> > > > > > > It seems > > >> > > > > > > > > > > > > like > > >> > > > > > > > > > > > > > > > > > > > > > > > the view you > > >> > > > > > > > > > > > > > > > > > > > > > would > > >> > > > > > > > > > > > > > > > > > > > > > > > get on the "current > state > > >> of the > > >> > > > > buffer" > > >> > > > > > > would > > >> > > > > > > > > > > > > always > > >> > > > > > > > > > > > > > > > > > > > > > > > be equivalent to the > view > > >> of the > > >> > > > > > > upstream table. > > >> > > > > > > > > > > > > > > > > > > > > > > > Let me try an example, > and > > >> maybe > > >> > > you > > >> > > > > can > > >> > > > > > > point out > > >> > > > > > > > > > > > > > > > > > > > > > > > the flaw in my > reasoning. > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Let's say we're doing > 10 ms > > >> > > windows > > >> > > > > with > > >> > > > > > > a grace > > >> > > > > > > > > > > > > > > > > > > > > > > > period of zero. Let's > also > > >> say > > >> > > we're > > >> > > > > > > computing a > > >> > > > > > > > > > > > > > > > > > > > > > > > windowed count, and > that we > > >> have > > >> > > a > > >> > > > > > > "final results" > > >> > > > > > > > > > > > > > > > > > > > > > > > suppression after the > count. > > >> > > Let's > > >> > > > > > > materialize the > > >> > > > > > > > > > > > > > > > > > > > > > > > count as "Count" and the > > >> > > suppressed > > >> > > > > > > result as "Final > > >> > > > > > > > > > > > > > > > > > > > > > > > Count". > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Suppose we get an input > > >> event: > > >> > > > > (time=10, > > >> > > > > > > key=A, > > >> > > > > > > > > > > > > > > > > > > > > > > > value=...) > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look > like: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > | | > > >> 10 > > >> > > | A > > >> > > > > > > | 1 | > > >> > > > > > > > > > > > > > > > > > > > > > > > The (internal) > suppression > > >> buffer > > >> > > > > will > > >> > > > > > > contain: > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > | | > > >> 10 > > >> > > | A > > >> > > > > > > | 1 | > > >> > > > > > > > > > > > > > > > > > > > > > > > The record is still > buffered > > >> > > because > > >> > > > > the > > >> > > > > > > window > > >> > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > > >> Count is > > >> > > an > > >> > > > > > > empty table: > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > --------------- > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Now, we get a second > event: > > >> > > (time=15, > > >> > > > > > > key=A, > > >> > > > > > > > > > > > > > > > > > > > > > > > value=...) > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look > like: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > | | > > >> 10 > > >> > > | A > > >> > > > > > > | 2 | > > >> > > > > > > > > > > > > > > > > > > > > > > > The (internal) > suppression > > >> buffer > > >> > > > > will > > >> > > > > > > contain: > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > | | > > >> 10 > > >> > > | A > > >> > > > > > > | 2 | > > >> > > > > > > > > > > > > > > > > > > > > > > > The record is still > buffered > > >> > > because > > >> > > > > the > > >> > > > > > > window > > >> > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > > >> Count is > > >> > > an > > >> > > > > > > empty table: > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > --------------- > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Finally, we get a third > > >> event: > > >> > > > > (time=20, > > >> > > > > > > key=A, > > >> > > > > > > > > > > > > > > > > > > > > > > > value=...) > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look > like: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > | | > > >> 10 > > >> > > | A > > >> > > > > > > | 2 | | > > >> > > > > > > > > > > > > > > > > > > > > > > > 20 | A | 1 | > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > The (internal) > suppression > > >> buffer > > >> > > > > will > > >> > > > > > > contain: > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > | | > > >> 20 > > >> > > | A > > >> > > > > > > | 1 | > > >> > > > > > > > > > > > > > > > > > > > > > > > Note that window 10 has > been > > >> > > flushed > > >> > > > > > > out, because > > >> > > > > > > > > > > > > > > > > > > > > > > > it's now closed. And > window > > >> 20 is > > >> > > > > > > buffered because > > >> > > > > > > > > > > > > it > > >> > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > > >> Count is > > >> > > now: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > | | > > >> 10 > > >> > > | A > > >> > > > > > > | 2 | > > >> > > > > > > > > > > > > > > > > > > > > > > > --------------- > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > Reading your email, I > can't > > >> > > figure > > >> > > > > out > > >> > > > > > > what value > > >> > > > > > > > > > > > > > > > > > > > > > > > there is in querying > > >> > > > > > > > > > > > > > > > > > > > > > the > > >> > > > > > > > > > > > > > > > > > > > > > > > internal suppression > buffer, > > >> > > since it > > >> > > > > > > only contains > > >> > > > > > > > > > > > > > > > > > > > > > > > exactly the same > > >> > > > > > > > > > > > > > > > > > > > > > value > > >> > > > > > > > > > > > > > > > > > > > > > > > as the upstream table, > for > > >> each > > >> > > key > > >> > > > > that > > >> > > > > > > is still > > >> > > > > > > > > > > > > > > > > > > > > > > > buffered. But it feels > > >> > > > > > > > > > > > > > > > > > > > > > like > > >> > > > > > > > > > > > > > > > > > > > > > > > you're saying you're > trying > > >> to do > > >> > > > > > > something > > >> > > > > > > > > > > > > different > > >> > > > > > > > > > > > > > > > > > > > > > > > than just query > > >> > > > > > > > > > > > > > > > > > > > > > the > > >> > > > > > > > > > > > > > > > > > > > > > > > windowed key and get > back > > >> the > > >> > > current > > >> > > > > > > count? > > >> > > > > > > > > > > > > > > > > > > > > > > > Thanks, -John > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 19, 2020, at > > >> 09:49, > > >> > > > > Dongjin > > >> > > > > > > Lee wrote: > > >> > > > > > > > > > > > > > > > > > > > > > > > > Hi John, > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > 'The intermediate > state > > >> of the > > >> > > > > > > suppression' in KIP > > >> > > > > > > > > > > > > > > > > > > > > > > > > does not mean the > > >> > > > > > > > > > > > > > > > > > > > > > > > state > > >> > > > > > > > > > > > > > > > > > > > > > > > > of upstream KTable - > > >> sure, the > > >> > > > > state > > >> > > > > > > of the > > >> > > > > > > > > > > > > > > > > > > > > > > > > upstream KTable can be > > >> > > > > > > > > > > > > > > > > > > > > > > > queried > > >> > > > > > > > > > > > > > > > > > > > > > > > > by materializing the > > >> operator > > >> > > > > > > immediately before > > >> > > > > > > > > > > > > > > > > > > > > > > > > the suppress as you > > >> > > > > > > > > > > > > > > > > > > > > > > > shown. > > >> > > > > > > > > > > > > > > > > > > > > > > > > What I meant in KIP > was > > >> the > > >> > > final > > >> > > > > > > state of the > > >> > > > > > > > > > > > > > > > > > > > > > > > > buffer, which is not > > >> > > > > > > > > > > > > > > > > > > > > > > > emitted > > >> > > > > > > > > > > > > > > > > > > > > > > > > yet. (I agree, the > current > > >> > > > > description > > >> > > > > > > may be > > >> > > > > > > > > > > > > > > > > > > > > > > > > confusing; it would be > > >> > > > > > > > > > > > > > > > > > > > > > > > better > > >> > > > > > > > > > > > > > > > > > > > > > > > > to change it with 'the > > >> current > > >> > > > > state > > >> > > > > > > of the > > >> > > > > > > > > > > > > > > > > > > > > > > > > suppression' or 'the > > >> > > > > > > > > > > > > > > > > > > > > > results > > >> > > > > > > > > > > > > > > > > > > > > > > > of > > >> > > > > > > > > > > > > > > > > > > > > > > > > the suppression', like > > >> the Jira > > >> > > > > issue > > >> > > > > > > > > > > > > > > > > > > > > > > > > < > > >> > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > > >> > > > > > > > > > > > > > > > > > > > > > > > > states.) > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > For a little bit more > > >> about the > > >> > > > > > > motivation, here > > >> > > > > > > > > > > > > is > > >> > > > > > > > > > > > > > > > > > > > > > > > > one of my > > >> > > > > > > > > > > > > > > > > > > > > > > > experience: I > > >> > > > > > > > > > > > > > > > > > > > > > > > > had to build a > monitoring > > >> > > > > application > > >> > > > > > > which > > >> > > > > > > > > > > > > > > > > > > > > > > > > collects signals from > IoT > > >> > > devices > > >> > > > > > > (say, a > > >> > > > > > > > > > > > > > > > > > > > > > > > > semiconductor > production > > >> > > line.) If > > >> > > > > the > > >> > > > > > > number of > > >> > > > > > > > > > > > > > > > > > > > > > > > collected > > >> > > > > > > > > > > > > > > > > > > > > > > > > signals within the > time > > >> window > > >> > > is > > >> > > > > much > > >> > > > > > > less than > > >> > > > > > > > > > > > > > > > > > > > > > > > > the expected, there > > >> > > > > > > > > > > > > > > > > > > > > > may > > >> > > > > > > > > > > > > > > > > > > > > > > > be > > >> > > > > > > > > > > > > > > > > > > > > > > > > some problems like > network > > >> > > hiccup > > >> > > > > in > > >> > > > > > > the systems. > > >> > > > > > > > > > > > > > > > > > > > > > > > > We wanted to build > > >> > > > > > > > > > > > > > > > > > > > > > the > > >> > > > > > > > > > > > > > > > > > > > > > > > > system in the form of > a > > >> > > dashboard, > > >> > > > > but > > >> > > > > > > could not > > >> > > > > > > > > > > > > by > > >> > > > > > > > > > > > > > > > > > > > > > > > > lack of > > >> > > > > > > > > > > > > > > > > > > > > > materializing > > >> > > > > > > > > > > > > > > > > > > > > > > > > feature. It was > precisely > > >> the > > >> > > case > > >> > > > > of > > >> > > > > > > querying > > >> > > > > > > > > > > > > only > > >> > > > > > > > > > > > > > > > > > > > > > > > > the final > > >> > > > > > > > > > > > > > > > > > > > > > results of > > >> > > > > > > > > > > > > > > > > > > > > > > > a > > >> > > > > > > > > > > > > > > > > > > > > > > > > windowed aggregation, > as > > >> the > > >> > > Jira > > >> > > > > issue > > >> > > > > > > > > > > > > > > > > > > > > > > > > < > > >> > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > > >> > > > > > > > > > > > > > > > > > > > > > > > > states. We > > >> > > > > > > > > > > > > > > > > > > > > > finally > > >> > > > > > > > > > > > > > > > > > > > > > > > ended > > >> > > > > > > > > > > > > > > > > > > > > > > > > in implementing the > > >> system in > > >> > > an > > >> > > > > email > > >> > > > > > > alerting > > >> > > > > > > > > > > > > > > > > > > > > > > > > system like this < > > >> > > > > > > > > >> https://www.confluent.io/blog/kafka-streams-take-on-watermarks-an > > >> > > > > > > > > > d-t > > >> > > > > > > > > > riggers/ > > >> > > > > > > > > > > > > > > > > and had to collect the keys and > windows of > > >> > > trouble > > >> > > > > by > > >> > > > > > > hand. > > >> > > > > > > > > > > > > > > > > > > > > > > > > I think these kinds > of use > > >> > > cases > > >> > > > > would > > >> > > > > > > be much > > >> > > > > > > > > > > > > > > > > > > > > > > > > common. Should it be > > >> described > > >> > > in > > >> > > > > the > > >> > > > > > > KIP much > > >> > > > > > > > > > > > > more > > >> > > > > > > > > > > > > > > > > > > > > > > > > in detail? > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Feb 15, 2020 > at > > >> 4:43 AM > > >> > > > > John > > >> > > > > > > Roesler > > >> > > > > > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> > > >> > > > > > > > > > > > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Hi Dongjin, > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP! > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Can you explain more > > >> about > > >> > > why > > >> > > > > the > > >> > > > > > > internal data > > >> > > > > > > > > > > > > > > > > > > > > > > > > > structures of > > >> > > > > > > > > > > > > > > > > > > > > > > > suppression > > >> > > > > > > > > > > > > > > > > > > > > > > > > > should be > queriable? The > > >> > > > > motivation > > >> > > > > > > just says > > >> > > > > > > > > > > > > > > > > > > > > > > > > > that users might > > >> > > > > > > > > > > > > > > > > > > > > > want to > > >> > > > > > > > > > > > > > > > > > > > > > > > do > > >> > > > > > > > > > > > > > > > > > > > > > > > > > it, which seems > like it > > >> could > > >> > > > > > > justify literally > > >> > > > > > > > > > > > > > > > > > > > > > > > > > anything :) > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > One design point of > > >> > > Suppression > > >> > > > > is > > >> > > > > > > that if you > > >> > > > > > > > > > > > > > > > > > > > > > > > > > wanted to query the > > >> > > > > > > > > > > > > > > > > > > > > > > > “final > > >> > > > > > > > > > > > > > > > > > > > > > > > > > state”, you can > > >> Materialize > > >> > > the > > >> > > > > > > suppress itself > > >> > > > > > > > > > > > > > > > > > > > > > > > > > (which is why it > > >> > > > > > > > > > > > > > > > > > > > > > needs > > >> > > > > > > > > > > > > > > > > > > > > > > > the > > >> > > > > > > > > > > > > > > > > > > > > > > > > > variant); if you > wanted > > >> to > > >> > > query > > >> > > > > the > > >> > > > > > > > > > > > > > > > > > > > > > > > > > “intermediate > state”, > > >> you can > > >> > > > > > > materialize the > > >> > > > > > > > > > > > > > > > > > > > > > > > > > operator immediately > > >> before > > >> > > the > > >> > > > > > > suppress. > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Example: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > ...count(Materialized.as(“intermediate”)) > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> .supress(untilWindowClosed(), > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Materialized.as(“final”)) > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > I’m not sure what > use > > >> case > > >> > > would > > >> > > > > > > require > > >> > > > > > > > > > > > > > > > > > > > > > > > > > actually fetching > from > > >> the > > >> > > > > internal > > >> > > > > > > buffers. > > >> > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, John > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 14, > 2020, at > > >> > > 07:55, > > >> > > > > > > Dongjin Lee > > >> > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to > reboot the > > >> > > > > discussion > > >> > > > > > > on KIP-508, > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > which aims to > > >> > > > > > > > > > > > > > > > > > > > > > support a > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Materialized > variant > > >> of > > >> > > > > > > KTable#suppress. It > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > was initially > > >> > > > > > > > > > > > > > > > > > > > > > submitted > > >> > > > > > > > > > > > > > > > > > > > > > > > > > several > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > months ago but > closed > > >> by > > >> > > the > > >> > > > > > > inactivity. > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > - KIP: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-508%3A+Make > > >> > > > > > > > > > +Su > > >> > > > > > > > > > ppression+State+Queriable > > >> > > > > > > > > > > > > > > > > - Jira: > > >> > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > All kinds of > feedback > > >> will > > >> > > be > > >> > > > > > > greatly > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated. > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > Best, Dongjin > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in > the > > >> > > > > mathematical > > >> > > > > > > world.* > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > *github: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > < > > >> http://goog_969573159/> > > >> > > > > > > > > > > > > github.com/dongjinleekr > > >> > > > > > > > > > <https://github.com/dongjinleekr>linkedin: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > < > > >> > > > > > > https://kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > > > > > > > speakerdeck: > > >> > > > > > > > > > speakerdeck.com/dongjin > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > < > > >> > > > > https://speakerdeck.com/dongjin>* > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > > >> > > mathematical > > >> > > > > > > world.* *github: > > >> > > > > > > > > > > > > > > > > > > > > > > > > < > http://goog_969573159/> > > >> > > > > > > github.com/dongjinleekr > > >> > > > > > > > > > > > > > > > > > > > > > > > > < > > >> > > https://github.com/dongjinleekr > > >> > > > > > > > linkedin: > > >> > > > > > > > > > > > > > > > > > > > > > > > > > >> kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > > > > > > > > > > > > > > > > > > < > > >> > > > > > > https://kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > > > > > > > speakerdeck: > > >> > > > > > > > > > speakerdeck.com/dongjin > > >> > > > > > > > > > > > > > > > > > > > > > > > > < > > >> > > https://speakerdeck.com/dongjin>* > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > > >> mathematical > > >> > > > > world.* > > >> > > > > > > *github: > > >> > > > > > > > > > > > > > > > > > > > > > > <http://goog_969573159/> > > >> > > > > > > github.com/dongjinleekr > > >> > > > > > > > > > > > > > > > > > > > > > > < > > >> https://github.com/dongjinleekr > > >> > > > > >linkedin: > > >> > > > > > > > > > > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > > > > > > > > > > > > > > > > < > > >> > > > > https://kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > speakerdeck: > > >> > > > > > > > > > > > > > > > > > > > > > speakerdeck.com/dongjin > > >> > > > > > > > > > > > > > > > > > > > > > > < > > >> https://speakerdeck.com/dongjin>* > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > mathematical > > >> > > world.* > > >> > > > > > > *github: > > >> > > > > > > > > > > > > > > > > > > > > <http://goog_969573159/> > > >> > > > > > > github.com/dongjinleekr > > >> > > > > > > > > > > > > > > > > > > > > < > https://github.com/dongjinleekr > > >> > > >linkedin: > > >> > > > > > > > > > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > > > > > > > > > > > > > > < > > >> > > https://kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > speakerdeck: > > >> > > > > > > > > > > > > > > > > > > > > speakerdeck.com/dongjin < > > >> > > > > > > https://speakerdeck.com/dongjin > > >> > > > > > > > > > > > > > * > > >> > > > > > > > > > > > > >> > > > > > > > > > > -- > > >> > > > > > > > > > > *Dongjin Lee* > > >> > > > > > > > > > > > > >> > > > > > > > > > > *A hitchhiker in the mathematical world.* > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > *github: <http://goog_969573159/> > > >> github.com/dongjinleekr > > >> > > > > > > > > > > <https://github.com/dongjinleekr>keybase: > > >> > > > > > > > > https://keybase.io/dongjinleekr > > >> > > > > > > > > > > <https://keybase.io/dongjinleekr>linkedin: > > >> > > > > > > > > kr.linkedin.com/in/dongjinleekr > > >> > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > >speakerdeck: > > >> > > > > > > > > speakerdeck.com/dongjin > > >> > > > > > > > > > > <https://speakerdeck.com/dongjin>* > > >> > > > > > > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > -- > > >> > > > *Dongjin Lee* > > >> > > > > > >> > > > *A hitchhiker in the mathematical world.* > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > >> > > > <https://github.com/dongjinleekr>keybase: > > >> > > https://keybase.io/dongjinleekr > > >> > > > <https://keybase.io/dongjinleekr>linkedin: > > >> > > kr.linkedin.com/in/dongjinleekr > > >> > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > >> > > speakerdeck.com/dongjin > > >> > > > <https://speakerdeck.com/dongjin>* > > >> > > > > > >> > > > > >> > > > >> > > > >> > -- > > >> > *Dongjin Lee* > > >> > > > >> > *A hitchhiker in the mathematical world.* > > >> > > > >> > > > >> > > > >> > > > >> > *github: <http://goog_969573159/>github.com/dongjinleekr > > >> > <https://github.com/dongjinleekr>keybase: > > >> https://keybase.io/dongjinleekr > > >> > <https://keybase.io/dongjinleekr>linkedin: > > >> kr.linkedin.com/in/dongjinleekr > > >> > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > >> speakerdeck.com/dongjin > > >> > <https://speakerdeck.com/dongjin>* > > >> > > > >> > > > > > > > > > -- > > > *Dongjin Lee* > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > <https://github.com/dongjinleekr>keybase: > https://keybase.io/dongjinleekr > > > <https://keybase.io/dongjinleekr>linkedin: > kr.linkedin.com/in/dongjinleekr > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > -- > > *Dongjin Lee* > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > <https://github.com/dongjinleekr>keybase: > https://keybase.io/dongjinleekr > > <https://keybase.io/dongjinleekr>linkedin: > kr.linkedin.com/in/dongjinleekr > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > speakerdeck.com/dongjin > > <https://speakerdeck.com/dongjin>* > > > -- *Dongjin Lee* *A hitchhiker in the mathematical world.* *github: <http://goog_969573159/>github.com/dongjinleekr <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin <https://speakerdeck.com/dongjin>*
