Thanks Dongjin, It typically is nicer to be able to see usage examples, so I'd certainly be in favor if you're willing to add it to the KIP.
I'm wondering if it's possible to implement the whole ReadOnlyKeyValueStore interface as proposed, if we really go ahead and just internally query into the suppression buffer as well as using the upstream ValueGetter. The reason is twofold: 1. The suppression buffer is ordered by arrival time, not by key. There is a by-key index, but it is also not ordered the same way that in-memory stores are ordered. Thus, we'd have a hard time implementing key-based range scans. 2. The internal ValueGetter interface only supports get-by- key lookups, so it would also need to be expanded to support range scans on the parent table. Neither of these problems are insurmountable, but I'm wondering if we _want_ to surmount them right now. Or should we instead just throw an UnsupportedOperationException on any API call that's inconvenient to implement right now? Then, we could get incremental value by first supporting (eg) key-based lookups and adding scans later. Or does this mean that our design so far is invalid, and we should really just make people provision a separate Materialized downstream? To pull this off, we'd need to first address KIP-300's challenges, though. I'm honestly not sure what the right call is here. Thanks, -John On Thu, 2020-10-01 at 01:50 +0900, Dongjin Lee wrote: > > It seems like it must be a ReadOnlyKeyValueStore. Does that sound right? > > Yes, it is. Would it be better to add a detailed description of how this > feature effects interactive query, with examples? > > Best, > Dongjin > > On Tue, Sep 29, 2020 at 10:31 AM John Roesler <vvcep...@apache.org> wrote: > > > Hi Dongjin, > > > > Thanks! Sorry, I missed your prior message. The proposed API looks good to > > me. > > > > I’m wondering if we should specify what kind of store view would be > > returned when querying the operation result. It seems like it must be a > > ReadOnlyKeyValueStore. Does that sound right? > > > > Thanks! > > John > > > > On Mon, Sep 28, 2020, at 10:06, Dongjin Lee wrote: > > > Hi John, > > > > > > I updated the KIP with the discussion above. The 'Public Interfaces' > > > section describes the new API, and the 'Rejected Alternatives' section > > > describes the reasoning about why we selected this API design and > > rejected > > > the other alternatives. > > > > > > Please have a look when you are free. And please note that the KIP freeze > > > for 2.7.0 is imminent. > > > > > > Thanks, > > > Dongjin > > > > > > On Mon, Sep 21, 2020 at 11:35 PM Dongjin Lee <dong...@apache.org> wrote: > > > > > > > Hi John, > > > > > > > > I updated the PR applying the API changes we discussed above. I am now > > > > updating the KIP document. > > > > > > > > Thanks, > > > > Dongjin > > > > > > > > On Sat, Sep 19, 2020 at 10:42 AM John Roesler <vvcep...@apache.org> > > wrote: > > > > > Hi Dongjin, > > > > > > > > > > Yes, that’s right. My the time of KIP-307, we had no choice but to > > add a > > > > > second name. But we do have a choice with Suppress. > > > > > > > > > > Thanks! > > > > > -John > > > > > > > > > > On Thu, Sep 17, 2020, at 13:14, Dongjin Lee wrote: > > > > > > Hi John, > > > > > > > > > > > > I just reviewed KIP-307. As far as I understood, ... > > > > > > > > > > > > 1. There was Materialized name initially. > > > > > > 2. With KIP-307, Named Operations were added. > > > > > > 3. Now we have two options for materializing suppression. If we take > > > > > > Materialized name here, we have two names for the same operation, > > which > > > > > is > > > > > > not feasible. > > > > > > > > > > > > Do I understand correctly? > > > > > > > > > > > > > Do you have a use case in mind for having two separate names for > > the > > > > > > operation and the view? > > > > > > > > > > > > No. I am now entirely convinced with your suggestion. > > > > > > > > > > > > I just started to update the draft implementation. If I understand > > > > > > correctly, please notify me; I will update the KIP by adding the > > > > > discussion > > > > > > above. > > > > > > > > > > > > Best, > > > > > > Dongjin > > > > > > > > > > > > On Thu, Sep 17, 2020 at 11:06 AM John Roesler <vvcep...@apache.org> > > > > > wrote: > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > Thanks for the reply. Yes, that’s correct, we added that method to > > > > > name > > > > > > > the operation. But the operation seems synonymous with the view > > > > > produced > > > > > > > the operation, right? > > > > > > > > > > > > > > During KIP-307, I remember thinking that it’s unfortunate the we > > had > > > > > to > > > > > > > have two different “name” concepts for the same thing just because > > > > > setting > > > > > > > the name on Materialized is equivalent both to making it > > queriable and > > > > > > > actually materializing it. > > > > > > > > > > > > > > If we were to reconsider the API, it would be nice to treat these > > > > > three as > > > > > > > orthogonal: > > > > > > > * specify a name > > > > > > > * flag to make the view queriable > > > > > > > * flag to materialize the view > > > > > > > > > > > > > > That was the context behind my suggestion. Do you have a use case > > in > > > > > mind > > > > > > > for having two separate names for the operation and the view? > > > > > > > > > > > > > > Thanks, > > > > > > > John > > > > > > > > > > > > > > On Wed, Sep 16, 2020, at 11:43, Dongjin Lee wrote: > > > > > > > > Hi John, > > > > > > > > > > > > > > > > It seems like the available alternatives in this point is clear: > > > > > > > > > > > > > > > > 1. Pass queriable name as a separate parameter (i.e., > > > > > > > > `KTable#suppress(Suppressed, String)`) > > > > > > > > 2. Make use of the Suppression processor name as a queryable > > name by > > > > > > > adding > > > > > > > > `enableQuery` optional flag to `Suppressed`. > > > > > > > > > > > > > > > > However, I doubt the second approach a little bit; As far as I > > > > > know, the > > > > > > > > processor name is introduced in KIP-307[^1] to make debugging > > > > > topology > > > > > > > easy > > > > > > > > and understandable. Since the processor name is an independent > > > > > concept > > > > > > > with > > > > > > > > the materialization, I feel the first approach is more natural > > and > > > > > > > > consistent. Is there any specific reason that you prefer the > > second > > > > > > > > approach? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dongjin > > > > > > > > > > > > > > > > [^1]: > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-307%3A+Allow+to+define+custom+processor+names+with+KStreams+DSL > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Sep 16, 2020 at 11:48 PM John Roesler < > > vvcep...@apache.org> > > > > > > > wrote: > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > > > > > Yes, that's where I was leaning. Although, I'd prefer adding > > > > > > > > > the option to Suppressed instead of adding a new argument to > > > > > > > > > the method call. > > > > > > > > > > > > > > > > > > What do you think about: > > > > > > > > > > > > > > > > > > class Suppressed<K> { > > > > > > > > > + public Suppressed<K> enableQuery(); > > > > > > > > > } > > > > > > > > > > > > > > > > > > Since Suppressed already has `withName(String)`, it seems > > > > > > > > > like all we need to add is a boolean flag. > > > > > > > > > > > > > > > > > > Does that seem sensible to you? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > -John > > > > > > > > > > > > > > > > > > On Wed, 2020-09-16 at 21:50 +0900, Dongjin Lee wrote: > > > > > > > > > > Hi John, > > > > > > > > > > > > > > > > > > > > > Although it's not great to have "special snowflakes" in > > the > > > > > API, > > > > > > > > > Choice B > > > > > > > > > > does seem safer in the short term. We would basically be > > > > > proposing a > > > > > > > > > > temporary API to make the suppressed view queriable without > > a > > > > > > > > > Materialized > > > > > > > > > > argument. > > > > > > > > > > > > > > > > > > > > Then, it seems like you prefer `KTable#suppress(Suppressed, > > > > > String)` > > > > > > > > > (i.e., > > > > > > > > > > queriable name only as a parameter) for this time, and > > refine > > > > > API > > > > > > > with > > > > > > > > > the > > > > > > > > > > other related KIPs later. > > > > > > > > > > > > > > > > > > > > Do I understand correctly? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > On Wed, Sep 16, 2020 at 2:17 AM John Roesler < > > > > > vvcep...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > > > > > > > > > Thanks for presenting these options. The concern that > > > > > > > > > > > Matthias brought up is a very deep problem that afflicts > > all > > > > > > > > > > > operations downstream of windowing operations. It's the > > same > > > > > > > > > > > thing that derailed KIP-300. For the larger context, I > > have > > > > > > > > > > > developed a couple of approaches to resolve this > > situation, > > > > > > > > > > > but I think it makes sense to finish up KIP-478 before > > > > > > > > > > > presenting them. > > > > > > > > > > > > > > > > > > > > > > However, I don't think that we need in particular to block > > > > > > > > > > > the current proposal on solving that long-running and deep > > > > > > > > > > > issue with the DSL. Instead, we should make a top-level > > > > > > > > > > > decision whether to: > > > > > > > > > > > > > > > > > > > > > > A: Make Suppress just like all the other KTable > > operations. > > > > > > > > > > > It will have the same pathological behavior that the > > keyset > > > > > > > > > > > is unbounded while the store implementation is only a > > > > > > > > > > > KeyValueStore. Again, this exact pathology currently > > affects > > > > > > > > > > > all KTable operations that follow from windowing > > operations. > > > > > > > > > > > For example, it applies to the current workaround that > > > > > > > > > > > Dongjin documented in the KIP: > > > > > > > > > > > suppress().filter(Materialized<KeyValueStore>). This is > > > > > > > > > > > Option 2 that Dongjin presented. > > > > > > > > > > > > > > > > > > > > > > B: Do something different with Suppress in order to side- > > > > > > > > > > > step the problem. For example, Suppress does not _need_ to > > > > > > > > > > > have a separate state store at all. If we just give > > people a > > > > > > > > > > > switch to make the operation queriable, we can implement a > > > > > > > > > > > ReadOnlyKeyValueStore interface by querying the > > "priorValue" > > > > > > > > > > > of the buffer first and then querying the upstream > > > > > > > > > > > ValueGetter. This broad category of "do something > > different > > > > > > > > > > > with Suppress" encompases Option 1 and Option 3 that > > Dongjin > > > > > > > > > > > presented. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Speaking personally, I think Choice A would be the most > > > > > > > > > > > obvious and least weird choice, but it presents a serious > > > > > > > > > > > risk of escalating the severity of the problem of > > unbounded > > > > > > > > > > > state. This is currently a risk that we're aware of, but > > has > > > > > > > > > > > not yet become a big problem in practice. As Matthias > > > > > > > > > > > pointed out, Suppress is far more likely to be used > > > > > > > > > > > downstream of windowed tables than other operations, so > > > > > > > > > > > having a Materialized<KVStore> overload has the > > significant > > > > > > > > > > > risk of getting people into a bad state. Note, broadly > > > > > > > > > > > advertising the workaround from the KIP would have the > > exact > > > > > > > > > > > same impact, so we should be careful about recommending > > it. > > > > > > > > > > > Although it's not great to have "special snowflakes" in > > the > > > > > > > > > > > API, Choice B does seem safer in the short term. We would > > > > > > > > > > > basically be proposing a temporary API to make the > > > > > > > > > > > suppressed view queriable without a Materialized argument. > > > > > > > > > > > Then, once we fix the main KIP-300 problem, we would look > > at > > > > > > > > > > > converging Suppress with the rest of the KTable > > > > > > > > > > > materialization APIs. > > > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > Thanks, > > > > > > > > > > > -John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 2020-09-16 at 00:01 +0900, Dongjin Lee wrote: > > > > > > > > > > > > Hi Matthias, > > > > > > > > > > > > > > > > > > > > > > > > Thank you very much for the detailed feedback. Here are > > my > > > > > > > opinions: > > > > > > > > > > > > > Because there is no final result for non-windowed > > > > > KTables, it > > > > > > > seems > > > > > > > > > > > that > > > > > > > > > > > > this new feature only make sense for the > > > > > windowed-aggregation > > > > > > > case? > > > > > > > > > > > > I think a little bit different. Of course, for windowed > > > > > KTable, > > > > > > > this > > > > > > > > > > > > feature provides the final state; for non-windowed > > KTables, > > > > > it > > > > > > > > > provides a > > > > > > > > > > > > view to the records received more than the predefined > > > > > waiting > > > > > > > time > > > > > > > > > ago - > > > > > > > > > > > > excluding the records that are waiting for more events. > > > > > > > > > > > > > > > > > > > > > > > > > Thus, the signature of `Materialized` should take a > > > > > > > `WindowStore` > > > > > > > > > > > instead > > > > > > > > > > > > of a `KeyValueStore`? > > > > > > > > > > > > > > > > > > > > > > > > I reviewed the implementation following your comments > > and > > > > > found > > > > > > > the > > > > > > > > > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > 1. `Materialized` instance includes the following: > > KeySerde, > > > > > > > > > ValueSerde, > > > > > > > > > > > > StoreSupplier, and Queriable Name. > > > > > > > > > > > > 2. The other `Materialized` method variants in `KTable` > > are > > > > > > > making > > > > > > > > > use of > > > > > > > > > > > > KeySerde, ValueSerde, and Queriable Name only. (That is, > > > > > > > > > StoreSupplier is > > > > > > > > > > > > ignored.) > > > > > > > > > > > > 3. `KTable#suppress(Suppressed, Materialized)` uses the > > > > > Queriable > > > > > > > > > Name > > > > > > > > > > > > only. StoreSupplier is also ignored. > > > > > > > > > > > > > > > > > > > > > > > > So, we have three choices for the method signature: > > > > > > > > > > > > > > > > > > > > > > > > 1. `KTable#suppress(Suppressed, String)` (i.e., passing > > the > > > > > > > Queriable > > > > > > > > > > > Name > > > > > > > > > > > > only): > > > > > > > > > > > > > > > > > > > > > > > > This is the simplest; however, it is inconsistent > > with the > > > > > > > other > > > > > > > > > > > > Materialized variant methods. > > > > > > > > > > > > > > > > > > > > > > > > 2. `KTable#suppress(Suppressed, Materialized<K, V, > > > > > > > KeyValueStore>)` > > > > > > > > > > > (i.e., > > > > > > > > > > > > current proposal) > > > > > > > > > > > > > > > > > > > > > > > > This approach is harmless at this point, for > > > > > StoreSupplier is > > > > > > > > > ignored; > > > > > > > > > > > > However, since suppression can be used to both of > > > > > > > `KeyValueStore` and > > > > > > > > > > > > `WindowStore`, this approach is not only weird but also > > > > > leaving > > > > > > > some > > > > > > > > > > > > potential risk to the future. (On second thoughts, I > > agree, > > > > > this > > > > > > > API > > > > > > > > > > > design > > > > > > > > > > > > is bad and dangerous.) > > > > > > > > > > > > > > > > > > > > > > > > 3. `KTable#suppress(Suppressed, Materialized<K, V, > > > > > StateStore>)` > > > > > > > > > > > > This approach embraces both of `KeyValueStore` and > > > > > > > `WindowStore` > > > > > > > > > cases. > > > > > > > > > > > > Since the concrete class type of `Suppressed` instance > > > > > differs > > > > > > > for > > > > > > > > > the > > > > > > > > > > > > `StateStore`[^1], it seems like we can validate the > > > > > arguments on > > > > > > > the > > > > > > > > > > > method > > > > > > > > > > > > call. (e.g., throw `IllegalArgumentException` if when > > > > > > > `Suppressed` > > > > > > > > > > > instance > > > > > > > > > > > > on `KeyValueStore` is given with `Materialized` > > instance of > > > > > > > > > > > `WindowStore`.) > > > > > > > > > > > > This approach not only breaks the API consistency but > > also > > > > > guards > > > > > > > > > from a > > > > > > > > > > > > miss-usage of the API. > > > > > > > > > > > > > > > > > > > > > > > > How do you think? I am now making a try on the third > > > > > approach. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > [^1]: `SuppressedInternal` for `KeyValueStore` and > > > > > > > > > > > > `FinalResultsSuppressionBuilder` for `WindowStore`. > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Sep 12, 2020 at 3:29 AM Matthias J. Sax < > > > > > > > mj...@apache.org> > > > > > > > > > > > wrote: > > > > > > > > > > > > > Thanks for updating the KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > I think there is still one open question. `suppress()` > > > > > can be > > > > > > > used > > > > > > > > > on > > > > > > > > > > > > > non-windowed KTable for rate control, as well as on a > > > > > > > > > windowed-KTable > > > > > > > > > > > > > (also for rate control, but actually mainly) for only > > > > > emitting > > > > > > > the > > > > > > > > > > > final > > > > > > > > > > > > > result of a windowed aggregation. For the non-windowed > > > > > case, we > > > > > > > > > use a > > > > > > > > > > > > > KeyValueStore while for the windowed cases, we use a > > > > > > > WindowStore. > > > > > > > > > > > > > Because there is no final result for non-windowed > > > > > KTables, it > > > > > > > seems > > > > > > > > > > > that > > > > > > > > > > > > > this new feature only make sense for the > > > > > windowed-aggregation > > > > > > > case? > > > > > > > > > > > > > Thus, the signature of `Materialized` should take a > > > > > > > `WindowStore` > > > > > > > > > > > > > instead of a `KeyValueStore`? > > > > > > > > > > > > > > > > > > > > > > > > > > If that's correct, I am wondering: > > > > > > > > > > > > > > > > > > > > > > > > > > - Can we guard from a miss-usage of the API if the > > > > > upstream > > > > > > > > > KTable is > > > > > > > > > > > > > not windowed (or maybe it's not necessary to guard)? > > > > > > > > > > > > > - Can we actually implement it? We had issues with > > > > > regard to > > > > > > > > > KIP-300 > > > > > > > > > > > to > > > > > > > > > > > > > materialize windowed-KTables? > > > > > > > > > > > > > > > > > > > > > > > > > > Would be worth to clarify upfront. Maybe, we even > > need a > > > > > POC > > > > > > > > > > > > > implementation to verify that it works? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 9/11/20 12:26 AM, Dongjin Lee wrote: > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here is the voting thread: > > > > > > > > > > > > > > > > https://lists.apache.org/thread.html/r5653bf2dafbb27b247bf20dbe6f070c151b3823d96c9c9ca94183e20%40%3Cdev.kafka.apache.org%3E > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 4:23 PM Dongjin Lee < > > > > > > > dong...@apache.org> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi John, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the feedback. I will open the Vote > > thread > > > > > now. > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Sep 11, 2020 at 2:00 AM John Roesler < > > > > > > > > > vvcep...@apache.org> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sorry for the delay. I'm glad you're still > > pushing > > > > > this > > > > > > > > > > > > > > > > forward. It would be nice to get this in to the > > 2.7 > > > > > > > release. > > > > > > > > > > > > > > > > I just took another look at the KIP, and it > > looks > > > > > good to > > > > > > > > > > > > > > > > me! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think this is ready for a vote. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > -John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, 2020-08-05 at 22:04 +0900, Dongjin Lee > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I updated the KIP > > > > > > > > > > > > > > > > > < > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-508%3A+Make+Suppression+State+Queriable > > > > > > > > > > > > > > > > > and the implementation, following the > > discussion > > > > > here. > > > > > > > > > > > > > > > > > You must be working hard preparing the > > release of > > > > > > > 2.6.0, so > > > > > > > > > > > please > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > look after your work is done. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Mar 8, 2020 at 12:20 PM John Roesler < > > > > > > > > > > > vvcep...@apache.org> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Thanks Matthias, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Good idea. I've changed the ticket name and > > > > > added a > > > > > > > note > > > > > > > > > > > > > > > > > > clarifying that this ticket is not the same > > as > > > > > https://issues.apache.org/jira/browse/KAFKA-7224 > > > > > > > > > > > > > > > > > > Incidentally, I learned that I never > > documented > > > > > my > > > > > > > > > reasons > > > > > > > > > > > > > > > > > > for abandoning my work on KAFKA-7224 ! I've > > now > > > > > > > updated > > > > > > > > > > > > > > > > > > that ticket, too, so your question had an > > > > > unexpected > > > > > > > > > > > side-benefit. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > -John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Mar 7, 2020, at 18:01, Matthias J. > > Sax > > > > > wrote: > > > > > > > > > > > > > > Thanks for clarification. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you maybe update the Jira ticket? Do we have a > > > > > ticket for > > > > > > > > > > > > > > spill-to-disk? Maybe link to it and explain that > > it's > > > > > two > > > > > > > > > different > > > > > > > > > > > > > > things? Maybe even rename the ticket to something > > more > > > > > > > clear, ie, > > > > > > > > > > > > > > "make suppress result queryable" or simliar? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 3/7/20 1:58 PM, John Roesler wrote: > > > > > > > > > > > > > > > > > > > > > Hey Matthias, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m sorry if the ticket was poorly > > > > > stated. The > > > > > > > > > ticket > > > > > > > > > > > is to add a > > > > > > > > > > > > > > DSL overload to pass a Materialized argument to > > > > > suppress. As > > > > > > > a > > > > > > > > > > > > > > > > > result, > > > > > > > > > > > > > > the result of the suppression would be queriable. > > > > > > > > > > > > > > > > > > > > > This is unrelated to “persistent > > buffer” > > > > > aka > > > > > > > > > > > “spill-to-disk”. > > > > > > > > > > > > > > > > > > > > > There was some confusion before about > > > > > whether > > > > > > > this > > > > > > > > > > > ticket could be > > > > > > > > > > > > > > implemented as “query the buffer”. Maybe it can, > > but not > > > > > > > > > trivially. > > > > > > > > > > > > > > The obvious way is just to add a new state store > > which > > > > > we > > > > > > > write > > > > > > > > > the > > > > > > > > > > > > > > results into just before we forward. I.e., it’s > > exactly > > > > > like > > > > > > > the > > > > > > > > > > > > > > materialized variant of any stateless KTable > > operation. > > > > > > > > > > > > > > > > > > > > > Thanks, John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Mar 7, 2020, at 15:32, > > Matthias > > > > > J. Sax > > > > > > > > > wrote: > > > > > > > > > > > Thanks for > > > > > > > > > > > > > > > > > > > > > the KIP Dongjin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I am still not sure if I can follow, > > what > > > > > might > > > > > > > > > also > > > > > > > > > > > be caused by > > > > > > > > > > > > > > > > > > > > > the backing JIRA ticket (maybe John > > can > > > > > > > clarify the > > > > > > > > > > > intent of the > > > > > > > > > > > > > > > > > > > > > ticket as he created it): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, suppress() only uses an > > > > > in-memory > > > > > > > buffer > > > > > > > > > > > and my > > > > > > > > > > > > > > > > > > > > > understanding of the Jira is, to add > > the > > > > > > > ability to > > > > > > > > > > > use a > > > > > > > > > > > > > > > > > > > > > persistent buffer (ie, spill to disk > > > > > backed by > > > > > > > > > > > RocksDB). > > > > > > > > > > > > > > > > > > > > > Adding a persistent buffer is > > completely > > > > > > > unrelated > > > > > > > > > to > > > > > > > > > > > allow > > > > > > > > > > > > > > > > > > > > > querying the buffer. In fact, one > > could > > > > > query > > > > > > > an > > > > > > > > > > > in-memory buffer, > > > > > > > > > > > > > > > > > > > > > too. However, querying the buffer > > does not > > > > > > > really > > > > > > > > > seem > > > > > > > > > > > to be > > > > > > > > > > > > > > > > > useful > > > > > > > > > > > > > > > > > > > > > as pointed out by John, as you can > > always > > > > > > > query the > > > > > > > > > > > upstream > > > > > > > > > > > > > > > > > KTable > > > > > > > > > > > > > > > > > > > > > store. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Also note that for the > > > > > emit-on-window-close > > > > > > > case > > > > > > > > > the > > > > > > > > > > > result is > > > > > > > > > > > > > > > > > > > > > deleted from the buffer when it is > > > > > emitted, and > > > > > > > > > thus > > > > > > > > > > > cannot be > > > > > > > > > > > > > > > > > > > > > queried any longe r. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you please clarify if you intend > > to > > > > > allow > > > > > > > > > spilling > > > > > > > > > > > to disk or > > > > > > > > > > > > > > > > > > > > > if you intent to enable IQ (even if I > > > > > don't > > > > > > > see why > > > > > > > > > > > querying make > > > > > > > > > > > > > > > > > > > > > sense, as the data is either upstream > > or > > > > > > > deleted). > > > > > > > > > > > Also, if you > > > > > > > > > > > > > > > > > > > > > want to enable IQ, why do we need all > > > > > those new > > > > > > > > > > > interfaces? The > > > > > > > > > > > > > > > > > > > > > result of a suppress() is a KTable > > that > > > > > is the > > > > > > > > > same as > > > > > > > > > > > any other > > > > > > > > > > > > > > > > > > > > > key-value/windowed/sessions store? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We should also have corresponding Jira > > > > > tickets > > > > > > > for > > > > > > > > > > > different cases > > > > > > > > > > > > > > > > > > > > > to avoid the confusion I am in atm :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Matthias > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 2/27/20 8:21 AM, John Roesler > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > No problem; glad we got it > > sorted > > > > > out. > > > > > > > > > > > > > > > > > > > > > > > > Thanks again for picking this > > up! > > > > > -John > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 26, 2020, at 09:24, > > > > > Dongjin > > > > > > > Lee > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > I was under the impression > > that > > > > > you > > > > > > > > > wanted > > > > > > > > > > > to expand the > > > > > > > > > > > > > > > > > > > > > > > > > > scope of the KIP > > > > > > > > > > > > > > > > > > > > > > > > > to additionally allow > > querying the > > > > > > > internal > > > > > > > > > > > buffer, not > > > > > > > > > > > > > > > > > > > > > > > > > just the result. Can you > > clarify > > > > > > > whether > > > > > > > > > you > > > > > > > > > > > are proposing > > > > > > > > > > > > > > > > > > > > > > > > > to allow querying the state > > of the > > > > > > > internal > > > > > > > > > > > buffer, the > > > > > > > > > > > > > > > > > > > > > > > > > result, or both? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sorry for the confusion. As we > > > > > already > > > > > > > > > talked > > > > > > > > > > > with, we > > > > > > > > > > > > > > > > > only > > > > > > > > > > > > > > > > > > > > > > > > > need to query the suppressed > > > > > output, > > > > > > > not > > > > > > > > > the > > > > > > > > > > > internal > > > > > > > > > > > > > > > > > > > > > > > > > buffer. The current > > > > > implementation is > > > > > > > > > wrong. > > > > > > > > > > > After > > > > > > > > > > > > > > > > > refining > > > > > > > > > > > > > > > > > > > > > > > > > the KIP and implementation > > > > > accordingly > > > > > > > I > > > > > > > > > will > > > > > > > > > > > notify you - > > > > > > > > > > > > > > > > > > > > > > > > > I must be confused, also. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 25, 2020 at 12:17 > > AM > > > > > John > > > > > > > > > Roesler > > > > > > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ah, I think I may have been > > > > > > > confused. I > > > > > > > > > 100% > > > > > > > > > > > agree that > > > > > > > > > > > > > > > > > > > > > > > > > > we need a materialized > > variant > > > > > for > > > > > > > > > > > suppress(). Then, you > > > > > > > > > > > > > > > > > > > > > > > > > > could do: ...suppress(..., > > > > > > > > > > > > > > > > > > > > > > > > > > > > Materialized.as(“final-count”)) > > > > > > > > > > > > > > > > > > > > > > > > > > If that’s your proposal, > > then > > > > > we are > > > > > > > on > > > > > > > > > the > > > > > > > > > > > same page. > > > > > > > > > > > > > > > > > > > > > > > > > > I was under the impression > > that > > > > > you > > > > > > > > > wanted > > > > > > > > > > > to expand the > > > > > > > > > > > > > > > > > > > > > > > > > > scope of the KIP to > > additionally > > > > > > > allow > > > > > > > > > > > querying the > > > > > > > > > > > > > > > > > > > > > > > > > > internal buffer, not just > > the > > > > > > > result. Can > > > > > > > > > > > you clarify > > > > > > > > > > > > > > > > > > > > > > > > > > whether you are proposing to > > > > > allow > > > > > > > > > querying > > > > > > > > > > > the state of > > > > > > > > > > > > > > > > > > > > > > > > > > the internal buffer, the > > > > > result, or > > > > > > > both? > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 20, 2020, at > > 08:41, > > > > > > > Dongjin > > > > > > > > > Lee > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi John, Thanks for your > > kind > > > > > > > > > explanation > > > > > > > > > > > with an > > > > > > > > > > > > > > > > > > > > > > > > > > > example. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But it feels like you're > > > > > saying > > > > > > > > > you're > > > > > > > > > > > trying to do > > > > > > > > > > > > > > > > > > > > > > > > > > > > something different > > > > > > > > > > > > > > > > > > > > > > > > > > > than just query the > > windowed > > > > > key > > > > > > > and > > > > > > > > > get > > > > > > > > > > > back the > > > > > > > > > > > > > > > > > > > > > > > > > > > current count? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, for example, what if > > we > > > > > need > > > > > > > to > > > > > > > > > > > retrieve the (all > > > > > > > > > > > > > > > > > > > > > > > > > > > or range) keys > > > > > > > > > > > > > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > a closed window? In this > > > > > example, > > > > > > > let's > > > > > > > > > > > imagine we > > > > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > > > > > > > > > > > > to retrieve only (key=A, > > > > > > > window=10), > > > > > > > > > not > > > > > > > > > > > (key=A, > > > > > > > > > > > > > > > > > > > > > > > > > > > window=20). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Of course, the value > > > > > accompanied > > > > > > > by a > > > > > > > > > > > flushed key is > > > > > > > > > > > > > > > > > > > > > > > > > > > exactly the same to the > > one > > > > > in the > > > > > > > > > > > upstream KTable; > > > > > > > > > > > > > > > > > > > > > > > > > > > However, if our intention > > is > > > > > not > > > > > > > > > pointing > > > > > > > > > > > out a > > > > > > > > > > > > > > > > > > > > > > > > > > > specific key but > > retrieving a > > > > > > > group of > > > > > > > > > > > unspecified > > > > > > > > > > > > > > > > > > > > > > > > > > > keys, we stuck > > > > > > > > > > > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > > > > > > > > > > > trouble - since we can't > > be > > > > > sure > > > > > > > which > > > > > > > > > key > > > > > > > > > > > is flushed > > > > > > > > > > > > > > > > > > > > > > > > > > > out beforehand. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One workaround would be > > > > > > > materializing > > > > > > > > > it > > > > > > > > > > > with > > > > > > > > > > > > > > > > > > > > > > > > > > > `suppressed.filter(e -> > > > > > > > > > > > > > > > > > > > > > > > > > > true, > > > > > Materialized.as("final-count"))`. > > > > > > > But I > > > > > > > > > > > think > > > > > > > > > > > > > > > > > providing > > > > > > > > > > > > > > > > > > > > > > > > > > > a materialized variant for > > > > > suppress > > > > > > > > > method > > > > > > > > > > > is better > > > > > > > > > > > > > > > > > > > > > > > > > > > than this workaround. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Feb 20, 2020 at > > 1:26 > > > > > AM > > > > > > > John > > > > > > > > > > > Roesler > > > > > > > > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the response, > > > > > Dongjin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'm sorry, but I'm > > still not > > > > > > > > > following. > > > > > > > > > > > It seems > > > > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > > > > > > > > > > > > the view you > > > > > > > > > > > > > > > > > > > > > > > > > > would > > > > > > > > > > > > > > > > > > > > > > > > > > > > get on the "current > > state > > > > > of the > > > > > > > > > buffer" > > > > > > > > > > > would > > > > > > > > > > > > > > > > > always > > > > > > > > > > > > > > > > > > > > > > > > > > > > be equivalent to the > > view > > > > > of the > > > > > > > > > > > upstream table. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me try an example, > > and > > > > > maybe > > > > > > > you > > > > > > > > > can > > > > > > > > > > > point out > > > > > > > > > > > > > > > > > > > > > > > > > > > > the flaw in my > > reasoning. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let's say we're doing > > 10 ms > > > > > > > windows > > > > > > > > > with > > > > > > > > > > > a grace > > > > > > > > > > > > > > > > > > > > > > > > > > > > period of zero. Let's > > also > > > > > say > > > > > > > we're > > > > > > > > > > > computing a > > > > > > > > > > > > > > > > > > > > > > > > > > > > windowed count, and > > that we > > > > > have > > > > > > > a > > > > > > > > > > > "final results" > > > > > > > > > > > > > > > > > > > > > > > > > > > > suppression after the > > count. > > > > > > > Let's > > > > > > > > > > > materialize the > > > > > > > > > > > > > > > > > > > > > > > > > > > > count as "Count" and the > > > > > > > suppressed > > > > > > > > > > > result as "Final > > > > > > > > > > > > > > > > > > > > > > > > > > > > Count". > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Suppose we get an input > > > > > event: > > > > > > > > > (time=10, > > > > > > > > > > > key=A, > > > > > > > > > > > > > > > > > > > > > > > > > > > > value=...) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look > > like: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > > > > > 10 > > > > > > > | A > > > > > > > > > > > | 1 | > > > > > > > > > > > > > > > > > > > > > > > > > > > > The (internal) > > suppression > > > > > buffer > > > > > > > > > will > > > > > > > > > > > contain: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > > > > > 10 > > > > > > > | A > > > > > > > > > > > | 1 | > > > > > > > > > > > > > > > > > > > > > > > > > > > > The record is still > > buffered > > > > > > > because > > > > > > > > > the > > > > > > > > > > > window > > > > > > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > > > > > Count is > > > > > > > an > > > > > > > > > > > empty table: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Now, we get a second > > event: > > > > > > > (time=15, > > > > > > > > > > > key=A, > > > > > > > > > > > > > > > > > > > > > > > > > > > > value=...) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look > > like: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > > > > > 10 > > > > > > > | A > > > > > > > > > > > | 2 | > > > > > > > > > > > > > > > > > > > > > > > > > > > > The (internal) > > suppression > > > > > buffer > > > > > > > > > will > > > > > > > > > > > contain: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > > > > > 10 > > > > > > > | A > > > > > > > > > > > | 2 | > > > > > > > > > > > > > > > > > > > > > > > > > > > > The record is still > > buffered > > > > > > > because > > > > > > > > > the > > > > > > > > > > > window > > > > > > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > > > > > Count is > > > > > > > an > > > > > > > > > > > empty table: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Finally, we get a third > > > > > event: > > > > > > > > > (time=20, > > > > > > > > > > > key=A, > > > > > > > > > > > > > > > > > > > > > > > > > > > > value=...) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Then, Count will look > > like: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > > > > > 10 > > > > > > > | A > > > > > > > > > > > | 2 | | > > > > > > > > > > > > > > > > > > > > > > > > > > > > 20 | A | 1 | > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The (internal) > > suppression > > > > > buffer > > > > > > > > > will > > > > > > > > > > > contain: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > > > > > 20 > > > > > > > | A > > > > > > > > > > > | 1 | > > > > > > > > > > > > > > > > > > > > > > > > > > > > Note that window 10 has > > been > > > > > > > flushed > > > > > > > > > > > out, because > > > > > > > > > > > > > > > > > > > > > > > > > > > > it's now closed. And > > window > > > > > 20 is > > > > > > > > > > > buffered because > > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > > > > > > > isn't closed yet. Final > > > > > Count is > > > > > > > now: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > window | key | value > > > > > 10 > > > > > > > | A > > > > > > > > > > > | 2 | > > > > > > > > > > > > > > > > > > > > > > > > > > > > --------------- > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Reading your email, I > > can't > > > > > > > figure > > > > > > > > > out > > > > > > > > > > > what value > > > > > > > > > > > > > > > > > > > > > > > > > > > > there is in querying > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > internal suppression > > buffer, > > > > > > > since it > > > > > > > > > > > only contains > > > > > > > > > > > > > > > > > > > > > > > > > > > > exactly the same > > > > > > > > > > > > > > > > > > > > > > > > > > value > > > > > > > > > > > > > > > > > > > > > > > > > > > > as the upstream table, > > for > > > > > each > > > > > > > key > > > > > > > > > that > > > > > > > > > > > is still > > > > > > > > > > > > > > > > > > > > > > > > > > > > buffered. But it feels > > > > > > > > > > > > > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > > > > > > > > > > > > you're saying you're > > trying > > > > > to do > > > > > > > > > > > something > > > > > > > > > > > > > > > > > different > > > > > > > > > > > > > > > > > > > > > > > > > > > > than just query > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > windowed key and get > > back > > > > > the > > > > > > > current > > > > > > > > > > > count? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, -John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Feb 19, 2020, at > > > > > 09:49, > > > > > > > > > Dongjin > > > > > > > > > > > Lee wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi John, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 'The intermediate > > state > > > > > of the > > > > > > > > > > > suppression' in KIP > > > > > > > > > > > > > > > > > > > > > > > > > > > > > does not mean the > > > > > > > > > > > > > > > > > > > > > > > > > > > > state > > > > > > > > > > > > > > > > > > > > > > > > > > > > > of upstream KTable - > > > > > sure, the > > > > > > > > > state > > > > > > > > > > > of the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > upstream KTable can be > > > > > > > > > > > > > > > > > > > > > > > > > > > > queried > > > > > > > > > > > > > > > > > > > > > > > > > > > > > by materializing the > > > > > operator > > > > > > > > > > > immediately before > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the suppress as you > > > > > > > > > > > > > > > > > > > > > > > > > > > > shown. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What I meant in KIP > > was > > > > > the > > > > > > > final > > > > > > > > > > > state of the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > buffer, which is not > > > > > > > > > > > > > > > > > > > > > > > > > > > > emitted > > > > > > > > > > > > > > > > > > > > > > > > > > > > > yet. (I agree, the > > current > > > > > > > > > description > > > > > > > > > > > may be > > > > > > > > > > > > > > > > > > > > > > > > > > > > > confusing; it would be > > > > > > > > > > > > > > > > > > > > > > > > > > > > better > > > > > > > > > > > > > > > > > > > > > > > > > > > > > to change it with 'the > > > > > current > > > > > > > > > state > > > > > > > > > > > of the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > suppression' or 'the > > > > > > > > > > > > > > > > > > > > > > > > > > results > > > > > > > > > > > > > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the suppression', like > > > > > the Jira > > > > > > > > > issue > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > states.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For a little bit more > > > > > about the > > > > > > > > > > > motivation, here > > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > > > > > > > > > > > > > > one of my > > > > > > > > > > > > > > > > > > > > > > > > > > > > experience: I > > > > > > > > > > > > > > > > > > > > > > > > > > > > > had to build a > > monitoring > > > > > > > > > application > > > > > > > > > > > which > > > > > > > > > > > > > > > > > > > > > > > > > > > > > collects signals from > > IoT > > > > > > > devices > > > > > > > > > > > (say, a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > semiconductor > > production > > > > > > > line.) If > > > > > > > > > the > > > > > > > > > > > number of > > > > > > > > > > > > > > > > > > > > > > > > > > > > collected > > > > > > > > > > > > > > > > > > > > > > > > > > > > > signals within the > > time > > > > > window > > > > > > > is > > > > > > > > > much > > > > > > > > > > > less than > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the expected, there > > > > > > > > > > > > > > > > > > > > > > > > > > may > > > > > > > > > > > > > > > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > > > > > > > > > > > > > > some problems like > > network > > > > > > > hiccup > > > > > > > > > in > > > > > > > > > > > the systems. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > We wanted to build > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > system in the form of > > a > > > > > > > dashboard, > > > > > > > > > but > > > > > > > > > > > could not > > > > > > > > > > > > > > > > > by > > > > > > > > > > > > > > > > > > > > > > > > > > > > > lack of > > > > > > > > > > > > > > > > > > > > > > > > > > materializing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > feature. It was > > precisely > > > > > the > > > > > > > case > > > > > > > > > of > > > > > > > > > > > querying > > > > > > > > > > > > > > > > > only > > > > > > > > > > > > > > > > > > > > > > > > > > > > > the final > > > > > > > > > > > > > > > > > > > > > > > > > > results of > > > > > > > > > > > > > > > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > windowed aggregation, > > as > > > > > the > > > > > > > Jira > > > > > > > > > issue > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > states. We > > > > > > > > > > > > > > > > > > > > > > > > > > finally > > > > > > > > > > > > > > > > > > > > > > > > > > > > ended > > > > > > > > > > > > > > > > > > > > > > > > > > > > > in implementing the > > > > > system in > > > > > > > an > > > > > > > > > email > > > > > > > > > > > alerting > > > > > > > > > > > > > > > > > > > > > > > > > > > > > system like this < > > > > > https://www.confluent.io/blog/kafka-streams-take-on-watermarks-an > > > > > > > > > > > > > > d-t > > > > > > > > > > > > > > riggers/ > > > > > > > > > > > > > > > > > > > > > and had to collect the keys and > > windows of > > > > > > > trouble > > > > > > > > > by > > > > > > > > > > > hand. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think these kinds > > of use > > > > > > > cases > > > > > > > > > would > > > > > > > > > > > be much > > > > > > > > > > > > > > > > > > > > > > > > > > > > > common. Should it be > > > > > described > > > > > > > in > > > > > > > > > the > > > > > > > > > > > KIP much > > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > > > > > > > > > > > > > > in detail? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Feb 15, 2020 > > at > > > > > 4:43 AM > > > > > > > > > John > > > > > > > > > > > Roesler > > > > > > > > > > > > > > > > > > > > > > > > > > > > > <vvcep...@apache.org> > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you explain more > > > > > about > > > > > > > why > > > > > > > > > the > > > > > > > > > > > internal data > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > structures of > > > > > > > > > > > > > > > > > > > > > > > > > > > > suppression > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > should be > > queriable? The > > > > > > > > > motivation > > > > > > > > > > > just says > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > that users might > > > > > > > > > > > > > > > > > > > > > > > > > > want to > > > > > > > > > > > > > > > > > > > > > > > > > > > > do > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > it, which seems > > like it > > > > > could > > > > > > > > > > > justify literally > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > anything :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > One design point of > > > > > > > Suppression > > > > > > > > > is > > > > > > > > > > > that if you > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wanted to query the > > > > > > > > > > > > > > > > > > > > > > > > > > > > “final > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > state”, you can > > > > > Materialize > > > > > > > the > > > > > > > > > > > suppress itself > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > (which is why it > > > > > > > > > > > > > > > > > > > > > > > > > > needs > > > > > > > > > > > > > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > variant); if you > > wanted > > > > > to > > > > > > > query > > > > > > > > > the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > “intermediate > > state”, > > > > > you can > > > > > > > > > > > materialize the > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > operator immediately > > > > > before > > > > > > > the > > > > > > > > > > > suppress. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Example: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ...count(Materialized.as(“intermediate”)) > > > > > .supress(untilWindowClosed(), > > > > > Materialized.as(“final”)) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m not sure what > > use > > > > > case > > > > > > > would > > > > > > > > > > > require > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > actually fetching > > from > > > > > the > > > > > > > > > internal > > > > > > > > > > > buffers. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 14, > > 2020, at > > > > > > > 07:55, > > > > > > > > > > > Dongjin Lee > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I'd like to > > reboot the > > > > > > > > > discussion > > > > > > > > > > > on KIP-508, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > which aims to > > > > > > > > > > > > > > > > > > > > > > > > > > support a > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Materialized > > variant > > > > > of > > > > > > > > > > > KTable#suppress. It > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > was initially > > > > > > > > > > > > > > > > > > > > > > > > > > submitted > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > several > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > months ago but > > closed > > > > > by > > > > > > > the > > > > > > > > > > > inactivity. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - KIP: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-508%3A+Make > > > > > > > > > > > > > > +Su > > > > > > > > > > > > > > ppression+State+Queriable > > > > > > > > > > > > > > > > > > > > > - Jira: > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-8403 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > All kinds of > > feedback > > > > > will > > > > > > > be > > > > > > > > > > > greatly > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > appreciated. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in > > the > > > > > > > > > mathematical > > > > > > > > > > > world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > http://goog_969573159/> > > > > > > > > > > > > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > <https://github.com/dongjinleekr>linkedin: > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > https://kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > > > > speakerdeck: > > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > > > > > > > mathematical > > > > > > > > > > > world.* *github: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > http://goog_969573159/> > > > > > > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > https://github.com/dongjinleekr > > > > > > > > > > > > linkedin: > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > > > https://kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > > > > speakerdeck: > > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > > > > > mathematical > > > > > > > > > world.* > > > > > > > > > > > *github: > > > > > > > > > > > > > > > > > > > > > > > > > > > <http://goog_969573159/> > > > > > > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > https://github.com/dongjinleekr > > > > > > > > > > linkedin: > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > > > https://kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > speakerdeck: > > > > > > > > > > > > > > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > > > > > > > > > > > > > -- *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the > > mathematical > > > > > > > world.* > > > > > > > > > > > *github: > > > > > > > > > > > > > > > > > > > > > > > > > <http://goog_969573159/> > > > > > > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > > > > > > > > > > > > < > > https://github.com/dongjinleekr > > > > > > > > linkedin: > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > > > > > > > > > > > < > > > > > > > https://kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > speakerdeck: > > > > > > > > > > > > > > > > > > > > > > > > > speakerdeck.com/dongjin < > > > > > > > > > > > https://speakerdeck.com/dongjin > > > > > > > > > > > > > > > > > > * > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/> > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > > <https://github.com/dongjinleekr>keybase: > > > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > > > > <https://keybase.io/dongjinleekr>linkedin: > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > > > speakerdeck: > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > > > > > <https://github.com/dongjinleekr>keybase: > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > <https://keybase.io/dongjinleekr>linkedin: > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > > > > > > speakerdeck.com/dongjin > > > > > > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > > > > > > > > -- > > > > > > *Dongjin Lee* > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > > > <https://github.com/dongjinleekr>keybase: > > > > > https://keybase.io/dongjinleekr > > > > > > <https://keybase.io/dongjinleekr>linkedin: > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > > > > speakerdeck.com/dongjin > > > > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > > > > > > -- > > > > *Dongjin Lee* > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > > <https://github.com/dongjinleekr>keybase: > > https://keybase.io/dongjinleekr > > > > <https://keybase.io/dongjinleekr>linkedin: > > kr.linkedin.com/in/dongjinleekr > > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > speakerdeck.com/dongjin > > > > <https://speakerdeck.com/dongjin>* > > > > > > > > > > -- > > > *Dongjin Lee* > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > <https://github.com/dongjinleekr>keybase: > > https://keybase.io/dongjinleekr > > > <https://keybase.io/dongjinleekr>linkedin: > > kr.linkedin.com/in/dongjinleekr > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > speakerdeck.com/dongjin > > > <https://speakerdeck.com/dongjin>* > > > > >
