PR submitted :) https://github.com/apache/kafka/pull/8520
On Mon, Apr 20, 2020 at 2:34 PM John Roesler <vvcep...@apache.org> wrote: > Hi Liam, > > That sounds like a good idea to me. In fact, I’d go so far as to say we > should just change the existing example to include a grace period, and not > bother with an extra example. That would put it front and center. > > A PR would be greatly appreciated! Thanks for the offer! > > Thanks, > John > > On Sun, Apr 19, 2020, at 19:58, Liam Clarke wrote: > > Hi Matthias, > > > > I think as an interim measure, if the windowing samples in the docs > showed > > an additional example where the grace period was set (with perhaps a > > comment about the current default grace period, and planned future > > changes?) it would make it sufficiently visible - happy to submit a PR > with > > those changes if it seems appropriate. > > > > Cheers, > > > > Liam Clarke-Hutchinson > > > > On Mon, Apr 20, 2020 at 12:12 PM Matthias J. Sax <mj...@apache.org> > wrote: > > > > > I would prefer to not make the grace-period a mandatory argument and > > > keep the API as-is. I understand the issue of backward compatibility, > > > but I would still argue that we should just change the default grace > > > period to 0 in the 3.0 release. It's a major release and thus it seems > > > to be fine. To prepare for this change, we could start to log a WARN > > > message, if a user does not set the grace period explicitly for now. > > > > > > Just my 2 ct. Thoughts? > > > > > > -Matthias > > > > > > On 4/19/20 7:40 AM, John Roesler wrote: > > > > Oh, man, that’s a good idea. > > > > > > > > I can propose to deprecate (not remove) the existing ‘of’ factory > method > > > and add one with a mandatory grace period. Not sure why I didn’t think > of > > > that before. Probably too caught up in looking for something “smart”. > > > > > > > > Thanks! > > > > John > > > > > > > > On Sun, Apr 19, 2020, at 02:27, Liam Clarke wrote: > > > >> Hi John, > > > >> > > > >> I can't really think of a way to make it more obvious without > breaking > > > >> backwards compatibility - e.g., obvious easy fix is that grace > period > > > is a > > > >> mandatory arg to TimeWindows, but that would definitely break > > > compatibility. > > > >> > > > >> Cheers, > > > >> > > > >> Liam Clarke-Hutchinson > > > >> > > > >> On Thu, Apr 16, 2020 at 1:59 AM John Roesler <vvcep...@apache.org> > > > wrote: > > > >> > > > >>> Boom, you got it, Liam! Nice debugging work. > > > >>> > > > >>> This is a pretty big bummer, but I had to do it that way for > > > >>> compatibility. I added a log message to try and help reduce the > risk, > > > but > > > >>> it’s still kind of a trap. > > > >>> > > > >>> I’d like to do a KIP at some point to consider changing the default > > > grace > > > >>> period, but haven’t done it because it’s not clear what the default > > > should > > > >>> be. > > > >>> > > > >>> Please let me know if you have any ideas! > > > >>> Thanks, > > > >>> -John > > > >>> > > > >>> > > > >>> On Tue, Apr 14, 2020, at 23:44, Liam Clarke wrote: > > > >>>> And the answer is to change > > > >>>> .windowedBy(TimeWindows.of(Duration.ofMillis(5000))) > > > >>>> and specify the grace period: > > > >>>> > > > >>>> > > > >>> > > > > windowedBy(TimeWindows.of(Duration.ofMillis(5000)).grace(Duration.ofMillis(100))) > > > >>>> > > > >>>> On Wed, Apr 15, 2020 at 4:34 PM Liam Clarke < > > > liam.cla...@adscale.co.nz> > > > >>>> wrote: > > > >>>> > > > >>>>> Okay, doing some debugging it looks like I'm seeing this > behaviour > > > >>> because > > > >>>>> it's picking up a grace duration of 86,395,000 ms in > > > >>>>> KTableImpl.buildSuppress, which would happen to be 5000 millis > (my > > > >>> window > > > >>>>> size) off 24 hours, so I've got some clues! > > > >>>>> > > > >>>>> On Wed, Apr 15, 2020 at 3:43 PM Liam Clarke < > > > liam.cla...@adscale.co.nz > > > >>>> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> Hi all, > > > >>>>>> > > > >>>>>> I have a case where I want to consume from a topic, count the > number > > > >>> of > > > >>>>>> certain ids in a given time period X, and emit a new record to a > > > >>> different > > > >>>>>> topic after that same time period X has elapsed containing the > > > >>> aggregated > > > >>>>>> value. > > > >>>>>> > > > >>>>>> I'm using suppress with Suppressed.untilWindowCloses, but > nothing is > > > >>> ever > > > >>>>>> emitted, nor is my peek placed after the suppress ever being > hit. > > > >>>>>> My code is in the below Gist - I've hardcoded the durations for > 5 > > > >>> seconds > > > >>>>>> after testing purposes: > > > >>>>>> > > > https://gist.github.com/LiamClarkeNZ/24121ccf0f09e4530749cbd92633fa46 > > > >>>>>> > > > >>>>>> I'm assuming I've misunderstood something drastically, and would > > > >>> greatly > > > >>>>>> appreciate a pointer on where I may have gone wrong. I'm > wondering > > > if > > > >>> I > > > >>>>>> need a larger retention on the persistent store? > > > >>>>>> > > > >>>>>> I understand that events have to arrive in order for windows to > > > >>> close, so > > > >>>>>> I've sent events after the window has expired to attempt to > move the > > > >>> window > > > >>>>>> on, and my first peek (before the suppression) is emitting as I > do: > > > >>>>>> > > > >>>>>> 1. 2020-04-15T03:36:48.569Z > e2442bef-72bf-4424-b94e-7e4743e03c5e - 1 > > > >>>>>> 1. 2020-04-15T03:37:11.682Z > e2442bef-72bf-4424-b94e-7e4743e03c5e - 1 > > > >>>>>> 1. 2020-04-15T03:39:18.882Z aqgzftnvyn - 1 > > > >>>>>> > > > >>>>>> > > > >>>>>> Any guidance greatfully appreciated. > > > >>>>>> > > > >>>>>> Kind regards, > > > >>>>>> > > > >>>>>> Liam Clarke > > > >>>>>> > > > >>>>> > > > >>>> > > > >>> > > > >> > > > > > > > > >
