I don't think there is a JIRA for this, but I agree that the use case is good and compatible with the model. Please do open a JIRA ticket in the beam-model component. Feel free to assign to me or leave unassigned.
On Thu, Dec 8, 2016 at 5:30 PM, Manu Zhang <[email protected]> wrote: > @Kenn and @Tyler, > > Given the use case is defined (hope I've explained it clearly), do we have > plans/jiras to add the new functionality ? > > Thanks, > Manu > > On Fri, Dec 2, 2016 at 12:21 PM Manu Zhang <[email protected]> > wrote: > >> @Kenn, >> >> 1. when the watermark jumps from 0 to 7, http://foo -> http://foo/bar -> >> http://foo will be emitted >> We can emit events with timestamps before watermark in the pane >> 2. http://foo -> http://foo/bizzle -> http://foo/bar -> http://foo will >> be emitted if it's within the allowed lateness >> which Beam already allows us to do. >> >> To elaborate on the use case, when users are visiting Amazon we want to >> offer them best recommendations. >> Thus, we would like to know what leads to their final decision and track >> the pages they visit until clicking the "Add to cart" button. >> It will be too late if we only send the results when they finish shopping. >> >> @Tyler, >> I don't think it's likely to happen for my use case. Think about a user >> jumping between pages like crazy. Meanwhile, we can control how fast >> watermark progresses as long as it meets the latency requirement. >> >> >> >> >> On Fri, Dec 2, 2016 at 11:45 AM Tyler Akidau <[email protected]> wrote: >> >> And one more question while we're at it: what if you have events >> happening every second within the window? Do you really want to emit a new >> pane every second as the watermark progresses (assuming it progresses >> relatively smoothly)? What if we're talking differences of event times of >> milliseconds? Is one pane per millisecond what you want? >> >> -Tyler >> >> On Fri, Dec 2, 2016 at 10:41 AM Kenneth Knowles <[email protected]> wrote: >> >> Thanks for laying out some details. >> >> On Thu, Dec 1, 2016 at 7:09 PM, Manu Zhang <[email protected]> >> wrote: >> >> Yes, the difficulty is to define that trigger. The existing triggers fire >> at the end of window. (I could be mistaken, which will be good news) >> >> >> You are not mistaken that the only existing event time trigger is the one >> that fires at the end of the window. The trigger you describe would be a >> new primitive trigger. It fits with the design, if we ensure monotonicity, >> etc. Actually implementing it in the backend is easy, of course. We >> actually had something like it, but didn't quite nail it down so we removed >> it until we had a solid use case and design for it. >> >> B and C which are not mutually exclusive >> More on my use case. Say a user visits http://foo at 1, http://foo/bar at >> 4 and back to http://foo at 5 all in a Session >> we would want to emit >> >> http://foo when the watermark passes 1 >> http://foo -> http://foo/bar when the watermark passes 4 >> http://foo -> http://foo/bar -> http://foo when the watermark passes 5 >> >> >> What would you want to emit when the watermark jumps from 0 to 7 and all >> three of the above are buffered? >> >> What would you want to emit when the watermark was at 9 and >> http://foo/bizzle came in with timestamp 3? >> >> Kenn >> >> >> >> >> On Fri, Dec 2, 2016 at 10:12 AM Ben Chambers <[email protected]> >> wrote: >> >> As a clarifying question: >> >> If you have three elements in the pane with timestamps [1, 4, 5], would >> you: >> A. want to emit that entire pane when the watermark passes 1 >> B. want to emit that entire pane when the watermark passes 5 >> C. emit a fragment of that pane containing only the first element when >> the watermark passes 1 >> >> On Thu, Dec 1, 2016 at 6:01 PM Tyler Akidau <[email protected]> wrote: >> >> So what you want is essentially a trigger that fires when the watermark >> has passed the event time of the oldest un-emitted element in the current >> pane? You could them presumably wrap this in a repeat to get the overall >> desired semantics, right? >> >> -Tyler >> >> >> On Fri, Dec 2, 2016 at 7:32 AM Manu Zhang <[email protected]> >> wrote: >> >> My use case is to track user trajectory based on page view event when >> they visit a website. The input would be like a list of >> PageView(userId, url, eventTimestamp) with watermarks (= eventTimestamp - >> duration). I'm trying Sessions with event time trigger. Note we can't wait >> for the end of session window due to latency requirement. Instead, we want >> to emit the user trajectories whenever a buffered PageView's event time is >> passed by watermark. >> >> On Fri, Dec 2, 2016 at 5:41 AM Lukasz Cwik <[email protected]> wrote: >> >> Can you provide more details about the problem your trying to solve with >> some examples showing input and the expected output? >> >> >> >> >> On Wed, Nov 30, 2016 at 11:08 PM, Manu Zhang <[email protected]> >> wrote: >> >> Hi, >> >> Recently I’m addressing a problem where users want to trigger after >> watermark past each element (i.e. in the middle of event-time window). I >> fail to find an existing trigger that does so. Any idea on model this >> problem with Beam ? >> >> Thanks, >> Manu Zhang >> >> >>
