Have you looked at using withTimestampCombiner(TimestampCombiner.EARLIEST)? This will hold the downstream watermark back to the beginning of the window (presumably the timestamp of the LOGIN event), so you can .call outputWithTimestamp using the CLICK GREEN timestamp without needing to set the allowed-lateness skew.
Reuven On Sat, Jan 11, 2020 at 1:50 PM Aaron Dixon <[email protected]> wrote: > I've just built a pipeline in Beam and after exploring several options for > my use case, I've ended up relying on the deprecated > .outputWithTimestamp() + DoFn#getAllowedTimestampSkew in what seems to me a > quite valid use case. So I suppose this is a vote for un-deprecating this > API (or a teachable moment in which I could be pointed to a more suitable > non-deprecated approach.) > > I'll stick with a previously simplification of my use case: > > I get these events from my users: > LOGIN > CLICK GREEN BUTTON > LOGOUT > > I capture user session duration (logout time *minus* login time) and I > want to perform a PER DAY average (i.e., my window is on CalendarDays) BUT > where the aggregation's timestamp is the time of the CLICK GREEN event. > > So once I calculate and emit a single user's session duration I need to > .outputWithTimestamp using the CLICK GREEN event's timestamp. This > involves, of course, outputting with a timestamp *before* the watermark. > > In most cases my users LOGOUT in the same day as the CLICK GREEN BUTTON > event, so even though I'm typically outputting a timestamp before the > watermark the CalendarDay window is not yet closed and so most user session > duration's do not affect a late aggregation for that CalendarDay. > > Only when a LOGOUT occurs on a day later than the CLICK GREEN event do I > have to contend with potentially late data contributing back to a prior > CalendarDay. > > In any case, I have .withAllowedLateness to allow me to make a call here > about what I'm willing tradeoff (keeping windows open vs. dropping data for > users with overly long sessions), etc. > > This here seems to be a simple scenario (it is effectively my real-world > scenario) and the .outputWithTimestamp + DoFn#getAllowedTimestampSkew seem > to cover it in a straightforward, effective way. > > However of course I don't like building production code on deprecated > capabilities -- so advice on alternatives (or perhaps a reconsideration of > this deprecation :) ) would be appreciated. > >
