OK. I also prefer the delete option. The main issue that remains is SQL, or
joins in general.
Kenn
On Wed, Feb 17, 2021 at 1:17 PM Robert Bradshaw wrote:
> On Wed, Feb 17, 2021 at 12:30 PM Kenneth Knowles wrote:
>
>> Slight correction: you mean default SlidingWindows? This is the only
>> nontr
On Wed, Feb 17, 2021 at 12:30 PM Kenneth Knowles wrote:
> Slight correction: you mean default SlidingWindows? This is the only
> nontrivial implementation I know of.
>
Yes.
> Sessions does not shift the timestamp because you can't really guess how
> far to shift it. Sessions with EARLIEST just
These are all marked experimental, but they are also all many years old.
However, this is exactly why they are experimental: they aren't really a
good solution that we are totally confident in.
Kenn
On Wed, Feb 17, 2021 at 12:30 PM Kenneth Knowles wrote:
> Slight correction: you mean default Sl
Slight correction: you mean default SlidingWindows? This is the only
nontrivial implementation I know of. Sessions does not shift the timestamp
because you can't really guess how far to shift it. Sessions with EARLIEST
just get hung entirely whenever there is a long-lived session.
I've thought abo
OK, so to move forward, shall we update the default Sessions to not do this
timestamp shifting, perhaps with a (deprecated) timestamp-shifting opt-in
variant to ease the transition for those that want the old (marked
experimental) behavior?
On Fri, Feb 12, 2021 at 9:12 AM Kenneth Knowles wrote:
All of this is right. Things have changed a lot. Nowadays the default will
work well, and we can caveat to users that EARLIEST may hold up downstream
output for overlapping windows.
I'm slightly concerned about the fact that EARLIEST is necessary for CoGBK
joins, unless there is some special consi
Yes, unless you manually set the timestamp combiner to earliest, which in
this case gives earliest + shifted.
On Fri, Feb 12, 2021 at 12:33 AM Reuven Lax wrote:
> The default now is end of window, right? Doesn't that alleviate the
> problem that the original change was supposed to fix?
>
> On Fr
The default now is end of window, right? Doesn't that alleviate the problem
that the original change was supposed to fix?
On Fri, Feb 12, 2021 at 12:25 AM Robert Bradshaw
wrote:
> The default timestamp combiner used to be earliest as well.
>
> On Fri, Feb 12, 2021 at 12:10 AM Reuven Lax wrote:
The default timestamp combiner used to be earliest as well.
On Fri, Feb 12, 2021 at 12:10 AM Reuven Lax wrote:
> IIRC, this was introduced because at the time users complained that
> sliding windows were virtually unusable for reasonably-sized windows.
> However this was before we allowed custom
IIRC, this was introduced because at the time users complained that sliding
windows were virtually unusable for reasonably-sized windows. However this
was before we allowed customizing the timestamp combiner, so maybe this is
less of a problem now?
On Thu, Feb 11, 2021 at 10:53 PM Robert Bradshaw
On Wed, Feb 10, 2021 at 8:03 PM Kenneth Knowles wrote:
>
>
> On Wed, Feb 10, 2021 at 2:24 PM Alex Amato wrote:
>
>>
>>
>> On Wed, Feb 10, 2021 at 12:14 PM Kenneth Knowles wrote:
>>
>>> On a PR (https://github.com/apache/beam/pull/13927) we got into a
>>> discussion of a very old and strange fea
On Wed, Feb 10, 2021 at 2:24 PM Alex Amato wrote:
>
>
> On Wed, Feb 10, 2021 at 12:14 PM Kenneth Knowles wrote:
>
>> On a PR (https://github.com/apache/beam/pull/13927) we got into a
>> discussion of a very old and strange feature of Beam that I think we should
>> revisit.
>>
>> The WindowFn has
On Wed, Feb 10, 2021 at 12:14 PM Kenneth Knowles wrote:
> On a PR (https://github.com/apache/beam/pull/13927) we got into a
> discussion of a very old and strange feature of Beam that I think we should
> revisit.
>
> The WindowFn has the ability to shift timestamps forward in order to
> unblock d
On a PR (https://github.com/apache/beam/pull/13927) we got into a
discussion of a very old and strange feature of Beam that I think we should
revisit.
The WindowFn has the ability to shift timestamps forward in order to
unblock downstream watermarks. Why? Specifically in this situation:
- aggreg
14 matches
Mail list logo
