Re: Throttle PTransform

2024-02-23 Thread Robert Bradshaw via dev
On Thu, Feb 22, 2024 at 10:16 AM Robert Bradshaw wrote: > > On Thu, Feb 22, 2024 at 9:37 AM Reuven Lax via dev > wrote: > > > > On Thu, Feb 22, 2024 at 9:26 AM Kenneth Knowles wrote: > >> > >> Wow I love your input Reuven. Of course "the source" that you are applying > >> backpressure to is of

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

2024-02-23 Thread Robert Bradshaw via dev
On Fri, Feb 23, 2024 at 3:54 PM Robert Burke wrote: > > While I'm currently on the other side of the fence, I would not be against > changing/requiring the semantics of ProcessingTime constructs to be "must > wait and execute" as such a solution, and enables the Proposed "batch" > process conti

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

2024-02-23 Thread Robert Burke
While I'm currently on the other side of the fence, I would not be against changing/requiring the semantics of ProcessingTime constructs to be "must wait and execute" as such a solution, and enables the Proposed "batch" process continuation throttling mechanism to work as hypothesized for both "

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

2024-02-23 Thread Robert Bradshaw via dev
Thanks for bringing this up. My position is that both batch and streaming should wait for processing time timers, according to local time (with the exception of tests that can accelerate this via faked clocks). Both ProcessContinuations delays and ProcessingTimeTimers are IMHO isomorphic, and can

Beam High Priority Issue Report (37)

2024-02-23 Thread beamactions
This is your daily summary of Beam's current high priority issues that may need attention. See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities. Unassigned P0 Issues: https://github.com/apache/beam/issues/30377 [Failing Test]: 404

Re: [DISCUSS] Processing time timers in "batch" (faster-than-wall-time [re]processing)

2024-02-23 Thread Jan Lukavský
For me it always helps to seek analogy in our physical reality. Stream processing actually has quite a good analogy for both event-time and processing-time - the simplest model for this being relativity theory. Event-time is the time at which events occur _at distant locations_. Due to finite a