Re: Output timestamp for Python event timers

2020-08-12 Thread Boyuan Zhang
Thanks for your help! I'll take a look at the PR. On Wed, Aug 12, 2020 at 2:27 AM Maximilian Michels wrote: > Thanks for your suggestions! > > It makes sense to complete the work on this feature by exposing it in > the Python API. We can do this as a next step. (There might be questions > on how

Re: Output timestamp for Python event timers

2020-08-12 Thread Maximilian Michels
Thanks for your suggestions! It makes sense to complete the work on this feature by exposing it in the Python API. We can do this as a next step. (There might be questions on how to do that exactly) For now, I'm concerned with getting the semantics right and unblocking users from stalling pi

Re: Output timestamp for Python event timers

2020-08-11 Thread Luke Cwik
+1 on what Boyuan said. It is important that the defaults for processing time domain differ from the defaults for the event time domain. On Tue, Aug 11, 2020 at 12:36 PM Yichi Zhang wrote: > +1 to expose set_output_timestamp and enrich python set timer api. > > On Tue, Aug 11, 2020 at 12:01 PM B

Re: Output timestamp for Python event timers

2020-08-11 Thread Yichi Zhang
+1 to expose set_output_timestamp and enrich python set timer api. On Tue, Aug 11, 2020 at 12:01 PM Boyuan Zhang wrote: > Hi Maximilian, > > It makes sense to set hold_timestamp as fire_timestamp when the > fire_timestamp is in the event time domain. Otherwise, the system may > advance the wate

Re: Output timestamp for Python event timers

2020-08-11 Thread Boyuan Zhang
Hi Maximilian, It makes sense to set hold_timestamp as fire_timestamp when the fire_timestamp is in the event time domain. Otherwise, the system may advance the watermark incorrectly. I think we can do something similar to Java FnApiRunner[1]: - Expose set_output_timestamp API to python timer

Output timestamp for Python event timers

2020-08-11 Thread Maximilian Michels
We ran into problems setting event time timers per-element in the Python SDK. Pipeline progress would stall. Turns out, although the Python SDK does not expose the timer output timestamp feature to the user, it sets the timer output timestamp to the current input timestamp of an element. Thi