Sorry, I made a typo above. I mean I prefer proposal (1) that only needs to set `table.exec.emit.allow-lateness` to handle late events. `table.exec.emit.late-fire.delay` can be optional which is 0s by default. `table.exec.state.ttl` will not affect window state anymore, so window state is still cleaned accurately by watermark.
We don't need to expose `table.exec.emit.late-fire.enabled` on docs and can remove it in the next version. Best, Jark On Thu, 1 Jul 2021 at 21:20, Jark Wu <imj...@gmail.com> wrote: > Thanks Jing for bringing up this topic, > > The emit strategy configs are annotated as Experiential and not public on > documentations. > However, I see this is a very useful feature which many users are looking > for. > I have posted these configs for many questions like "how to handle late > events in SQL". > Thus, I think it's time to make the configuration public and explicitly > document it. In the long > term, we would like to propose an EMIT syntax for SQL, but until then we > can get more > valuable feedback from users when they are using the configs. > > Regarding the exposed configuration, I prefer proposal (2). > But it would be better not to expose `table.exec.emit.late-fire.enabled` > on docs and we can > remove it in the next version. > > Best, > Jark > > > On Tue, 29 Jun 2021 at 11:09, JING ZHANG <beyond1...@gmail.com> wrote: > >> When WindowAggregate works upon Changelog which contains update messages, >> UPDATE BEFORE message may be dropped as a late message. [1] >> >> In order to handle late UB message, user needs to set *all* the >> following 3 parameters: >> >> (1) enable late fire by setting >> >> table.exec.emit.late-fire.enabled : true >> >> (2) set per record emit behavior for late records by setting >> >> table.exec.emit.late-fire.delay : 0 s >> >> (3) keep window state for extra time after window is fired by setting >> >> table.exec.emit.allow-lateness : 1 h// 或者table.exec.state.ttl: 1h >> >> >> The solution has two disadvantages: >> >> (1) Users may not realize that UB messages may be dropped as a late >> event, so they will not set related parameters. >> >> (2) When users look for a solution to solve the dropped UB messages >> problem, the current solution is a bit inconvenient for them because they >> need to set all the 3 parameters. Besides, some configurations have overlap >> ability. >> >> >> Now there are two proposals to simplify the 3 parameters a little. >> >> (1) Users only need set table.exec.emit.allow-lateness (just like the >> behavior on Datastream, user only need set allow-lateness), framework could >> atom set `table.exec.emit.late-fire.enabled` to true and set >> `table.exec.emit.late-fire.delay` to 0s. >> >> And in the later version, we deprecate `table.exec.emit.late-fire.delay` >> and `table.exec.emit.late-fire.enabled`. >> >> >> (2) Users need set `table.exec.emit.late-fire.enabled` to true and set >> `table.exec.state.ttl`, framework could atom set >> `table.exec.emit.late-fire.delay` to 0s. >> >> And in the later version, we deprecate `table.exec.emit.late-fire.delay` >> and `table.exec.emit.allow-lateness `. >> >> >> Please let me know what you think about the issue. >> >> Thank you. >> >> [1] https://issues.apache.org/jira/browse/FLINK-22781 >> >> >> Best regards, >> JING ZHANG >> >> >> >>
