Re: Watermark generating mechanism in Flink SQL
Hi Hunk, there is documentation about watermarking in FlinkSQL [1]. There is also a FlinkSQL cookbook entry about watermarking [2]. Essentially, you define the watermark strategy in your CREATE TABLE statement and specify the lateness for a given event (not the period in which watermarks are automatically generated!). You have to apply the `WATERMARK FOR` phrase on a column that is declared as a time attribute [3]. Watermarks are based on event time, i.e. based on an event being processed that provides the event time. Your idea of generating them "every 5 seconds" does not work out of the box because a watermark wouldn't be generated if the source idles for more than 5 seconds (in case of your specific example). Sending periodic dummy events extrapolating the current event time would be a way to work around this issue. Keep in mind that mixing processing time (what you would do if you create a watermark based on the system's current time rather than relying on events) and event time is usually not advised. I hope that helps. Best, Matthias [1] https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/create/#watermark [2] https://github.com/ververica/flink-sql-cookbook/blob/main/aggregations-and-analytics/02_watermarks/02_watermarks.md [3] https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/concepts/time_attributes/#event-time On Tue, Oct 18, 2022 at 5:32 AM wang <24248...@163.com> wrote: > Hi dear engineers, > > I have one question about watermark generating mechanism in Flink SQL. > There are two mechanisms called *Periodic Watermarks* and *Punctuated > Watermarks, *I want to use* Periodic Watermarks* with interval 5 seconds > (meaning watermarks will be generated every 5 seconds), how should I set in > Flink sql? thanks in advance! > > Regards, > Hunk >
Re: Watermark generating mechanism in Flink SQL
Hi Hunk, there is documentation about watermarking in FlinkSQL [1]. There is also a FlinkSQL cookbook entry about watermarking [2]. Essentially, you define the watermark strategy in your CREATE TABLE statement and specify the lateness for a given event (not the period in which watermarks are automatically generated!). You have to apply the `WATERMARK FOR` phrase on a column that is declared as a time attribute [3]. Watermarks are based on event time, i.e. based on an event being processed that provides the event time. Your idea of generating them "every 5 seconds" does not work out of the box because a watermark wouldn't be generated if the source idles for more than 5 seconds (in case of your specific example). Sending periodic dummy events extrapolating the current event time would be a way to work around this issue. Keep in mind that mixing processing time (what you would do if you create a watermark based on the system's current time rather than relying on events) and event time is usually not advised. I hope that helps. Best, Matthias [1] https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/create/#watermark [2] https://github.com/ververica/flink-sql-cookbook/blob/main/aggregations-and-analytics/02_watermarks/02_watermarks.md [3] https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/concepts/time_attributes/#event-time On Tue, Oct 18, 2022 at 5:32 AM wang <24248...@163.com> wrote: > Hi dear engineers, > > I have one question about watermark generating mechanism in Flink SQL. > There are two mechanisms called *Periodic Watermarks* and *Punctuated > Watermarks, *I want to use* Periodic Watermarks* with interval 5 seconds > (meaning watermarks will be generated every 5 seconds), how should I set in > Flink sql? thanks in advance! > > Regards, > Hunk >
Watermark generating mechanism in Flink SQL
Hi dear engineers, I have one question about watermark generating mechanism in Flink SQL. There are two mechanisms called Periodic Watermarks and Punctuated Watermarks, I want to use Periodic Watermarks with interval 5 seconds (meaning watermarks will be generated every 5 seconds), how should I set in Flink sql? thanks in advance! Regards, Hunk
Watermark generating mechanism in Flink SQL
Hi dear engineers, I have one question about watermark generating mechanism in Flink SQL. There are two mechanisms called Periodic Watermarks and Punctuated Watermarks, I want to use Periodic Watermarks with interval 5 seconds (meaning watermarks will be generated every 5 seconds), how should I set in Flink sql? thanks in advance! Regards, Hunk
