Re: Re: How to define event time and Watermark for intermediary joined table in FlinkSQL?

2022-05-02 Thread liuxiangcao
Thanks for the jira link. Actually my comment in the initial email "In Java DataStream API, you can easily do so within flink topology without having to create a separate kafka topic: " is incorrect. I took a closer look and realized Flink Java DataStream also does not support redefining Timesta

Re:Re: How to define event time and Watermark for intermediary joined table in FlinkSQL?

2022-04-29 Thread Xuyang
I think it's not a good idea to defining a watermark on a view, because currently the view is only a set of SQL query text in Flink , and a query should not contain a watermark definition. You can see the discussion here: https://issues.apache.org/jira/browse/FLINK-22804 Maybe you can open a jir

Re: How to define event time and Watermark for intermediary joined table in FlinkSQL?

2022-04-28 Thread liuxiangcao
Hi Shengkai, Thank you for the reply. The UDF getEventTimeInNS uses timestamps of both streamA and streamB to calculate the true event time for streamB events. For illustrating purpose, we can consider it to be like this: public Long eval( Long baseTimeStampFromA, Long timestamp

Re: How to define event time and Watermark for intermediary joined table in FlinkSQL?

2022-04-21 Thread Shengkai Fang
Hi, The watermark of the join operator is the minimum of the watermark of the input streams. ``` JoinOperator.watermark = min(left.watermark, right.watermark); ``` I think it's enough for most cases. Could you share more details about the logic in the UDF getEventTimeInNS? I think the better s

How to define event time and Watermark for intermediary joined table in FlinkSQL?

2022-04-15 Thread liuxiangcao
Hi Flink community, *Here is the context: * Theoretically, I would like to write following query but it won't work since we can only define the WATERMARK in a table DDL: INSERT into tableC select tableA.field1 SUM(1) as `count`, time_ltz AS getEventTimeInNS(tableA.timestamp, tab