Hi Kui,
Thanks for driving the discussion. It's quite useful to introduce Watermark
options. I have some questions:
What kind of hints is "WATERMARK_PARAMS"?
Currently, we have two kinds of hints in Flink: Dynamic Table Options &
Query Hints. As described in the Flip, "WATERMARK_PARAMS" is more like
Dynamic Table Options. So two questions arise here:
1) Are these watermark options to be exposed as connector WITH options? Aa
described in SQL Hints doc[1], "Dynamic Table Options allow to specify or
override table options dynamically", which implies that these options can
also be configured in WITH options.
2) Do we really need a new hint name like 'WATERMARK_PARAMS', table
options use "OPTIONS" as hint name, like '/*+
OPTIONS('csv.ignore-parse-errors'='true') */', maybe we can enrich more
table option keys for watermark, e.g., /*+
OPTIONS('watermark.emit-strategy'='ON_PERIODIC') */.
[1]
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/queries/hints/
On Wed, Feb 22, 2023 at 10:22 AM kui yuan <catye...@gmail.com> wrote:
> Hi devs,
>
>
> I'd like to start a discussion thread for FLIP-296[1]. This comes from an
> offline discussion with @Yun Tang, and we hope to enrich table API & SQL to
> support many watermark-related features which were only implemented at the
> datastream API level.
>
>
> Basically, we want to introduce watermark options in table API & SQL via
> SQL hint named 'WATERMARK_PARAMS' to support features:
>
> 1、Configurable watermark emit strategy
>
> 2、Dealing with idle sources
>
> 3、Watermark alignment
>
>
> Last but not least, thanks to Qingsheng and Jing Zhang for the initial
> reviews.
>
>
> Looking forward to your thoughts and any feedback is appreciated!
>
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240884405
>
>
> Best
>
> Yuan Kui
>
