Hi all,
I have updated the flip according to the discussion, and we will extend the
watermark-related features with both table options and 'OPTIONS' hint, like
this:
```
-- configure in table options
CREATE TABLE user_actions (
...
user_action_time TIMESTAMP(3),
WATERMARK FOR user_action_time AS user_action_time - INTERVAL '5' SECOND
) WITH (
'watermark.emit.strategy'='ON_PERIODIC',
...
);
-- use 'OPTIONS' hint
select ... from source_table /*+ OPTIONS('watermark.emit.strategy'=
'ON_PERIODIC') */
```
Does everybody have any other questions?
Best
Kui Yuan
kui yuan <catye...@gmail.com> 于2023年2月23日周四 20:05写道：
> Hi all,
>
> Thanks for all suggestions.
>
> We will extend the watermark-related features in SQL layer with dynamic
> table options and 'OPTIONS' hint, just as everyone expects. I will modify
> Flip-296 as discussed.
>
> @Martijn As far as I know, there is no hint interface in the table API,
> so we can't use hint in table API directly. if we need to extend the hint
> interface in the table API, maybe we need another flip. However, if we
> extend the watermark-related features in the dynamic table options, maybe
> we are able to use them indirectly in the table API like this[1]:
>
> ```
> // register a table named "Orders"
> tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product STRING,
> amount INT) WITH ('watermark.emit.strategy'='ON_EVENT'...)");
> ```
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/create/
>
> Best
> Kui Yuan
>
> Yun Tang <myas...@live.com> 于2023年2月23日周四 17:46写道：
>
>> Thanks for the warm discussions!
>>
>> I had an offline discussion with Kui about the replies. I think I could
>> give some explanations on the original intention to introduce another
>> WATERMARK_PARAMS. If we take a look at the current datastream API, the
>> watermark strategy does not belong to any specific connector. And we
>> thought the dynamic table options were more like the configurations within
>> some specific connector.
>>
>> From the review comments, I think most people feel good to make it part
>> of the dynamic table options. I think this is fine if we give more clear
>> scope definition of the dynamic table options here. And I also agree with
>> Jingsong's concern about adding SQL syntax which is the most concerning
>> part before launching this discussion.
>>
>> For Martijn's concern, if we accept to make the watermark-related options
>> part of dynamic table options, the problem becomes another topic: how to
>> support the dynamic table options in table API, which is deserved to create
>> another FLIP.
>>
>> Best
>> Yun Tang
>> ________________________________
>> From: Martijn Visser <martijnvis...@apache.org>
>> Sent: Thursday, February 23, 2023 17:14
>> To: dev@flink.apache.org <dev@flink.apache.org>
>> Subject: Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL
>>
>> Hi,
>>
>> While I can understand that there's a desire to first focus on solving
>> this
>> problem for SQL, I do wonder if we should ignore the Table API at this
>> point. If we could include the syntax for the Table API, it potentially
>> could also be implemented by another contributor without needing to create
>> another FLIP. If we don't design it right now, my concern is that this
>> will
>> increase sparsity for the Table API which ultimately hurts adoption.
>>
>> With regards to the syntax, I have a preference to solve this via the
>> connector options (e.g. like you can currently specify things as
>> scan.startup.specific-offsets or scan.bounded.mode for the Kafka
>> connector). You could still use the dynamic table options to override/add
>> them.
>>
>> Best regards,
>>
>> Martijn
>>
>> On Thu, Feb 23, 2023 at 7:21 AM Shammon FY <zjur...@gmail.com> wrote:
>>
>> > Hi kui
>> >
>> > Thanks for your answer and +1 to yuxia too
>> >
>> > > we should not bind the watermark-related options to a connector to
>> ensure
>> > semantic clarity.
>> >
>> > In my opinion, adding watermark-related options to a connector is much
>> more
>> > clear. Currently users can define simple watermark strategy in DDL,
>> adding
>> > more configuration items in connector options is easy to understand
>> >
>> > Best,
>> > Shammon
>> >
>> >
>> > On Thu, Feb 23, 2023 at 10:52 AM Jingsong Li <jingsongl...@gmail.com>
>> > wrote:
>> >
>> > > Thanks for your proposal.
>> > >
>> > > +1 to yuxia, consider watermark-related hints as option hints.
>> > >
>> > > Personally, I am cautious about adding SQL syntax, WATERMARK_PARAMS is
>> > > also SQL syntax to some extent.
>> > >
>> > > We can use OPTIONS to meet this requirement if possible.
>> > >
>> > > Best,
>> > > Jingsong
>> > >
>> > > On Thu, Feb 23, 2023 at 10:41 AM yuxia <luoyu...@alumni.sjtu.edu.cn>
>> > > wrote:
>> > > >
>> > > > Hi, Yuan Kui.
>> > > > Thanks for driving it.
>> > > > IMO, the 'OPTIONS' hint may be not only specific to the connector
>> > > options. Just as a reference, we also have `sink.parallelism`[1] as a
>> > > connector options. It enables
>> > > > user to specific the writer's parallelism dynamically per-query.
>> > > >
>> > > > Personally, I perfer to consider watermark-related hints as option
>> > > hints. So, user can define a default watermark strategy for the table,
>> > and
>> > > if user dosen't needed to changes it, they need to do nothing in their
>> > > query instead of specific it ervery time.
>> > > >
>> > > > [1]
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-master/zh/docs/connectors/table/filesystem/#sink-parallelism
>> > > >
>> > > > Best regards,
>> > > > Yuxia
>> > > >
>> > > > Best regards,
>> > > > Yuxia
>> > > >
>> > > > ----- 原始邮件 -----
>> > > > 发件人: "kui yuan" <catye...@gmail.com>
>> > > > 收件人: "dev" <dev@flink.apache.org>
>> > > > 抄送: "Jark Wu" <imj...@gmail.com>
>> > > > 发送时间: 星期三, 2023年 2 月 22日 下午 10:08:11
>> > > > 主题: Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL
>> > > >
>> > > > Hi all,
>> > > >
>> > > > Thanks for the lively discussion and I will respond to these
>> questions
>> > > one
>> > > > by one. However, there are also some common questions and I will
>> answer
>> > > > together.
>> > > >
>> > > > @郑 Thanks for your reply. The features mentioned in this flip are
>> only
>> > > for
>> > > > those source connectors that implement the SupportsWatermarkPushDown
>> > > > interface, generating watermarks in other graph locations is not in
>> the
>> > > > scope of this discussion. Perhaps another flip can be proposed
>> later to
>> > > > implement this feature.
>> > > >
>> > > > @Shammon Thanks for your reply. In Flip-296, a rejected alternative
>> is
>> > > > adding watermark related options in the connector options，we believe
>> > that
>> > > > we should not bind the watermark-related options to a connector to
>> > ensure
>> > > > semantic clarity.
>> > > >
>> > > > > What will happen if we add watermark related options in `the
>> > connector
>> > > > > options`? Will the connector ignore these options or throw an
>> > > exception?
>> > > > > How can we support this?
>> > > >
>> > > > If user defines different watermark configurations for one table in
>> two
>> > > > places, I tend to prefer the first place would prevail, but we can
>> > also
>> > > > throw exception or just print logs to prompt the user, which are
>> > > > implementation details.
>> > > >
>> > > > > If one table is used by two operators with different watermark
>> > params,
>> > > > > what will happen?
>> > > >
>> > > > @Martijn Thanks for your reply. I'm sorry that we are not
>> particularly
>> > > > accurate, this hint is mainly for SQL, not table API.
>> > > >
>> > > > > While the FLIP talks about watermark options for Table API & SQL,
>> I
>> > > only
>> > > > > see proposed syntax for SQL, not for the Table API. What is your
>> > > proposal
>> > > > > for the Table API
>> > > >
>> > > > @Jane Thanks for your reply. For the first question, If the user
>> uses
>> > > this
>> > > > hint on those sourse that does not implement the
>> > > SupportsWatermarkPushDown
>> > > > interface, it will be completely invalid. The task will run as
>> normal
>> > as
>> > > if
>> > > > the hint had not been used.
>> > > >
>> > > > > What's the behavior if there are multiple table sources, among
>> which
>> > > > > some do not support `SupportsWatermarkPushDown`?
>> > > >
>> > > > @Jane feedback that 'WATERMARK_PARAMS' is difficult to remember,
>> > perhaps
>> > > > the naming issue can be put to the end of the discussion, because
>> more
>> > > > people like @Martijn @Shuo are considering whether these
>> configurations
>> > > > should be put into the DDL or the 'OPTIONS' hint. Here's what I
>> > > > think, Putting these configs into DDL or putting them into 'OPTIONS'
>> > hint
>> > > > is actually the same thing, because the 'OPTIONS' hint is mainly
>> used
>> > to
>> > > > configure the properties of conenctor. The reason why I want to use
>> a
>> > new
>> > > > hint is to make sure the semantics clear, in my opinion the
>> > configuration
>> > > > of watermark should not be mixed up with connector. However, a new
>> hint
>> > > > does make it more difficult to use to some extent, for example,
>> when a
>> > > user
>> > > > uses both 'OPTIONS' hint and 'WATERMARK_PARAMS' hint. For this
>> point,
>> > > maby
>> > > > it is more appropriate to use uniform 'OPTIONS' hint.
>> > > > On the other hand, if we enrich more watermark option keys in
>> 'OPTIONS'
>> > > > hints, The question will be what we treat the definatrions of
>> > 'OPTIONS'
>> > > > hint, is this only specific to the connector options or could be
>> more?
>> > > > Maybe @Jark could share more insights here. In my opion, 'OPTIONS'
>> is
>> > > only
>> > > > related to the connector options, which is not like the gernal
>> > watermark
>> > > > options.
>> > > >
>> > > >
>> > > >
>> > > > Shuo Cheng <njucs...@gmail.com> 于2023年2月22日周三 19:17写道：
>> > > >
>> > > > > Hi Kui,
>> > > > >
>> > > > > Thanks for driving the discussion. It's quite useful to introduce
>> > > Watermark
>> > > > > options. I have some questions:
>> > > > >
>> > > > > What kind of hints is "WATERMARK_PARAMS"?
>> > > > > Currently, we have two kinds of hints in Flink: Dynamic Table
>> > Options &
>> > > > > Query Hints. As described in the Flip, "WATERMARK_PARAMS" is more
>> > like
>> > > > > Dynamic Table Options. So two questions arise here:
>> > > > >
>> > > > > 1) Are these watermark options to be exposed as connector WITH
>> > > options? Aa
>> > > > > described in SQL Hints doc[1], "Dynamic Table Options allow to
>> > > specify or
>> > > > > override table options dynamically", which implies that these
>> options
>> > > can
>> > > > > also be configured in WITH options.
>> > > > >
>> > > > > 2) Do we really need a new hint name like 'WATERMARK_PARAMS',
>> table
>> > > > > options use "OPTIONS" as hint name, like '/*+
>> > > > > OPTIONS('csv.ignore-parse-errors'='true') */', maybe we can enrich
>> > more
>> > > > > table option keys for watermark, e.g., /*+
>> > > > > OPTIONS('watermark.emit-strategy'='ON_PERIODIC') */.
>> > > > >
>> > > > >
>> > > > > [1]
>> > > > >
>> > > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/queries/hints/
>> > > > >
>> > > > > On Wed, Feb 22, 2023 at 10:22 AM kui yuan <catye...@gmail.com>
>> > wrote:
>> > > > >
>> > > > > > Hi devs,
>> > > > > >
>> > > > > >
>> > > > > > I'd like to start a discussion thread for FLIP-296[1]. This
>> comes
>> > > from an
>> > > > > > offline discussion with @Yun Tang, and we hope to enrich table
>> API
>> > &
>> > > SQL
>> > > > > to
>> > > > > > support many watermark-related features which were only
>> implemented
>> > > at
>> > > > > the
>> > > > > > datastream API level.
>> > > > > >
>> > > > > >
>> > > > > > Basically, we want to introduce watermark options in table API &
>> > SQL
>> > > via
>> > > > > > SQL hint named 'WATERMARK_PARAMS' to support features:
>> > > > > >
>> > > > > > 1、Configurable watermark emit strategy
>> > > > > >
>> > > > > > 2、Dealing with idle sources
>> > > > > >
>> > > > > > 3、Watermark alignment
>> > > > > >
>> > > > > >
>> > > > > > Last but not least, thanks to Qingsheng and Jing Zhang for the
>> > > initial
>> > > > > > reviews.
>> > > > > >
>> > > > > >
>> > > > > > Looking forward to your thoughts and any feedback is
>> appreciated!
>> > > > > >
>> > > > > >
>> > > > > > [1]
>> > > > > >
>> > > > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240884405
>> > > > > >
>> > > > > >
>> > > > > > Best
>> > > > > >
>> > > > > > Yuan Kui
>> > > > > >
>> > > > >
>> > >
>> >
>>
>
