t;>>>>>> ExecNode becomes more complex, see WatermarkPushDownSpec
> >>>>>>>>>>>
> >>>>>>>>>>> Best
> >>>>>>>>>>> Kui Yuan
> >>>>>>>>>>>
> >&
nks for driving it.
IMO, the 'OPTIONS' hint may be not only specific to
the
connector
options. Just as a reference, we also have
`sink.parallelism`[1]
as
a
connector options. It enables
user to specific the writer's parallelism dynamically
per-query.
Personally, I perfer to consid
t; > > >>>>> 'scan.watermark.alignment.group'='alignment-group-1',
> > > > > > >>>>> 'scan.watermark.alignment.max-drift'='1min',
> > > > > > >>>>> 'scan.watermark.alignmen
t; >>>>>> kui yuan 于2023年2月23日周四 20:05写道：
> > > > > >>>>>>
> > > > > >>>>>>> Hi all,
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks for all suggestions.
> > &
[1]
> > > > >>>>>
> > > > >>>>>
> > > > >>>
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.11/dev/table/connectors/kafka.html#sink-partitioner
> > > > >>&
+Factory
> > > >>>>> [4]
> > > >>>>>
> > > >>>>>
> > > >>>
> > >
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/a
> > > > >>>>>
> > > > >>>>> Regards,
> > > > >>>>> Timo
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> [1]
> > > > &g
; > >>>>> [3]
> > > >>>>>
> > > >>>>>
> > > >>>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory
> > > >>>>> [4]
> &
have updated the flip according to the discussion, and we will
> > >>> extend
> > >>>>> the
> > >>>>>> watermark-related features with both table options and 'OPTIONS'
> > >>> hint,
> > >>>>> like
> > >>
27;='ON_PERIODIC',
> >>>>>> ...
> >>>>>> );
> >>>>>>
> >>>>>> -- use 'OPTIONS' hint
> >>>>>> select ... from source_table /*+ OPTIONS('watermark.emit.s
concern, if we accept to make the watermark-related
options
part of dynamic table options, the problem becomes another topic:
how
to
support the dynamic table options in table API, which is deserved
to
create
another FLIP.
Best
Yun Tang
From: Martijn Visser
Se
we can't use hint in table API directly. if we need to extend the
>> >> hint
>> >> >> interface in the table API, maybe we need another flip. However, if
>> we
>> >> >> extend the watermark-related features in the dynamic table options,
>&
register a table named "Orders"
> >> >> tableEnv.executeSql("CREATE TABLE Orders (`user` BIGINT, product
> >> STRING,
> >> >> amount INT) WITH ('watermark.emit.strategy'='ON_EVENT'...)");
> >> >> ```
;> >> Kui Yuan
>> >>
>> >> Yun Tang 于2023年2月23日周四 17:46写道：
>> >>
>> >>> Thanks for the warm discussions!
>> >>>
>> >>> I had an offline discussion with Kui about the replies. I think I
>> could
>
e
> >>>> clear. Currently users can define simple watermark strategy in DDL,
> >>> adding
> >>>> more configuration items in connector options is easy to understand
> >>>>
> >>>> Best,
> >>>> Sham
artijn's concern, if we accept to make the watermark-related options
part of dynamic table options, the problem becomes another topic: how to
support the dynamic table options in table API, which is deserved to create
another FLIP.
Best
Yun Tang
From: Martijn
ermark-related options
>> part of dynamic table options, the problem becomes another topic: how to
>> support the dynamic table options in table API, which is deserved to create
>> another FLIP.
>>
>> Best
>> Yun Tang
>>
>
sonally, I perfer to consider watermark-related hints as option
> > > hints. So, user can define a default watermark strategy for the table,
> > and
> > > if user dosen't needed to changes it, they need to do nothing in their
> > > query instead of specific it ervery
't needed to changes it, they need to do nothing in their
> > query instead of specific it ervery time.
> > >
> > > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/zh/docs/connectors/table/filesystem/#sink-parallelism
> > >
> > &g
to changes it, they need to do nothing in their
> > query instead of specific it ervery time.
> > >
> > > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/zh/docs/connectors/table/filesystem/#sink-parallelism
> > >
> > > Best regards,
> >
ttps://nightlies.apache.org/flink/flink-docs-master/zh/docs/connectors/table/filesystem/#sink-parallelism
> >
> > Best regards,
> > Yuxia
> >
> > Best regards,
> > Yuxia
> >
> > - 原始邮件 -
> > 发件人: "kui yuan"
> > 收件人: "dev"
stead
> of specific it ervery time.
>
> [1]
> https://nightlies.apache.org/flink/flink-docs-master/zh/docs/connectors/table/filesystem/#sink-parallelism
>
> Best regards,
> Yuxia
>
> Best regards,
> Yuxia
>
> - 原始邮件 -
> 发件人: "kui yuan"
> 收件人: "
-docs-master/zh/docs/connectors/table/filesystem/#sink-parallelism
Best regards,
Yuxia
Best regards,
Yuxia
- 原始邮件 -
发件人: "kui yuan"
收件人: "dev"
抄送: "Jark Wu"
发送时间: 星期三, 2023年 2 月 22日 下午 10:08:11
主题: Re: [DISCUSS] FLIP-296: Watermark options for table API &
Hi all,
Thanks for the lively discussion and I will respond to these questions one
by one. However, there are also some common questions and I will answer
together.
@郑 Thanks for your reply. The features mentioned in this flip are only for
those source connectors that implement the SupportsWaterm
Hi Kui,
Thanks for driving the discussion. It's quite useful to introduce Watermark
options. I have some questions:
What kind of hints is "WATERMARK_PARAMS"?
Currently, we have two kinds of hints in Flink: Dynamic Table Options &
Query Hints. As described in the Flip, "WATERMARK_PARAMS" is more l
Hi Kui,
Thanks for bringing this to the discussion. Toward the FLIP, I have several
questions.
1. What's the behavior if there are multiple table sources, among which
some do not support `SupportsWatermarkPushDown`?
> So the features that this flip intends to support are only for those
> sources
Hi Yuan Kui,
Thanks for creating the FLIP. A couple of questions / remarks
1. While the FLIP talks about watermark options for Table API & SQL, I only
see proposed syntax for SQL, not for the Table API. What is your proposal
for the Table API?
2. A rejected alternative is adding watermark relate
Hi kui
Thanks for initializing this discussion. I have two questions
1. What will happen if we add watermark related options in `the connector
options`? Will the connector ignore these options or throw an exception?
How can we support this?
2. If one table is used by two operators with different
Hi kui,
There is a scenario using watermark that cannot be implemented through SQL now：
We only support specify rowtime column in table DDL，if the rowtime field is
generated by join dimension table , it cannot be implemented
Can we consider implement through HINTS like :
Select * from t1 join t2
Sent: Wednesday, February 22, 2023 10:22
To: dev@flink.apache.org
Subject: [DISCUSS] FLIP-296: Watermark options for table API & SQL
Hi devs,
I'd like to start a discussion thread for FLIP-296[1]. This comes from an
offline discussion with @Yun Tang, and we hope to enrich table API & S
Hi devs,
I'd like to start a discussion thread for FLIP-296[1]. This comes from an
offline discussion with @Yun Tang, and we hope to enrich table API & SQL to
support many watermark-related features which were only implemented at the
datastream API level.
Basically, we want to introduce waterma
31 matches
Mail list logo
