Re: [DISCUSS] FLIP-424: Asynchronous State APIs

[weijie guo]

Hi Zakelly,

Thanks for the proposal! I like this idea and I can see the performance
improvements it brings.

In the previous reply you mentioned “these APIs are in some newly
introduced classes, which are located in a different package name with the
original one”. I can see the benefits of this. To be honest, there is a lot
of historical burdens with the old state API, maybe this is a chance to
break free. If I understand you correctly, the new State(V2) interface will
still support synchronous API, right? But I didn't see that in the FLIP.



Best regards,

Weijie


Zakelly Lan  于2024年3月13日周三 13:03写道：

> Hi Jing,
>
> The deprecation and removal of original APIs is beyond the scope of current
> FLIP, but I do add/highlight such information under "Compatibility,
> Deprecation, and Migration Plan" section.
>
>
> Best,
> Zakelly
>
> On Wed, Mar 13, 2024 at 9:18 AM Yunfeng Zhou 
> wrote:
>
> > Hi Zakelly,
> >
> > Thanks for your responses. I agree with it that we can keep the design
> > as it is for now and see if others have any better ideas for these
> > questions.
> >
> > Best,
> > Yunfeng
> >
> > On Tue, Mar 12, 2024 at 5:23 PM Zakelly Lan 
> wrote:
> > >
> > > Hi Xuannan,
> > >
> > > Thanks for your comments, I modified the FLIP accordingly.
> > >
> > > Hi Yunfeng,
> > >
> > > Thanks for sharing your opinions!
> > >
> > >> Could you provide some hint on use cases where users need to mix sync
> > >> and async state operations in spite of the performance regression?
> > >> This information might help address our concerns on design. If the
> > >> mixed usage is simply something not recommended, I would prefer to
> > >> prohibit such usage from API.
> > >
> > > In fact, there is no scenario where users MUST use the sync APIs, but
> it
> > is much easier to use for those who are not familiar with asynchronous
> > programming. If they want to migrate their job from Flink 1.x to 2.0
> > leveraging some benefits from asynchronous APIs, they may try the mixed
> > usage. It is not user-friendly to directly throw exceptions at runtime, I
> > think our better approach is to warn users and recommend avoiding this. I
> > added an example in this FLIP.
> > >
> > > Well, I do not insist on allowing mixed usage of APIs if others reach
> an
> > agreement that we won't support that . I think the most important is to
> > keep the API easy to use and understand, thus I propose a unified state
> > declaration and explicit meaning in method name. WDYT?
> > >
> > >> Sorry I missed the new sink API. I do still think that it would be
> > >> better to make the package name more informative, and ".v2." does not
> > >> contain information for new Flink users who did not know the v1 of
> > >> state API. Unlike internal implementation and performance
> > >> optimization, API will hardly be compromised for now and updated in
> > >> future, so I still suggest we improve the package name now if
> > >> possible. But given the existing practice of sink v2 and
> > >> AbstractStreamOperatorV2, the current package name would be acceptable
> > >> to me if other reviewers of this FLIP agrees on it.
> > >
> > > Actually, I don't like 'v2' either. So if there is another good name,
> > I'd be happy to apply. This is a compromise to the current situation.
> Maybe
> > we could refine this after the retirement of original state APIs.
> > >
> > >
> > > Thanks & Best,
> > > Zakelly
> > >
> > >
> > > On Tue, Mar 12, 2024 at 1:42 PM Yunfeng Zhou <
> > flink.zhouyunf...@gmail.com> wrote:
> > >>
> > >> Hi Zakelly,
> > >>
> > >> Thanks for the quick response!
> > >>
> > >> > Actually splitting APIs into two sets ... warn them in runtime.
> > >>
> > >> Could you provide some hint on use cases where users need to mix sync
> > >> and async state operations in spite of the performance regression?
> > >> This information might help address our concerns on design. If the
> > >> mixed usage is simply something not recommended, I would prefer to
> > >> prohibit such usage from API.
> > >>
> > >> > In fact ... .sink2`.
> > >>
> > >> Sorry I missed the new sink API. I do still think that it would be
> > >> better to make the package name more informative, and ".v2." does not
> > >> contain information for new Flink users who did not know the v1 of
> > >> state API. Unlike internal implementation and performance
> > >> optimization, API will hardly be compromised for now and updated in
> > >> future, so I still suggest we improve the package name now if
> > >> possible. But given the existing practice of sink v2 and
> > >> AbstractStreamOperatorV2, the current package name would be acceptable
> > >> to me if other reviewers of this FLIP agrees on it.
> > >>
> > >> Best,
> > >> Yunfeng
> > >>
> > >> On Mon, Mar 11, 2024 at 5:27 PM Zakelly Lan 
> > wrote:
> > >> >
> > >> > Hi Yunfeng,
> > >> >
> > >> > Thanks for your comments!
> > >> >
> > >> > +1 for JingGe's suggestion to introduce an AsyncState API, instead
> of
> > >> > > having both get() and asyncGet() in the sam

[jira] [Created] (FLINK-34660) AutoRescalingITCase#testCheckpointRescalingInKeyedState AssertionError

[Hangxiang Yu (Jira)]

Hangxiang Yu created FLINK-34660:


 Summary: AutoRescalingITCase#testCheckpointRescalingInKeyedState 
AssertionError
 Key: FLINK-34660
 URL: https://issues.apache.org/jira/browse/FLINK-34660
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Reporter: Hangxiang Yu


[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=58249&view=ms.vss-test-web.build-test-results-tab&runId=4036370&resultId=100718&paneView=debug]

 
{code:java}
expected:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000), (1,3), 
(1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000), (0,82000), 
(1,8), (1,0), (1,16000), (1,24000), (1,4), (1,56000), (1,64000), 
(0,12000), (0,28000), (0,52000), (0,6), (0,68000), (0,76000), (1,18000), 
(1,26000), (1,34000), (1,42000), (1,58000), (0,6000), (0,14000), (0,22000), 
(0,38000), (0,46000), (0,62000), (0,7), (1,4000), (1,2), (1,36000), 
(1,44000)]> but was:<[(0,8000), (0,32000), (0,48000), (0,72000), (1,78000), 
(1,3), (1,54000), (0,2000), (0,1), (0,5), (0,66000), (0,74000), 
(0,82000), (0,23000), (0,31000), (1,8), (1,0), (1,16000), (1,24000), 
(1,4), (1,56000), (1,64000), (0,12000), (0,28000), (0,52000), (0,6), 
(0,68000), (0,76000), (1,18000), (1,26000), (1,34000), (1,42000), (1,58000), 
(0,6000), (0,14000), (0,22000), (0,19000), (0,35000), (1,4000), (1,2), 
(1,36000), (1,44000)]> {code}
 

This maybe related to FLINK-34624 as we could see from the log:
{code:java}
03:31:02,073 [ main] INFO 
org.apache.flink.runtime.testutils.PseudoRandomValueSelector [] - Randomly 
selected true for state.changelog.enabled
03:31:02,163 [jobmanager-io-thread-2] INFO 
org.apache.flink.state.changelog.AbstractChangelogStateBackend [] - 
ChangelogStateBackend is used, delegating EmbeddedRocksDBStateBackend. {code}
FLINK-34624 disables changelog since it doesn't support local rescaling 
currently.

Even if disabling changelog for AutoRescalingITCase manually, 
randomization may still be applied to it.
We should apply randomization only when it's not pre-defined.
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-437: Support ML Models in Flink SQL

[Jark Wu]

Hi Minge, Chris, Hao,

Thanks for proposing this interesting idea. I think this is a nice step
towards
the AI world for Apache Flink. I don't know much about AI/ML, so I may have
some stupid questions.

1. Could you tell more about why polymorphism table function (PTF) doesn't
work and do we have plan to use PTF as model functions?

2. What kind of object does the model map to in SQL? A relation or a data
type?
It looks like a data type because we use it as a parameter of the table
function.
If it is a data type, how does it cooperate with type inference[1]?

3. What built-in model functions will we support? How to define a
user-defined model function?

4. What built-in model types will we support? How to define a user-defined
model type?

5. Regarding the remote model, what providers will we support? Can users
implement
3rd-party providers except OpenAI?

Best,
Jark

[1]:
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#type-inference




On Wed, 13 Mar 2024 at 05:55, Hao Li  wrote:

> Hi, Dev
>
>
> Mingge, Chris and I would like to start a discussion about FLIP-437:
> Support ML Models in Flink SQL.
>
> This FLIP is proposing to support machine learning models in Flink SQL
> syntax so that users can CRUD models with Flink SQL and use models on Flink
> to do prediction with Flink data. The FLIP also proposes new model entities
> and changes to catalog interface to support model CRUD operations in
> catalog.
>
> For more details, see FLIP-437 [1]. Looking forward to your feedback.
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-437%3A+Support+ML+Models+in+Flink+SQL
>
> Thanks,
> Minge, Chris & Hao
>

Re: Flink's treatment to "hadoop" and "yarn" configuration overrides seems unintuitive

[Venkatakrishnan Sowrirajan]

Thanks for your response. Sorry for the late reply, Ferenc.

Yes, totally agree with you on deprecating this behavior as part of 1.20.
Let me follow it up with a FLIP to deprecate the current behavior and with
a proposed solution. We can discuss further in the [DISCUSS] thread of the
FLIP.

Regards
Venkata krishnan


On Mon, Feb 26, 2024 at 11:25 PM Ferenc Csaky 
wrote:

> Thanks for the more shared details Venkata, I did not used spark widely
> myself, but if there are more examples
> like follows that approach it makes sense to comply and be
> consistent.
>
> Regarding the planned release schedule on the 2.0 wiki page [1] the
> expected releases are 1.19 -> 1.20 -> 2.0. I am not sure about how
> realistic is that or there are any chance there will be a 1.21, but even if
> not, deprecating the current behavior for even 1.20 would not hurt IMO.
>
> WDYT?
>
> Looking for other opinios as well of course.
>
> Regards,
> Ferenc
>
> [1]
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/2.0*Release__;Kw!!IKRxdwAv5BmarQ!ZafwWMtIzLzlT2CxT8XSct-ZjnETPvL0KFMGi13v1KWCEE7GXBq4NXZK-kjhRG1AZfiU0aALIPsSkEDcwzDHxrhej3L0i2YI$
>
>
> On Tuesday, February 27th, 2024 at 07:08, Venkatakrishnan Sowrirajan <
> vsowr...@asu.edu> wrote:
>
> >
> >
> > Thanks for sharing your thoughts, Ferenc.
> >
> > Just my 2 cents, coming from Spark background that uses "spark.hadoop."
> > prefix to handle all Hadoop configs, I prefer the "flink.hadoop." prefix
> > and "flink.yarn." prefix. Generally, users use both these systems and I
> > prefer to be consistent that way. Having said that, Flink doesn't need to
> > be consistent with Spark so I am fine with the other approach as well.
> >
> > I believe in order to make backwards incompatible changes, Flink needs
> the
> > change to be in deprecated status for at least 2 minor versions which
> means
> > we will already have 2.0, therefore this can probably go in 3.0 only.
> >
> > It is still good to deprecate the current behavior and fix with the right
> > behavior and get rid of this in 3.0 totally.
> >
> > Looking for more thoughts from others in the community to make sure that
> I
> > don't miss anything. Once the discussion settles, I can start a FLIP with
> > the new proposal.
> >
> > Thanks
> > Venkat
> >
> >
> > On Mon, Feb 26, 2024, 1:09 AM Ferenc Csaky ferenc.cs...@pm.me.invalid
> >
> > wrote:
> >
> > > Hi Venkata krishnan,
> > >
> > > Thanks for starting a discussion on this topic. I completely
> > > agree with you on that, this behavior can create confusion and
> > > cause debugging sessions that could be spared with aligning how Flink
> > > parses external properties.
> > >
> > > Personally, I find the Yarn props prefixing more intuitive, but
> > > I do not have strong opinions other than prefixing configs for
> > > external systems should follow the same semantics and behavior.
> > >
> > > It would make sense to align these in Flink 2.0 IMO, but I would
> > > be curious about other opinions.
> > >
> > > On Saturday, February 24th, 2024 at 07:36, Venkatakrishnan Sowrirajan <
> > > vsowr...@asu.edu> wrote:
> > >
> > > > Gentle ping on the ^^ question to surface this back up again. Any
> > > > thoughts?
> > > >
> > > > Regards
> > > > Venkata krishnan
> > > >
> > > > On Fri, Feb 16, 2024 at 7:32 PM Venkatakrishnan Sowrirajan
> > > > vsowr...@asu.edu
> > > >
> > > > wrote:
> > > >
> > > > > Hi Flink devs,
> > > > >
> > > > > Flink supports overriding "hadoop" and "yarn" configuration. As
> part of
> > > > > the override mechanism, users have to prefix `hadoop` configs with
> "
> > > > > flink.hadoop." and the prefix will be removed, while with `yarn`
> > > > > configs
> > > > > users have to prefix it with "flink.yarn." but "flink." only is
> > > > > removed,
> > > > > not "flink.yarn.".
> > > > >
> > > > > Following is an example:
> > > > >
> > > > > 1. "Hadoop" config
> > > > >
> > > > > Hadoop config key = hadoop.tmp.dir => Flink config =
> > > > > flink.hadoop.hadoop.tmp.dir => Hadoop's configuration object would
> have
> > > > > hadoop.tmp.dir*.*
> > > > >
> > > > > 2. "YARN" config
> > > > >
> > > > > YARN config key = yarn.application.classpath => Flink config =
> > > > > flink.yarn.yarn.application.classpath => YARN's configuration
> object
> > > > > would have yarn.yarn.application.classpath*.*
> > > > >
> > > > > Although this is documented
> > >
> > >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/*flink-yarn-__;Iw!!IKRxdwAv5BmarQ!ewtlUgGysGDiWgKPr9D1bsGDp-jLagZqppUvXAtqvbO5lNMg7QTr4y5L4OL-hTFPO1qTR1nvh4ALBEQtm0RnE6X1WTEkp0Sb$
> > > 
> > >
> > > > > properly, it feels unintuitive and it tripped me, took quite a
> while to
> > > > > understand why the above YARN configuration override was not
> working as
> > > > > expected. Is this something that should be fixed? The problem with
> > > > > fixing
> > > > > it is, it will become backwards incompatible. Therefore, can this
> be
> > > > 

Re: Question around Flink's AdaptiveBatchScheduler

[Venkatakrishnan Sowrirajan]

Thanks for the response Lijie and Junrui. Sorry for the late reply. Few
follow up questions.

> Source can actually ignore this limit
because it has no upstream, but this will lead to semantic inconsistency.

Lijie, can you please elaborate on the above comment further? What do you
mean when you say it will lead to "semantic inconsistency"?

> Secondly, we first need to limit the max parallelism of (downstream)
vertex, and then we can decide how many subpartitions (upstream vertex)
should produce. The limit should be effective, otherwise some downstream
tasks will have no data to process.

This makes sense in the context of any other vertices other than the source
vertex. As you mentioned above ("Source can actually ignore this limit
because it has no upstream"), therefore I feel "
jobmanager.adaptive-batch-scheduler.default-source-parallelism" need not be
upper bounded by "jobmanager.adaptive-batch-scheduler.max-parallelism".

Regards
Venkata krishnan


On Thu, Feb 29, 2024 at 2:11 AM Junrui Lee  wrote:

> Hi Venkat,
>
> As Lijie mentioned,  in Flink, the parallelism is required to be less than
> or equal to the maximum parallelism. The config option
> jobmanager.adaptive-batch-scheduler.max-parallelism and
> jobmanager.adaptive-batch-scheduler.default-source-parallelism will be set
> as the source's parallelism and max-parallelism, respectively. Therefore,
> the check failed situation you encountered is in line with the
> expectations.
>
> Best,
> Junrui
>
> Lijie Wang  于2024年2月29日周四 17:35写道：
>
> > Hi Venkat,
> >
> > >> default-source-parallelism config should be independent from the
> > max-parallelism
> >
> > Actually, it's not.
> >
> > Firstly, it's obvious that the parallelism should be less than or equal
> to
> > the max parallelism(both literally and execution). The
> > "jobmanager.adaptive-batch-scheduler.max-parallelism" will be used as the
> > max parallelism for a vertex if you don't set max parallelism for it
> > individually (Just like the source in your case).
> >
> > Secondly, we first need to limit the max parallelism of (downstream)
> > vertex, and then we can decide how many subpartitions (upstream vertex)
> > should produce. The limit should be effective, otherwise some downstream
> > tasks will have no data to process. Source can actually ignore this limit
> > because it has no upstream, but this will lead to semantic inconsistency.
> >
> > Best,
> > Lijie
> >
> > Venkatakrishnan Sowrirajan  于2024年2月29日周四 05:49写道：
> >
> > > Hi Flink devs,
> > >
> > > With Flink's AdaptiveBatchScheduler
> > > <
> > >
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/elastic_scaling/*adaptive-batch-scheduler__;Iw!!IKRxdwAv5BmarQ!fwD4qP-fTEiqJH9CC3AHgXbR5MJPGm7ll1dYwElK-zrtujWDWio6_yvBa4rHlaZHP_lefLs4bZQAISrg5BrHLw$
> > > >
> > > (Note:
> > > this is different from AdaptiveScheduler
> > > <
> > >
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/elastic_scaling/*adaptive-scheduler__;Iw!!IKRxdwAv5BmarQ!fwD4qP-fTEiqJH9CC3AHgXbR5MJPGm7ll1dYwElK-zrtujWDWio6_yvBa4rHlaZHP_lefLs4bZQAISqUzURivw$
> > > >),
> > > the scheduler automatically determines the correct number of downstream
> > > tasks required to process the shuffle generated by the upstream vertex.
> > >
> > > I have a question regarding the current behavior. There are 2 configs
> > which
> > > are in interplay here.
> > > 1. jobmanager.adaptive-batch-scheduler.default-source-parallelism
> > > <
> > >
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/*jobmanager-adaptive-batch-scheduler-default-source-parallelism__;Iw!!IKRxdwAv5BmarQ!fwD4qP-fTEiqJH9CC3AHgXbR5MJPGm7ll1dYwElK-zrtujWDWio6_yvBa4rHlaZHP_lefLs4bZQAISoOTMiiCA$
> > > >
> > >  - The default parallelism of data source.
> > > 2. jobmanager.adaptive-batch-scheduler.max-parallelism
> > > <
> > >
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/*jobmanager-adaptive-batch-scheduler-max-parallelism__;Iw!!IKRxdwAv5BmarQ!fwD4qP-fTEiqJH9CC3AHgXbR5MJPGm7ll1dYwElK-zrtujWDWio6_yvBa4rHlaZHP_lefLs4bZQAISpOw_L_Eg$
> > > >
> > > -
> > > Upper bound of allowed parallelism to set adaptively.
> > >
> > > Currently, if "
> > > jobmanager.adaptive-batch-scheduler.default-source-parallelism
> > > <
> > >
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/*jobmanager-adaptive-batch-scheduler-default-source-parallelism__;Iw!!IKRxdwAv5BmarQ!fwD4qP-fTEiqJH9CC3AHgXbR5MJPGm7ll1dYwElK-zrtujWDWio6_yvBa4rHlaZHP_lefLs4bZQAISoOTMiiCA$
> > > >"
> > > is greater than "jobmanager.adaptive-batch-scheduler.max-parallelism
> > > <
> > >
> >
> https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/config/*jobmanager-adaptive-batch-scheduler-max-parallelism__;Iw!!IKRxdwAv5BmarQ!f

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

[Zakelly Lan]

Hi Jing,

The deprecation and removal of original APIs is beyond the scope of current
FLIP, but I do add/highlight such information under "Compatibility,
Deprecation, and Migration Plan" section.


Best,
Zakelly

On Wed, Mar 13, 2024 at 9:18 AM Yunfeng Zhou 
wrote:

> Hi Zakelly,
>
> Thanks for your responses. I agree with it that we can keep the design
> as it is for now and see if others have any better ideas for these
> questions.
>
> Best,
> Yunfeng
>
> On Tue, Mar 12, 2024 at 5:23 PM Zakelly Lan  wrote:
> >
> > Hi Xuannan,
> >
> > Thanks for your comments, I modified the FLIP accordingly.
> >
> > Hi Yunfeng,
> >
> > Thanks for sharing your opinions!
> >
> >> Could you provide some hint on use cases where users need to mix sync
> >> and async state operations in spite of the performance regression?
> >> This information might help address our concerns on design. If the
> >> mixed usage is simply something not recommended, I would prefer to
> >> prohibit such usage from API.
> >
> > In fact, there is no scenario where users MUST use the sync APIs, but it
> is much easier to use for those who are not familiar with asynchronous
> programming. If they want to migrate their job from Flink 1.x to 2.0
> leveraging some benefits from asynchronous APIs, they may try the mixed
> usage. It is not user-friendly to directly throw exceptions at runtime, I
> think our better approach is to warn users and recommend avoiding this. I
> added an example in this FLIP.
> >
> > Well, I do not insist on allowing mixed usage of APIs if others reach an
> agreement that we won't support that . I think the most important is to
> keep the API easy to use and understand, thus I propose a unified state
> declaration and explicit meaning in method name. WDYT?
> >
> >> Sorry I missed the new sink API. I do still think that it would be
> >> better to make the package name more informative, and ".v2." does not
> >> contain information for new Flink users who did not know the v1 of
> >> state API. Unlike internal implementation and performance
> >> optimization, API will hardly be compromised for now and updated in
> >> future, so I still suggest we improve the package name now if
> >> possible. But given the existing practice of sink v2 and
> >> AbstractStreamOperatorV2, the current package name would be acceptable
> >> to me if other reviewers of this FLIP agrees on it.
> >
> > Actually, I don't like 'v2' either. So if there is another good name,
> I'd be happy to apply. This is a compromise to the current situation. Maybe
> we could refine this after the retirement of original state APIs.
> >
> >
> > Thanks & Best,
> > Zakelly
> >
> >
> > On Tue, Mar 12, 2024 at 1:42 PM Yunfeng Zhou <
> flink.zhouyunf...@gmail.com> wrote:
> >>
> >> Hi Zakelly,
> >>
> >> Thanks for the quick response!
> >>
> >> > Actually splitting APIs into two sets ... warn them in runtime.
> >>
> >> Could you provide some hint on use cases where users need to mix sync
> >> and async state operations in spite of the performance regression?
> >> This information might help address our concerns on design. If the
> >> mixed usage is simply something not recommended, I would prefer to
> >> prohibit such usage from API.
> >>
> >> > In fact ... .sink2`.
> >>
> >> Sorry I missed the new sink API. I do still think that it would be
> >> better to make the package name more informative, and ".v2." does not
> >> contain information for new Flink users who did not know the v1 of
> >> state API. Unlike internal implementation and performance
> >> optimization, API will hardly be compromised for now and updated in
> >> future, so I still suggest we improve the package name now if
> >> possible. But given the existing practice of sink v2 and
> >> AbstractStreamOperatorV2, the current package name would be acceptable
> >> to me if other reviewers of this FLIP agrees on it.
> >>
> >> Best,
> >> Yunfeng
> >>
> >> On Mon, Mar 11, 2024 at 5:27 PM Zakelly Lan 
> wrote:
> >> >
> >> > Hi Yunfeng,
> >> >
> >> > Thanks for your comments!
> >> >
> >> > +1 for JingGe's suggestion to introduce an AsyncState API, instead of
> >> > > having both get() and asyncGet() in the same State class. As a
> >> > > supplement to its benefits, this design could help avoid having
> users
> >> > > to use sync and async API in a mixed way (unless they create both a
> >> > > State and an AsyncState from the same state descriptor), which is
> >> > > supposed to bring suboptimal performance according to the FLIP's
> >> > > description.
> >> >
> >> >
> >> > Actually splitting APIs into two sets of classes also brings some
> >> > difficulties. In this case, users must explicitly define their usage
> before
> >> > actually doing state access. It is a little strange that the user can
> >> > define a sync and an async version of State with the same name, while
> they
> >> > cannot allocate two async States with the same name.
> >> > Another reason for distinguishing API by their method name instead of
> class
> >> > n

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

[Yanfei Lei]

Hi Jing,
Thanks for the reply and follow up.

> What is the benefit for users to build a chain of mails instead of just one 
> mail(it is still async)?

Just to make sure we're on the same page, I try to paraphrase your question:
A `then()` call will be encapsulated as a callback mail. Your question
is whether we can call then() as little as possible to reduce the
overhead of encapsulating it into a mail.

In general, whether to call `then()` depends on the user's data
dependencies. The operations in a chain of `then()` are strictly
ordered.



The following is an example without data dependencies, if written in
the form of a `then` chain:
stateA.update(1).then(stateB.update(2).then(stateC.update(3)));

The execution order is:
```
stateA update 1 -> stateB update 2-> stateC update 3
```

If written in the form without `then()` call, they will be placed in a
"mail/mailboxDefaultAction", and each state request will still be
executed asynchronously:
```
stateA.update(1);
stateB.update(2);
stateC.update(3);
```

The order in which they are executed is undefined and may be:
```
- stateA update 1 -> stateB update 2-> stateC update 3
- stateB update 2 -> stateC update 3-> stateA update 1
- stateC update 3 -> stateA update 1-> stateB update 2
...
```
And the final results are "stateA = 1, stateB = 2, stateC = 3". In
this case, the two ways of writing are equivalent.



If there are data dependencies, for example:
```
stateA.update(1).then(stateA.update(2))
```

Then the execution order is:
```
stateA update 1 -> stateA update 2
```

If written in the form without `then()` call:
```
stateA.update(1);
stateA.update(2);
```

The order in which they are executed is undefined and may be:
```
- stateA update 1 -> stateA update 2
- stateA update 2-> stateA update 1
```
The final result may be "stateA = 1" *OR* "stateA = 2". In this case,
the way without `then()` chain to limit the execution order, and the
results may be wrong.

In summary, how many mails are encapsulated depends on how the user
writes the code, and how the user writes the code depends on their
data dependencies. [1][2] may be helpful for asynchronous programming
practice.


> I was wondering if exceptions in the mail chain would have an impact on the 
> reference counting?

We will catch exceptions that can be handled, they don't have impacts
on the reference counting.
For exceptions that cannot be handled, we will directly fail the job.

> Maybe a UT to cover all kinds of cases, i.e. happy paths and unhappy paths, 
> would make sense.

Nice suggestions, we will add a UT to cover those cases.


[1] 
https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletableFuture.html
[2] https://www.codingjunkie.net/completable-futures-part1/

Jing Ge  于2024年3月13日周三 07:05写道：
>
> Hi Yanfei,
>
> Thanks for your clarification! Now I got a much clear picture and I am
> still trying to understand your thoughts for some of those questions:
>
>
> > How many mails are encapsulated depends on how the user writes the
> > code. The statements in a `then()` will be wrapped into a mail.
> > StateFuture is a restricted version of CompletableFuture, their basic
> > semantics are consistent.
> >
>
> Conceptually, users can write a chain of many async calls, i.e. many then()
> calls. And all these calls for Record A must be executed in order, while
> Record B should stay at the Blocking buffer. What is the benefit for users
> to build a chain of mails instead of just one mail(it is still async)? Is
> there any best practices or guidelines to teach/tell users when and how
> many async calls in a chain could/should be built?
>
> > The challenge arises in determining when all the processing logic
> associated with Record A is fully executed. To address this, we have
> adopted a reference counting mechanism that tracks ongoing operations
> (either processing input or executing a callback) related to a single
> record.
>
> > We describe this in the "Error handling"[2] section. This FLIP also
> > adopts the design from FLIP-368, ensuring that all state interfaces
> > throw unchecked exceptions and, consequently, do not declare any
> > exceptions in their signatures. In cases where an exception occurs
> > while accessing the state, the job should fail.
> >
>
> My question was not about how exceptions will be defined. I am not sure how
> unchecked exceptions handling will be implemented. I was wondering if
> exceptions in the mail chain would have an impact on the reference
> counting? E.g. in Fig 5, if an exception happened in the value(),
> update(int), or function within then(), any -1 counting might be missed?
> Maybe a UT to cover all kinds of cases, i.e. happy paths and unhappy paths,
> would make sense.
>
> Best regards,
> Jing
>
> On Mon, Mar 11, 2024 at 3:58 AM Yanfei Lei  wrote:
>
> > Hi Jing,
> >
> > Thanks for your thoughtful feedback!
> >
> > > does it mean that there will be three mails for Read, Update, and Output
> > ?
> >
> > With this example, there are two mai

Re: [VOTE] Release 1.19.0, release candidate #2

[Qingsheng Ren]

 +1 (binding)

- Verified signature and checksum
- Verified no binary in source
- Built from source
- Tested reading and writing Kafka with SQL client and Kafka connector 3.1.0
- Verified source code tag
- Reviewed release note
- Reviewed web PR

Thanks to all release managers and contributors for the awesome work!

Best,
Qingsheng

On Wed, Mar 13, 2024 at 1:23 AM Matthias Pohl
 wrote:

> I want to share an update on FLINK-34227 [1]: It's still not clear what's
> causing the test instability. So far, we agreed in today's release sync [2]
> that it's not considered a blocker because it is observed in 1.18 nightly
> builds and it only appears in the GitHub Actions workflow. But I still have
> a bit of a concern that this is something that was introduced in 1.19 and
> backported to 1.18 after the 1.18.1 release (because the test instability
> started to appear more regularly in March; with one occurrence in January).
> Additionally, I have no reason to believe, yet, that the instability is
> caused by some GHA-related infrastructure issue.
>
> So, if someone else has some capacity to help looking into it; that would
> be appreciated. I will continue my investigation tomorrow.
>
> Best,
> Matthias
>
> [1] https://issues.apache.org/jira/browse/FLINK-34227
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/1.19+Release#id-1.19Release-03/12/2024
>
> On Tue, Mar 12, 2024 at 12:50 PM Benchao Li  wrote:
>
> > +1 (non-binding)
> >
> > - checked signature and checksum: OK
> > - checkout copyright year in notice file: OK
> > - diffed source distribution with tag, make sure there is no
> > unexpected files: OK
> > - build from source : OK
> > - start a local cluster, played with jdbc connector: OK
> >
> > weijie guo  于2024年3月12日周二 16:55写道：
> > >
> > > +1 (non-binding)
> > >
> > > - Verified signature and checksum
> > > - Verified source distribution does not contains binaries
> > > - Build from source code and submit a word-count job successfully
> > >
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Jane Chan  于2024年3月12日周二 16:38写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > - Verify that the source distributions do not contain any binaries;
> > > > - Build the source distribution to ensure all source files have
> Apache
> > > > headers;
> > > > - Verify checksum and GPG signatures;
> > > >
> > > > Best,
> > > > Jane
> > > >
> > > > On Tue, Mar 12, 2024 at 4:08 PM Xuannan Su 
> > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - Verified signature and checksum
> > > > > - Verified that source distribution does not contain binaries
> > > > > - Built from source code successfully
> > > > > - Reviewed the release announcement PR
> > > > >
> > > > > Best regards,
> > > > > Xuannan
> > > > >
> > > > > On Tue, Mar 12, 2024 at 2:18 PM Hang Ruan 
> > > > wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > - Verified signatures and checksums
> > > > > > - Verified that source does not contain binaries
> > > > > > - Build source code successfully
> > > > > > - Reviewed the release note and left a comment
> > > > > >
> > > > > > Best,
> > > > > > Hang
> > > > > >
> > > > > > Feng Jin  于2024年3月12日周二 11:23写道：
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > - Verified signatures and checksums
> > > > > > > - Verified that source does not contain binaries
> > > > > > > - Build source code successfully
> > > > > > > - Run a simple sql query successfully
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng Jin
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 12, 2024 at 11:09 AM Ron liu 
> > wrote:
> > > > > > >
> > > > > > > > +1 (non binding)
> > > > > > > >
> > > > > > > > quickly verified:
> > > > > > > > - verified that source distribution does not contain binaries
> > > > > > > > - verified checksums
> > > > > > > > - built source code successfully
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Ron
> > > > > > > >
> > > > > > > > Jeyhun Karimov  于2024年3月12日周二 01:00写道：
> > > > > > > >
> > > > > > > > > +1 (non binding)
> > > > > > > > >
> > > > > > > > > - verified that source distribution does not contain
> binaries
> > > > > > > > > - verified signatures and checksums
> > > > > > > > > - built source code successfully
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Jeyhun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb <
> > > > decordea...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1 (non binding)
> > > > > > > > > >
> > > > > > > > > > - verified signatures and checksums
> > > > > > > > > > - ASF headers are present in all expected file
> > > > > > > > > > - No unexpected binaries files found in the source
> > > > > > > > > > - Build successful locally
> > > > > > > > > > - tested basic word count example
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Bests,
> > > > > > > 

[jira] [Created] (FLINK-34659) How to implement global sort in latest flink datastream API

[Junyao Huang (Jira)]

Junyao Huang created FLINK-34659:


 Summary: How to implement global sort in latest flink datastream 
API
 Key: FLINK-34659
 URL: https://issues.apache.org/jira/browse/FLINK-34659
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Affects Versions: 1.18.1
Reporter: Junyao Huang
 Attachments: image-2024-03-13-11-21-57-846.png

[https://nightlies.apache.org/flink/flink-docs-master/zh/docs/dev/datastream/dataset_migration/#%E7%AC%AC%E4%B8%89%E7%B1%BB]
 
{{DataStream dataStream = // [...]// assign subtask ID to all 
recordsDataStream> dataStream1 = dataStream.map(new 
AddSubtaskIDMapFunction());dataStream1.keyBy(value -> value.f0)   
.window(EndOfStreamWindows.get())   .apply(new WindowFunction<>(){  
 // implement user-defined map partition or sort partition logic   
});}}

{{}}

{{will this cause OOM in streaming execution mode?}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Jinzhong Li]

Hi Jing,

Thanks for your feedback!

> I am fine with @Internal and once we
> use @Internal, we should add it to all classes within a module to keep the
> consistency.

I think the scope to keep consistency within a module is all interfaces,
not all classes.
For example, in changelog-statebackend module or flink-runtime/state/heap
package (hashmap-statebackend), all interfaces are annotated, and classes'
annotations are optional.
The changes in this FLIP(Flip-420) also covers all interfaces of the
rocksdb statebackend module.

Best,
Jinzhong


On Wed, Mar 13, 2024 at 7:30 AM Jing Ge  wrote:

> Hi Jinzhong,
>
> Thanks for your clarification!
>
>
> > As mentioned in Flink's “API compatibility guarantees”[1], "any API
> without
> > such an annotation is considered internal to Flink, with no guarantees
> > being provided."
> > I think these interfaces are treated as internal interfaces regardless of
> > whether they are marked @Internal.
> >
> > So, it's acceptable for me to either leave it unmarked or mark it as
> > @Internal.
> >
>
> It is a question of consistency. I am fine with @Internal and once we
> use @Internal, we should add it to all classes within a module to keep the
> consistency. Otherwise, when developers read some interfaces
> have @Internal, some other classes have no annotations. It turns out there
> are different meanings of with or without @Internal. But it is not a big
> deal. If no other one has the same concern. I will not block you :-)
>
>
>
> > As for FLIP-197, I found that the proposal in this flip has not actually
> > been implemented. Because FLINK-25352 [2] has not been resolved，it looks
> > like we can't add more info into the PublicEvolving annotation.
> >
> >
> Fair enough! Thanks for the hint!
>
> Best regards,
> Jing
>
>
>
> On Mon, Mar 11, 2024 at 9:03 AM Jinzhong Li 
> wrote:
>
> > Hi Jing.
> >
> > Thanks for your suggestion.
> >
> > >>  I was wondering if SingleStateIterator and RocksDBRestoreOperation
> are
> > exposed to users even if they are interfaces.
> >
> > I agree that very very few users implement these two interfaces, except
> for
> > some advanced users who implement their own StateBackend based on
> > rocksdbStateBackend. (The "StateBackend" and
> "EmbeddedRocksDBStateBackend"
> > are PublicEvolving interface)
> >
> > >> Only adding @Internal annotation to these two interfaces could be
> > considered as an implicit hint that users can replace the default
> behaviour
> > with their own implementation.
>
>
>
> > As mentioned in Flink's “API compatibility guarantees”[1], "any API
> without
> > such an annotation is considered internal to Flink, with no guarantees
> > being provided."
> > I think these interfaces are treated as internal interfaces regardless of
> > whether they are marked @Internal.
> >
> > So, it's acceptable for me to either leave it unmarked or mark it as
> > @Internal.
> >
>
>
>
> > >> Another suggestion is that we should start following FLIP-197[1](which
> > is already accepted by the community) to add more info into the
> > PublicEvolving annotation in order to kick off the graduation process.
> >
> >
>
> > As for FLIP-197, I found that the proposal in this flip has not actually
> > been implemented. Because FLINK-25352 [2] has not been resolved，it looks
> > like we can't add more info into the PublicEvolving annotation.
> >
> >
>
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
> > [2] https://issues.apache.org/jira/browse/FLINK-25352
> >
> > Best,
> > Jinzhong
> >
> >
> > On Mon, Mar 11, 2024 at 12:29 AM Jing Ge 
> > wrote:
> >
> > > Hi Jinzhong,
> > >
> > > Sorry for the late reply. The key guideline of adding visibility
> > annotation
> > > is whether the interface/class is a customer-facing API. I was
> wondering
> > if
> > > SingleStateIterator and RocksDBRestoreOperation are exposed to users
> even
> > > if they are interfaces, i.e. users can use their own implementation
> > > classes. The flink-statebackend-rocksdb module has many more classes
> than
> > > the FLIP described. Only adding @Internal annotation to these two
> > > interfaces could be considered as an implicit hint that users can
> replace
> > > the default behaviour with their own implementation.
> > >
> > > Another suggestion is that we should start following FLIP-197[1](which
> is
> > > already accepted by the community) to add more info into the
> > PublicEvolving
> > > annotation in order to kick off the graduation process. WDYT?
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> > >
> > > On Mon, Feb 26, 2024 at 2:22 PM Jinzhong Li 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks for all the feedback. It seems there are no more questions
> > > > unaddressed.  I would like to open the voting thread after three
> days.
> > > >
> > > > Please let me know if you have any conc

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

[Yunfeng Zhou]

Hi Zakelly,

Thanks for your responses. I agree with it that we can keep the design
as it is for now and see if others have any better ideas for these
questions.

Best,
Yunfeng

On Tue, Mar 12, 2024 at 5:23 PM Zakelly Lan  wrote:
>
> Hi Xuannan,
>
> Thanks for your comments, I modified the FLIP accordingly.
>
> Hi Yunfeng,
>
> Thanks for sharing your opinions!
>
>> Could you provide some hint on use cases where users need to mix sync
>> and async state operations in spite of the performance regression?
>> This information might help address our concerns on design. If the
>> mixed usage is simply something not recommended, I would prefer to
>> prohibit such usage from API.
>
> In fact, there is no scenario where users MUST use the sync APIs, but it is 
> much easier to use for those who are not familiar with asynchronous 
> programming. If they want to migrate their job from Flink 1.x to 2.0 
> leveraging some benefits from asynchronous APIs, they may try the mixed 
> usage. It is not user-friendly to directly throw exceptions at runtime, I 
> think our better approach is to warn users and recommend avoiding this. I 
> added an example in this FLIP.
>
> Well, I do not insist on allowing mixed usage of APIs if others reach an 
> agreement that we won't support that . I think the most important is to keep 
> the API easy to use and understand, thus I propose a unified state 
> declaration and explicit meaning in method name. WDYT?
>
>> Sorry I missed the new sink API. I do still think that it would be
>> better to make the package name more informative, and ".v2." does not
>> contain information for new Flink users who did not know the v1 of
>> state API. Unlike internal implementation and performance
>> optimization, API will hardly be compromised for now and updated in
>> future, so I still suggest we improve the package name now if
>> possible. But given the existing practice of sink v2 and
>> AbstractStreamOperatorV2, the current package name would be acceptable
>> to me if other reviewers of this FLIP agrees on it.
>
> Actually, I don't like 'v2' either. So if there is another good name, I'd be 
> happy to apply. This is a compromise to the current situation. Maybe we could 
> refine this after the retirement of original state APIs.
>
>
> Thanks & Best,
> Zakelly
>
>
> On Tue, Mar 12, 2024 at 1:42 PM Yunfeng Zhou  
> wrote:
>>
>> Hi Zakelly,
>>
>> Thanks for the quick response!
>>
>> > Actually splitting APIs into two sets ... warn them in runtime.
>>
>> Could you provide some hint on use cases where users need to mix sync
>> and async state operations in spite of the performance regression?
>> This information might help address our concerns on design. If the
>> mixed usage is simply something not recommended, I would prefer to
>> prohibit such usage from API.
>>
>> > In fact ... .sink2`.
>>
>> Sorry I missed the new sink API. I do still think that it would be
>> better to make the package name more informative, and ".v2." does not
>> contain information for new Flink users who did not know the v1 of
>> state API. Unlike internal implementation and performance
>> optimization, API will hardly be compromised for now and updated in
>> future, so I still suggest we improve the package name now if
>> possible. But given the existing practice of sink v2 and
>> AbstractStreamOperatorV2, the current package name would be acceptable
>> to me if other reviewers of this FLIP agrees on it.
>>
>> Best,
>> Yunfeng
>>
>> On Mon, Mar 11, 2024 at 5:27 PM Zakelly Lan  wrote:
>> >
>> > Hi Yunfeng,
>> >
>> > Thanks for your comments!
>> >
>> > +1 for JingGe's suggestion to introduce an AsyncState API, instead of
>> > > having both get() and asyncGet() in the same State class. As a
>> > > supplement to its benefits, this design could help avoid having users
>> > > to use sync and async API in a mixed way (unless they create both a
>> > > State and an AsyncState from the same state descriptor), which is
>> > > supposed to bring suboptimal performance according to the FLIP's
>> > > description.
>> >
>> >
>> > Actually splitting APIs into two sets of classes also brings some
>> > difficulties. In this case, users must explicitly define their usage before
>> > actually doing state access. It is a little strange that the user can
>> > define a sync and an async version of State with the same name, while they
>> > cannot allocate two async States with the same name.
>> > Another reason for distinguishing API by their method name instead of class
>> > name is that users typically use the State instances to access state but
>> > forget their type/class. For example:
>> > ```
>> > SyncState a = getState(xxx);
>> > AsyncState b = getAsyncState(xxx);
>> > //...
>> > a.update(1);
>> > b.update(1);
>> > ```
>> > Users are likely to think there is no difference between the `a.update(1)`
>> > and `b.update(1)`, since they may forget the type for `a` and `b`. Thus I
>> > proposed to distinguish the behavior in method names.
>> > As for th

Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Jing Ge]

+1 (binding) Thanks!

Best regards,
Jing

On Sun, Mar 10, 2024 at 5:32 PM Jing Ge  wrote:

> Hi Jinzhong,
>
> Thanks for driving this topic and sorry for just joining the discussion
> now. I replied in your discussion thread. Would you like to take a look
> and let's keep the discussion there? I will come back to this thread and
> vote once the discussion is done. Thanks!
>
> Best regards,
> Jing
>
> On Thu, Mar 7, 2024 at 4:39 AM Zakelly Lan  wrote:
>
>> +1 non-binding
>>
>> Thanks for proposing this.
>>
>> Best,
>> Zakelly
>>
>> On Thu, Mar 7, 2024 at 10:13 AM Yanfei Lei  wrote:
>>
>> > +1(binding) for this vote.
>> >
>> > Hangxiang Yu  于2024年3月7日周四 09:54写道：
>> > >
>> > > +1 (binding)
>> > >
>> > > On Thu, Mar 7, 2024 at 9:34 AM Yun Tang  wrote:
>> > >
>> > > > > +1 for this FLIP.
>> > > > Sorry for not being clear in my previous reply, it's a binding vote.
>> > > >
>> > > > Best
>> > > > Yun Tang
>> > > > 
>> > > > From: Jeyhun Karimov 
>> > > > Sent: Thursday, March 7, 2024 4:40
>> > > > To: dev@flink.apache.org 
>> > > > Subject: Re: [VOTE] FLIP-420: Add API annotations for RocksDB
>> > StateBackend
>> > > > user-facing classes
>> > > >
>> > > > Hi Jinzhong,
>> > > >
>> > > > Thanks for the FLIP.
>> > > >
>> > > > +1 (non-binding)
>> > > >
>> > > > Regards,
>> > > > Jeyhun
>> > > >
>> > > > On Wed, Mar 6, 2024 at 5:09 PM Yun Tang  wrote:
>> > > >
>> > > > > +1 for this FLIP.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Best
>> > > > > Yun Tang
>> > > > > 
>> > > > > From: Jinzhong Li 
>> > > > > Sent: Wednesday, March 6, 2024 20:29
>> > > > > To: dev@flink.apache.org 
>> > > > > Subject: [VOTE] FLIP-420: Add API annotations for RocksDB
>> > StateBackend
>> > > > > user-facing classes
>> > > > >
>> > > > > Hi All,
>> > > > >
>> > > > > I'd like to start a vote on the FLIP-420: Add API annotations for
>> > RocksDB
>> > > > > StateBackend user-facing classes[1].
>> > > > > The discussion thread is here [2].
>> > > > >
>> > > > > The vote will be open for at least 72 hours unless there is an
>> > objection
>> > > > or
>> > > > > not enough votes.
>> > > > >
>> > > > >
>> > > > > [1]https://cwiki.apache.org/confluence/x/JQs4EQ
>> > > > > [2]
>> https://lists.apache.org/thread/4t71lz2j2ft8hf90ylvtomynhr2qthoo
>> > > > >
>> > > > >
>> > > > > Best,
>> > > > > Jinzhong Li
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best,
>> > > Hangxiang.
>> >
>> >
>> >
>> > --
>> > Best,
>> > Yanfei
>> >
>>
>

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Jing Ge]

Hi Jinzhong,

Thanks for your clarification!


> As mentioned in Flink's “API compatibility guarantees”[1], "any API without
> such an annotation is considered internal to Flink, with no guarantees
> being provided."
> I think these interfaces are treated as internal interfaces regardless of
> whether they are marked @Internal.
>
> So, it's acceptable for me to either leave it unmarked or mark it as
> @Internal.
>

It is a question of consistency. I am fine with @Internal and once we
use @Internal, we should add it to all classes within a module to keep the
consistency. Otherwise, when developers read some interfaces
have @Internal, some other classes have no annotations. It turns out there
are different meanings of with or without @Internal. But it is not a big
deal. If no other one has the same concern. I will not block you :-)



> As for FLIP-197, I found that the proposal in this flip has not actually
> been implemented. Because FLINK-25352 [2] has not been resolved，it looks
> like we can't add more info into the PublicEvolving annotation.
>
>
Fair enough! Thanks for the hint!

Best regards,
Jing



On Mon, Mar 11, 2024 at 9:03 AM Jinzhong Li 
wrote:

> Hi Jing.
>
> Thanks for your suggestion.
>
> >>  I was wondering if SingleStateIterator and RocksDBRestoreOperation are
> exposed to users even if they are interfaces.
>
> I agree that very very few users implement these two interfaces, except for
> some advanced users who implement their own StateBackend based on
> rocksdbStateBackend. (The "StateBackend" and "EmbeddedRocksDBStateBackend"
> are PublicEvolving interface)
>
> >> Only adding @Internal annotation to these two interfaces could be
> considered as an implicit hint that users can replace the default behaviour
> with their own implementation.



> As mentioned in Flink's “API compatibility guarantees”[1], "any API without
> such an annotation is considered internal to Flink, with no guarantees
> being provided."
> I think these interfaces are treated as internal interfaces regardless of
> whether they are marked @Internal.
>
> So, it's acceptable for me to either leave it unmarked or mark it as
> @Internal.
>



> >> Another suggestion is that we should start following FLIP-197[1](which
> is already accepted by the community) to add more info into the
> PublicEvolving annotation in order to kick off the graduation process.
>
>

> As for FLIP-197, I found that the proposal in this flip has not actually
> been implemented. Because FLINK-25352 [2] has not been resolved，it looks
> like we can't add more info into the PublicEvolving annotation.
>
>

> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
> [2] https://issues.apache.org/jira/browse/FLINK-25352
>
> Best,
> Jinzhong
>
>
> On Mon, Mar 11, 2024 at 12:29 AM Jing Ge 
> wrote:
>
> > Hi Jinzhong,
> >
> > Sorry for the late reply. The key guideline of adding visibility
> annotation
> > is whether the interface/class is a customer-facing API. I was wondering
> if
> > SingleStateIterator and RocksDBRestoreOperation are exposed to users even
> > if they are interfaces, i.e. users can use their own implementation
> > classes. The flink-statebackend-rocksdb module has many more classes than
> > the FLIP described. Only adding @Internal annotation to these two
> > interfaces could be considered as an implicit hint that users can replace
> > the default behaviour with their own implementation.
> >
> > Another suggestion is that we should start following FLIP-197[1](which is
> > already accepted by the community) to add more info into the
> PublicEvolving
> > annotation in order to kick off the graduation process. WDYT?
> >
> > Best regards,
> > Jing
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> >
> > On Mon, Feb 26, 2024 at 2:22 PM Jinzhong Li 
> > wrote:
> >
> > > Hi all,
> > >
> > > Thanks for all the feedback. It seems there are no more questions
> > > unaddressed.  I would like to open the voting thread after three days.
> > >
> > > Please let me know if you have any concerns, thanks!
> > >
> > > Best,
> > > Jinzhong Li
> > >
> > > On Mon, Feb 26, 2024 at 11:29 AM Yanfei Lei 
> wrote:
> > >
> > > > @Yun Tang
> > > > Thanks for the information, +1 for marking
> > > > `ConfigurableRocksDBOptionsFactory` as `PublicEvolving `.
> > > >
> > > > Best,
> > > > Yanfei
> > > >
> > > > Yun Tang  于2024年2月23日周五 19:54写道：
> > > > >
> > > > > Hi Jinzhong,
> > > > >
> > > > > Thanks for driving this topic, and +1 for fixing the lack of
> > > annotation.
> > > > >
> > > > > @Yanfei the `ConfigurableRocksDBOptionsFactory` interface is
> > introduced
> > > > for user extension, you can refer to the doc[1], which shows an
> example
> > > of
> > > > how to use this interface.
> > > > >
> > > > >
> > > > > [1]
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/large_state_tuning/#tuning-rocksdb-memory
> 

[jira] [Created] (FLINK-34658) Scala API unusable on Flink 1.18.1/Java 17 Docker image

[Matthew Ernst (Jira)]

Matthew Ernst created FLINK-34658:
-

 Summary: Scala API unusable on Flink 1.18.1/Java 17 Docker image
 Key: FLINK-34658
 URL: https://issues.apache.org/jira/browse/FLINK-34658
 Project: Flink
  Issue Type: Bug
  Components: API / Scala
Affects Versions: 1.18.1, 1.18.0
 Environment: This bug has been reproduced under macOS (Intel x64) and 
Linux (AMD 64) on a Flink cluster running in session mode.
Reporter: Matthew Ernst


The Scala API should still work in Flink 1.18. The official Docker image for 
Flink 1.18.1 on Java 17 ("flink:1.18.1-scala_2.12-java17") causes jobs using 
the Scala API to immediately throw a ReflectiveOperationException. Jobs using 
the Scala API still work correctly on the Java 11 image 
("flink:1.18.1-scala_2.12-java11").

The problem happens because the flink-scala JAR file included in the image 
("flink-scala_2.12-1.18.1.jar") has been built with an old Scala compiler that 
has a [compatibility bug with Java 
17|https://github.com/scala/bug/issues/12419]. Rebuilding the flink-scala JAR 
file with the Scala compiler set to 2.12.15 or later fixes the bug. At my day 
job I cannot use Java 11 for a particular Flink job due to dependency on a Java 
library that uses [Java records|https://openjdk.org/jeps/395] (introduced in 
Java 16).

I have created a github repository with an example application and a longer 
description of the bug and how to fix it with a newer Scala compiler version: 
https://github.com/mattbernst/scala-java17-flink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

[Jing Ge]

Hi Zakelly,

Thanks for your clarification! I would suggest explicitly adding
description(better highlight) into the FLIP that the original State API
will be deprecated. My gut feeling is that it is very important for anyone,
who will review the new design, to understand the long-term intention.

Best regards,
Jing

On Mon, Mar 11, 2024 at 8:01 AM Zakelly Lan  wrote:

> Hi Jing,
>
> Thanks for your comments!
>
> Sorry for not making this clear. Actually these APIs are in some newly
> introduced classes, which are located in a different package name with the
> original ones. I suggest we name it "State API V2" and the package name
> will be 'org.apache.flink.api.common.state.v2'. They work closely with the
> Datastream V2 and are annotated with @Experimental in first few versions,
> and will be promoted to @PublicEvolving alongside the DataStream V2. I
> agree that the name of interfaces should add 'async', and we will also
> support synchronous APIs in these new API classes. This approach allows us
> to:
>
>1. Have enough flexibility for rapid development with @Experimental
>annotation.
>2. Provide both sync and async style APIs for greater ease of use.
>3. Release ourselves from legacy constraints that prevent altering
>interface signatures, and we can do something like reorganizing the
>exceptions as FLIP-368[1] proposed.
>
> State API V2 will become the exclusive API set available to users when
> working with DataStream API V2. We may discuss the deprecation of original
> ones in future.
>
> WDYT?
>
>
> Best,
> Zakelly
>
>
> On Sun, Mar 10, 2024 at 8:34 PM Jing Ge 
> wrote:
>
>> Hi Zakelly,
>>
>> Thanks for your proposal. The FLIP looks in good shape. +1 for it! I'd
>> like
>> to ask some questions to understand your thoughts more precisely.
>>
>> 1. StateFuture is a new interface. At first glance, it should
>> be @Experimental. But according to our API Arch rule[1], it should be at
>> least @PublicEvolving, if it will be used by any existing PublicEvloving
>> classes. You might want to add this info to your FLIP, if we want to go
>> with this option.
>>
>> 2. The return types of methods in State and related sub-interfaces are
>> StateFuture. Since the old State interfaces already have those
>> methods
>> and Java does not allow method overload with the same method but different
>> return types. Do you want to change the old methods or use new interfaces?
>> My understanding is that, according to the description in the
>> "Compatibility, Deprecation, and Migration Plan'' section in the FLIP, new
>> interfaces will be defined alongside the old interfaces. I guess the
>> long-term intention of this FLIP is not to deprecate the synchronous State
>> API. Both State APIs will be supported for different scenarios. In this
>> case, does it make sense to:
>>
>> 2.1 annotated all new interfaces with @Experimental to have the
>> flexibility for further modifications?
>> 2.2 use different names e.g. AsyncState etc. to avoid potential
>> human mistakes while coding(e.g. import wrong package by mistake etc.) and
>> ease the job development with state?
>>
>> Best regards,
>> Jing
>>
>>
>> [1]
>>
>> https://github.com/apache/flink/blob/d6a4eb966fbc47277e07b79e7c64939a62eb1d54/flink-architecture-tests/flink-architecture-tests-production/src/main/java/org/apache/flink/architecture/rules/ApiAnnotationRules.java#L99
>>
>> On Thu, Mar 7, 2024 at 9:49 AM Zakelly Lan  wrote:
>>
>> > Hi devs,
>> >
>> > I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
>> > State Storage and Management[1], which is a joint work of Yuan Mei,
>> Zakelly
>> > Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
>> >
>> >  - FLIP-424: Asynchronous State APIs [2]
>> >
>> > This FLIP introduces new APIs for asynchronous state access.
>> >
>> > Please make sure you have read the FLIP-423[1] to know the whole story,
>> and
>> > we'll discuss the details of FLIP-424[2] under this mail. For the
>> > discussion of overall architecture or topics related with multiple
>> > sub-FLIPs, please post in the previous mail[3].
>> >
>> > Looking forward to hearing from you!
>> >
>> > [1] https://cwiki.apache.org/confluence/x/R4p3EQ
>> > [2] https://cwiki.apache.org/confluence/x/SYp3EQ
>> > [3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
>> >
>> >
>> > Best,
>> > Zakelly
>> >
>>
>

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

[Jing Ge]

Hi Yanfei,

Thanks for your clarification! Now I got a much clear picture and I am
still trying to understand your thoughts for some of those questions:


> How many mails are encapsulated depends on how the user writes the
> code. The statements in a `then()` will be wrapped into a mail.
> StateFuture is a restricted version of CompletableFuture, their basic
> semantics are consistent.
>

Conceptually, users can write a chain of many async calls, i.e. many then()
calls. And all these calls for Record A must be executed in order, while
Record B should stay at the Blocking buffer. What is the benefit for users
to build a chain of mails instead of just one mail(it is still async)? Is
there any best practices or guidelines to teach/tell users when and how
many async calls in a chain could/should be built?

> The challenge arises in determining when all the processing logic
associated with Record A is fully executed. To address this, we have
adopted a reference counting mechanism that tracks ongoing operations
(either processing input or executing a callback) related to a single
record.

> We describe this in the "Error handling"[2] section. This FLIP also
> adopts the design from FLIP-368, ensuring that all state interfaces
> throw unchecked exceptions and, consequently, do not declare any
> exceptions in their signatures. In cases where an exception occurs
> while accessing the state, the job should fail.
>

My question was not about how exceptions will be defined. I am not sure how
unchecked exceptions handling will be implemented. I was wondering if
exceptions in the mail chain would have an impact on the reference
counting? E.g. in Fig 5, if an exception happened in the value(),
update(int), or function within then(), any -1 counting might be missed?
Maybe a UT to cover all kinds of cases, i.e. happy paths and unhappy paths,
would make sense.

Best regards,
Jing

On Mon, Mar 11, 2024 at 3:58 AM Yanfei Lei  wrote:

> Hi Jing,
>
> Thanks for your thoughtful feedback!
>
> > does it mean that there will be three mails for Read, Update, and Output
> ?
>
> With this example, there are two mails. The Read is processed by
> `mailboxDefaultAction`[1], and the Update and Output are encapsulated
> as mail.
>
> > does it make sense to encapsulate one mail instead of 3 mails with more
> overhead?
>
>

> How many mails are encapsulated depends on how the user writes the
> code. The statements in a `then()` will be wrapped into a mail.
> StateFuture is a restricted version of CompletableFuture, their basic
> semantics are consistent.
>
>

> > Would you like to add more description for cases when exceptions
> happened? E.g. when reading or/and updating State throws IOExceptions.
>
>

> We describe this in the "Error handling"[2] section. This FLIP also
> adopts the design from FLIP-368, ensuring that all state interfaces
> throw unchecked exceptions and, consequently, do not declare any
> exceptions in their signatures. In cases where an exception occurs
> while accessing the state, the job should fail.
>



> > Is it correct to understand that AEC is stateless?
>
> Great perspective, yes, it can be understood that way.
> AEC is a task-level component. When the job fails or is restarted, the
> runtime status in AEC will be reset.
> In fact, we have considered taking a snapshot of the status in AEC and
> storing it in a checkpoint like "unaligned checkpoint", but since
> callback cannot be serialized, this idea is not feasible for the time
> being.
>
> > would you like to add Pseudo-code for the inFilghtReocordNum decrement
> to help us understand the logic better?
>
> This part of the code is a bit scattered, we will try to abstract a
> pseudo-code. You can first refer to the RecordContext-related code [3]
> in the PoC to understand it.
>
> [1]
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/MailboxProcessor.java#L81
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-425%3A+Asynchronous+Execution+Model#FLIP425:AsynchronousExecutionModel-ErrorHandling
> [3]
> https://github.com/ververica/flink-poc/blob/disagg-poc-2/flink-runtime/src/main/java/org/apache/flink/runtime/state/async/RecordContext.java#L77
>
>
> Best,
> Yanfei
>
> Jing Ge  于2024年3月10日周日 23:47写道：
> >
> > Hi Yanfei,
> >
> > Thanks for your proposal! The FLIP contains a lot of great new ideas. I'd
> > like to ask some questions to make sure we are on the same page.
> >
> > > For the asynchronous interface, Record A should run with Read, Update
> and
> > Output, while Record B should stay at the Blocking buffer.
> >
> > 1. With this example, does it mean that there will be three mails for
> Read,
> > Update, and Output ?
> > 2. If yes, since the Read, Update, and Output have to be executed before
> > Record B, does it make sense to encapsulate one mail instead of 3 mails
> > with more overhead? There must be some thoughts behind the design. Look
> > forward to it.
> >
> >

Additional metadata available for Kafka serdes

[Balint Bene]

Hello! Looking to get some guidance for a problem around the Flink formats
used for Kafka.

Flink currently uses common serdes interfaces across all formats. However,
some data formats used in Kafka require headers for serdes.  It's the same
problem for serialization and deserialization, so I'll just use
DynamicKafkaDeserialationSchema

as
an example. It has access to the Kafka record headers, but it can't pass
them to the DeserializationSchema

implemented
by the format since the interface is generic.

If it were possible to pass the headers, then open source formats such as
Apicurio could be supported. Unlike the Confluent formats which store the
metadata (schema ID) appended to the serialized bytes in the key and value,
the Apicurio formats store their metadata in the record headers.

I have bandwidth to work on this, but it would be great to have direction
from the community. I have a simple working prototype that's able to load a
custom version of the format with a modified interface that can accept the
headers (I just put the entire Apache Kafka ConsumerRecord

/ProducerRecord

for simplicity). The issues I foresee is that the class-loader

exists in the Flink repo along with interfaces for the formats, but these
changes are specific to Kafka. This solution could require migrating
formats to the Flink-connector-kafka repo which is a decent amount of work.

Feedback is appreciated!
Thanks
Balint

[DISCUSS] FLIP-437: Support ML Models in Flink SQL

[Hao Li]

Hi, Dev


Mingge, Chris and I would like to start a discussion about FLIP-437:
Support ML Models in Flink SQL.

This FLIP is proposing to support machine learning models in Flink SQL
syntax so that users can CRUD models with Flink SQL and use models on Flink
to do prediction with Flink data. The FLIP also proposes new model entities
and changes to catalog interface to support model CRUD operations in
catalog.

For more details, see FLIP-437 [1]. Looking forward to your feedback.


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-437%3A+Support+ML+Models+in+Flink+SQL

Thanks,
Minge, Chris & Hao

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

[Kevin Lam]

Hi Anupam,

Thanks again for your work on contributing this feature back.

Sounds good re: the refactoring/re-organizing.

Regarding the schema-id, in my opinion this should NOT be a configuration
option on the format. We should be able to deterministically map the Flink
type to the ProtoSchema and register that with the Schema Registry.

I think it can make sense to provide the `subject` as a parameter, so that
the serialization format can look up existing schemas.

This would be used in something I mentioned something related above

> Another topic I had is Protobuf's field ids. Ideally in Flink it would be
> nice if we are idiomatic about not renumbering them in incompatible ways,
> similar to what's discussed on the Schema Registry issue here:
> https://github.com/confluentinc/schema-registry/issues/2551


When we construct the ProtobufSchema from the Flink LogicalType, we
shouldn't renumber the field ids in an incompatible way, so one approach
would be to use the subject to look up the most recent version, and use
that to evolve the field ids correctly.


On Tue, Mar 12, 2024 at 2:33 AM Anupam Aggarwal 
wrote:

> Hi Kevin,
>
> Thanks for starting the discussion on this.
> I will be working on contributing this feature back (This was developed by
> Dawid Wysakowicz and others at Confluent).
>
> I have opened a (very initial) draft PR here
> https://github.com/apache/flink/pull/24482 with our current
> implementation.
> Thanks for the feedback on the PR, I haven’t gotten around to
> re-organizing/refactoring the classes yet, but it would be inline with some
> of your comments.
>
> On the overall approach there are some slight variations from the initial
> proposal.
> Our implementation relies on an explicit schema-id being passed through the
> config. This could help in cases where one Flink type could potentially map
> to multiple proto types.
> We could make the schema-Id optional and fall back to deriving it from the
> rowType (during serialization) if not present?
>
> The message index handling is still TBD. I am thinking of replicating logic
> in AbstractKafkaProtobufSerializer.java
> <
> https://github.com/confluentinc/schema-registry/blob/342c8a9d3854d4253d785214f5dcfb1b6cc59a06/protobuf-serializer/src/main/java/io/confluent/kafka/serializers/protobuf/AbstractKafkaProtobufSerializer.java#L157
> >
>  (|Deserializer).
> Please let me know if this makes sense / or in case you have any other
> feedback.
>
> Thanks
> Anupam
>
> On Thu, Feb 29, 2024 at 8:54 PM Kevin Lam 
> wrote:
>
> > Hey Robert,
> >
> > Awesome thanks, that timeline works for me. Sounds good re: deciding on
> > FLIP once we have the PR, and thanks for looking into the field ids.
> >
> > Looking forward to it!
> >
> > On Thu, Feb 29, 2024 at 5:09 AM Robert Metzger 
> > wrote:
> >
> > > Hey Kevin,
> > >
> > > Thanks a lot. Then let's contribute the Confluent implementation to
> > > apache/flink. We can't start working on this immediately because of a
> > team
> > > event next week, but within the next two weeks, we will start working
> on
> > > this.
> > > It probably makes sense for us to open a pull request of what we have
> > > already, so that you can start reviewing and maybe also contributing to
> > the
> > > PR.
> > > I hope this timeline works for you!
> > >
> > > Let's also decide if we need a FLIP once the code is public.
> > > We will look into the field ids.
> > >
> > >
> > > On Tue, Feb 27, 2024 at 8:56 PM Kevin Lam
>  > >
> > > wrote:
> > >
> > > > Hey Robert,
> > > >
> > > > Thanks for your response. I have a partial implementation, just for
> the
> > > > decoding portion.
> > > >
> > > > The code I have is pretty rough and doesn't do any of the refactors I
> > > > mentioned, but the decoder logic does pull the schema from the schema
> > > > registry and use that to deserialize the DynamicMessage before
> > converting
> > > > it to RowData using a DynamicMessageToRowDataConverter class. For the
> > > other
> > > > aspects, I would need to start from scratch for the encoder.
> > > >
> > > > Would be very happy to see you drive the contribution back to open
> > source
> > > > from Confluent, or collaborate on this.
> > > >
> > > > Another topic I had is Protobuf's field ids. Ideally in Flink it
> would
> > be
> > > > nice if we are idiomatic about not renumbering them in incompatible
> > ways,
> > > > similar to what's discussed on the Schema Registry issue here:
> > > > https://github.com/confluentinc/schema-registry/issues/2551
> > > >
> > > >
> > > > On Tue, Feb 27, 2024 at 5:51 AM Robert Metzger 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > +1 to support the format in Flink.
> > > > >
> > > > > @Kevin: Do you already have an implementation for this inhouse that
> > you
> > > > are
> > > > > looking to upstream, or would you start from scratch?
> > > > > I'm asking because my current employer, Confluent, has a Protobuf
> > > Schema
> > > > > registry implementation for Flink, and I could help drive
> > c

Re: [DISCUSS] Manual savepoint triggering in flink-kubernetes-operator

[Gyula Fóra]

That would be great Mate! If you could draw up a FLIP for this that would
be nice as this is a rather large change that will have a significant
impact for existing users.

If possible it would be good to provide some backward compatibility /
transition period while we preserve the current content of the status so
it's easy to migrate to the new savepoint CRs.

Cheers,
Gyula

On Tue, Mar 12, 2024 at 9:22 PM Mate Czagany  wrote:

> Hi,
>
> I really like this idea as well, I think it would be a great improvement
> compared to how manual savepoints currently work, and suits Kubernetes
> workflows a lot better.
>
> If there are no objections, I can investigate it during the next few weeks
> and see how this could be implemented in the current code.
>
> Cheers,
> Mate
>
> Gyula Fóra  ezt írta (időpont: 2024. márc. 12., K,
> 16:01):
>
> > That's definitely a good improvement Robert and we should add it at some
> > point. At the point in time when this was implemented we went with the
> > current simpler / more lightweight approach.
> > However if anyone is interested in working on this / contributing this
> > improvement I would personally support it.
> >
> > Gyula
> >
> > On Tue, Mar 12, 2024 at 3:53 PM Robert Metzger 
> > wrote:
> >
> > > Have you guys considered making savepoints a first class citizen in the
> > > Kubernetes operator?
> > > E.g. to trigger a savepoint, you create a "FlinkSavepoint" CR, the K8s
> > > operator picks up that resource and tries to create a savepoint
> > > indefinitely until the savepoint has been successfully created. We
> report
> > > the savepoint status and location in the "status" field.
> > >
> > > We could even add an (optional) finalizer to delete the physical
> > savepoint
> > > from the savepoint storage once the "FlinkSavepoint" CR has been
> deleted.
> > > optional: the savepoint spec could contain a field "retain
> > > physical savepoint" or something, that controls the delete behavior.
> > >
> > >
> > > On Thu, Mar 3, 2022 at 4:02 AM Yang Wang 
> wrote:
> > >
> > > > I agree that we could start with the annotation approach and collect
> > the
> > > > feedback at the same time.
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Őrhidi Mátyás  于2022年3月2日周三 20:06写道：
> > > >
> > > > > Thank you for your feedback!
> > > > >
> > > > > The annotation on the
> > > > >
> > > > > @ControllerConfiguration(generationAwareEventProcessing = false)
> > > > > FlinkDeploymentController
> > > > >
> > > > > already enables the event triggering based on metadata changes. It
> > was
> > > > set
> > > > > earlier to support some failure scenarios. (It can be used for
> > example
> > > to
> > > > > manually reenable the reconcile loop when it got stuck in an error
> > > phase)
> > > > >
> > > > > I will go ahead and propose a PR using annotations then.
> > > > >
> > > > > Cheers,
> > > > > Matyas
> > > > >
> > > > > On Wed, Mar 2, 2022 at 12:47 PM Yang Wang 
> > > wrote:
> > > > >
> > > > > > I also like the annotation approach since it is more natural.
> > > > > > But I am not sure about whether the meta data change will trigger
> > an
> > > > > event
> > > > > > in java-operator-sdk.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Yang
> > > > > >
> > > > > > Gyula Fóra  于2022年3月2日周三 16:29写道：
> > > > > >
> > > > > > > Thanks Matyas,
> > > > > > >
> > > > > > > From a user perspective I think the annotation is pretty nice
> and
> > > > user
> > > > > > > friendly so I personally prefer that approach.
> > > > > > >
> > > > > > > You said:
> > > > > > >  "It seems, the java-operator-sdk handles the changes of the
> > > > .metadata
> > > > > > and
> > > > > > > .spec fields of custom resources differently."
> > > > > > >
> > > > > > > What implications does this have on the above mentioned 2
> > > approaches?
> > > > > > Does
> > > > > > > it make one more difficult than the other?
> > > > > > >
> > > > > > > Cheers
> > > > > > > Gyula
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 1, 2022 at 1:52 PM Őrhidi Mátyás <
> > > > matyas.orh...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All!
> > > > > > > >
> > > > > > > > I'd like to start a quick discussion about the way we allow
> > users
> > > > to
> > > > > > > > trigger savepoints manually in the operator [FLINK-26181]
> > > > > > > > . There
> are
> > > > > > existing
> > > > > > > > solutions already for this functionality in other operators,
> > for
> > > > > > example:
> > > > > > > > - counter based
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/spotify/flink-on-k8s-operator/blob/master/docs/savepoints_guide.md#2-taking-savepoints-by-updating-the-flinkcluster-custom-resource
> > > > > > > > >
> > > > > > > > - annotation based
> > > > > > > > <
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/spotify/flink-on-k8s-operator/blob/master/docs

Re: [DISCUSS] Manual savepoint triggering in flink-kubernetes-operator

[Mate Czagany]

Hi,

I really like this idea as well, I think it would be a great improvement
compared to how manual savepoints currently work, and suits Kubernetes
workflows a lot better.

If there are no objections, I can investigate it during the next few weeks
and see how this could be implemented in the current code.

Cheers,
Mate

Gyula Fóra  ezt írta (időpont: 2024. márc. 12., K,
16:01):

> That's definitely a good improvement Robert and we should add it at some
> point. At the point in time when this was implemented we went with the
> current simpler / more lightweight approach.
> However if anyone is interested in working on this / contributing this
> improvement I would personally support it.
>
> Gyula
>
> On Tue, Mar 12, 2024 at 3:53 PM Robert Metzger 
> wrote:
>
> > Have you guys considered making savepoints a first class citizen in the
> > Kubernetes operator?
> > E.g. to trigger a savepoint, you create a "FlinkSavepoint" CR, the K8s
> > operator picks up that resource and tries to create a savepoint
> > indefinitely until the savepoint has been successfully created. We report
> > the savepoint status and location in the "status" field.
> >
> > We could even add an (optional) finalizer to delete the physical
> savepoint
> > from the savepoint storage once the "FlinkSavepoint" CR has been deleted.
> > optional: the savepoint spec could contain a field "retain
> > physical savepoint" or something, that controls the delete behavior.
> >
> >
> > On Thu, Mar 3, 2022 at 4:02 AM Yang Wang  wrote:
> >
> > > I agree that we could start with the annotation approach and collect
> the
> > > feedback at the same time.
> > >
> > > Best,
> > > Yang
> > >
> > > Őrhidi Mátyás  于2022年3月2日周三 20:06写道：
> > >
> > > > Thank you for your feedback!
> > > >
> > > > The annotation on the
> > > >
> > > > @ControllerConfiguration(generationAwareEventProcessing = false)
> > > > FlinkDeploymentController
> > > >
> > > > already enables the event triggering based on metadata changes. It
> was
> > > set
> > > > earlier to support some failure scenarios. (It can be used for
> example
> > to
> > > > manually reenable the reconcile loop when it got stuck in an error
> > phase)
> > > >
> > > > I will go ahead and propose a PR using annotations then.
> > > >
> > > > Cheers,
> > > > Matyas
> > > >
> > > > On Wed, Mar 2, 2022 at 12:47 PM Yang Wang 
> > wrote:
> > > >
> > > > > I also like the annotation approach since it is more natural.
> > > > > But I am not sure about whether the meta data change will trigger
> an
> > > > event
> > > > > in java-operator-sdk.
> > > > >
> > > > >
> > > > > Best,
> > > > > Yang
> > > > >
> > > > > Gyula Fóra  于2022年3月2日周三 16:29写道：
> > > > >
> > > > > > Thanks Matyas,
> > > > > >
> > > > > > From a user perspective I think the annotation is pretty nice and
> > > user
> > > > > > friendly so I personally prefer that approach.
> > > > > >
> > > > > > You said:
> > > > > >  "It seems, the java-operator-sdk handles the changes of the
> > > .metadata
> > > > > and
> > > > > > .spec fields of custom resources differently."
> > > > > >
> > > > > > What implications does this have on the above mentioned 2
> > approaches?
> > > > > Does
> > > > > > it make one more difficult than the other?
> > > > > >
> > > > > > Cheers
> > > > > > Gyula
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 1, 2022 at 1:52 PM Őrhidi Mátyás <
> > > matyas.orh...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi All!
> > > > > > >
> > > > > > > I'd like to start a quick discussion about the way we allow
> users
> > > to
> > > > > > > trigger savepoints manually in the operator [FLINK-26181]
> > > > > > > . There are
> > > > > existing
> > > > > > > solutions already for this functionality in other operators,
> for
> > > > > example:
> > > > > > > - counter based
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/spotify/flink-on-k8s-operator/blob/master/docs/savepoints_guide.md#2-taking-savepoints-by-updating-the-flinkcluster-custom-resource
> > > > > > > >
> > > > > > > - annotation based
> > > > > > > <
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/spotify/flink-on-k8s-operator/blob/master/docs/savepoints_guide.md#3-taking-savepoints-by-attaching-annotation-to-the-flinkcluster-custom-resource
> > > > > > > >
> > > > > > >
> > > > > > > We could implement any of these or both or come up with our own
> > > > > approach.
> > > > > > > It seems, the java-operator-sdk handles the changes of the
> > > .metadata
> > > > > and
> > > > > > > .spec fields of custom resources differently. For further info
> > see
> > > > the
> > > > > > > chapter Generation Awareness and Event Filtering in the docs
> > > > > > > .
> > > > > > >
> > > > > > > Let me know what you think.
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Matyas
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> 

[jira] [Created] (FLINK-34657) Implement Lineage Graph for streaming API use cases

[Zhenqiu Huang (Jira)]

Zhenqiu Huang created FLINK-34657:
-

 Summary: Implement Lineage Graph for streaming API use cases
 Key: FLINK-34657
 URL: https://issues.apache.org/jira/browse/FLINK-34657
 Project: Flink
  Issue Type: Sub-task
Reporter: Zhenqiu Huang






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-435: Introduce a New Dynamic Table for Simplifying Data Pipelines

[Timo Walther]

Hi Lincoln & Ron,

thanks for proposing this FLIP. I think a design similar to what you 
propose has been in the heads of many people, however, I'm wondering how 
this will fit into the bigger picture.


I haven't deeply reviewed the FLIP yet, but would like to ask some 
initial questions:


Flink has introduced the concept of Dynamic Tables many years ago. How 
does the term "Dynamic Table" fit into Flink's regular tables and also 
how does it relate to Table API?


I fear that adding the DYNAMIC TABLE keyword could cause confusion for 
users, because a term for regular CREATE TABLE (that can be "kind of 
dynamic" as well and is backed by a changelog) is then missing. Also 
given that we call our connectors for those tables, DynamicTableSource 
and DynamicTableSink.


In general, I find it contradicting that a TABLE can be "paused" or 
"resumed". From an English language perspective, this does sound 
incorrect. In my opinion (without much research yet), a continuous 
updating trigger should rather be modelled as a CREATE MATERIALIZED VIEW 
(which users are familiar with?) or a new concept such as a CREATE TASK 
(that can be paused and resumed?).


How do you envision re-adding the functionality of a statement set, that 
fans out to multiple tables? This is a very important use case for data 
pipelines.


Since the early days of Flink SQL, we were discussing `SELECT STREAM * 
FROM T EMIT 5 MINUTES`. Your proposal seems to rephrase STREAM and EMIT, 
into other keywords DYNAMIC TABLE and FRESHNESS. But the core 
functionality is still there. I'm wondering if we should widen the scope 
(maybe not part of this FLIP but a new FLIP) to follow the standard more 
closely. Making `SELECT * FROM t` bounded by default and use new syntax 
for the dynamic behavior. Flink 2.0 would be the perfect time for this, 
however, it would require careful discussions. What do you think?


Regards,
Timo


On 11.03.24 08:23, Ron liu wrote:

Hi, Dev


Lincoln Lee and I would like to start a discussion about FLIP-435:
Introduce a  New Dynamic Table for Simplifying Data Pipelines.


This FLIP is designed to simplify the development of data processing
pipelines. With Dynamic Tables with uniform SQL statements and
freshness, users can define batch and streaming transformations to
data in the same way, accelerate ETL pipeline development, and manage
task scheduling automatically.


For more details, see FLIP-435 [1]. Looking forward to your feedback.


[1]


Best,

Lincoln & Ron

Re: [VOTE] Release 1.19.0, release candidate #2

[Matthias Pohl]

I want to share an update on FLINK-34227 [1]: It's still not clear what's
causing the test instability. So far, we agreed in today's release sync [2]
that it's not considered a blocker because it is observed in 1.18 nightly
builds and it only appears in the GitHub Actions workflow. But I still have
a bit of a concern that this is something that was introduced in 1.19 and
backported to 1.18 after the 1.18.1 release (because the test instability
started to appear more regularly in March; with one occurrence in January).
Additionally, I have no reason to believe, yet, that the instability is
caused by some GHA-related infrastructure issue.

So, if someone else has some capacity to help looking into it; that would
be appreciated. I will continue my investigation tomorrow.

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-34227
[2]
https://cwiki.apache.org/confluence/display/FLINK/1.19+Release#id-1.19Release-03/12/2024

On Tue, Mar 12, 2024 at 12:50 PM Benchao Li  wrote:

> +1 (non-binding)
>
> - checked signature and checksum: OK
> - checkout copyright year in notice file: OK
> - diffed source distribution with tag, make sure there is no
> unexpected files: OK
> - build from source : OK
> - start a local cluster, played with jdbc connector: OK
>
> weijie guo  于2024年3月12日周二 16:55写道：
> >
> > +1 (non-binding)
> >
> > - Verified signature and checksum
> > - Verified source distribution does not contains binaries
> > - Build from source code and submit a word-count job successfully
> >
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Jane Chan  于2024年3月12日周二 16:38写道：
> >
> > > +1 (non-binding)
> > >
> > > - Verify that the source distributions do not contain any binaries;
> > > - Build the source distribution to ensure all source files have Apache
> > > headers;
> > > - Verify checksum and GPG signatures;
> > >
> > > Best,
> > > Jane
> > >
> > > On Tue, Mar 12, 2024 at 4:08 PM Xuannan Su 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > - Verified signature and checksum
> > > > - Verified that source distribution does not contain binaries
> > > > - Built from source code successfully
> > > > - Reviewed the release announcement PR
> > > >
> > > > Best regards,
> > > > Xuannan
> > > >
> > > > On Tue, Mar 12, 2024 at 2:18 PM Hang Ruan 
> > > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - Verified signatures and checksums
> > > > > - Verified that source does not contain binaries
> > > > > - Build source code successfully
> > > > > - Reviewed the release note and left a comment
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Feng Jin  于2024年3月12日周二 11:23写道：
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > - Verified signatures and checksums
> > > > > > - Verified that source does not contain binaries
> > > > > > - Build source code successfully
> > > > > > - Run a simple sql query successfully
> > > > > >
> > > > > > Best,
> > > > > > Feng Jin
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 12, 2024 at 11:09 AM Ron liu 
> wrote:
> > > > > >
> > > > > > > +1 (non binding)
> > > > > > >
> > > > > > > quickly verified:
> > > > > > > - verified that source distribution does not contain binaries
> > > > > > > - verified checksums
> > > > > > > - built source code successfully
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Ron
> > > > > > >
> > > > > > > Jeyhun Karimov  于2024年3月12日周二 01:00写道：
> > > > > > >
> > > > > > > > +1 (non binding)
> > > > > > > >
> > > > > > > > - verified that source distribution does not contain binaries
> > > > > > > > - verified signatures and checksums
> > > > > > > > - built source code successfully
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Jeyhun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb <
> > > decordea...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non binding)
> > > > > > > > >
> > > > > > > > > - verified signatures and checksums
> > > > > > > > > - ASF headers are present in all expected file
> > > > > > > > > - No unexpected binaries files found in the source
> > > > > > > > > - Build successful locally
> > > > > > > > > - tested basic word count example
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Bests,
> > > > > > > > > Samrat
> > > > > > > > >
> > > > > > > > > On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy <
> > > > hamdy10...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Lincoln
> > > > > > > > > > +1 (non-binding) from me
> > > > > > > > > >
> > > > > > > > > > - Verified Checksums & Signatures
> > > > > > > > > > - Verified Source dists don't contain binaries
> > > > > > > > > > - Built source successfully
> > > > > > > > > > - reviewed web PR
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best Regards
> > > > > > > > > > Ahmed Hamdy
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, 11 Mar 2024 at 15:18, Linco

Re: Default scale and precision SQL data types

[Timo Walther]

Hi Sergei,

please check with the SQL standard before. Most of these values have 
been derived from the standard. I don't like the default of TIMESTAMP(6) 
for timestamps but this is what the standard dictates. Same for not 
allowing VARCHAR(0) or VARCHAR defaulting to VARCHAR(1).


Changes to the type system affect a lot of downstream projects and 
people. Be aware that the FLIP might not be accepted.


Regards,
Timo


On 07.03.24 10:05, lorenzo.affe...@ververica.com.INVALID wrote:

Hello Sergei!

The proposal makes a lot of sense, and Martijn is right as well.
Are you willing to drive the FLIP effort?
Do you need any assistance with that?
On Mar 4, 2024 at 01:48 +0100, Martijn Visser , wrote:

Hi,

I think it would first require a FLIP, given it touches on the core type
system of SQL.

Best regards,

Martijn

On Sat, Mar 2, 2024 at 5:34 PM Sergei Morozov  wrote:


Hi there,

org.apache.flink.table.api.DataTypes allows the creation of temporal data
types by specifying precision (e.g. TIME(3)) or omitting it (TIME()). The
ability to omit precision for temporal types was introduced in
apache/flink@36fef44
<
https://github.com/apache/flink/commit/36fef4457a7f1de47989c8a2485581bcf8633b32



.

Unfortunately, this isn't possible for other data types (e.g. CHAR,
DECIMAL).
Even though they define defaults for length, precision, and scale, their
values have to be passed to the method explicitly.

Would a PR be accepted which will introduce the methods for the remaining
types similar to the temporal ones?

Thanks.

Re: [DISCUSS] Manual savepoint triggering in flink-kubernetes-operator

[Gyula Fóra]

That's definitely a good improvement Robert and we should add it at some
point. At the point in time when this was implemented we went with the
current simpler / more lightweight approach.
However if anyone is interested in working on this / contributing this
improvement I would personally support it.

Gyula

On Tue, Mar 12, 2024 at 3:53 PM Robert Metzger  wrote:

> Have you guys considered making savepoints a first class citizen in the
> Kubernetes operator?
> E.g. to trigger a savepoint, you create a "FlinkSavepoint" CR, the K8s
> operator picks up that resource and tries to create a savepoint
> indefinitely until the savepoint has been successfully created. We report
> the savepoint status and location in the "status" field.
>
> We could even add an (optional) finalizer to delete the physical savepoint
> from the savepoint storage once the "FlinkSavepoint" CR has been deleted.
> optional: the savepoint spec could contain a field "retain
> physical savepoint" or something, that controls the delete behavior.
>
>
> On Thu, Mar 3, 2022 at 4:02 AM Yang Wang  wrote:
>
> > I agree that we could start with the annotation approach and collect the
> > feedback at the same time.
> >
> > Best,
> > Yang
> >
> > Őrhidi Mátyás  于2022年3月2日周三 20:06写道：
> >
> > > Thank you for your feedback!
> > >
> > > The annotation on the
> > >
> > > @ControllerConfiguration(generationAwareEventProcessing = false)
> > > FlinkDeploymentController
> > >
> > > already enables the event triggering based on metadata changes. It was
> > set
> > > earlier to support some failure scenarios. (It can be used for example
> to
> > > manually reenable the reconcile loop when it got stuck in an error
> phase)
> > >
> > > I will go ahead and propose a PR using annotations then.
> > >
> > > Cheers,
> > > Matyas
> > >
> > > On Wed, Mar 2, 2022 at 12:47 PM Yang Wang 
> wrote:
> > >
> > > > I also like the annotation approach since it is more natural.
> > > > But I am not sure about whether the meta data change will trigger an
> > > event
> > > > in java-operator-sdk.
> > > >
> > > >
> > > > Best,
> > > > Yang
> > > >
> > > > Gyula Fóra  于2022年3月2日周三 16:29写道：
> > > >
> > > > > Thanks Matyas,
> > > > >
> > > > > From a user perspective I think the annotation is pretty nice and
> > user
> > > > > friendly so I personally prefer that approach.
> > > > >
> > > > > You said:
> > > > >  "It seems, the java-operator-sdk handles the changes of the
> > .metadata
> > > > and
> > > > > .spec fields of custom resources differently."
> > > > >
> > > > > What implications does this have on the above mentioned 2
> approaches?
> > > > Does
> > > > > it make one more difficult than the other?
> > > > >
> > > > > Cheers
> > > > > Gyula
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 1, 2022 at 1:52 PM Őrhidi Mátyás <
> > matyas.orh...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi All!
> > > > > >
> > > > > > I'd like to start a quick discussion about the way we allow users
> > to
> > > > > > trigger savepoints manually in the operator [FLINK-26181]
> > > > > > . There are
> > > > existing
> > > > > > solutions already for this functionality in other operators, for
> > > > example:
> > > > > > - counter based
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/spotify/flink-on-k8s-operator/blob/master/docs/savepoints_guide.md#2-taking-savepoints-by-updating-the-flinkcluster-custom-resource
> > > > > > >
> > > > > > - annotation based
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/spotify/flink-on-k8s-operator/blob/master/docs/savepoints_guide.md#3-taking-savepoints-by-attaching-annotation-to-the-flinkcluster-custom-resource
> > > > > > >
> > > > > >
> > > > > > We could implement any of these or both or come up with our own
> > > > approach.
> > > > > > It seems, the java-operator-sdk handles the changes of the
> > .metadata
> > > > and
> > > > > > .spec fields of custom resources differently. For further info
> see
> > > the
> > > > > > chapter Generation Awareness and Event Filtering in the docs
> > > > > > .
> > > > > >
> > > > > > Let me know what you think.
> > > > > >
> > > > > > Cheers,
> > > > > > Matyas
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Manual savepoint triggering in flink-kubernetes-operator

[Robert Metzger]

Have you guys considered making savepoints a first class citizen in the
Kubernetes operator?
E.g. to trigger a savepoint, you create a "FlinkSavepoint" CR, the K8s
operator picks up that resource and tries to create a savepoint
indefinitely until the savepoint has been successfully created. We report
the savepoint status and location in the "status" field.

We could even add an (optional) finalizer to delete the physical savepoint
from the savepoint storage once the "FlinkSavepoint" CR has been deleted.
optional: the savepoint spec could contain a field "retain
physical savepoint" or something, that controls the delete behavior.


On Thu, Mar 3, 2022 at 4:02 AM Yang Wang  wrote:

> I agree that we could start with the annotation approach and collect the
> feedback at the same time.
>
> Best,
> Yang
>
> Őrhidi Mátyás  于2022年3月2日周三 20:06写道：
>
> > Thank you for your feedback!
> >
> > The annotation on the
> >
> > @ControllerConfiguration(generationAwareEventProcessing = false)
> > FlinkDeploymentController
> >
> > already enables the event triggering based on metadata changes. It was
> set
> > earlier to support some failure scenarios. (It can be used for example to
> > manually reenable the reconcile loop when it got stuck in an error phase)
> >
> > I will go ahead and propose a PR using annotations then.
> >
> > Cheers,
> > Matyas
> >
> > On Wed, Mar 2, 2022 at 12:47 PM Yang Wang  wrote:
> >
> > > I also like the annotation approach since it is more natural.
> > > But I am not sure about whether the meta data change will trigger an
> > event
> > > in java-operator-sdk.
> > >
> > >
> > > Best,
> > > Yang
> > >
> > > Gyula Fóra  于2022年3月2日周三 16:29写道：
> > >
> > > > Thanks Matyas,
> > > >
> > > > From a user perspective I think the annotation is pretty nice and
> user
> > > > friendly so I personally prefer that approach.
> > > >
> > > > You said:
> > > >  "It seems, the java-operator-sdk handles the changes of the
> .metadata
> > > and
> > > > .spec fields of custom resources differently."
> > > >
> > > > What implications does this have on the above mentioned 2 approaches?
> > > Does
> > > > it make one more difficult than the other?
> > > >
> > > > Cheers
> > > > Gyula
> > > >
> > > >
> > > >
> > > > On Tue, Mar 1, 2022 at 1:52 PM Őrhidi Mátyás <
> matyas.orh...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All!
> > > > >
> > > > > I'd like to start a quick discussion about the way we allow users
> to
> > > > > trigger savepoints manually in the operator [FLINK-26181]
> > > > > . There are
> > > existing
> > > > > solutions already for this functionality in other operators, for
> > > example:
> > > > > - counter based
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/spotify/flink-on-k8s-operator/blob/master/docs/savepoints_guide.md#2-taking-savepoints-by-updating-the-flinkcluster-custom-resource
> > > > > >
> > > > > - annotation based
> > > > > <
> > > > >
> > > >
> > >
> >
> https://github.com/spotify/flink-on-k8s-operator/blob/master/docs/savepoints_guide.md#3-taking-savepoints-by-attaching-annotation-to-the-flinkcluster-custom-resource
> > > > > >
> > > > >
> > > > > We could implement any of these or both or come up with our own
> > > approach.
> > > > > It seems, the java-operator-sdk handles the changes of the
> .metadata
> > > and
> > > > > .spec fields of custom resources differently. For further info see
> > the
> > > > > chapter Generation Awareness and Event Filtering in the docs
> > > > > .
> > > > >
> > > > > Let me know what you think.
> > > > >
> > > > > Cheers,
> > > > > Matyas
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-34656) Generated code for `ITEM` operator should return null when getting element of a null map/array/row

[yisha zhou (Jira)]

yisha zhou created FLINK-34656:
--

 Summary: Generated code for `ITEM` operator should return null 
when getting element of a null map/array/row
 Key: FLINK-34656
 URL: https://issues.apache.org/jira/browse/FLINK-34656
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.20.0
Reporter: yisha zhou


In FieldAccessFromTableITCase we can find that the expected result of f0[1] is 
null when f0 is a null array. 

However, behavior in generated code for ITEM is not consistent with case above. 
The main code is:

 
{code:java}
val arrayAccessCode =
  s"""
 |${array.code}
 |${index.code}
 |boolean $nullTerm = ${array.nullTerm} || ${index.nullTerm} ||
 |   $idxStr < 0 || $idxStr >= ${array.resultTerm}.size() || $arrayIsNull;
 |$resultTypeTerm $resultTerm = $nullTerm ? $defaultTerm : $arrayGet;
 |""".stripMargin {code}
If `array.nullTerm` is true, a default value of element type will be returned, 
for example -1 for null bigint array.

The reason why FieldAccessFromTableITCase can get expected result is that the 
ReduceExpressionsRule generated an expression code for that case like:
{code:java}
boolean isNull$0 = true || false ||
   ((int) 1) - 1 < 0 || ((int) 1) - 1 >= 
((org.apache.flink.table.data.ArrayData) null).size() || 
((org.apache.flink.table.data.ArrayData) null).isNullAt(((int) 1) - 1);
long result$0 = isNull$0 ? -1L : ((org.apache.flink.table.data.ArrayData) 
null).getLong(((int) 1) - 1);
if (isNull$0) {
  out.setField(0, null);
} else {
  out.setField(0, result$0);
} {code}
The reduced expr will be a null literal.
 

I think the behaviors for getting element of a null value should be unified.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 1.19.0, release candidate #2

[Benchao Li]

+1 (non-binding)

- checked signature and checksum: OK
- checkout copyright year in notice file: OK
- diffed source distribution with tag, make sure there is no
unexpected files: OK
- build from source : OK
- start a local cluster, played with jdbc connector: OK

weijie guo  于2024年3月12日周二 16:55写道：
>
> +1 (non-binding)
>
> - Verified signature and checksum
> - Verified source distribution does not contains binaries
> - Build from source code and submit a word-count job successfully
>
>
> Best regards,
>
> Weijie
>
>
> Jane Chan  于2024年3月12日周二 16:38写道：
>
> > +1 (non-binding)
> >
> > - Verify that the source distributions do not contain any binaries;
> > - Build the source distribution to ensure all source files have Apache
> > headers;
> > - Verify checksum and GPG signatures;
> >
> > Best,
> > Jane
> >
> > On Tue, Mar 12, 2024 at 4:08 PM Xuannan Su  wrote:
> >
> > > +1 (non-binding)
> > >
> > > - Verified signature and checksum
> > > - Verified that source distribution does not contain binaries
> > > - Built from source code successfully
> > > - Reviewed the release announcement PR
> > >
> > > Best regards,
> > > Xuannan
> > >
> > > On Tue, Mar 12, 2024 at 2:18 PM Hang Ruan 
> > wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > - Verified signatures and checksums
> > > > - Verified that source does not contain binaries
> > > > - Build source code successfully
> > > > - Reviewed the release note and left a comment
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Feng Jin  于2024年3月12日周二 11:23写道：
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > - Verified signatures and checksums
> > > > > - Verified that source does not contain binaries
> > > > > - Build source code successfully
> > > > > - Run a simple sql query successfully
> > > > >
> > > > > Best,
> > > > > Feng Jin
> > > > >
> > > > >
> > > > > On Tue, Mar 12, 2024 at 11:09 AM Ron liu  wrote:
> > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > quickly verified:
> > > > > > - verified that source distribution does not contain binaries
> > > > > > - verified checksums
> > > > > > - built source code successfully
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Ron
> > > > > >
> > > > > > Jeyhun Karimov  于2024年3月12日周二 01:00写道：
> > > > > >
> > > > > > > +1 (non binding)
> > > > > > >
> > > > > > > - verified that source distribution does not contain binaries
> > > > > > > - verified signatures and checksums
> > > > > > > - built source code successfully
> > > > > > >
> > > > > > > Regards,
> > > > > > > Jeyhun
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb <
> > decordea...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non binding)
> > > > > > > >
> > > > > > > > - verified signatures and checksums
> > > > > > > > - ASF headers are present in all expected file
> > > > > > > > - No unexpected binaries files found in the source
> > > > > > > > - Build successful locally
> > > > > > > > - tested basic word count example
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Bests,
> > > > > > > > Samrat
> > > > > > > >
> > > > > > > > On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy <
> > > hamdy10...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Lincoln
> > > > > > > > > +1 (non-binding) from me
> > > > > > > > >
> > > > > > > > > - Verified Checksums & Signatures
> > > > > > > > > - Verified Source dists don't contain binaries
> > > > > > > > > - Built source successfully
> > > > > > > > > - reviewed web PR
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best Regards
> > > > > > > > > Ahmed Hamdy
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, 11 Mar 2024 at 15:18, Lincoln Lee <
> > > lincoln.8...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Robin,
> > > > > > > > > >
> > > > > > > > > > Thanks for helping verifying the release note[1],
> > FLINK-14879
> > > > > > should
> > > > > > > > not
> > > > > > > > > > have been included, after confirming this
> > > > > > > > > > I moved all unresolved non-blocker issues left over from
> > > 1.19.0
> > > > > to
> > > > > > > > 1.20.0
> > > > > > > > > > and reconfigured the release note [1].
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Lincoln Lee
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353282
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Robin Moffatt  于2024年3月11日周一
> > > > > 19:36写道：
> > > > > > > > > >
> > > > > > > > > > > Looking at the release notes [1] it lists `DESCRIBE
> > > DATABASE`
> > > > > > > > > > (FLINK-14879)
> > > > > > > > > > > and `DESCRIBE CATALOG` (FLINK-14690).
> > > > > > > > > > > When I try these in 1.19 RC2 the behaviour is as in
> > 1.18.1,
> > > > > i.e.
> > > > > > it
> > 

[jira] [Created] (FLINK-34655) Autoscaler doesn't work for flink 1.15

[Rui Fan (Jira)]

Rui Fan created FLINK-34655:
---

 Summary: Autoscaler doesn't work for flink 1.15
 Key: FLINK-34655
 URL: https://issues.apache.org/jira/browse/FLINK-34655
 Project: Flink
  Issue Type: Bug
  Components: Autoscaler
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.8.0


flink-ubernetes-operator is committed to supporting the latest 4 flink minor 
versions, and autoscaler is a part of flink-ubernetes-operator. Currently,  the 
latest 4 flink minor versions are 1.15, 1.16, 1.17 and 1.18.

But autoscaler doesn't work for  flink 1.15.

h2. Root cause: 

* FLINK-28310 added some properties in IOMetricsInfo in flink-1.16
* IOMetricsInfo is a part of JobDetailsInfo
* JobDetailsInfo is necessary for autoscaler [1]
* flink's RestClient doesn't allow miss any property during deserializing the 
json

That means that the RestClient after 1.15 cannot fetch JobDetailsInfo for 1.15 
jobs.

h2. How to fix it properly?

Flink side support ignore unknown properties.

FLINK-33268 already do it. But I try run autoscaler with flink-1.15 job, it 
still doesn't work. Because the IOMetricsInfo added some properties, they are 
primitive type.

It should disable DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES as well. 
(Not sure whether it should be a seperate FLIP or it can be a part of FLIP-401 
[2].)


h2. How to fix it in the short term?

1. Copy the latest RestMapperUtils and RestClient from master branch (It 
includes FLINK-33268) to flink-autoscaler module. (The copied class will be 
loaded first)
2. Disable DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES in 
RestMapperUtils#flexibleObjectMapper in copied class.

Based on these 2 steps, flink-1.15 works well with autoscaler. (I try it 
locally).


After DeserializationFeature.FAIL_ON_NULL_FOR_PRIMITIVES in 
RestMapperUtils#flexibleObjectMapper is disabled, and the corresponding code is 
released in flink side. flink-ubernetes-operator can remove these 2 copied 
classes.

[1] 
https://github.com/apache/flink-kubernetes-operator/blob/ede1a610b3375d31a2e82287eec67ace70c4c8df/flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingMetricCollector.java#L109
[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-401%3A+REST+API+JSON+response+deserialization+unknown+field+tolerance



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Add "Special Thanks" Page on the Flink Website

[Jark Wu]

I have created a JIRA issue and opened a pull request for this:
https://github.com/apache/flink-web/pull/725.

Best,
Jark

On Tue, 12 Mar 2024 at 16:56, Jark Wu  wrote:

> Thank you all for your feedback. If there are no other concerns or
> objections,
> I'm going to create a pull request to add the Special Thanks page.
>
> Further feedback and sponsors to be added are still welcome!
>
> Best,
> Jark
>
> On Mon, 11 Mar 2024 at 23:09, Maximilian Michels  wrote:
>
>> Hi Jark,
>>
>> Thanks for clarifying. At first sight, such a page indicated general
>> sponsorship. +1 for a Thank You page to list specific monetary
>> contributions to the project for resources which are actively used or
>> were actively used in the past.
>>
>> Cheers,
>> Max
>>
>> On Fri, Mar 8, 2024 at 11:55 AM Martijn Visser 
>> wrote:
>> >
>> > Hi all,
>> >
>> > I'm +1 on it. As long as we follow the ASF rules on this, we can thank
>> > those that are/have made contributions.
>> >
>> > Best regards,
>> >
>> > Martijn
>> >
>> > On Wed, Mar 6, 2024 at 7:45 AM Jark Wu  wrote:
>> >
>> > > Hi Matthias,
>> > >
>> > > Thanks for your comments! Please see my reply inline.
>> > >
>> > > > What do we do if we have enough VMs? Do we still allow
>> > > companies to add more VMs to the pool even though it's not adding any
>> > > value?
>> > >
>> > > The ASF policy[1] makes it very clear: "Project Thanks pages are to
>> show
>> > > appreciation
>> > > for goods that the project truly needs, not just for goods that
>> someone
>> > > wants to donate."
>> > > Therefore, the community should reject new VMs if it is enough.
>> > >
>> > >
>> > > > The community lacks the openly accessible tools to monitor the VM
>> usage
>> > > independently
>> > > as far as I know (the Azure Pipelines project is owned by Ververica
>> right
>> > > now).
>> > >
>> > > The Azure pipeline account is sponsored by Ververica, and is managed
>> by the
>> > > community.
>> > > AFAIK, Chesnay and Robert both have admin permissions [2] to the Azure
>> > > pipeline project.
>> > > Others can contact the managers to get access to the environment.
>> > >
>> > > > I figured that there could be a chance for us to
>> > > rely on Apache-provided infrastructure entirely with our current
>> workload
>> > > when switching over from Azure Pipelines.
>> > >
>> > > That sounds great. We can return back the VMs and mark the donations
>> as
>> > > historical
>> > > on the Thank Page once the new GitHub Actions CI is ready.
>> > >
>> > > > I am fine with creating a Thank You page to acknowledge the
>> financial
>> > > contributions from Alibaba and Ververica in the past (since Apache
>> allows
>> > > historical donations) considering that the contributions of the two
>> > > companies go way back in time and are quite significant in my
>> opinion. I
>> > > suggest focusing on the past for now because of the option to migrate
>> to
>> > > Apache infrastructure midterm.
>> > >
>> > > Sorry, do you mean we only mention past donations for now?
>> > > IIUC, the new GitHub Actions might be ready after the end of v1.20,
>> which
>> > > probably be in half a year.
>> > > I'm worried that if we say the sponsorship is ongoing until now (but
>> it's
>> > > not), it will confuse
>> > > people and disrespect the sponsor.
>> > >
>> > > Besides, I'm not sure whether the new GitHub Actions CI will replace
>> the
>> > > machines for running
>> > > flink-ci mirrors [3] and the flink benchmarks [4]. If not, I think
>> it's
>> > > inappropriate to say they are
>> > > historical donations.
>> > >
>> > > Furthermore, we are collecting all kinds of donations. I just noticed
>> that
>> > > AWS donated [5] service costs
>> > > for flink-connector-aws tests that hit real AWS services. This is an
>> > > ongoing donation and I think it's not
>> > > good to mark it as a historical donation. (Thanks for the donation,
>> AWS,
>> > > @Danny
>> > > Cranmer  @HongTeoh!
>> > > We should add it to the Thank Page!)
>> > >
>> > > Best,
>> > > Jark
>> > >
>> > >
>> > > [1]: https://www.apache.org/foundation/marks/linking#projectthanks
>> > > [2]:
>> > >
>> > >
>> https://cwiki.apache.org/confluence/display/FLINK/Continuous+Integration#ContinuousIntegration-Contacts
>> > >
>> > > [3]:
>> > >
>> > >
>> https://cwiki.apache.org/confluence/display/FLINK/Continuous+Integration#ContinuousIntegration-Repositories
>> > >
>> > > [4]: https://lists.apache.org/thread/bkw6ozoflgltwfwmzjtgx522hyssfko6
>> > >
>> > > [5]: https://issues.apache.org/jira/browse/INFRA-24474
>> > >
>> > > On Wed, 6 Mar 2024 at 17:58, Matthias Pohl  wrote:
>> > >
>> > > > Thanks for starting this discussion. I see the value of such a page
>> if we
>> > > > want to encourage companies to sponsor CI infrastructure in case we
>> need
>> > > > this infrastructure (as Yun Tang pointed out). The question is,
>> though:
>> > > Do
>> > > > we need more VMs? The amount of commits to master is constantly
>> > > decreasing
>> > > > since its peak in 2019/2020 [1]. Did we observe shortage

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

[Zakelly Lan]

Hi Weijie,

Thanks for your reply!

Overall I'd be fine with the builder pattern, but it is a little bit long
when carrying explicit 'build()' and declaring the builder. Keeping the
StateDeclaration immutable is OK, but it is a little bit inconvenient for
overriding the undefined options by job configuration at runtime. I'd
suggest providing some methods responsible for rebuilding a new
StateDeclaration with new configurable options, just like the
ConfigOptions#defaultValue does. Well, this is just a suggestion, I'm not
going to insist on it.


Best,
Zakelly

On Tue, Mar 12, 2024 at 2:07 PM weijie guo 
wrote:

> Hi Zakelly,
>
> > But still, from a user's point of view,  state can be characterized along
> two relatively independent dimensions, how states redistribute and the data
> structure. Thus I still suggest a chained-like configuration API that
> configures one aspect on each call.
>
>
> I think the chained-like style is a good suggestion. But I'm not going to
> introduce any mutable-like API to StateDeclaration (even though we can
> achieve immutability by returning a new object). For this reason, I decided
> to use the builder pattern, which also has the benefit of chaining calls
> and allows us to support further configurations such as setTTL in the
> future. For ease of use, we'll also provide some shortcuts to avoid having
> to go through a long build chain each time. Of course, I have updated the
> the FLIP about this part.
>
>
>
> Best regards,
>
> Weijie
>
>
> weijie guo  于2024年3月12日周二 14:00写道：
>
> > Hi Hangxiang,
> >
> > > So these operators only define all states they may use which could be
> > explained by the caller, right ?
> >
> > Yes, you're right.
> >
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > weijie guo  于2024年3月12日周二 13:59写道：
> >
> >> Hi Max,
> >>
> >> > In this thread it looks like the plan is to remove the old state
> >> declaration API. I think we should consider keeping the old APIs to
> >> avoid breaking too many jobs.
> >>
> >> We're not plan to remove any old apis, which means that changes made in
> >> V2 won't affect any V1 DataStream jobs. But V2 is limited to the new
> state
> >> declaration API, and users who want to migrate to DataStream V2 will
> need
> >> to rewrite their jobs anyway.
> >>
> >> Best regards,
> >>
> >> Weijie
> >>
> >>
> >> Hangxiang Yu  于2024年3月12日周二 10:26写道：
> >>
> >>> Hi, Weijie.
> >>> Thanks for your answer!
> >>>
> >>> > No, Introducing and declaring new state
> >>> > at runtime is something we want to explicitly disallow.
> >>>
> >>> I just thinked about how some operators define their useState() when
> >>> their
> >>> real used states may be changed at runtime, e.g. different state types
> >>> for
> >>> different state sizes.
> >>> So these operators only define all states they may use which could be
> >>> explained by the caller, right ?
> >>>
> >>> On Mon, Mar 11, 2024 at 10:57 PM Maximilian Michels 
> >>> wrote:
> >>>
> >>> > The FLIP mentions: "The contents described in this FLIP are all new
> >>> > APIs and do not involve compatibility issues."
> >>> >
> >>> > In this thread it looks like the plan is to remove the old state
> >>> > declaration API. I think we should consider keeping the old APIs to
> >>> > avoid breaking too many jobs. The new APIs will still be beneficial
> >>> > for new jobs, e.g. for SQL jobs.
> >>> >
> >>> > -Max
> >>> >
> >>> > On Fri, Mar 8, 2024 at 4:39 AM Zakelly Lan 
> >>> wrote:
> >>> > >
> >>> > > Hi Weijie,
> >>> > >
> >>> > > Thanks for your answer! Well I get your point. Since partitions are
> >>> > > first-class citizens, and redistribution means how states migrate
> >>> when
> >>> > > partitions change, I'd be fine with deemphasizing the concept of
> >>> > > keyed/operator state if we highlight the definition of partition in
> >>> the
> >>> > > document. Keeping `RedistributionMode` under `StateDeclaration` is
> >>> also
> >>> > > fine with me, as I guess it is only for internal usage.
> >>> > > But still, from a user's point of view,  state can be characterized
> >>> along
> >>> > > two relatively independent dimensions, how states redistribute and
> >>> the
> >>> > data
> >>> > > structure. Thus I still suggest a chained-like configuration API
> that
> >>> > > configures one aspect on each call, such as:
> >>> > > ```
> >>> > > # Keyed stream, no redistribution mode specified, the state will go
> >>> with
> >>> > > partition (no redistribution).  Keyed state
> >>> > > StateDeclaration a = States.declare(name).listState(type);
> >>> > >
> >>> > > # Keyed stream, redistribution strategy specified, the state
> follows
> >>> the
> >>> > > specified redistribute strategy.   Operator state
> >>> > > StateDeclaration b =
> >>> > > States.declare(name).listState(type).redistributeBy(strategy);
> >>> > >
> >>> > > # Non-keyed stream, redistribution strategy *must be* specified.
> >>> > > StateDeclaration c =
> >>> > > States.declare(name).listState(type).redistributeBy(strategy);
> >>> > >
> >>> > > # B

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

[Zakelly Lan]

Hi Xuannan,

Thanks for your comments, I modified the FLIP accordingly.

Hi Yunfeng,

Thanks for sharing your opinions!

Could you provide some hint on use cases where users need to mix sync
> and async state operations in spite of the performance regression?
> This information might help address our concerns on design. If the
> mixed usage is simply something not recommended, I would prefer to
> prohibit such usage from API.

In fact, there is no scenario where users MUST use the sync APIs, but it is
much easier to use for those who are not familiar with asynchronous
programming. If they want to migrate their job from Flink 1.x to 2.0
leveraging some benefits from asynchronous APIs, they may try the mixed
usage. It is not user-friendly to directly throw exceptions at runtime, I
think our better approach is to warn users and recommend avoiding this. I
added an example in this FLIP.

Well, I do not insist on allowing mixed usage of APIs if others reach an
agreement that we won't support that . I think the most important is to
keep the API easy to use and understand, thus I propose a unified state
declaration and explicit meaning in method name. WDYT?

Sorry I missed the new sink API. I do still think that it would be
> better to make the package name more informative, and ".v2." does not
> contain information for new Flink users who did not know the v1 of
> state API. Unlike internal implementation and performance
> optimization, API will hardly be compromised for now and updated in
> future, so I still suggest we improve the package name now if
> possible. But given the existing practice of sink v2 and
> AbstractStreamOperatorV2, the current package name would be acceptable
> to me if other reviewers of this FLIP agrees on it.

Actually, I don't like 'v2' either. So if there is another good name, I'd
be happy to apply. This is a compromise to the current situation. Maybe we
could refine this after the retirement of original state APIs.


Thanks & Best,
Zakelly


On Tue, Mar 12, 2024 at 1:42 PM Yunfeng Zhou 
wrote:

> Hi Zakelly,
>
> Thanks for the quick response!
>
> > Actually splitting APIs into two sets ... warn them in runtime.
>
> Could you provide some hint on use cases where users need to mix sync
> and async state operations in spite of the performance regression?
> This information might help address our concerns on design. If the
> mixed usage is simply something not recommended, I would prefer to
> prohibit such usage from API.
>
> > In fact ... .sink2`.
>
> Sorry I missed the new sink API. I do still think that it would be
> better to make the package name more informative, and ".v2." does not
> contain information for new Flink users who did not know the v1 of
> state API. Unlike internal implementation and performance
> optimization, API will hardly be compromised for now and updated in
> future, so I still suggest we improve the package name now if
> possible. But given the existing practice of sink v2 and
> AbstractStreamOperatorV2, the current package name would be acceptable
> to me if other reviewers of this FLIP agrees on it.
>
> Best,
> Yunfeng
>
> On Mon, Mar 11, 2024 at 5:27 PM Zakelly Lan  wrote:
> >
> > Hi Yunfeng,
> >
> > Thanks for your comments!
> >
> > +1 for JingGe's suggestion to introduce an AsyncState API, instead of
> > > having both get() and asyncGet() in the same State class. As a
> > > supplement to its benefits, this design could help avoid having users
> > > to use sync and async API in a mixed way (unless they create both a
> > > State and an AsyncState from the same state descriptor), which is
> > > supposed to bring suboptimal performance according to the FLIP's
> > > description.
> >
> >
> > Actually splitting APIs into two sets of classes also brings some
> > difficulties. In this case, users must explicitly define their usage
> before
> > actually doing state access. It is a little strange that the user can
> > define a sync and an async version of State with the same name, while
> they
> > cannot allocate two async States with the same name.
> > Another reason for distinguishing API by their method name instead of
> class
> > name is that users typically use the State instances to access state but
> > forget their type/class. For example:
> > ```
> > SyncState a = getState(xxx);
> > AsyncState b = getAsyncState(xxx);
> > //...
> > a.update(1);
> > b.update(1);
> > ```
> > Users are likely to think there is no difference between the
> `a.update(1)`
> > and `b.update(1)`, since they may forget the type for `a` and `b`. Thus I
> > proposed to distinguish the behavior in method names.
> > As for the suboptimal performance with mixed usage of sync and async, my
> > proposal is to warn them in runtime.
> >
> > I noticed that the FLIP proposes to place the newly introduced API in
> > > the package "org.apache.flink.api.common.state.v2", which seems a
> > > little strange to me as there has not been such a naming pattern
> > > ".v2." for packages in Flink.
> >
> >
> >

[jira] [Created] (FLINK-34654) Add "Special Thanks" Page on the Flink Website

[Jark Wu (Jira)]

Jark Wu created FLINK-34654:
---

 Summary: Add "Special Thanks" Page on the Flink Website
 Key: FLINK-34654
 URL: https://issues.apache.org/jira/browse/FLINK-34654
 Project: Flink
  Issue Type: New Feature
  Components: Project Website
Reporter: Jark Wu
Assignee: Jark Wu


This issue aims to add a "Special Thanks" page on the Flink website 
(https://flink.apache.org/) to honor and appreciate the companies and 
organizations that have sponsored machines or services for our project.

Discussion thread: 
https://lists.apache.org/thread/y5g0nd5t8h2ql4gq7d0kb9tkwv1wkm1j



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Add "Special Thanks" Page on the Flink Website

[Jark Wu]

Thank you all for your feedback. If there are no other concerns or
objections,
I'm going to create a pull request to add the Special Thanks page.

Further feedback and sponsors to be added are still welcome!

Best,
Jark

On Mon, 11 Mar 2024 at 23:09, Maximilian Michels  wrote:

> Hi Jark,
>
> Thanks for clarifying. At first sight, such a page indicated general
> sponsorship. +1 for a Thank You page to list specific monetary
> contributions to the project for resources which are actively used or
> were actively used in the past.
>
> Cheers,
> Max
>
> On Fri, Mar 8, 2024 at 11:55 AM Martijn Visser 
> wrote:
> >
> > Hi all,
> >
> > I'm +1 on it. As long as we follow the ASF rules on this, we can thank
> > those that are/have made contributions.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Wed, Mar 6, 2024 at 7:45 AM Jark Wu  wrote:
> >
> > > Hi Matthias,
> > >
> > > Thanks for your comments! Please see my reply inline.
> > >
> > > > What do we do if we have enough VMs? Do we still allow
> > > companies to add more VMs to the pool even though it's not adding any
> > > value?
> > >
> > > The ASF policy[1] makes it very clear: "Project Thanks pages are to
> show
> > > appreciation
> > > for goods that the project truly needs, not just for goods that someone
> > > wants to donate."
> > > Therefore, the community should reject new VMs if it is enough.
> > >
> > >
> > > > The community lacks the openly accessible tools to monitor the VM
> usage
> > > independently
> > > as far as I know (the Azure Pipelines project is owned by Ververica
> right
> > > now).
> > >
> > > The Azure pipeline account is sponsored by Ververica, and is managed
> by the
> > > community.
> > > AFAIK, Chesnay and Robert both have admin permissions [2] to the Azure
> > > pipeline project.
> > > Others can contact the managers to get access to the environment.
> > >
> > > > I figured that there could be a chance for us to
> > > rely on Apache-provided infrastructure entirely with our current
> workload
> > > when switching over from Azure Pipelines.
> > >
> > > That sounds great. We can return back the VMs and mark the donations as
> > > historical
> > > on the Thank Page once the new GitHub Actions CI is ready.
> > >
> > > > I am fine with creating a Thank You page to acknowledge the financial
> > > contributions from Alibaba and Ververica in the past (since Apache
> allows
> > > historical donations) considering that the contributions of the two
> > > companies go way back in time and are quite significant in my opinion.
> I
> > > suggest focusing on the past for now because of the option to migrate
> to
> > > Apache infrastructure midterm.
> > >
> > > Sorry, do you mean we only mention past donations for now?
> > > IIUC, the new GitHub Actions might be ready after the end of v1.20,
> which
> > > probably be in half a year.
> > > I'm worried that if we say the sponsorship is ongoing until now (but
> it's
> > > not), it will confuse
> > > people and disrespect the sponsor.
> > >
> > > Besides, I'm not sure whether the new GitHub Actions CI will replace
> the
> > > machines for running
> > > flink-ci mirrors [3] and the flink benchmarks [4]. If not, I think it's
> > > inappropriate to say they are
> > > historical donations.
> > >
> > > Furthermore, we are collecting all kinds of donations. I just noticed
> that
> > > AWS donated [5] service costs
> > > for flink-connector-aws tests that hit real AWS services. This is an
> > > ongoing donation and I think it's not
> > > good to mark it as a historical donation. (Thanks for the donation,
> AWS,
> > > @Danny
> > > Cranmer  @HongTeoh!
> > > We should add it to the Thank Page!)
> > >
> > > Best,
> > > Jark
> > >
> > >
> > > [1]: https://www.apache.org/foundation/marks/linking#projectthanks
> > > [2]:
> > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Continuous+Integration#ContinuousIntegration-Contacts
> > >
> > > [3]:
> > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Continuous+Integration#ContinuousIntegration-Repositories
> > >
> > > [4]: https://lists.apache.org/thread/bkw6ozoflgltwfwmzjtgx522hyssfko6
> > >
> > > [5]: https://issues.apache.org/jira/browse/INFRA-24474
> > >
> > > On Wed, 6 Mar 2024 at 17:58, Matthias Pohl  wrote:
> > >
> > > > Thanks for starting this discussion. I see the value of such a page
> if we
> > > > want to encourage companies to sponsor CI infrastructure in case we
> need
> > > > this infrastructure (as Yun Tang pointed out). The question is,
> though:
> > > Do
> > > > we need more VMs? The amount of commits to master is constantly
> > > decreasing
> > > > since its peak in 2019/2020 [1]. Did we observe shortage of CI
> runners in
> > > > the past years? What do we do if we have enough VMs? Do we still
> allow
> > > > companies to add more VMs to the pool even though it's not adding any
> > > > value? Then it becomes a marketing tool for companies. The community
> > > lacks
> > > > the openly accessible tools to monitor the VM usage ind

Re: [VOTE] Release 1.19.0, release candidate #2

[weijie guo]

+1 (non-binding)

- Verified signature and checksum
- Verified source distribution does not contains binaries
- Build from source code and submit a word-count job successfully


Best regards,

Weijie


Jane Chan  于2024年3月12日周二 16:38写道：

> +1 (non-binding)
>
> - Verify that the source distributions do not contain any binaries;
> - Build the source distribution to ensure all source files have Apache
> headers;
> - Verify checksum and GPG signatures;
>
> Best,
> Jane
>
> On Tue, Mar 12, 2024 at 4:08 PM Xuannan Su  wrote:
>
> > +1 (non-binding)
> >
> > - Verified signature and checksum
> > - Verified that source distribution does not contain binaries
> > - Built from source code successfully
> > - Reviewed the release announcement PR
> >
> > Best regards,
> > Xuannan
> >
> > On Tue, Mar 12, 2024 at 2:18 PM Hang Ruan 
> wrote:
> > >
> > > +1 (non-binding)
> > >
> > > - Verified signatures and checksums
> > > - Verified that source does not contain binaries
> > > - Build source code successfully
> > > - Reviewed the release note and left a comment
> > >
> > > Best,
> > > Hang
> > >
> > > Feng Jin  于2024年3月12日周二 11:23写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > - Verified signatures and checksums
> > > > - Verified that source does not contain binaries
> > > > - Build source code successfully
> > > > - Run a simple sql query successfully
> > > >
> > > > Best,
> > > > Feng Jin
> > > >
> > > >
> > > > On Tue, Mar 12, 2024 at 11:09 AM Ron liu  wrote:
> > > >
> > > > > +1 (non binding)
> > > > >
> > > > > quickly verified:
> > > > > - verified that source distribution does not contain binaries
> > > > > - verified checksums
> > > > > - built source code successfully
> > > > >
> > > > >
> > > > > Best,
> > > > > Ron
> > > > >
> > > > > Jeyhun Karimov  于2024年3月12日周二 01:00写道：
> > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > - verified that source distribution does not contain binaries
> > > > > > - verified signatures and checksums
> > > > > > - built source code successfully
> > > > > >
> > > > > > Regards,
> > > > > > Jeyhun
> > > > > >
> > > > > >
> > > > > > On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb <
> decordea...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > +1 (non binding)
> > > > > > >
> > > > > > > - verified signatures and checksums
> > > > > > > - ASF headers are present in all expected file
> > > > > > > - No unexpected binaries files found in the source
> > > > > > > - Build successful locally
> > > > > > > - tested basic word count example
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Bests,
> > > > > > > Samrat
> > > > > > >
> > > > > > > On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy <
> > hamdy10...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Lincoln
> > > > > > > > +1 (non-binding) from me
> > > > > > > >
> > > > > > > > - Verified Checksums & Signatures
> > > > > > > > - Verified Source dists don't contain binaries
> > > > > > > > - Built source successfully
> > > > > > > > - reviewed web PR
> > > > > > > >
> > > > > > > >
> > > > > > > > Best Regards
> > > > > > > > Ahmed Hamdy
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, 11 Mar 2024 at 15:18, Lincoln Lee <
> > lincoln.8...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Robin,
> > > > > > > > >
> > > > > > > > > Thanks for helping verifying the release note[1],
> FLINK-14879
> > > > > should
> > > > > > > not
> > > > > > > > > have been included, after confirming this
> > > > > > > > > I moved all unresolved non-blocker issues left over from
> > 1.19.0
> > > > to
> > > > > > > 1.20.0
> > > > > > > > > and reconfigured the release note [1].
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Lincoln Lee
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353282
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Robin Moffatt  于2024年3月11日周一
> > > > 19:36写道：
> > > > > > > > >
> > > > > > > > > > Looking at the release notes [1] it lists `DESCRIBE
> > DATABASE`
> > > > > > > > > (FLINK-14879)
> > > > > > > > > > and `DESCRIBE CATALOG` (FLINK-14690).
> > > > > > > > > > When I try these in 1.19 RC2 the behaviour is as in
> 1.18.1,
> > > > i.e.
> > > > > it
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > supported:
> > > > > > > > > >
> > > > > > > > > > ```
> > > > > > > > > > [INFO] Execute statement succeed.
> > > > > > > > > >
> > > > > > > > > > Flink SQL> show catalogs;
> > > > > > > > > > +-+
> > > > > > > > > > |catalog name |
> > > > > > > > > > +-+
> > > > > > > > > > |   c_new |
> > > > > > > > > > | default_catalog |
> > > > > > > > > > +-+
> > > > > > > > > > 2 rows in set
> > > > > > > > > >
> > > > > > > > > > Flink SQL> DESCRIBE CATALOG c_new;
> > > > > > > > > > [ERROR] Could not execute

Re: [VOTE] Release 1.19.0, release candidate #2

[Rui Fan]

+1 (non-binding)

- Verified signature
- Verified checksum
- Built source code successfully
- Reviewed the release PR and some comments are updated

Best,
Rui

On Tue, Mar 12, 2024 at 4:08 PM Xuannan Su  wrote:

> +1 (non-binding)
>
> - Verified signature and checksum
> - Verified that source distribution does not contain binaries
> - Built from source code successfully
> - Reviewed the release announcement PR
>
> Best regards,
> Xuannan
>
> On Tue, Mar 12, 2024 at 2:18 PM Hang Ruan  wrote:
> >
> > +1 (non-binding)
> >
> > - Verified signatures and checksums
> > - Verified that source does not contain binaries
> > - Build source code successfully
> > - Reviewed the release note and left a comment
> >
> > Best,
> > Hang
> >
> > Feng Jin  于2024年3月12日周二 11:23写道：
> >
> > > +1 (non-binding)
> > >
> > > - Verified signatures and checksums
> > > - Verified that source does not contain binaries
> > > - Build source code successfully
> > > - Run a simple sql query successfully
> > >
> > > Best,
> > > Feng Jin
> > >
> > >
> > > On Tue, Mar 12, 2024 at 11:09 AM Ron liu  wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > quickly verified:
> > > > - verified that source distribution does not contain binaries
> > > > - verified checksums
> > > > - built source code successfully
> > > >
> > > >
> > > > Best,
> > > > Ron
> > > >
> > > > Jeyhun Karimov  于2024年3月12日周二 01:00写道：
> > > >
> > > > > +1 (non binding)
> > > > >
> > > > > - verified that source distribution does not contain binaries
> > > > > - verified signatures and checksums
> > > > > - built source code successfully
> > > > >
> > > > > Regards,
> > > > > Jeyhun
> > > > >
> > > > >
> > > > > On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb 
> > > > wrote:
> > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > - verified signatures and checksums
> > > > > > - ASF headers are present in all expected file
> > > > > > - No unexpected binaries files found in the source
> > > > > > - Build successful locally
> > > > > > - tested basic word count example
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Bests,
> > > > > > Samrat
> > > > > >
> > > > > > On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy <
> hamdy10...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Lincoln
> > > > > > > +1 (non-binding) from me
> > > > > > >
> > > > > > > - Verified Checksums & Signatures
> > > > > > > - Verified Source dists don't contain binaries
> > > > > > > - Built source successfully
> > > > > > > - reviewed web PR
> > > > > > >
> > > > > > >
> > > > > > > Best Regards
> > > > > > > Ahmed Hamdy
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 11 Mar 2024 at 15:18, Lincoln Lee <
> lincoln.8...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Robin,
> > > > > > > >
> > > > > > > > Thanks for helping verifying the release note[1], FLINK-14879
> > > > should
> > > > > > not
> > > > > > > > have been included, after confirming this
> > > > > > > > I moved all unresolved non-blocker issues left over from
> 1.19.0
> > > to
> > > > > > 1.20.0
> > > > > > > > and reconfigured the release note [1].
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lincoln Lee
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353282
> > > > > > > >
> > > > > > > >
> > > > > > > > Robin Moffatt  于2024年3月11日周一
> > > 19:36写道：
> > > > > > > >
> > > > > > > > > Looking at the release notes [1] it lists `DESCRIBE
> DATABASE`
> > > > > > > > (FLINK-14879)
> > > > > > > > > and `DESCRIBE CATALOG` (FLINK-14690).
> > > > > > > > > When I try these in 1.19 RC2 the behaviour is as in 1.18.1,
> > > i.e.
> > > > it
> > > > > > is
> > > > > > > > not
> > > > > > > > > supported:
> > > > > > > > >
> > > > > > > > > ```
> > > > > > > > > [INFO] Execute statement succeed.
> > > > > > > > >
> > > > > > > > > Flink SQL> show catalogs;
> > > > > > > > > +-+
> > > > > > > > > |catalog name |
> > > > > > > > > +-+
> > > > > > > > > |   c_new |
> > > > > > > > > | default_catalog |
> > > > > > > > > +-+
> > > > > > > > > 2 rows in set
> > > > > > > > >
> > > > > > > > > Flink SQL> DESCRIBE CATALOG c_new;
> > > > > > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > > > > > org.apache.calcite.sql.validate.SqlValidatorException:
> Column
> > > > > 'c_new'
> > > > > > > not
> > > > > > > > > found in any table
> > > > > > > > >
> > > > > > > > > Flink SQL> show databases;
> > > > > > > > > +--+
> > > > > > > > > |database name |
> > > > > > > > > +--+
> > > > > > > > > | default_database |
> > > > > > > > > +--+
> > > > > > > > > 1 row in set
> > > > > > > > >
> > > > > > > > > Flink SQL> DESCRIBE DATABASE default_database;
> > > > > > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > > > > > org.apache.calci

Re: [VOTE] Release 1.19.0, release candidate #2

[Jane Chan]

+1 (non-binding)

- Verify that the source distributions do not contain any binaries;
- Build the source distribution to ensure all source files have Apache
headers;
- Verify checksum and GPG signatures;

Best,
Jane

On Tue, Mar 12, 2024 at 4:08 PM Xuannan Su  wrote:

> +1 (non-binding)
>
> - Verified signature and checksum
> - Verified that source distribution does not contain binaries
> - Built from source code successfully
> - Reviewed the release announcement PR
>
> Best regards,
> Xuannan
>
> On Tue, Mar 12, 2024 at 2:18 PM Hang Ruan  wrote:
> >
> > +1 (non-binding)
> >
> > - Verified signatures and checksums
> > - Verified that source does not contain binaries
> > - Build source code successfully
> > - Reviewed the release note and left a comment
> >
> > Best,
> > Hang
> >
> > Feng Jin  于2024年3月12日周二 11:23写道：
> >
> > > +1 (non-binding)
> > >
> > > - Verified signatures and checksums
> > > - Verified that source does not contain binaries
> > > - Build source code successfully
> > > - Run a simple sql query successfully
> > >
> > > Best,
> > > Feng Jin
> > >
> > >
> > > On Tue, Mar 12, 2024 at 11:09 AM Ron liu  wrote:
> > >
> > > > +1 (non binding)
> > > >
> > > > quickly verified:
> > > > - verified that source distribution does not contain binaries
> > > > - verified checksums
> > > > - built source code successfully
> > > >
> > > >
> > > > Best,
> > > > Ron
> > > >
> > > > Jeyhun Karimov  于2024年3月12日周二 01:00写道：
> > > >
> > > > > +1 (non binding)
> > > > >
> > > > > - verified that source distribution does not contain binaries
> > > > > - verified signatures and checksums
> > > > > - built source code successfully
> > > > >
> > > > > Regards,
> > > > > Jeyhun
> > > > >
> > > > >
> > > > > On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb 
> > > > wrote:
> > > > >
> > > > > > +1 (non binding)
> > > > > >
> > > > > > - verified signatures and checksums
> > > > > > - ASF headers are present in all expected file
> > > > > > - No unexpected binaries files found in the source
> > > > > > - Build successful locally
> > > > > > - tested basic word count example
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Bests,
> > > > > > Samrat
> > > > > >
> > > > > > On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy <
> hamdy10...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Lincoln
> > > > > > > +1 (non-binding) from me
> > > > > > >
> > > > > > > - Verified Checksums & Signatures
> > > > > > > - Verified Source dists don't contain binaries
> > > > > > > - Built source successfully
> > > > > > > - reviewed web PR
> > > > > > >
> > > > > > >
> > > > > > > Best Regards
> > > > > > > Ahmed Hamdy
> > > > > > >
> > > > > > >
> > > > > > > On Mon, 11 Mar 2024 at 15:18, Lincoln Lee <
> lincoln.8...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Robin,
> > > > > > > >
> > > > > > > > Thanks for helping verifying the release note[1], FLINK-14879
> > > > should
> > > > > > not
> > > > > > > > have been included, after confirming this
> > > > > > > > I moved all unresolved non-blocker issues left over from
> 1.19.0
> > > to
> > > > > > 1.20.0
> > > > > > > > and reconfigured the release note [1].
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Lincoln Lee
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353282
> > > > > > > >
> > > > > > > >
> > > > > > > > Robin Moffatt  于2024年3月11日周一
> > > 19:36写道：
> > > > > > > >
> > > > > > > > > Looking at the release notes [1] it lists `DESCRIBE
> DATABASE`
> > > > > > > > (FLINK-14879)
> > > > > > > > > and `DESCRIBE CATALOG` (FLINK-14690).
> > > > > > > > > When I try these in 1.19 RC2 the behaviour is as in 1.18.1,
> > > i.e.
> > > > it
> > > > > > is
> > > > > > > > not
> > > > > > > > > supported:
> > > > > > > > >
> > > > > > > > > ```
> > > > > > > > > [INFO] Execute statement succeed.
> > > > > > > > >
> > > > > > > > > Flink SQL> show catalogs;
> > > > > > > > > +-+
> > > > > > > > > |catalog name |
> > > > > > > > > +-+
> > > > > > > > > |   c_new |
> > > > > > > > > | default_catalog |
> > > > > > > > > +-+
> > > > > > > > > 2 rows in set
> > > > > > > > >
> > > > > > > > > Flink SQL> DESCRIBE CATALOG c_new;
> > > > > > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > > > > > org.apache.calcite.sql.validate.SqlValidatorException:
> Column
> > > > > 'c_new'
> > > > > > > not
> > > > > > > > > found in any table
> > > > > > > > >
> > > > > > > > > Flink SQL> show databases;
> > > > > > > > > +--+
> > > > > > > > > |database name |
> > > > > > > > > +--+
> > > > > > > > > | default_database |
> > > > > > > > > +--+
> > > > > > > > > 1 row in set
> > > > > > > > >
> > > > > > > > > Flink SQL> DESCRIBE DATABASE default_database;
> > > > > > > > > [ERROR] Could not execute 

Re: [VOTE] Release 1.19.0, release candidate #2

[Xuannan Su]

+1 (non-binding)

- Verified signature and checksum
- Verified that source distribution does not contain binaries
- Built from source code successfully
- Reviewed the release announcement PR

Best regards,
Xuannan

On Tue, Mar 12, 2024 at 2:18 PM Hang Ruan  wrote:
>
> +1 (non-binding)
>
> - Verified signatures and checksums
> - Verified that source does not contain binaries
> - Build source code successfully
> - Reviewed the release note and left a comment
>
> Best,
> Hang
>
> Feng Jin  于2024年3月12日周二 11:23写道：
>
> > +1 (non-binding)
> >
> > - Verified signatures and checksums
> > - Verified that source does not contain binaries
> > - Build source code successfully
> > - Run a simple sql query successfully
> >
> > Best,
> > Feng Jin
> >
> >
> > On Tue, Mar 12, 2024 at 11:09 AM Ron liu  wrote:
> >
> > > +1 (non binding)
> > >
> > > quickly verified:
> > > - verified that source distribution does not contain binaries
> > > - verified checksums
> > > - built source code successfully
> > >
> > >
> > > Best,
> > > Ron
> > >
> > > Jeyhun Karimov  于2024年3月12日周二 01:00写道：
> > >
> > > > +1 (non binding)
> > > >
> > > > - verified that source distribution does not contain binaries
> > > > - verified signatures and checksums
> > > > - built source code successfully
> > > >
> > > > Regards,
> > > > Jeyhun
> > > >
> > > >
> > > > On Mon, Mar 11, 2024 at 3:08 PM Samrat Deb 
> > > wrote:
> > > >
> > > > > +1 (non binding)
> > > > >
> > > > > - verified signatures and checksums
> > > > > - ASF headers are present in all expected file
> > > > > - No unexpected binaries files found in the source
> > > > > - Build successful locally
> > > > > - tested basic word count example
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Bests,
> > > > > Samrat
> > > > >
> > > > > On Mon, 11 Mar 2024 at 7:33 PM, Ahmed Hamdy 
> > > > wrote:
> > > > >
> > > > > > Hi Lincoln
> > > > > > +1 (non-binding) from me
> > > > > >
> > > > > > - Verified Checksums & Signatures
> > > > > > - Verified Source dists don't contain binaries
> > > > > > - Built source successfully
> > > > > > - reviewed web PR
> > > > > >
> > > > > >
> > > > > > Best Regards
> > > > > > Ahmed Hamdy
> > > > > >
> > > > > >
> > > > > > On Mon, 11 Mar 2024 at 15:18, Lincoln Lee 
> > > > > wrote:
> > > > > >
> > > > > > > Hi Robin,
> > > > > > >
> > > > > > > Thanks for helping verifying the release note[1], FLINK-14879
> > > should
> > > > > not
> > > > > > > have been included, after confirming this
> > > > > > > I moved all unresolved non-blocker issues left over from 1.19.0
> > to
> > > > > 1.20.0
> > > > > > > and reconfigured the release note [1].
> > > > > > >
> > > > > > > Best,
> > > > > > > Lincoln Lee
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353282
> > > > > > >
> > > > > > >
> > > > > > > Robin Moffatt  于2024年3月11日周一
> > 19:36写道：
> > > > > > >
> > > > > > > > Looking at the release notes [1] it lists `DESCRIBE DATABASE`
> > > > > > > (FLINK-14879)
> > > > > > > > and `DESCRIBE CATALOG` (FLINK-14690).
> > > > > > > > When I try these in 1.19 RC2 the behaviour is as in 1.18.1,
> > i.e.
> > > it
> > > > > is
> > > > > > > not
> > > > > > > > supported:
> > > > > > > >
> > > > > > > > ```
> > > > > > > > [INFO] Execute statement succeed.
> > > > > > > >
> > > > > > > > Flink SQL> show catalogs;
> > > > > > > > +-+
> > > > > > > > |catalog name |
> > > > > > > > +-+
> > > > > > > > |   c_new |
> > > > > > > > | default_catalog |
> > > > > > > > +-+
> > > > > > > > 2 rows in set
> > > > > > > >
> > > > > > > > Flink SQL> DESCRIBE CATALOG c_new;
> > > > > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > > > > org.apache.calcite.sql.validate.SqlValidatorException: Column
> > > > 'c_new'
> > > > > > not
> > > > > > > > found in any table
> > > > > > > >
> > > > > > > > Flink SQL> show databases;
> > > > > > > > +--+
> > > > > > > > |database name |
> > > > > > > > +--+
> > > > > > > > | default_database |
> > > > > > > > +--+
> > > > > > > > 1 row in set
> > > > > > > >
> > > > > > > > Flink SQL> DESCRIBE DATABASE default_database;
> > > > > > > > [ERROR] Could not execute SQL statement. Reason:
> > > > > > > > org.apache.calcite.sql.validate.SqlValidatorException: Column
> > > > > > > > 'default_database' not found in
> > > > > > > > any table
> > > > > > > > ```
> > > > > > > >
> > > > > > > > Is this an error in the release notes, or my mistake in
> > > > interpreting
> > > > > > > them?
> > > > > > > >
> > > > > > > > thanks, Robin.
> > > > > > > >
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353282
> > > > > > > >
> > > > > > > > On Thu, 7 Mar 2024

Re: [DISCUSS] FLIP-434: Support optimizations for pre-partitioned data sources

[Jane Chan]

Hi Jeyhun,

Thank you for leading the discussion. I'm generally +1 with this proposal,
along with some questions. Please see my comments below.

1. Concerning the `sourcePartitions()` method, the partition information
returned during the optimization phase may not be the same as the partition
information during runtime execution. For long-running jobs, partitions may
be continuously created. Is this FLIP equipped to handle scenarios?

2. Regarding the `RemoveRedundantShuffleRule` optimization rule, I
understand that it is also necessary to verify whether the hash key within
the Exchange node is consistent with the partition key defined in the table
source that implements `SupportsPartitioning`.

3. Could you elaborate on the desired physical plan and integration with
`CompiledPlan` to enhance the overall functionality?

Best,
Jane

On Tue, Mar 12, 2024 at 11:11 AM Jim Hughes 
wrote:

> Hi Jeyhun,
>
> I like the idea!  Given FLIP-376[1], I wonder if it'd make sense to
> generalize FLIP-434 to be about "pre-divided" data to cover "buckets" and
> "partitions" (and maybe even situations where a data source is partitioned
> and bucketed).
>
> Separate from that, the page mentions TPC-H Q1 as an example.  For a join,
> any two tables joined on the same bucket key should provide a concrete
> example of a join.  Systems like Kafka Streams/ksqlDB call this
> "co-partitioning"; for those systems, it is a requirement placed on the
> input sources.  For Flink, with FLIP-434, the proposed planner rule
> could remove the shuffle.
>
> Definitely a fun idea; I look forward to hearing more!
>
> Cheers,
>
> Jim
>
>
> 1.
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-376%3A+Add+DISTRIBUTED+BY+clause
> 2.
>
> https://docs.ksqldb.io/en/latest/developer-guide/joins/partition-data/#co-partitioning-requirements
>
> On Sun, Mar 10, 2024 at 3:38 PM Jeyhun Karimov 
> wrote:
>
> > Hi devs,
> >
> > I’d like to start a discussion on FLIP-434: Support optimizations for
> > pre-partitioned data sources [1].
> >
> > The FLIP introduces taking advantage of pre-partitioned data sources for
> > SQL/Table API (it is already supported as experimental feature in
> > DataStream API [2]).
> >
> >
> > Please find more details in the FLIP wiki document [1].
> > Looking forward to your feedback.
> >
> > Regards,
> > Jeyhun
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-434%3A+Support+optimizations+for+pre-partitioned+data+sources
> > [2]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/experimental/
> >
>

[jira] [Created] (FLINK-34653) Support table merging with route in Flink CDC

[Qingsheng Ren (Jira)]

Qingsheng Ren created FLINK-34653:
-

 Summary: Support table merging with route in Flink CDC
 Key: FLINK-34653
 URL: https://issues.apache.org/jira/browse/FLINK-34653
 Project: Flink
  Issue Type: New Feature
  Components: Flink CDC
Reporter: Qingsheng Ren
Assignee: Qingsheng Ren
 Fix For: 3.1.0


Currently route in Flink CDC only supports very simple table id replacing. It 
should support more complex table merging strategies. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)