Re: [DISUCSS] Deprecate multiple APIs in 1.18

[Xintong Song]

Thanks all for the discussion.

It seems to me there's a consensus on marking the following as deprecated
in 1.18:
- DataSet API
- SourceFunction
- Queryable State
- All Scala APIs

More time is needed for deprecating SinkFunction.

I'll leave this discussion open for a few more days. And if there's no
objections, I'll create JIRA tickets accordingly.

Best,

Xintong



On Wed, Jul 5, 2023 at 1:34 PM Xintong Song  wrote:

> Thanks for the input, Jing. I'd also be +1 for option 1.
>
> Best,
>
> Xintong
>
>
>
> On Mon, Jul 3, 2023 at 7:20 PM Jing Ge  wrote:
>
>> Hi Xingtong,
>>
>> Option 1, secure plan would be:
>>
>> 1. graduate kafka, File, JDBC connectors to @Public
>> 2. graduate SinkV2 to @Public
>> 3. remove SinkFunction.
>>
>> Option 2, risky plan but at a fast pace:
>>
>> 1. graduate SinkV2 to @Public and expecting more maintenance effort since
>> there are many known and unsolved issues.
>> 2. remove SinkFunction.
>> 3. It depends on the connectors' contributors whether connectors can
>> upgrade to Flink 2.0, since we moved forward with SinkV2 API without
>> taking
>> care of implementations in external connectors.
>>
>> I am ok with both of them and personally prefer option 1.
>>
>> Best regards,
>> Jing
>>
>>
>> On Fri, Jun 30, 2023 at 3:41 AM Xintong Song 
>> wrote:
>>
>> > I see. Thanks for the explanation. I may have not looked into this
>> deeply
>> > enough, and would trust the decision from you and the community members
>> who
>> > participated in the discussion & vote.
>> >
>> > Best,
>> >
>> > Xintong
>> >
>> >
>> >
>> > On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov <
>> > alexander.fedu...@gmail.com> wrote:
>> >
>> > > > However, I'm not sure about 2.
>> > >
>> > > I am not aware of a bylaw that states the specific requirements in
>> order
>> > to
>> > > mark something as @Deprecated. My understanding from the discussion
>> and
>> > the
>> > > vote was that the community recognizes the necessity to make it
>> explicit
>> > > that
>> > > the usage of the SourceFunction API is discouraged. This can actually
>> > > stimulate
>> > > authors of connectors that rely on this very specific and non-baseline
>> > > functionality to contribute extensions to the new Source API
>> themselves
>> > in
>> > > order to
>> > > close the gap. ExternallyInducedSource, for instance, was driven by
>> > Pravega
>> > > to
>> > > begin with, since it was only needed for their purposes [1]. We are
>> not
>> > > removing
>> > > anything - until 2.0 everything will continue to work and we can work
>> on
>> > > resolving the limitations until then, I personally don't see a big
>> issue
>> > > here.
>> > >
>> > > >Do you think it is feasible to resolve them by the feature freeze
>> date
>> > of
>> > > 1.18?
>> > > No, these are rather complex additions that would probably require
>> > FLIP(s).
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration
>> > >
>> > > On Thu, 29 Jun 2023 at 14:25, Xintong Song 
>> > wrote:
>> > >
>> > > > Thanks for the explanation, Alex.
>> > > >
>> > > > Not blocking the deprecation on 1 & 3 makes sense to me. However,
>> I'm
>> > not
>> > > > sure about 2.
>> > > >
>> > > > It sounds to me that, without FLINK-28051 & FLINK-28054, some of the
>> > > > connectors cannot migrate to the new Source API, or at least further
>> > > > investigation is needed to understand the situation. If this is the
>> > case,
>> > > > we probably should not deprecate the API until these issues are
>> > resolved.
>> > > > Do you think it is feasible to resolve them by the feature freeze
>> date
>> > of
>> > > > 1.18?
>> > > >
>> > > > Best,
>> > > >
>> > > > Xintong
>> > > >
>> > > >
>> > > >
>> > > > On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov <
>> > > > alexander.fedu...@gmail.com> wrote:
>> > > >
>> > > > > @Xintong
>> > > > > The original discussion [1] and vote [2] converged on the idea
>> that
>> > it
>> > > is
>> > > > > better
>> > > > > to make it clear to the users that they should stop using
>> > > SourceFunction
>> > > > > since it
>> > > > > is going away. The longer we do not have this indication, the more
>> > user
>> > > > > implementations will be based on it and the more pain will be
>> induced
>> > > > when
>> > > > > we
>> > > > > finally drop it. Users now have an alternative API that they
>> should
>> > use
>> > > > and
>> > > > > which
>> > > > > is fully functional, from that perspective nothing blocks marking
>> it
>> > > > > @Deprecated.
>> > > > > As for the remaining work items - there are primarily three kinds:
>> > > > >
>> > > > > 1. Where Flink internally uses SourceFunction, without exposing
>> this
>> > > fact
>> > > > > to the
>> > > > >outside world:
>> > > > >- FLINK-28050 [3]
>> > > > >- FLINK-28229 [4]
>> > > > >- FLINK-28048 [5]
>> > > > >
>> > > > > 2. Very specific edge cases that might not be covered by the
>> Source
>> > API
>> > > > as
>> > > > > is:
>> > > > >- 

Re: [DISUCSS] Deprecate multiple APIs in 1.18

[Xintong Song]

Thanks for the input, Jing. I'd also be +1 for option 1.

Best,

Xintong



On Mon, Jul 3, 2023 at 7:20 PM Jing Ge  wrote:

> Hi Xingtong,
>
> Option 1, secure plan would be:
>
> 1. graduate kafka, File, JDBC connectors to @Public
> 2. graduate SinkV2 to @Public
> 3. remove SinkFunction.
>
> Option 2, risky plan but at a fast pace:
>
> 1. graduate SinkV2 to @Public and expecting more maintenance effort since
> there are many known and unsolved issues.
> 2. remove SinkFunction.
> 3. It depends on the connectors' contributors whether connectors can
> upgrade to Flink 2.0, since we moved forward with SinkV2 API without taking
> care of implementations in external connectors.
>
> I am ok with both of them and personally prefer option 1.
>
> Best regards,
> Jing
>
>
> On Fri, Jun 30, 2023 at 3:41 AM Xintong Song 
> wrote:
>
> > I see. Thanks for the explanation. I may have not looked into this deeply
> > enough, and would trust the decision from you and the community members
> who
> > participated in the discussion & vote.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov <
> > alexander.fedu...@gmail.com> wrote:
> >
> > > > However, I'm not sure about 2.
> > >
> > > I am not aware of a bylaw that states the specific requirements in
> order
> > to
> > > mark something as @Deprecated. My understanding from the discussion and
> > the
> > > vote was that the community recognizes the necessity to make it
> explicit
> > > that
> > > the usage of the SourceFunction API is discouraged. This can actually
> > > stimulate
> > > authors of connectors that rely on this very specific and non-baseline
> > > functionality to contribute extensions to the new Source API themselves
> > in
> > > order to
> > > close the gap. ExternallyInducedSource, for instance, was driven by
> > Pravega
> > > to
> > > begin with, since it was only needed for their purposes [1]. We are not
> > > removing
> > > anything - until 2.0 everything will continue to work and we can work
> on
> > > resolving the limitations until then, I personally don't see a big
> issue
> > > here.
> > >
> > > >Do you think it is feasible to resolve them by the feature freeze date
> > of
> > > 1.18?
> > > No, these are rather complex additions that would probably require
> > FLIP(s).
> > >
> > > [1]
> > >
> > >
> >
> https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration
> > >
> > > On Thu, 29 Jun 2023 at 14:25, Xintong Song 
> > wrote:
> > >
> > > > Thanks for the explanation, Alex.
> > > >
> > > > Not blocking the deprecation on 1 & 3 makes sense to me. However, I'm
> > not
> > > > sure about 2.
> > > >
> > > > It sounds to me that, without FLINK-28051 & FLINK-28054, some of the
> > > > connectors cannot migrate to the new Source API, or at least further
> > > > investigation is needed to understand the situation. If this is the
> > case,
> > > > we probably should not deprecate the API until these issues are
> > resolved.
> > > > Do you think it is feasible to resolve them by the feature freeze
> date
> > of
> > > > 1.18?
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov <
> > > > alexander.fedu...@gmail.com> wrote:
> > > >
> > > > > @Xintong
> > > > > The original discussion [1] and vote [2] converged on the idea that
> > it
> > > is
> > > > > better
> > > > > to make it clear to the users that they should stop using
> > > SourceFunction
> > > > > since it
> > > > > is going away. The longer we do not have this indication, the more
> > user
> > > > > implementations will be based on it and the more pain will be
> induced
> > > > when
> > > > > we
> > > > > finally drop it. Users now have an alternative API that they should
> > use
> > > > and
> > > > > which
> > > > > is fully functional, from that perspective nothing blocks marking
> it
> > > > > @Deprecated.
> > > > > As for the remaining work items - there are primarily three kinds:
> > > > >
> > > > > 1. Where Flink internally uses SourceFunction, without exposing
> this
> > > fact
> > > > > to the
> > > > >outside world:
> > > > >- FLINK-28050 [3]
> > > > >- FLINK-28229 [4]
> > > > >- FLINK-28048 [5]
> > > > >
> > > > > 2. Very specific edge cases that might not be covered by the Source
> > API
> > > > as
> > > > > is:
> > > > >- FLINK-28054 [6]
> > > > >- FLINK-28051 [7]
> > > > >
> > > > > 3. Usability improvements - something that was easily doable with
> > > > > SourceFunction,
> > > > >but requires deep knowledge of the new, significantly more
> > complex,
> > > > > Source API
> > > > >to achieve:
> > > > >- FLINK-28056 [8]
> > > > >
> > > > > In my mind, none of those are blockers for proceeding with adding
> the
> > > > > @Deprecated
> > > > > annotation:
> > > > > (1) is a simple case of encapsulation, internals should not concern
> > the
> > > > API
> > > > > users
> > > > > (2) is really only relevant for 

[jira] [Created] (FLINK-32539) Archunit violations started to fail in test_misc

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-32539:
---

 Summary: Archunit violations started to fail in test_misc
 Key: FLINK-32539
 URL: https://issues.apache.org/jira/browse/FLINK-32539
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin
Assignee: Sergey Nuyanzin


blocker since now it fails on every build
to reproduce jdk 8 is required
{noformat}
mvn clean install -DskipTests
mvn verify -pl flink-architecture-tests/flink-architecture-tests-production/ 
-Darchunit.freeze.store.default.allowStoreUpdate=false
{code}
It seems the reason is FLINK-27415
where it was removed line 
{code:java}
checkArgument(fileLength > 0L);
{code}
at the same time it was mentioned in achunit violations  and now should be 
removed as well



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

[宇航 李]

Hi Aitozi,

I think it is necessary to add the following description in FLIP to express the 
difference between user-defined asynchronous table function and 
AsyncTableFunction:

User-defined asynchronous table functions allow complex parameters (e.g., Row 
type) to be passed to function, which is important in RPC, rather than using 
‘join … on ...'. 

Thanks,
Awake.


On 2023/06/26 02:31:59 Aitozi wrote:
> Hi Lincoln,
> Thanks for your confirmation. I have updated the consensus to the FLIP
> doc.
> If there are no other comments, I'd like to restart the vote process in [1]
> today.
> 
> https://lists.apache.org/thread/7g5n2vshosom2dj9bp7x4n01okrnx4xx
> 
> Thanks,
> Aitozi.
> 
> Lincoln Lee  于2023年6月21日周三 22:29写道：
> 
> > Hi Aitozi,
> >
> > Thanks for your updates!
> >
> > By the design of hints, the hints after select clause belong to the query
> > hints category, and this new hint is also a kind of join hints[1].
> > Join table function is one of the join type defined by flink sql joins[2],
> > all existing join hints[1] omit the 'join' keyword,
> > so I would prefer the 'ASYNC_TABLE_FUNC' (which is actually the one for
> > 'ASYNC_TABLE_FUNC_JOIN').
> >
> > Since a short Chinese holiday is coming, I suggest waiting for other
> > people's responses before continuing to vote (next monday?)
> >
> > Btw, I discussed with @fudian offline about pyflink support, there should
> > be no known issues, so you can create a subtask with pyflink support after
> > the vote passed.
> >
> > [1]
> >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/hints/#join-hints
> > [2]
> >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Aitozi  于2023年6月18日周日 21:18写道：
> >
> > > Hi all,
> > > Sorry for the late reply, I have a discussion with Lincoln offline,
> > > mainly about
> > > the naming of the hints option. Thanks Lincoln for the valuable
> > > suggestions.
> > >
> > > Let me answer the last email inline.
> > >
> > > >For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call
> > as
> > > an example?
> > >
> > > Sure, will give an example when adding the doc of async udtf and will
> > > update the FLIP simultaneously
> > >
> > > >For the name of this query hint, 'LATERAL' (include its internal
> > options)
> > > don't show any relevance to async, but I haven't thought of a suitable
> > name
> > > at the moment,
> > >
> > > After some discussion with Lincoln, We prefer to choose one of the
> > > `ASYNC_TABLE_FUNC` and `ASYNC_LATERAL`.
> > > Besides, In my opinion the keyword `lateral`'s use scenario is wider than
> > > the table function join, but in this case we only want to config
> > > the async table function, So I'm a bit more lean to the
> > `ASYNC_TABLE_FUNC`.
> > > Looking forward to some inputs if you guys have
> > > some better suggestion on the naming.
> > >
> > > For the usage of the hints config option, I have updated the section
> > > of ConfigOption, you can refer to the FLIP
> > > for more details.
> > >
> > > >Also, the terms 'correlate join' and 'lateral join' are not the same as
> > in
> > > the current joins page[1], so maybe it would be better if we unified them
> > > into  'join table function'
> > >
> > > Yes, we should unified to the 'join table function', updated.
> > >
> > > Best,
> > > Aitozi
> > >
> > > Lincoln Lee  于2023年6月15日周四 09:15写道：
> > >
> > > > Hi Aitozi,
> > > >
> > > > Thanks for your reply!  Gives sql users more flexibility to get
> > > > asynchronous processing capabilities via lateral join table function +1
> > > for
> > > > this
> > > >
> > > > For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call
> > > as
> > > > an example?
> > > >
> > > > For the name of this query hint, 'LATERAL' (include its internal
> > options)
> > > > don't show any relevance to async, but I haven't thought of a suitable
> > > name
> > > > at the moment,
> > > > maybe we need to highlight the async keyword directly, we can also see
> > if
> > > > others have better candidates
> > > >
> > > > For the hint option "timeout = '180s'" should be "'timeout' = '180s'",
> > > > seems a typo in the flip. And use upper case for all keywords in sql
> > > > examples.
> > > > Also, the terms 'correlate join' and 'lateral join' are not the same as
> > > in
> > > > the current joins page[1], so maybe it would be better if we unified
> > them
> > > > into  'join table function'
> > > >
> > > > [1]
> > > >
> > > >
> > >
> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sql/queries/joins/#table-function
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Aitozi  于2023年6月14日周三 16:11写道：
> > > >
> > > > > Hi Lincoln
> > > > >
> > > > > Very thanks for your valuable question. I will try to answer your
> > > > > questions inline.
> > > > >
> > > > > >Does the async udtf bring any additional benefits besides a
> > > > > lighter implementation？

[jira] [Created] (FLINK-32538) CI build failed because node is corrupted when compiling

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32538:
-

 Summary: CI build failed because node is corrupted when compiling
 Key: FLINK-32538
 URL: https://issues.apache.org/jira/browse/FLINK-32538
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI, Tests
Affects Versions: 1.18.0
Reporter: Yuxin Tan


[ERROR] The archive file 
/__w/3/.m2/repository/com/github/eirslett/node/16.13.2/node-16.13.2-linux-x64.tar.gz
 is corrupted and will be deleted. Please try the build again. 
 
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50896=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=10984]

[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50919=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=10984]



[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50925=logs=52b61abe-a3cc-5bde-cc35-1bbe89bb7df5=54421a62-0c80-5aad-3319-094ff69180bb=10984]

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Release 2.0 Work Items

[Xintong Song]

Hi ConradJam,

I think Chesnay has already put his name as the Contributor for the two
tasks you listed. Maybe you can reach out to him to see if you can
collaborate on this.

In general, I don't think contributing to a release 2.0 issue is much
different from contributing to a regular issue. We haven't yet created JIRA
tickets for all the listed tasks because many of them needs further
discussions and / or FLIPs to decide whether and how they should be
performed.

Best,

Xintong



On Mon, Jul 3, 2023 at 10:37 PM ConradJam  wrote:

> Hi Community:
>   I see some tasks in the 2.0 list that haven't been assigned yet. I want
> to take the initiative to take on some tasks that I can complete. How do I
> apply to the community for this part of the task? I am interested in the
> following parts of FLINK-32377
> , do I need to create
> issuse myself and point it to myself?
>
> - the current timestamp, which is problematic w.r.t. caching and testing,
> while providing no value.
> - Remove JarRequestBody#programArgs in favor of #programArgsList.
>
> [1] FLINK-32377 
> https://issues.apache.org/jira/browse/FLINK-32377
>
> Teoh, Hong  于2023年6月30日周五 00:53写道：
>
>
> Teoh, Hong  于2023年6月30日周五 00:53写道：
>
> > Thanks Xintong for driving the effort.
> >
> > I’d add a +1 to reworking configs, as suggested by @Jark and @Chesnay,
> > especially the types. We have various configs that encode Time /
> MemorySize
> > that are Long instead!
> >
> > Regards,
> > Hong
> >
> >
> >
> > > On 29 Jun 2023, at 16:19, Yuan Mei  wrote:
> > >
> > > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> > >
> > >
> > >
> > > Thanks for driving this effort, Xintong!
> > >
> > > To Chesnay
> > >> I'm curious as to why the "Disaggregated State Management" item is
> > >> marked as a must-have; will it require changes that break something?
> > >> What prevents it from being added in 2.1?
> > >
> > > As to "Disaggregated State Management".
> > >
> > > We plan to provide a new type of state backend to support DFS as
> primary
> > > storage.
> > > To achieve this, we at least need to include two parts of amends (not
> > > entirely sure yet, since we are still in the designing and prototype
> > phase)
> > >
> > > 1. Statebackend Change
> > > 2. State Access Change
> > >
> > > Not all of the interfaces related are `@Internal`. Some of the
> interfaces
> > > like `StateBackend` is `@PublicEvolving`
> > > So, you are right in the sense that "Disaggregated State Management"
> > itself
> > > probably does not need to be a "Must Have"
> > >
> > > But I was hoping changes that related to public APIs can be finalized
> and
> > > merged in Flink 2.0 (I will fix the wiki accordingly).
> > >
> > > I also agree with Jark that 2.0 is a good chance to rework the default
> > > value of configurations.
> > >
> > > Best
> > > Yuan
> > >
> > >
> > > On Thu, Jun 29, 2023 at 8:43 PM Chesnay Schepler 
> > wrote:
> > >
> > >> Something else configuration-related is that there are a bunch of
> > >> options where the type isn't quite correct (e.g., a String where it
> > >> could be an enum, a string where it should be an int or something).
> > >> Could do a pass over those as well.
> > >>
> > >> On 29/06/2023 13:50, Jark Wu wrote:
> > >>> Hi,
> > >>>
> > >>> I think one more thing we need to consider to do in 2.0 is changing
> the
> > >>> default value of configuration to improve out-of-box user experience.
> > >>>
> > >>> Currently, in order to run a Flink job, users may need to set
> > >>> a bunch of configurations, such as minibatch, checkpoint interval,
> > >>> exactly-once,
> > >>> incremental-checkpoint, etc. It's very verbose and hard to use for
> > >>> beginners.
> > >>> Most of them can have a universally applicable value.  Because
> changing
> > >> the
> > >>> default value is a breaking change. I think It's worth considering
> > >> changing
> > >>> them in 2.0.
> > >>>
> > >>> What do you think?
> > >>>
> > >>> Best,
> > >>> Jark
> > >>>
> > >>>
> > >>> On Wed, 28 Jun 2023 at 14:10, Sergey Nuyanzin 
> > >> wrote:
> > >>>
> >  Hi Chesnay
> > 
> > > "Move Calcite rules from Scala to Java": I would hope that this
> would
> > >> be
> > > an entirely internal change, and could thus be an incremental
> process
> > > independent of major releases.
> > > What is the actual scale of this item; how much are we actually
> >  re-writing?
> > 
> >  Thanks for asking
> >  yes, you're right, that should be internal change.
> >  Yeah I was also thinking about incremental change (rule by rule or
> >  reasonable small group of rules).
> >  And yes, this could be an independent (on major release) activity
> > 
> >  The problem is actually for children of RelOptRule.
> >  Currently I see 60+ such 

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

[Dong Lin]

Hi Piotr,

Any suggestion on how we can practically move forward to address the target
use-case?

My understanding is that the current proposal does not have any
correctness/performance issues. And it allows the extension to support all
the extra use-case without having to throw away the proposed APIs.

If you prefer to have a better solution with simpler APIs and yet same or
better correctness/performance for the target use-case, could you please
kindly explain its API design so that we can continue the discussion?


Best,
Dong

On Mon, Jul 3, 2023 at 6:39 PM Dong Lin  wrote:

> Hi Piotr,
>
> Please see my comments inline.
>
> On Mon, Jul 3, 2023 at 5:19 PM Piotr Nowojski 
> wrote:
>
>> Hi Dong,
>>
>> Starting from the end:
>>
>> > It seems that the only benefit of this approach is to avoid"
>> > adding SplitEnumeratorContext#setIsProcessingBacklog."
>>
>> Yes, that's the major benefit of this counter-proposal.
>>
>> > In the target use-case, user still want to do checkpoint (though at a"
>> > larger interval) when there is backlog. And HybridSource need to know
>> the"
>> > expected checkpoint interval during backlog in order to determine
>> whether"
>> > it should keep throwing CheckpointException. Thus, we still need to add"
>> > execution.checkpointing.interval-during-backlog for user to specify
>> this"
>> > information."
>> >
>> > The downside of this approach is that it is hard to enforce the"
>> > semantics specified by execution.checkpointing.interval-during-backlog.
>> For"
>> > example, suppose execution.checkpointing.interval =3 minute and"
>> > execution.checkpointing.interval-during-backlog = 7 minutes. During the"
>> > backlog phase, checkpoint coordinator will still trigger the checkpoint"
>> > once every 3 minutes. HybridSource will need to reject 2 out of the 3"
>> > checkpoint invocation, and the effective checkpoint interval will be 9"
>> > minutes."
>>
>> Does it really matter what's the exact value of the longer interval? Can
>> not we
>> hard-code it to be 5x or 10x of the base checkpoint interval? If there is
>> a
>> notice
>> able overhead from the base interval slowing down records processing rate,
>> reducing this interval by a factor of 5x or 10x, would fix performance
>> issue for
>> vast majority of users. So a source could just skip 4 out of 5 or 9 out of
>> 10
>> checkpoints.
>>
>
> Yes, I think the exact value of the longer interval matters.
>
> The main reason we need two intervals is for jobs which have two-phase
> commit sink. The short interval typically represents the interval that a
> user can accept for the two-phase commit sink to buffer data (since it can
> only emit data when checkpoint is triggered). And the long interval
> typically represents the maximum amount of duplicate work (in terms of
> time) that a job need to re-do after failover.
>
> Since there is no intrinsic relationship between the data buffer interval
> (related to processing latency) and the failover boundary, I don't think we
> can hardcode it to be 5x or 10x of the base checkpoint interval.
>
>
>> Alternatively we could introduce a config option like:
>>
>> execution.checkpointing.long-interval
>>
>> that might be re-used in the future, with more fancy algorithms, but I
>> don't see
>> much value in doing that.
>
>
>> > Overall, I think the solution is a bit hacky. I think it is preferred
>> to"
>> > throw exception only when there is indeed error. If we don't need to
>> check"
>> > a checkpoint, it is preferred to not trigger the checkpoint in the
>> first"
>> > place. And I think adding SplitEnumeratorContext#setIsProcessingBacklog
>> is"
>> > probably not that much of a big deal."
>>
>> Yes it's hacky, but at least it doesn't require extending the Public API
>> for a
>> quite limited solution, that only targets one or two sources that are
>> rarely used.
>>
>
> I am not sure it is fair to say MySQL CDC source is "rarely used".
> ververica/flink-cdc-connectors GitHub repo has 4K + starts. Also, note that
> the proposed feature can be useful for CDC sources with an internal
> "backlog phase". Its usage is not limited to just the two sources mentioned
> in the FLIP.
>
>
>>
>> 
>>
>> About the idea of emitting "RecordAttributes(isBacklog=..)". I have a
>> feeling that
>> this is overly complicated and would require every operator/function to
>> handle that.
>>
>> Yes it would cover even more use cases, at the cost of complicating the
>> system by
>> a lot. IMO it looks like something we could do if there would indeed by a
>> high
>> demand of such a feature, after we provide some baseline generic solution,
>> that
>> doesn't require any configuration.
>>
>> I have a feeling that by just statically looking at the shape of the job
>> graph and how
>> it is connected, we could deduce almost the same things.
>>
>
> Note that pretty much every FLIP will address my use-case at the cost
> of complicating the system. I understand you have the feeling that this
> complexity is not 

Re: [DISCUSS] Deprecate SourceFunction APIs

[Flavio Pompermaier]

Hi all,
I've tried to migrate my very simple Elasticsearch SourceFunction  (that
use scroll API and produce batch of documents) to new Source API, but I
gave up because it's too complicated. It should much simpler to migrate
that function to a bounded or unbounded source.
Before removing completely SourceFunction and Dataset I think it would be
better to provide a more detailed migration guide.
At least simplify the creation of a bounded Dataset...I still didn't give a
look at DataGeneratorSource though.
A review of the current online documentation is mandatory IMO.

Best,
Flavio


On Mon, Jul 3, 2023 at 5:58 PM Alexander Fedulov <
alexander.fedu...@gmail.com> wrote:

> I am happy to announce that the blocker has been resolved and
> SourceFunction
> is now marked as @Deprecated [1].
>
> The work continues to remove the dependencies on the SourceFunction API in
> Flink internals in order to prepare for dropping it completely in Flink
> 2.0.
>
> I'd like to get some opinions on an open question I currently have:
> StreamExecutionEnvironment#fromCollection() methods need to be modified to
> use
> the new FLIP-27 DataGeneratorSource [2]. This presents an issue because
> ITCases in DataGeneratorSource rely on StreamExecutionEnvironment, so we
> end
> up with a circular dependency.
>
> I see two main options here:
>
> 1. Split the tests from the DataGeneratorSource into a separate module
> called
>flink-connector-datagen-tests
>This is a rather straightforward solution that breaks the cycle, but so
> far
>we managed to avoid such workarounds and I'd like to know if anyone has
> a
>strong opinion against it
>
> 2. Move #fromCollection() methods into flink-connector-datagen, so
>StreamExecutionEnvironment#fromCollection() becomes
>DataGeneratorSource#fromCollection()
>While this deviates from the familiar pattern, it should be acceptable
> given
>the major version change.The key question here is whether we should also
>introduce a dependency from flink-connector-datagen to
> flink-streaming-java.
>This dependency does not exist in other connectors, but it would enhance
>usability. Without it, the user code would look somewhat like
>this:
>
>Collection data = ...;
>DataGeneratorSource collectionSource =
>DataGeneratorSource.fromCollection(data);
>DataStreamSource source = env.fromSource(collectionSource,
>WatermarkStrategy.forMonotonousTimestamps(), "Collection source")
>.forceNonParallel();
>
>Especially the necessity for the forceNonParallel()/setParallelism(1)
> call is
>   concerning because it is easy to forget.
>
>With the dependency, we can hide the internal details and achieve an API
>   closer to the current #fromCollection() implementation:
>
>Collection data = ...;
>DataStreamSource source =
>DataGeneratorSource.fromCollection(env, data);
>
> I would appreciate hearing your thoughts and suggestions on this matter.
>
> [1] https://github.com/apache/flink/pull/20049
> [2] https://github.com/apache/flink/pull/22850
>
> Best,
> Alex
>
>
>
>
> On Wed, 21 Jun 2023 at 19:27, Alexander Fedulov <
> alexander.fedu...@gmail.com>
> wrote:
>
> > I'd like to revive the efforts to deprecate the SourceFunction API.
> >
> > It would be great to get a review for this PR:
> > https://github.com/apache/flink/pull/21774
> >
> > It immediately unblocks marking the actual SourceFunction as deprecated.
> > https://github.com/apache/flink/pull/20049
> >
> > There is also this work thread related
> > to StreamExecutionEnvironment#fromCollection() methods.
> > The discussion seem to have stalled:
> > https://github.com/apache/flink/pull/21028
> >
> > Thanks,
> > Alex
> >
> > On 2022/06/15 19:30:31 Alexander Fedulov wrote:
> > > Thank you all for your valuable input and participation in the
> discussion
> > >
> > > The vote is open now [1]
> > >
> > > [1] https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v
> > >
> > > Best,
> > > Alexander Fedulov
> >
> >

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

[Jing Ge]

Hi Xuannan, Hi Dong

Thanks for the Proposal! After reading the FLIP, I'd like to ask some
questions:

1. Naming convention for boolean variables. It is recommended to follow
JavaBean [1], i.e. objectReuseCompliant as the variable name with
isObjectReuseCompliant() and setObjectReuseCompliant() as the methods' name.


2.

   -

   *If pipeline.object-reuse is set to true, records emitted by this
   operator will be re-used.*
   -

   *Otherwise, if getIsObjectReuseCompliant() returns true, records emitted
   by this operator will be re-used.*
   -

   *Otherwise, records emitted by this operator will be deep-copied before
   being given to the next operator in the chain.*


If I understand you correctly,  the hard coding objectReusedCompliant
should have higher priority over the configuration, the checking logic
should be:

   -

   *If getIsObjectReuseCompliant() returns true, records emitted by this
   operator will be re-used.*
   -

   *Otherwise, if pipeline.object-reuse is set to true, records emitted by
   this operator will be re-used.*
   -

   *Otherwise, records emitted by this operator will be deep-copied before
   being given to the next operator in the chain.*


The results are the same but the checking logics are different, but there
are some additional thoughts, which lead us to the next question.

3. Current design lets specific operators enable object reuse and ignore
the global config. There could be another thought, on the contrary: if an
operator has hard coded the objectReuseCompliant as false, i.e. disable
object reuse on purpose, records should not be reused even if the global
config pipeline.object-reused is set to true, which turns out that the
objectReuseCompliant could be a triple value logic: ture: force object
reusing; false: force deep-copying; unknown: depends on
pipeline.object-reuse config.


Best regards,
Jing


[1] https://en.wikipedia.org/wiki/JavaBeans

On Mon, Jul 3, 2023 at 4:25 AM Xuannan Su  wrote:

> Hi all,
>
> Dong(cc'ed) and I are opening this thread to discuss our proposal to
> add operator attribute to allow operator to specify support for
> object-reuse [1].
>
> Currently, the default configuration for pipeline.object-reuse is set
> to false to avoid data corruption, which can result in suboptimal
> performance. We propose adding APIs that operators can utilize to
> inform the Flink runtime whether it is safe to reuse the emitted
> records. This enhancement would enable Flink to maximize its
> performance using the default configuration.
>
> Please refer to the FLIP document for more details about the proposed
> design and implementation. We welcome any feedback and opinions on
> this proposal.
>
> Best regards,
>
> Dong and Xuannan
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255073749
>

Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems

[Ran Tao]

Thanks Dian Fu for sharing the externalized connector usage within
flink-python.

Looking forward to the solution to this problem, and if possible, can
participate in the contribution of this problem,
and thank you Martijn for the dependency usage that the externalized
connector should follow.

Best Regards,
Ran Tao
https://github.com/chucheng92


Dian Fu  于2023年7月4日周二 19:37写道：

> Thanks Ran Tao for proposing this discussion and Martijn for sharing
> the thought.
>
> >  While flink-python now fails the CI, it shouldn't actually depend on the
> externalized connectors. I'm not sure what PyFlink does with it, but if
> belongs to the connector code,
>
> For each DataStream connector, there is a corresponding Python wrapper
> and also some test cases in PyFlink. In theory, we should move that
> wrapper into each connector repository. In the past, we have not done
> that when externalizing the connectors since it may introduce some
> burden when releasing since it means that we have to publish each
> connector to PyPI separately.
>
> To resolve this problem, I guess we can move the connector support in
> PyFlink into the external connector repository.
>
> Regards,
> Dian
>
>
> On Mon, Jul 3, 2023 at 11:08 PM Ran Tao  wrote:
> >
> > @Martijn
> > thanks for clear explanations.
> >
> > If we follow the line you specified (Connectors shouldn't rely on
> > dependencies that may or may not be
> > available in Flink itself)
> > It seems that we should add a certain dependency if we need(such as
> > commons-io, commons-collection) in connector pom explicitly.
> > And bundle it in sql-connector uber jar.
> >
> > Then there is only one thing left that we need to make flink-python test
> > not depend on the released flink-connector.
> > Maybe we should check it out and decouple it like you suggested.
> >
> > Best Regards,
> > Ran Tao
> > https://github.com/chucheng92
> >
> >
> > Martijn Visser  于2023年7月3日周一 22:06写道：
> >
> > > Hi Ran Tao,
> > >
> > > Thanks for opening this topic. I think there's a couple of things at
> hand:
> > > 1. Connectors shouldn't rely on dependencies that may or may not be
> > > available in Flink itself, like we've seen with flink-shaded. That
> avoids a
> > > tight coupling between Flink and connectors, which is exactly what we
> try
> > > to avoid.
> > > 2. When following that line, that would also be applicable for things
> like
> > > commons-collections and commons-io. If a connector wants to use them,
> it
> > > should make sure that it bundles those artifacts itself.
> > > 3. While flink-python now fails the CI, it shouldn't actually depend
> on the
> > > externalized connectors. I'm not sure what PyFlink does with it, but if
> > > belongs to the connector code, that code should also be moved to the
> > > individual connector repo. If it's just a generic test, we could
> consider
> > > creating a generic test against released connector versions to
> determine
> > > compatibility.
> > >
> > > I'm curious about the opinions of others as well.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Mon, Jul 3, 2023 at 3:37 PM Ran Tao  wrote:
> > >
> > > > I have an issue here that needs to upgrade commons-collections[1]
> (this
> > > is
> > > > an example), but PR ci fails because flink-python test cases depend
> on
> > > > flink-sql-connector-kafka, but kafka-sql-connector is a small jar,
> does
> > > not
> > > > include this dependency, so flink ci cause exception[2]. Current my
> > > > solution is [3]. But even if this PR is done, the upgrade of flink
> still
> > > > requires kafka-connector released.
> > > >
> > > > This issue leads to deeper problems. Although the connectors have
> been
> > > > externalized, many UTs of flink-python depend on these connectors,
> and a
> > > > basic agreement of externalized connectors is that other dependencies
> > > > cannot be introduced explicitly, which means the externalized
> connectors
> > > > use dependencies inherited from flink. In this way, when flink main
> > > > upgrades some dependencies, it is easy to fail when executing
> > > flink-python
> > > > test cases，because flink no longer has this class, and the connector
> does
> > > > not contain it. It's circular problem.
> > > >
> > > > Unless, the connector self-consistently includes all dependencies,
> which
> > > is
> > > > uncontrollable.
> > > > (only a few connectors include all jars in shade phase)
> > > >
> > > > In short, the current flink-python module's dependencies on the
> connector
> > > > leads to an incomplete process of externalization and decoupling,
> which
> > > > will lead to circular dependencies when flink upgrade or change some
> > > > dependencies.
> > > >
> > > > I don't know if I made it clear. I hope to get everyone's opinions on
> > > what
> > > > better solutions we should adopt for similar problems in the future.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/FLINK-30274
> > > > [2]
> > > >
> > > >
> > >
> 

[jira] [Created] (FLINK-32537) Add compatibility annotation for REST API classes

[Hong Liang Teoh (Jira)]

Hong Liang Teoh created FLINK-32537:
---

 Summary: Add compatibility annotation for REST API classes
 Key: FLINK-32537
 URL: https://issues.apache.org/jira/browse/FLINK-32537
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / REST
Affects Versions: 1.17.1, 1.16.2
Reporter: Hong Liang Teoh
 Fix For: 1.18.0


*Why*

We want to standardise the class labelling for Flink classes. Currently, the 
compatibility annotations like @Public, @PublicEvolving, @Internal are not 
present for REST API classes.

 

*What*

We should be added @Internal for most Flink classes, unless they change the 
REST API variables, so we know clearly which components will change our REST 
API when changed



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-307: Flink connector Redshift

[Martijn Visser]

Hi Samrat,

The dependency between JDBC and AWS worries me a lot: we're already seeing
that coupling causes a lot of issues down the line. Why can't we decouple
these?

Best regards,

Martijn

On Tue, Jul 4, 2023 at 3:35 PM Samrat Deb  wrote:

> Hi Leonard,
>
> Sorry for the late reply.
>
> > 1 Reusing the capabilities of JDBC and Filesystem in the Redshift
> connector generally makes sense to me. However, since all of them are
> managed in different repositories and depend on Flink dependency, could you
> explain how you establish the versioning, release, and dependency
> management process?
>
> We intend to maintain the current release cycle for the Flink Connector
> AWS. However, we will be incorporating a dependency of Flink Connector JDBC
> to most recent version, which will synchronize between the flink connector
> AWS and flink connector JDBC connectors releases for new Flink version
> support. Additionally, the Flink Connector Redshift will exclusively
> utilize the public API, minimizing the occurrence of immediate breaking
> changes.
>
> > 2 Some configuration option names can be improved to match the naming
> style of existing configuration options, for example:
> table -> table-name
> query -> scan.query
> aws-iam-role -> aws.iam-role
> read.mode -> scan.read.mode: similar to scan.startup.mode , and maybe we
> will have lookup.read.mode
> write.mode -> sink.write.mode
>
> Updated the FLIP-307 [1]
>
> > 3 The source of Redshift connector supports JDBC queries, IIUC, we can
> also support the LookupTableSource as well?
>
> we will support `LookupTableSource` in redshift connector source
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift
>
> Bests,
> Samrat
>
> On Wed, Jun 21, 2023 at 4:47 PM Leonard Xu  wrote:
>
> > Thanks Samrat for driving this FLIP.
> >
> > Since the community has already built a set of basic components for the
> > connector, I only have three comments.
> >
> > 1 Reusing the capabilities of JDBC and Filesystem in the Redshift
> > connector generally makes sense to me. However, since all of them are
> > managed in different repositories and depend on Flink dependency, could
> you
> > explain how you establish the versioning, release, and dependency
> > management process?
> >
> > 2 Some configuration option names can be improved to match the naming
> > style of existing configuration options, for example:
> > table -> table-name
> > query -> scan.query
> > aws-iam-role -> aws.iam-role
> > read.mode -> scan.read.mode: similar to scan.startup.mode , and maybe we
> > will have lookup.read.mode
> > write.mode -> sink.write.mode
> >
> > 3 The source of Redshift connector supports JDBC queries, IIUC, we can
> > also support the LookupTableSource as well?
> >
> > Best,
> > Leonard
> >
> > > On Jun 21, 2023, at 4:57 PM, Samrat Deb  wrote:
> > >
> > > Hi Martijn,
> > >
> > > Thank you for sharing your thoughts on the matter.
> > > I understand that you don't have a strong opinion on whether to support
> > > exactly-once processing from the beginning or at a later stage.
> > > For initial implementation I will go ahead with at-least-once
> semantics.
> > >
> > >> The only consideration that I could think of is that
> > > if you start with at-least-once, you could consider using the ASync
> API,
> > > but I don't think the ASync API yet supports exactly-once.
> > >
> > > Noted. It's a valid consideration to start compatibility with the Async
> > > API.
> > >
> > > Bests,
> > > Samrat
> > >
> > >
> > > On Mon, Jun 19, 2023 at 5:28 PM Martijn Visser <
> martijnvis...@apache.org
> > >
> > > wrote:
> > >
> > >> Hi Samrat,
> > >>
> > >> I have no strong opinion on whether to support exactly-once from the
> > start
> > >> or potentially later. The only consideration that I could think of is
> > that
> > >> if you start with at-least-once, you could consider using the ASync
> API,
> > >> but I don't think the ASync API yet supports exactly-once.
> > >>
> > >> Thanks,
> > >>
> > >> Martijn
> > >>
> > >> On Fri, Jun 9, 2023 at 7:22 PM Jing Ge 
> > wrote:
> > >>
> > >>> Hi Samrat,
> > >>>
> > >>> The FLIP looks good, thanks!
> > >>>
> > >>> Best regards,
> > >>> Jing
> > >>>
> > >>>
> > >>> On Tue, Jun 6, 2023 at 8:16 PM Samrat Deb 
> > wrote:
> > >>>
> >  Hi Jing,
> > 
> > > I would suggest adding that information into the
> >  FLIP.
> > 
> >  Updated now, please review the new version of flip whenever time.
> > 
> > > +1 Looking forward to your PR :-)
> >  I will request for your review once m ready with PR :-)
> > 
> >  Bests,
> >  Samrat
> > 
> >  On Tue, Jun 6, 2023 at 11:43 PM Samrat Deb 
> > >>> wrote:
> > 
> > > Hi Martijn,
> > >
> > >> If I understand this correctly, the Redshift sink
> > > would not be able to support exactly-once, is that correct?
> > >
> > > As I delve deeper into the study of Redshift's capabilities, I have
> > > discovered that it 

Re: [DISCUSS] FLIP-307: Flink connector Redshift

[Samrat Deb]

Hi Leonard,

Sorry for the late reply.

> 1 Reusing the capabilities of JDBC and Filesystem in the Redshift
connector generally makes sense to me. However, since all of them are
managed in different repositories and depend on Flink dependency, could you
explain how you establish the versioning, release, and dependency
management process?

We intend to maintain the current release cycle for the Flink Connector
AWS. However, we will be incorporating a dependency of Flink Connector JDBC
to most recent version, which will synchronize between the flink connector
AWS and flink connector JDBC connectors releases for new Flink version
support. Additionally, the Flink Connector Redshift will exclusively
utilize the public API, minimizing the occurrence of immediate breaking
changes.

> 2 Some configuration option names can be improved to match the naming
style of existing configuration options, for example:
table -> table-name
query -> scan.query
aws-iam-role -> aws.iam-role
read.mode -> scan.read.mode: similar to scan.startup.mode , and maybe we
will have lookup.read.mode
write.mode -> sink.write.mode

Updated the FLIP-307 [1]

> 3 The source of Redshift connector supports JDBC queries, IIUC, we can
also support the LookupTableSource as well?

we will support `LookupTableSource` in redshift connector source

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-307%3A++Flink+Connector+Redshift

Bests,
Samrat

On Wed, Jun 21, 2023 at 4:47 PM Leonard Xu  wrote:

> Thanks Samrat for driving this FLIP.
>
> Since the community has already built a set of basic components for the
> connector, I only have three comments.
>
> 1 Reusing the capabilities of JDBC and Filesystem in the Redshift
> connector generally makes sense to me. However, since all of them are
> managed in different repositories and depend on Flink dependency, could you
> explain how you establish the versioning, release, and dependency
> management process?
>
> 2 Some configuration option names can be improved to match the naming
> style of existing configuration options, for example:
> table -> table-name
> query -> scan.query
> aws-iam-role -> aws.iam-role
> read.mode -> scan.read.mode: similar to scan.startup.mode , and maybe we
> will have lookup.read.mode
> write.mode -> sink.write.mode
>
> 3 The source of Redshift connector supports JDBC queries, IIUC, we can
> also support the LookupTableSource as well?
>
> Best,
> Leonard
>
> > On Jun 21, 2023, at 4:57 PM, Samrat Deb  wrote:
> >
> > Hi Martijn,
> >
> > Thank you for sharing your thoughts on the matter.
> > I understand that you don't have a strong opinion on whether to support
> > exactly-once processing from the beginning or at a later stage.
> > For initial implementation I will go ahead with at-least-once semantics.
> >
> >> The only consideration that I could think of is that
> > if you start with at-least-once, you could consider using the ASync API,
> > but I don't think the ASync API yet supports exactly-once.
> >
> > Noted. It's a valid consideration to start compatibility with the Async
> > API.
> >
> > Bests,
> > Samrat
> >
> >
> > On Mon, Jun 19, 2023 at 5:28 PM Martijn Visser  >
> > wrote:
> >
> >> Hi Samrat,
> >>
> >> I have no strong opinion on whether to support exactly-once from the
> start
> >> or potentially later. The only consideration that I could think of is
> that
> >> if you start with at-least-once, you could consider using the ASync API,
> >> but I don't think the ASync API yet supports exactly-once.
> >>
> >> Thanks,
> >>
> >> Martijn
> >>
> >> On Fri, Jun 9, 2023 at 7:22 PM Jing Ge 
> wrote:
> >>
> >>> Hi Samrat,
> >>>
> >>> The FLIP looks good, thanks!
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>>
> >>> On Tue, Jun 6, 2023 at 8:16 PM Samrat Deb 
> wrote:
> >>>
>  Hi Jing,
> 
> > I would suggest adding that information into the
>  FLIP.
> 
>  Updated now, please review the new version of flip whenever time.
> 
> > +1 Looking forward to your PR :-)
>  I will request for your review once m ready with PR :-)
> 
>  Bests,
>  Samrat
> 
>  On Tue, Jun 6, 2023 at 11:43 PM Samrat Deb 
> >>> wrote:
> 
> > Hi Martijn,
> >
> >> If I understand this correctly, the Redshift sink
> > would not be able to support exactly-once, is that correct?
> >
> > As I delve deeper into the study of Redshift's capabilities, I have
> > discovered that it does support "merge into" operations [1] and some
> > merge into examples [2].
> > This opens up the possibility of implementing exactly-once semantics
> >>> with
> > the connector.
> > However, I believe it would be prudent to start with a more focused
> >>> scope
> > for the initial phase of implementation and defer the exact-once
> >>> support
> > for subsequent iterations.
> >
> > Before finalizing the approach, I would greatly appreciate your
> >>> thoughts
> > and suggestions on this matter.
> > Should we 

[jira] [Created] (FLINK-32536) Python tests fail with Arrow DirectBuffer exception

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-32536:


 Summary: Python tests fail with Arrow DirectBuffer exception
 Key: FLINK-32536
 URL: https://issues.apache.org/jira/browse/FLINK-32536
 Project: Flink
  Issue Type: Sub-task
  Components: API / Python, Tests
Affects Versions: 1.18.0
Reporter: Chesnay Schepler


https://dev.azure.com/chesnay/flink/_build/results?buildId=3674=logs=fba17979-6d2e-591d-72f1-97cf42797c11=727942b6-6137-54f7-1ef9-e66e706ea068

{code}
2023-07-04T12:54:15.5296754Z Jul 04 12:54:15 E   
py4j.protocol.Py4JJavaError: An error occurred while calling 
z:org.apache.flink.table.runtime.arrow.ArrowUtils.collectAsPandasDataFrame.
2023-07-04T12:54:15.5299579Z Jul 04 12:54:15 E   : 
java.lang.RuntimeException: Arrow depends on DirectByteBuffer.(long, int) 
which is not available. Please set the system property 
'io.netty.tryReflectionSetAccessible' to 'true'.
2023-07-04T12:54:15.5302307Z Jul 04 12:54:15 E  at 
org.apache.flink.table.runtime.arrow.ArrowUtils.checkArrowUsable(ArrowUtils.java:184)
2023-07-04T12:54:15.5302859Z Jul 04 12:54:15 E  at 
org.apache.flink.table.runtime.arrow.ArrowUtils.collectAsPandasDataFrame(ArrowUtils.java:546)
2023-07-04T12:54:15.5303177Z Jul 04 12:54:15 E  at 
jdk.internal.reflect.GeneratedMethodAccessor287.invoke(Unknown Source)
2023-07-04T12:54:15.5303515Z Jul 04 12:54:15 E  at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2023-07-04T12:54:15.5303929Z Jul 04 12:54:15 E  at 
java.base/java.lang.reflect.Method.invoke(Method.java:568)
2023-07-04T12:54:15.5307338Z Jul 04 12:54:15 E  at 
org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
2023-07-04T12:54:15.5309888Z Jul 04 12:54:15 E  at 
org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
2023-07-04T12:54:15.5310306Z Jul 04 12:54:15 E  at 
org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
2023-07-04T12:54:15.5337220Z Jul 04 12:54:15 E  at 
org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
2023-07-04T12:54:15.5341859Z Jul 04 12:54:15 E  at 
org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
2023-07-04T12:54:15.5342363Z Jul 04 12:54:15 E  at 
org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
2023-07-04T12:54:15.5344866Z Jul 04 12:54:15 E  at 
java.base/java.lang.Thread.run(Thread.java:833)
{code}

{code}
2023-07-04T12:54:15.5663559Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::BatchPandasConversionTests::test_empty_to_pandas
2023-07-04T12:54:15.5663891Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::BatchPandasConversionTests::test_from_pandas
2023-07-04T12:54:15.5664299Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::BatchPandasConversionTests::test_to_pandas
2023-07-04T12:54:15.5664655Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::BatchPandasConversionTests::test_to_pandas_for_retract_table
2023-07-04T12:54:15.5665003Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::StreamPandasConversionTests::test_empty_to_pandas
2023-07-04T12:54:15.5665360Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::StreamPandasConversionTests::test_from_pandas
2023-07-04T12:54:15.5665704Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::StreamPandasConversionTests::test_to_pandas
2023-07-04T12:54:15.5666045Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::StreamPandasConversionTests::test_to_pandas_for_retract_table
2023-07-04T12:54:15.5666415Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_conversion.py::StreamPandasConversionTests::test_to_pandas_with_event_time
2023-07-04T12:54:15.5666840Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_udaf.py::BatchPandasUDAFITTests::test_group_aggregate_function
2023-07-04T12:54:15.5667189Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_udaf.py::BatchPandasUDAFITTests::test_group_aggregate_with_aux_group
2023-07-04T12:54:15.5667526Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_udaf.py::BatchPandasUDAFITTests::test_group_aggregate_without_keys
2023-07-04T12:54:15.5667882Z Jul 04 12:54:15 FAILED 
pyflink/table/tests/test_pandas_udaf.py::BatchPandasUDAFITTests::test_over_window_aggregate_function
2023-07-04T12:54:15.5668242Z Jul 04 12:54:15 FAILED 

[jira] [Created] (FLINK-32535) CheckpointingStatisticsHandler periodically returns NullArgumentException after job restarts

[Hong Liang Teoh (Jira)]

Hong Liang Teoh created FLINK-32535:
---

 Summary: CheckpointingStatisticsHandler periodically returns 
NullArgumentException after job restarts
 Key: FLINK-32535
 URL: https://issues.apache.org/jira/browse/FLINK-32535
 Project: Flink
  Issue Type: Bug
  Components: Runtime / REST
Affects Versions: 1.17.1, 1.16.2
Reporter: Hong Liang Teoh
 Fix For: 1.18.0


*What*

When making requests to /checkpoints REST API after a job restart, we see 500 
for a short period of time. We should handle this gracefully in the 
CheckpointingStatisticsHandler.

 

*How to replicate*
 * Checkpointing interval 1s
 * Job is constantly restarting
 * Make constant requests to /checkpoints REST API.

 

Stack trace:

{{org.apache.commons.math3.exception.NullArgumentException: input array}}
{{    at 
org.apache.commons.math3.util.MathArrays.verifyValues(MathArrays.java:1753)}}
{{    at 
org.apache.commons.math3.stat.descriptive.AbstractUnivariateStatistic.test(AbstractUnivariateStatistic.java:158)}}
{{    at 
org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:272)}}
{{    at 
org.apache.commons.math3.stat.descriptive.rank.Percentile.evaluate(Percentile.java:241)}}
{{    at 
org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics$CommonMetricsSnapshot.getPercentile(DescriptiveStatisticsHistogramStatistics.java:159)}}
{{    at 
org.apache.flink.runtime.metrics.DescriptiveStatisticsHistogramStatistics.getQuantile(DescriptiveStatisticsHistogramStatistics.java:53)}}
{{    at 
org.apache.flink.runtime.checkpoint.StatsSummarySnapshot.getQuantile(StatsSummarySnapshot.java:108)}}
{{    at 
org.apache.flink.runtime.rest.messages.checkpoints.StatsSummaryDto.valueOf(StatsSummaryDto.java:81)}}
{{    at 
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:133)}}
{{    at 
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)}}
{{    at 
org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)}}
{{    at 
org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:62)}}
{{    at 
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642)}}
{{    at 
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:478)}}
{{    at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)}}
{{    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)}}
{{    at 
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)}}
{{    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)}}
{{    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)}}
{{    at java.base/java.lang.Thread.run(Thread.java:829)\n}}

 

See graphs here for tests. The dips in the green line correspond to the 
failures immediately after a job restart.

!https://user-images.githubusercontent.com/35062175/250529297-908a6714-ea15-4aac-a7fc-332589da2582.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems

[Dian Fu]

Thanks Ran Tao for proposing this discussion and Martijn for sharing
the thought.

>  While flink-python now fails the CI, it shouldn't actually depend on the
externalized connectors. I'm not sure what PyFlink does with it, but if
belongs to the connector code,

For each DataStream connector, there is a corresponding Python wrapper
and also some test cases in PyFlink. In theory, we should move that
wrapper into each connector repository. In the past, we have not done
that when externalizing the connectors since it may introduce some
burden when releasing since it means that we have to publish each
connector to PyPI separately.

To resolve this problem, I guess we can move the connector support in
PyFlink into the external connector repository.

Regards,
Dian


On Mon, Jul 3, 2023 at 11:08 PM Ran Tao  wrote:
>
> @Martijn
> thanks for clear explanations.
>
> If we follow the line you specified (Connectors shouldn't rely on
> dependencies that may or may not be
> available in Flink itself)
> It seems that we should add a certain dependency if we need(such as
> commons-io, commons-collection) in connector pom explicitly.
> And bundle it in sql-connector uber jar.
>
> Then there is only one thing left that we need to make flink-python test
> not depend on the released flink-connector.
> Maybe we should check it out and decouple it like you suggested.
>
> Best Regards,
> Ran Tao
> https://github.com/chucheng92
>
>
> Martijn Visser  于2023年7月3日周一 22:06写道：
>
> > Hi Ran Tao,
> >
> > Thanks for opening this topic. I think there's a couple of things at hand:
> > 1. Connectors shouldn't rely on dependencies that may or may not be
> > available in Flink itself, like we've seen with flink-shaded. That avoids a
> > tight coupling between Flink and connectors, which is exactly what we try
> > to avoid.
> > 2. When following that line, that would also be applicable for things like
> > commons-collections and commons-io. If a connector wants to use them, it
> > should make sure that it bundles those artifacts itself.
> > 3. While flink-python now fails the CI, it shouldn't actually depend on the
> > externalized connectors. I'm not sure what PyFlink does with it, but if
> > belongs to the connector code, that code should also be moved to the
> > individual connector repo. If it's just a generic test, we could consider
> > creating a generic test against released connector versions to determine
> > compatibility.
> >
> > I'm curious about the opinions of others as well.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Mon, Jul 3, 2023 at 3:37 PM Ran Tao  wrote:
> >
> > > I have an issue here that needs to upgrade commons-collections[1] (this
> > is
> > > an example), but PR ci fails because flink-python test cases depend on
> > > flink-sql-connector-kafka, but kafka-sql-connector is a small jar, does
> > not
> > > include this dependency, so flink ci cause exception[2]. Current my
> > > solution is [3]. But even if this PR is done, the upgrade of flink still
> > > requires kafka-connector released.
> > >
> > > This issue leads to deeper problems. Although the connectors have been
> > > externalized, many UTs of flink-python depend on these connectors, and a
> > > basic agreement of externalized connectors is that other dependencies
> > > cannot be introduced explicitly, which means the externalized connectors
> > > use dependencies inherited from flink. In this way, when flink main
> > > upgrades some dependencies, it is easy to fail when executing
> > flink-python
> > > test cases，because flink no longer has this class, and the connector does
> > > not contain it. It's circular problem.
> > >
> > > Unless, the connector self-consistently includes all dependencies, which
> > is
> > > uncontrollable.
> > > (only a few connectors include all jars in shade phase)
> > >
> > > In short, the current flink-python module's dependencies on the connector
> > > leads to an incomplete process of externalization and decoupling, which
> > > will lead to circular dependencies when flink upgrade or change some
> > > dependencies.
> > >
> > > I don't know if I made it clear. I hope to get everyone's opinions on
> > what
> > > better solutions we should adopt for similar problems in the future.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-30274
> > > [2]
> > >
> > >
> > https://user-images.githubusercontent.com/11287509/250120404-d12b60f4-7ff3-457e-a2c4-8cd415bb5ca2.png
> > >
> > >
> > >
> > https://user-images.githubusercontent.com/11287509/250120522-6b096a4f-83f0-4287-b7ad-d46b9371de4c.png
> > > [3] https://github.com/apache/flink-connector-kafka/pull/38
> > >
> > > Best Regards,
> > > Ran Tao
> > > https://github.com/chucheng92
> > >
> >

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

[David Morávek]

> waiting 2 min between 2 requirements push seems ok to me

This depends on the workload. Would you care if the cost of rescaling were
close to zero (which is for most out-of-the-box workloads)? In that case,
it would be desirable to rescale more frequently, for example, if TMs join
incrementally.

Creating a value that covers everything is impossible unless it's
self-tuning, so I'd prefer having a smooth experience for people trying
things out (just imagine doing a demo at the conference) and having them
opt-in for longer cooldowns.


One idea to keep the timeouts lower while getting more balance would be
restarting the cooldown period when new resources or requirements are
received. This would also bring the cooldown's behavior closer to the
resource-stabilization timeout. Would that make sense?

> Depends on how you implement it. If you ignore all of shouldRescale, yes,
but you shouldn't do that in the first place.

That sounds great; let's go ahead and outline this in the FLIP.

Best,
D.


On Tue, Jul 4, 2023 at 12:30 PM Etienne Chauchot 
wrote:

> Hi all,
>
> Thanks David for your feedback. My comments are inline
>
> Le 04/07/2023 à 09:16, David Morávek a écrit :
> >> They will struggle if they add new resources and nothing happens for 5
> > minutes.
> >
> > The same applies if they start playing with FLIP-291 APIs. I'm wondering
> if
> > the cooldown makes sense there since it was the user's deliberate choice
> to
> > push new requirements. 樂
>
>
> Sure, but remember that the initial rescale is always done immediately.
> Only the time between 2 rescales is controlled by the cooldown period. I
> don't see a user adding resources every 10s (your proposed default
> value) or even with, let's say 2 min, waiting 2 min between 2
> requirements push seems ok to me.
>
>
> >
> > Best,
> > D.
> >
> > On Tue, Jul 4, 2023 at 9:11 AM David Morávek  wrote:
> >
> >> The FLIP reads sane to me. I'm unsure about the default values, though;
> 5
> >> minutes of wait time between rescales feels rather strict, and we should
> >> rethink it to provide a better out-of-the-box experience.
> >>
> >> I'd focus on newcomers trying AS / Reactive Mode out. They will struggle
> >> if they add new resources and nothing happens for 5 minutes. I'd suggest
> >> defaulting to
> >> *jobmanager.adaptive-scheduler.resource-stabilization-timeout* (which
> >> defaults to 10s).
>
>
> If users add resources, the re-scale will happen right away. It is only
> for next additions that they will have to wait for the coolDown period
> to end.
>
> But anyway, we could lower the default value, I just took what Robert
> suggested in the ticket.
>
>
> >>
> >> I'm still struggling to grasp max internal (force rescale). Ignoring
> `AdaptiveScheduler#shouldRescale()`
> >> condition seems rather dangerous. Wouldn't a simple case where you add a
> >> new TM and remove it before the max interval is reached (so there is
> >> nothing to do) result in an unnecessary job restart?
>
> With current behavior (on master) : adding the TM will result in
> restarting if the number of slots added leads to job parallelism
> increase of more than 2. Then removing it can have 2 consequences:
> either it is removed before the resource-stabilisation timeout and there
> will be no restart. Or it is removed after this timeout (the job is in
> Running state) and it will entail another restart and parallelism decrease.
>
> With the proposed behavior: what the scaling-interval.max will change is
> only on the resource addition part: when the TM is added, if the time
> since last rescale > scaling-interval.max, then a restart and
> parallelism increase will be done even if it leads to a parallelism
> increase < 2. The rest regarding TM removal does not change.
>
> => So, the real difference with the current behavior is ** if the slots
> addition was too little ** : in the current behavior nothing happens. In
> the new behavior nothing happens unless the addition arrives after
> scaling-interval.max.
>
>
> Best
>
> Etienne
>
> >>
> >> Best,
> >> D.
> >>
> >> On Thu, Jun 29, 2023 at 3:43 PM Etienne Chauchot
> >> wrote:
> >>
> >>> Thanks Chesnay for your feedback. I have updated the FLIP. I'll start a
> >>> vote thread.
> >>>
> >>> Best
> >>>
> >>> Etienne
> >>>
> >>> Le 28/06/2023 à 11:49, Chesnay Schepler a écrit :
> > we should schedule a check that will rescale if
>  min-parallelism-increase is met. Then, what it the use of
>  scaling-interval.max timeout in that context ?
> 
>  To force a rescale if min-parallelism-increase is not met (but we
>  could still run above the current parallelism).
> 
>  min-parallelism-increase is a trade-off between the cost of rescaling
>  vs the performance benefit of the parallelism increase. Over time the
>  balance tips more and more in favor of the parallelism increase, hence
>  we should eventually rescale anyway even if the minimum isn't met, or
>  at least give users the option to do so.
> 

[jira] [Created] (FLINK-32534) exit code 137 (i.e. OutOfMemoryError) in flink-tests module

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-32534:
---

 Summary: exit code 137 (i.e. OutOfMemoryError) in flink-tests 
module
 Key: FLINK-32534
 URL: https://issues.apache.org/jira/browse/FLINK-32534
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.17.2
Reporter: Sergey Nuyanzin


this build 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50816=logs=5c8e7682-d68f-54d1-16a2-a09310218a49=86f654fa-ab48-5c1a-25f4-7e7f6afb9bba=8056
is failing
{noformat}
Jul 03 11:07:41 Finished 
org.apache.flink.test.checkpointing.UnalignedCheckpointRescaleITCase#shouldRescaleUnalignedCheckpoint[upscale
 keyed_different_parallelism from 7 to 12, sourceSleepMs = 0, buffersPerChannel 
= 0].
##[error]Exit code 137 returned from process: file name '/bin/docker', 
arguments 'exec -i -u 1004  -w /home/agent05_azpcontainer 
20ef21acb85dd401bac092e74118bbc65dae54ceb7e28fee9007fd15987f7d27 
/__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
Finishing: Test - tests
{noformat}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32533) exit code 137 (i.e. OutOfMemoryError) in python module

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-32533:
---

 Summary: exit code 137 (i.e. OutOfMemoryError) in python module
 Key: FLINK-32533
 URL: https://issues.apache.org/jira/browse/FLINK-32533
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.17.2
Reporter: Sergey Nuyanzin


This build 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50839=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=c67e71ed-6451-5d26-8920-5a8cf9651901=24053

is failing like
{noformat}
Jul 03 15:31:56 pyflink/datastream/tests/test_data_stream.py 
... [ 42%]
##[error]Exit code 137 returned from process: file name '/bin/docker', 
arguments 'exec -i -u 1004  -w /home/agent05_azpcontainer 
262ff527fd71a48209f73ba1736f66a946df80f8d1494b6193462778435a66e2 
/__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
Finishing: Test - python

{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32532) exit code 137 (i.e. OutOfMemoryError) in flink-s3-fs-hadoop module

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-32532:
---

 Summary: exit code 137 (i.e. OutOfMemoryError) in 
flink-s3-fs-hadoop module
 Key: FLINK-32532
 URL: https://issues.apache.org/jira/browse/FLINK-32532
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hadoop Compatibility
Affects Versions: 1.16.3
Reporter: Sergey Nuyanzin


This build 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50840=logs=4eda0b4a-bd0d-521a-0916-8285b9be9bb5=2ff6d5fa-53a6-53ac-bff7-fa524ea361a9=16093

is failing like
{noformat}
Jul 03 15:33:35 [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time 
elapsed: 15.267 s - in org.apache.flink.fs.s3hadoop.HadoopS3FileSystemITCase
Jul 03 15:33:45 [ERROR] Picked up JAVA_TOOL_OPTIONS: 
-XX:+HeapDumpOnOutOfMemoryError
##[error]Exit code 137 returned from process: file name '/bin/docker', 
arguments 'exec -i -u 1000  -w /home/agent01_azpcontainer 
3e9ac5dd969222db5673644f5c729d323f624390f9dbc3238a1c99b1b3c4679b 
/__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
Finishing: Test - connect_1
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

[Etienne Chauchot]

Hi all,

Thanks David for your feedback. My comments are inline

Le 04/07/2023 à 09:16, David Morávek a écrit :

They will struggle if they add new resources and nothing happens for 5

minutes.

The same applies if they start playing with FLIP-291 APIs. I'm wondering if
the cooldown makes sense there since it was the user's deliberate choice to
push new requirements. 樂



Sure, but remember that the initial rescale is always done immediately. 
Only the time between 2 rescales is controlled by the cooldown period. I 
don't see a user adding resources every 10s (your proposed default 
value) or even with, let's say 2 min, waiting 2 min between 2 
requirements push seems ok to me.





Best,
D.

On Tue, Jul 4, 2023 at 9:11 AM David Morávek  wrote:


The FLIP reads sane to me. I'm unsure about the default values, though; 5
minutes of wait time between rescales feels rather strict, and we should
rethink it to provide a better out-of-the-box experience.

I'd focus on newcomers trying AS / Reactive Mode out. They will struggle
if they add new resources and nothing happens for 5 minutes. I'd suggest
defaulting to
*jobmanager.adaptive-scheduler.resource-stabilization-timeout* (which
defaults to 10s).



If users add resources, the re-scale will happen right away. It is only 
for next additions that they will have to wait for the coolDown period 
to end.


But anyway, we could lower the default value, I just took what Robert 
suggested in the ticket.





I'm still struggling to grasp max internal (force rescale). Ignoring 
`AdaptiveScheduler#shouldRescale()`
condition seems rather dangerous. Wouldn't a simple case where you add a
new TM and remove it before the max interval is reached (so there is
nothing to do) result in an unnecessary job restart?


With current behavior (on master) : adding the TM will result in 
restarting if the number of slots added leads to job parallelism 
increase of more than 2. Then removing it can have 2 consequences: 
either it is removed before the resource-stabilisation timeout and there 
will be no restart. Or it is removed after this timeout (the job is in 
Running state) and it will entail another restart and parallelism decrease.


With the proposed behavior: what the scaling-interval.max will change is 
only on the resource addition part: when the TM is added, if the time 
since last rescale > scaling-interval.max, then a restart and 
parallelism increase will be done even if it leads to a parallelism 
increase < 2. The rest regarding TM removal does not change.


=> So, the real difference with the current behavior is ** if the slots 
addition was too little ** : in the current behavior nothing happens. In 
the new behavior nothing happens unless the addition arrives after 
scaling-interval.max.



Best

Etienne



Best,
D.

On Thu, Jun 29, 2023 at 3:43 PM Etienne Chauchot
wrote:


Thanks Chesnay for your feedback. I have updated the FLIP. I'll start a
vote thread.

Best

Etienne

Le 28/06/2023 à 11:49, Chesnay Schepler a écrit :

we should schedule a check that will rescale if

min-parallelism-increase is met. Then, what it the use of
scaling-interval.max timeout in that context ?

To force a rescale if min-parallelism-increase is not met (but we
could still run above the current parallelism).

min-parallelism-increase is a trade-off between the cost of rescaling
vs the performance benefit of the parallelism increase. Over time the
balance tips more and more in favor of the parallelism increase, hence
we should eventually rescale anyway even if the minimum isn't met, or
at least give users the option to do so.


I meant the opposite: not having only the cooldown but having only

the stabilization time. I must have missed something because what I
wonder is: if every rescale entails a restart of the pipeline and
every restart entails passing in waiting for resources state, then why
introduce a cooldown when there is already at each rescale a stable
resource timeout ?

It is technically correct that the stable resource timeout can be used
to limit the number of rescale operations per interval, however during
that time the job isn't running, in contrast to the cooldown.

Having both just gives you a lot more flexibility.
"I want at most 1 rescale operation per hour, and wait at most 1
minute for resource to stabilize when a rescale happens".
You can't express this with only one of the options.

On 20/06/2023 14:41, Etienne Chauchot wrote:

Hi Chesnay,

Thanks for your feedback. Comments inline

Le 16/06/2023 à 17:24, Chesnay Schepler a écrit :

1) Options specific to the adaptive scheduler should start with
"jobmanager.adaptive-scheduler".


ok



2)
There isn't /really /a notion of a "scaling event". The scheduler is
informed about new/lost slots and job failures, and reacts
accordingly by maybe rescaling the job.
(sure, you can think of these as events, but you can think of
practically everything as events)

There shouldn't be a queue for events. All the scheduler should have

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

[Chesnay Schepler]

I think the cooldown still makes sense with FLIP-291 APIs.

If you want to fully control the parallelism and rescale timings then 
you can set the cooldown to zero.
If you don't want complete control but just the target parallelism from 
time to time, then the cooldown within Flink still makes sense imo 
because it can account for all scale up operations, which an external 
scaler would struggle with (because it doesn't actually know when a 
scale up happened).


> Wouldn't a simple case where you add a new TM and remove it before 
the max interval is reached (so there is nothing to do) result in an 
unnecessary job restart?


Depends on how you implement it. If you ignore all of shouldRescale, 
yes, but you shouldn't do that in the first place.


Within shouldRescale() the SlotAllocater wouldn't provide us with a new 
parallelism alternative and we wouldn't ask the RescaleController, which 
is the bit we actually want to override.


On 04/07/2023 09:16, David Morávek wrote:
> They will struggle if they add new resources and nothing happens for 
5 minutes.


The same applies if they start playing with FLIP-291 APIs. I'm 
wondering if the cooldown makes sense there since it was the user's 
deliberate choice to push new requirements. 樂


Best,
D.

On Tue, Jul 4, 2023 at 9:11 AM David Morávek  wrote:

The FLIP reads sane to me. I'm unsure about the default values,
though; 5 minutes of wait time between rescales feels rather
strict, and we should rethink it to provide a better
out-of-the-box experience.

I'd focus on newcomers trying AS / Reactive Mode out. They will
struggle if they add new resources and nothing happens for 5
minutes. I'd suggest defaulting to
/jobmanager.adaptive-scheduler.resource-stabilization-timeout/ (which
defaults to 10s).

I'm still struggling to grasp max internal (force rescale).
Ignoring `AdaptiveScheduler#shouldRescale()` condition seems
rather dangerous. Wouldn't a simple case where you add a new TM
and remove it before the max interval is reached (so there is
nothing to do) result in an unnecessary job restart?

Best,
D.

On Thu, Jun 29, 2023 at 3:43 PM Etienne Chauchot
 wrote:

Thanks Chesnay for your feedback. I have updated the FLIP.
I'll start a
vote thread.

Best

Etienne

Le 28/06/2023 à 11:49, Chesnay Schepler a écrit :
> > we should schedule a check that will rescale if
> min-parallelism-increase is met. Then, what it the use of
> scaling-interval.max timeout in that context ?
>
> To force a rescale if min-parallelism-increase is not met
(but we
> could still run above the current parallelism).
>
> min-parallelism-increase is a trade-off between the cost of
rescaling
> vs the performance benefit of the parallelism increase. Over
time the
> balance tips more and more in favor of the parallelism
increase, hence
> we should eventually rescale anyway even if the minimum
isn't met, or
> at least give users the option to do so.
>
> > I meant the opposite: not having only the cooldown but
having only
> the stabilization time. I must have missed something because
what I
> wonder is: if every rescale entails a restart of the
pipeline and
> every restart entails passing in waiting for resources
state, then why
> introduce a cooldown when there is already at each rescale a
stable
> resource timeout ?
>
> It is technically correct that the stable resource timeout
can be used
> to limit the number of rescale operations per interval,
however during
> that time the job isn't running, in contrast to the cooldown.
>
> Having both just gives you a lot more flexibility.
> "I want at most 1 rescale operation per hour, and wait at
most 1
> minute for resource to stabilize when a rescale happens".
> You can't express this with only one of the options.
>
> On 20/06/2023 14:41, Etienne Chauchot wrote:
>> Hi Chesnay,
>>
>> Thanks for your feedback. Comments inline
>>
>> Le 16/06/2023 à 17:24, Chesnay Schepler a écrit :
>>> 1) Options specific to the adaptive scheduler should start
with
>>> "jobmanager.adaptive-scheduler".
>>
>>
>> ok
>>
>>
>>> 2)
>>> There isn't /really /a notion of a "scaling event". The
scheduler is
>>> informed about new/lost slots and job failures, and reacts
>>> accordingly by maybe rescaling the job.
>>> (sure, you can think of these as events, but you can think of
>>> practically everything as events)
>>>
>>> There shouldn't be a queue for 

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Benchao Li]

Congratulations!

Feng Jin  于2023年7月4日周二 16:17写道：

> Congratulations!
>
> Best,
> Feng Jin
>
>
>
> On Tue, Jul 4, 2023 at 4:13 PM Yuxin Tan  wrote:
>
> > Congratulations!
> >
> > Best,
> > Yuxin
> >
> >
> > Dunn Bangui  于2023年7月4日周二 16:04写道：
> >
> > > Congratulations!
> > >
> > > Best,
> > > Bangui Dunn
> > >
> > > Yangze Guo  于2023年7月4日周二 15:59写道：
> > >
> > > > Congrats everyone!
> > > >
> > > > Best,
> > > > Yangze Guo
> > > >
> > > > On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote:
> > > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Rui Fan
> > > > >
> > > > > On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu  wrote:
> > > > >
> > > > > > Congratulations everyone!
> > > > > >
> > > > > > Thanks,
> > > > > > Zhu
> > > > > >
> > > > > > Hang Ruan  于2023年7月4日周二 14:06写道：
> > > > > > >
> > > > > > > Congratulations!
> > > > > > >
> > > > > > > Best,
> > > > > > > Hang
> > > > > > >
> > > > > > > Jingsong Li  于2023年7月4日周二 13:47写道：
> > > > > > >
> > > > > > > > Congratulations!
> > > > > > > >
> > > > > > > > Thank you! All of the Flink community!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jingsong
> > > > > > > >
> > > > > > > > On Tue, Jul 4, 2023 at 1:24 PM tison 
> > > wrote:
> > > > > > > > >
> > > > > > > > > Congrats and with honor :D
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > tison.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > > > > > > > >
> > > > > > > > > > Congratulations!--
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Mang Zhang
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > > > > > > > >Congrats everyone
> > > > > > > > > > >
> > > > > > > > > > >Best,
> > > > > > > > > > >Ron
> > > > > > > > > > >
> > > > > > > > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > > > > > > > >
> > > > > > > > > > >> Congrats everyone!
> > > > > > > > > > >>
> > > > > > > > > > >> Best,
> > > > > > > > > > >> Jark
> > > > > > > > > > >>
> > > > > > > > > > >> > 2023年7月3日 22:37，Yuval Itzchakov 
> > 写道：
> > > > > > > > > > >> >
> > > > > > > > > > >> > Congrats team!
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > > > > > > > u...@flink.apache.org
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> >> Congratulations!
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> Best regards,
> > > > > > > > > > >> >> Jing
> > > > > > > > > > >> >>
> > > > > > > > > > >> >>
> > > > > > > > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > > > > > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> >>> Congratulations!
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> Best regards,
> > > > > > > > > > >> >>> Yuxia
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> 发件人: "Pushpa Ramakrishnan" <
> > > > pushpa.ramakrish...@icloud.com
> > > > > > > >  > > > > > > > > > >> pushpa.ramakrish...@icloud.com>>
> > > > > > > > > > >> >>> 收件人: "Xintong Song"   > > > > > > > > > >> tonysong...@gmail.com>>
> > > > > > > > > > >> >>> 抄送: "dev"  > > > > > dev@flink.apache.org>>,
> > > > > > > > > > >> "User"  > > u...@flink.apache.org
> > > > >>
> > > > > > > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > > > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023
> > > SIGMOD
> > > > > > Systems
> > > > > > > > > > Award
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> > > > > > tonysong...@gmail.com
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> 
> > > > > > > > > > >> >>> Dear Community,
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> I'm pleased to share this good news with everyone.
> > As
> > > > some
> > > > > > of
> > > > > > > > you
> > > > > > > > > > may
> > > > > > > > > > >> have already heard, Apache Flink has won the 2023
> SIGMOD
> > > > Systems
> > > > > > > > Award
> > > > > > > > > > [1].
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > > > > > > data-processing."
> > > > > > > > > > --
> > > > > > > > > > >> SIGMOD Awards Committee
> > > > > > > > > > >> >>>
> > > > > > > > > > >> >>> SIGMOD is one of the most influential data
> > management
> > > > > > research
> > > > > > > > > > >> conferences in the world. The Systems Award is awarded
> > to
> > > an
> > > > > > > > individual
> > > > > > > > > > or
> > > > > > > > > > >> set of individuals to recognize the development of a
> > > > software or
> > > > > > > > > > hardware
> > > > > > > > > > >> system whose technical contributions have had
> > significant
> > > > > > impact on
> > > 

[jira] [Created] (FLINK-32531) "Recovery of missing job deployments" feature not working as described

[Ivan Stoiev (Jira)]

Ivan Stoiev created FLINK-32531:
---

 Summary: "Recovery of missing job deployments" feature not working 
as described
 Key: FLINK-32531
 URL: https://issues.apache.org/jira/browse/FLINK-32531
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Reporter: Ivan Stoiev


Maybe it's not a bug, but some documentation issue.

[Docs|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/#recovery-of-missing-job-deployments]
 says:
??When HA is enabled, the operator can recover the Flink cluster deployments in 
cases when it was accidentally deleted by the user or some external process.??

My use cases are stadalone jobs with savepoint as {{{}upgradeMode{}}}, and we 
want to ensure it's state, even when explicity deleted by the user. Starting a 
job withouth its previous state is something that we need to avoid at all costs.

In our tests (using Flink 1.16, kubernetes HA and operator 1.4.0), when we 
delete the FlinkDeployment object, all HA metadata is removed. And when 
applying the same FlinkDeployment object, flink always start withouth state.

Looking at the code and related issues it seems the normal behaviour, but the 
docs lead us to wrong conclusions. Is there something that we are missing here? 
Is there some configuration/situation that could replicate the behaviour 
described on docs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32530) array_position semantic should align with array_contains instead of spark

[Jacky Lau (Jira)]

Jacky Lau created FLINK-32530:
-

 Summary: array_position semantic should align with array_contains 
instead of spark
 Key: FLINK-32530
 URL: https://issues.apache.org/jira/browse/FLINK-32530
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.18.0
Reporter: Jacky Lau
 Fix For: 1.18.0


when i supports array_contains to calcite 
https://issues.apache.org/jira/browse/CALCITE-5707 i found the spark and 
flink's behavior is different.
{code:java}
spark: array_contains(array[1, null], null) -> null
flink: array_contains(array[1, null], null) -> true  {code}
so array_remove is also different(the array_remove is  supported by me, which 
aligns with flink).
{code:java}
spark: array_remove(array[1, null], null) -> null 
flink: array_remove(array[1, null], null) -> 1  {code}
while array_position is align with spark, i think it is not correct.
{code:java}
spark: array_position(array[1, null], null) -> 2 
flink: array_position(array[1, null], null) -> 2   {code}
and i test on postgre which is also 2
{code:java}
postgre:
select array_position(ARRAY[1, null], null); {code}
so the semantic should only follow one way to handle null element. so i think 
it should be changed



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

[Etienne Chauchot]

Hi all,

@David, I just saw your comments on the [DISCUSSION] thread. I'll close 
this vote for now and open a new voting thread when the discussion 
reaches a consensus.


Best

Etienne

Le 04/07/2023 à 10:49, Etienne Chauchot a écrit :


Hi David,

Indeed, this thread was a proper separate [VOTE] thread 

You withdrew your -1 vote but did not cast a new vote. Can you do so ?

Thanks

Etienne

Le 04/07/2023 à 08:52, David Morávek a écrit :

Hmm, sorry for the confusion; it seems that Google is playing games with me
(I see this chained under the old [DISCUSS] thread), but it seems correct
in the mail archive [1] :/

Just ignore the -1 above.

[1]https://lists.apache.org/thread/22fovrkmzcvzblcohhtsp5t96vd64obj

On Tue, Jul 4, 2023 at 8:49 AM David Morávek  wrote:


The vote closes within 6 hours and, as for now, there was no vote. This

is a very short FLIP, that takes a few minutes to read.


Maybe because there should have been a dedicated voting thread (marked as
[VOTE]), this one was hidden and hard to notice.

We should restart the vote with proper mechanics to allow everyone to
participate.

Soft -1 from my side until there is a proper voting thread.

Best,
D.

On Mon, Jul 3, 2023 at 4:40 PM ConradJam  wrote:


+1 (no-binding)

Etienne Chauchot  于2023年7月3日周一 15:57写道：


Hi all,

The vote closes within 6 hours and, as for now, there was no vote. This
is a very short FLIP, that takes a few minutes to read.

Please cast your vote so that the development could start.

Thanks.

Best

Etienne

Le 29/06/2023 à 15:51, Etienne Chauchot a écrit :

Hi all,

Thanks for all the feedback about the FLIP-322: Cooldown period for
adaptive scheduler [1].

This FLIP was discussed in [2].

I'd like to start a vote for it. The vote will be open for at least 72
hours (until July 3rd 14:00 GMT) unless there is an objection or
insufficient votes.

[1]


https://cwiki.apache.org/confluence/display/FLINK/FLIP-322+Cooldown+period+for+adaptive+scheduler

[2]https://lists.apache.org/thread/qvgxzhbp9rhlsqrybxdy51h05zwxfns6

Best,

Etienne

[jira] [Created] (FLINK-32529) Optional startup probe for JM deployment

[Gyula Fora (Jira)]

Gyula Fora created FLINK-32529:
--

 Summary: Optional startup probe for JM deployment
 Key: FLINK-32529
 URL: https://issues.apache.org/jira/browse/FLINK-32529
 Project: Flink
  Issue Type: New Feature
  Components: Kubernetes Operator
Reporter: Gyula Fora
Assignee: Gyula Fora


There are certain cases where the JM enters a startup crash loop for example 
due to incorrect HA config setup. With the current operator logic these cases 
require manual user intervention as we don't have HA metadata available for the 
last checkpoint and it also seems like the JM actually started already.

To solve this properly we suggest adding a default JM startup probe that 
queries the rest api (/config) endpoint. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

[Etienne Chauchot]

Hi David,

Indeed, this thread was a proper separate [VOTE] thread 

You withdrew your -1 vote but did not cast a new vote. Can you do so ?

Thanks

Etienne

Le 04/07/2023 à 08:52, David Morávek a écrit :

Hmm, sorry for the confusion; it seems that Google is playing games with me
(I see this chained under the old [DISCUSS] thread), but it seems correct
in the mail archive [1] :/

Just ignore the -1 above.

[1]https://lists.apache.org/thread/22fovrkmzcvzblcohhtsp5t96vd64obj

On Tue, Jul 4, 2023 at 8:49 AM David Morávek  wrote:


The vote closes within 6 hours and, as for now, there was no vote. This

is a very short FLIP, that takes a few minutes to read.


Maybe because there should have been a dedicated voting thread (marked as
[VOTE]), this one was hidden and hard to notice.

We should restart the vote with proper mechanics to allow everyone to
participate.

Soft -1 from my side until there is a proper voting thread.

Best,
D.

On Mon, Jul 3, 2023 at 4:40 PM ConradJam  wrote:


+1 (no-binding)

Etienne Chauchot  于2023年7月3日周一 15:57写道：


Hi all,

The vote closes within 6 hours and, as for now, there was no vote. This
is a very short FLIP, that takes a few minutes to read.

Please cast your vote so that the development could start.

Thanks.

Best

Etienne

Le 29/06/2023 à 15:51, Etienne Chauchot a écrit :

Hi all,

Thanks for all the feedback about the FLIP-322: Cooldown period for
adaptive scheduler [1].

This FLIP was discussed in [2].

I'd like to start a vote for it. The vote will be open for at least 72
hours (until July 3rd 14:00 GMT) unless there is an objection or
insufficient votes.

[1]


https://cwiki.apache.org/confluence/display/FLINK/FLIP-322+Cooldown+period+for+adaptive+scheduler

[2]https://lists.apache.org/thread/qvgxzhbp9rhlsqrybxdy51h05zwxfns6

Best,

Etienne

[jira] [Created] (FLINK-32528) The RexCall a = a,if a's datatype is nullable, and when a is null, a = a is null, it isn't true in BinaryComparisonExprReducer

[LakeShen (Jira)]

LakeShen created FLINK-32528:


 Summary: The RexCall a = a,if a's datatype is nullable, and when a 
is null, a = a is null, it isn't true in BinaryComparisonExprReducer
 Key: FLINK-32528
 URL: https://issues.apache.org/jira/browse/FLINK-32528
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.17.0
Reporter: LakeShen






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32527) Build failure on Windows

[Fangbin Sun (Jira)]

Fangbin Sun created FLINK-32527:
---

 Summary: Build failure on Windows
 Key: FLINK-32527
 URL: https://issues.apache.org/jira/browse/FLINK-32527
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Reporter: Fangbin Sun


 
{code:java}
[INFO] --- maven-antrun-plugin:1.8:run (deployment-crd-compatibility-check) @ 
flink-kubernetes-operator-api ---
[INFO] Executing tasksmain:
     [java] 2023-07-04 16:07:45,348 o.a.f.k.o.a.v.CrdCompatibilityChecker [INFO 
] [.] New schema: 
file://E:\project\open\flink-operator-main\flink-kubernetes-operator/helm/flink-kubernetes-operator/crds/flinkdeployments.flink.apache.org-v1.yml
     [java] 2023-07-04 16:07:45,350 o.a.f.k.o.a.v.CrdCompatibilityChecker [INFO 
] [.] Old schema: 
https://raw.githubusercontent.com/apache/flink-kubernetes-operator/release-1.4.0/helm/flink-kubernetes-operator/crds/flinkdeployments.flink.apache.org-v1.yml
     [java] Exception in thread "main" java.net.UnknownHostException: E
     [java]     at 
java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:220)
     [java]     at java.base/java.net.Socket.connect(Socket.java:609)
     [java]     at 
java.base/sun.net.ftp.impl.FtpClient.doConnect(FtpClient.java:1062)
     [java]     at 
java.base/sun.net.ftp.impl.FtpClient.tryConnect(FtpClient.java:1024)
     [java]     at 
java.base/sun.net.ftp.impl.FtpClient.connect(FtpClient.java:1119)
     [java]     at 
java.base/sun.net.ftp.impl.FtpClient.connect(FtpClient.java:1105)
     [java]     at 
java.base/sun.net.www.protocol.ftp.FtpURLConnection.connect(FtpURLConnection.java:312)
     [java]     at 
java.base/sun.net.www.protocol.ftp.FtpURLConnection.getInputStream(FtpURLConnection.java:418)
     [java]     at java.base/java.net.URL.openStream(URL.java:1165)
     [java]     at 
com.fasterxml.jackson.core.TokenStreamFactory._optimizedStreamFromURL(TokenStreamFactory.java:262)
     [java]     at 
com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:400)
     [java]     at 
com.fasterxml.jackson.dataformat.yaml.YAMLFactory.createParser(YAMLFactory.java:15)
     [java]     at 
com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3268)
     [java]     at 
org.apache.flink.kubernetes.operator.api.validation.CrdCompatibilityChecker.getSchema(CrdCompatibilityChecker.java:66)
     [java]     at 
org.apache.flink.kubernetes.operator.api.validation.CrdCompatibilityChecker.main(CrdCompatibilityChecker.java:60)
[INFO] 
[INFO] Reactor Summary for Flink Kubernetes: 1.6-SNAPSHOT:
[INFO] 
[INFO] Flink Kubernetes: .. SUCCESS [  4.028 s]
[INFO] Flink Kubernetes Standalone  SUCCESS [  8.140 s]
[INFO] Flink Kubernetes Operator Api .. FAILURE [ 31.335 s]
[INFO] Flink Kubernetes Operator .. SKIPPED
[INFO] Flink Kubernetes Operator Autoscaler ... SKIPPED
[INFO] Flink Kubernetes Webhook ... SKIPPED
[INFO] Flink Kubernetes Docs .. SKIPPED
[INFO] Flink SQL Runner Example ... SKIPPED
[INFO] Flink Beam Example . SKIPPED
[INFO] Flink Kubernetes Client Code Example ... SKIPPED
[INFO] Flink Autoscaler Test Job .. SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO]  
{code}
This is due to `JsonNode readTree(URL source)` can not handle the file schema 
on windows as below:

 
{code:java}
protected InputStream _optimizedStreamFromURL(URL url) throws IOException {
if ("file".equals(url.getProtocol())) {
String host = url.getHost();
if (host == null || host.length() == 0) {
String path = url.getPath();
if (path.indexOf(37) < 0) {
return new FileInputStream(url.getPath());
}
}
}

return url.openStream();
} {code}
we should use `JsonNode readTree(File file)` instead.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Feng Jin]

Congratulations!

Best,
Feng Jin



On Tue, Jul 4, 2023 at 4:13 PM Yuxin Tan  wrote:

> Congratulations!
>
> Best,
> Yuxin
>
>
> Dunn Bangui  于2023年7月4日周二 16:04写道：
>
> > Congratulations!
> >
> > Best,
> > Bangui Dunn
> >
> > Yangze Guo  于2023年7月4日周二 15:59写道：
> >
> > > Congrats everyone!
> > >
> > > Best,
> > > Yangze Guo
> > >
> > > On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote:
> > > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Rui Fan
> > > >
> > > > On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu  wrote:
> > > >
> > > > > Congratulations everyone!
> > > > >
> > > > > Thanks,
> > > > > Zhu
> > > > >
> > > > > Hang Ruan  于2023年7月4日周二 14:06写道：
> > > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > Best,
> > > > > > Hang
> > > > > >
> > > > > > Jingsong Li  于2023年7月4日周二 13:47写道：
> > > > > >
> > > > > > > Congratulations!
> > > > > > >
> > > > > > > Thank you! All of the Flink community!
> > > > > > >
> > > > > > > Best,
> > > > > > > Jingsong
> > > > > > >
> > > > > > > On Tue, Jul 4, 2023 at 1:24 PM tison 
> > wrote:
> > > > > > > >
> > > > > > > > Congrats and with honor :D
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > tison.
> > > > > > > >
> > > > > > > >
> > > > > > > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > > > > > > >
> > > > > > > > > Congratulations!--
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Mang Zhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > > > > > > >Congrats everyone
> > > > > > > > > >
> > > > > > > > > >Best,
> > > > > > > > > >Ron
> > > > > > > > > >
> > > > > > > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > > > > > > >
> > > > > > > > > >> Congrats everyone!
> > > > > > > > > >>
> > > > > > > > > >> Best,
> > > > > > > > > >> Jark
> > > > > > > > > >>
> > > > > > > > > >> > 2023年7月3日 22:37，Yuval Itzchakov 
> 写道：
> > > > > > > > > >> >
> > > > > > > > > >> > Congrats team!
> > > > > > > > > >> >
> > > > > > > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > > > > > > u...@flink.apache.org
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >> Congratulations!
> > > > > > > > > >> >>
> > > > > > > > > >> >> Best regards,
> > > > > > > > > >> >> Jing
> > > > > > > > > >> >>
> > > > > > > > > >> >>
> > > > > > > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > > > > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >>> Congratulations!
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> Best regards,
> > > > > > > > > >> >>> Yuxia
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> 发件人: "Pushpa Ramakrishnan" <
> > > pushpa.ramakrish...@icloud.com
> > > > > > >  > > > > > > > > >> pushpa.ramakrish...@icloud.com>>
> > > > > > > > > >> >>> 收件人: "Xintong Song"  > > > > > > > > >> tonysong...@gmail.com>>
> > > > > > > > > >> >>> 抄送: "dev"  > > > > dev@flink.apache.org>>,
> > > > > > > > > >> "User"  > u...@flink.apache.org
> > > >>
> > > > > > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023
> > SIGMOD
> > > > > Systems
> > > > > > > > > Award
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> > > > > tonysong...@gmail.com
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> 
> > > > > > > > > >> >>> Dear Community,
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> I'm pleased to share this good news with everyone.
> As
> > > some
> > > > > of
> > > > > > > you
> > > > > > > > > may
> > > > > > > > > >> have already heard, Apache Flink has won the 2023 SIGMOD
> > > Systems
> > > > > > > Award
> > > > > > > > > [1].
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > > > > > data-processing."
> > > > > > > > > --
> > > > > > > > > >> SIGMOD Awards Committee
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> SIGMOD is one of the most influential data
> management
> > > > > research
> > > > > > > > > >> conferences in the world. The Systems Award is awarded
> to
> > an
> > > > > > > individual
> > > > > > > > > or
> > > > > > > > > >> set of individuals to recognize the development of a
> > > software or
> > > > > > > > > hardware
> > > > > > > > > >> system whose technical contributions have had
> significant
> > > > > impact on
> > > > > > > the
> > > > > > > > > >> theory or practice of large-scale data management
> systems.
> > > > > Winning
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > >> award indicates the high recognition of Flink's
> > > technological
> > > > > > > > > advancement
> > > > > > > > > >> and industry influence from academia.
> > > > > > > > > >> >>>
> > > > > > > > > >> >>> As an 

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Yuxin Tan]

Congratulations!

Best,
Yuxin


Dunn Bangui  于2023年7月4日周二 16:04写道：

> Congratulations!
>
> Best,
> Bangui Dunn
>
> Yangze Guo  于2023年7月4日周二 15:59写道：
>
> > Congrats everyone!
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Congratulations!
> > >
> > > Best,
> > > Rui Fan
> > >
> > > On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu  wrote:
> > >
> > > > Congratulations everyone!
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > Hang Ruan  于2023年7月4日周二 14:06写道：
> > > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Jingsong Li  于2023年7月4日周二 13:47写道：
> > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > Thank you! All of the Flink community!
> > > > > >
> > > > > > Best,
> > > > > > Jingsong
> > > > > >
> > > > > > On Tue, Jul 4, 2023 at 1:24 PM tison 
> wrote:
> > > > > > >
> > > > > > > Congrats and with honor :D
> > > > > > >
> > > > > > > Best,
> > > > > > > tison.
> > > > > > >
> > > > > > >
> > > > > > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > > > > > >
> > > > > > > > Congratulations!--
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Mang Zhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > > > > > >Congrats everyone
> > > > > > > > >
> > > > > > > > >Best,
> > > > > > > > >Ron
> > > > > > > > >
> > > > > > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > > > > > >
> > > > > > > > >> Congrats everyone!
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Jark
> > > > > > > > >>
> > > > > > > > >> > 2023年7月3日 22:37，Yuval Itzchakov  写道：
> > > > > > > > >> >
> > > > > > > > >> > Congrats team!
> > > > > > > > >> >
> > > > > > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > > > > > u...@flink.apache.org
> > > > > > > > >> > wrote:
> > > > > > > > >> >> Congratulations!
> > > > > > > > >> >>
> > > > > > > > >> >> Best regards,
> > > > > > > > >> >> Jing
> > > > > > > > >> >>
> > > > > > > > >> >>
> > > > > > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > > > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > > > >> > wrote:
> > > > > > > > >> >>> Congratulations!
> > > > > > > > >> >>>
> > > > > > > > >> >>> Best regards,
> > > > > > > > >> >>> Yuxia
> > > > > > > > >> >>>
> > > > > > > > >> >>> 发件人: "Pushpa Ramakrishnan" <
> > pushpa.ramakrish...@icloud.com
> > > > > >  > > > > > > > >> pushpa.ramakrish...@icloud.com>>
> > > > > > > > >> >>> 收件人: "Xintong Song"  > > > > > > > >> tonysong...@gmail.com>>
> > > > > > > > >> >>> 抄送: "dev"  > > > dev@flink.apache.org>>,
> > > > > > > > >> "User"  u...@flink.apache.org
> > >>
> > > > > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023
> SIGMOD
> > > > Systems
> > > > > > > > Award
> > > > > > > > >> >>>
> > > > > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > > > > >> >>>
> > > > > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> > > > tonysong...@gmail.com
> > > > > > > > >> > wrote:
> > > > > > > > >> >>>
> > > > > > > > >> >>> 
> > > > > > > > >> >>> Dear Community,
> > > > > > > > >> >>>
> > > > > > > > >> >>> I'm pleased to share this good news with everyone. As
> > some
> > > > of
> > > > > > you
> > > > > > > > may
> > > > > > > > >> have already heard, Apache Flink has won the 2023 SIGMOD
> > Systems
> > > > > > Award
> > > > > > > > [1].
> > > > > > > > >> >>>
> > > > > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > > > > data-processing."
> > > > > > > > --
> > > > > > > > >> SIGMOD Awards Committee
> > > > > > > > >> >>>
> > > > > > > > >> >>> SIGMOD is one of the most influential data management
> > > > research
> > > > > > > > >> conferences in the world. The Systems Award is awarded to
> an
> > > > > > individual
> > > > > > > > or
> > > > > > > > >> set of individuals to recognize the development of a
> > software or
> > > > > > > > hardware
> > > > > > > > >> system whose technical contributions have had significant
> > > > impact on
> > > > > > the
> > > > > > > > >> theory or practice of large-scale data management systems.
> > > > Winning
> > > > > > of
> > > > > > > > the
> > > > > > > > >> award indicates the high recognition of Flink's
> > technological
> > > > > > > > advancement
> > > > > > > > >> and industry influence from academia.
> > > > > > > > >> >>>
> > > > > > > > >> >>> As an open-source project, Flink wouldn't have come
> > this far
> > > > > > without
> > > > > > > > >> the wide, active and supportive community behind it. Kudos
> > to
> > > > all
> > > > > > of us
> > > > > > > > who
> > > > > > > > >> helped make this happen, including the over 1,400
> > contributors
> > > > and
> > > > > > many
> > > > > > > > >> others who contributed in ways beyond code.
> > > > > > > > >> >>>
> > > > > > > > 

[jira] [Created] (FLINK-32526) Update apache parquet to 1.31.1

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-32526:
---

 Summary: Update apache parquet to 1.31.1
 Key: FLINK-32526
 URL: https://issues.apache.org/jira/browse/FLINK-32526
 Project: Flink
  Issue Type: Technical Debt
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.18.0
Reporter: Sergey Nuyanzin


Now 1.13.1 is available
https://parquet.apache.org/blog/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Dunn Bangui]

Congratulations!

Best,
Bangui Dunn

Yangze Guo  于2023年7月4日周二 15:59写道：

> Congrats everyone!
>
> Best,
> Yangze Guo
>
> On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Congratulations!
> >
> > Best,
> > Rui Fan
> >
> > On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu  wrote:
> >
> > > Congratulations everyone!
> > >
> > > Thanks,
> > > Zhu
> > >
> > > Hang Ruan  于2023年7月4日周二 14:06写道：
> > > >
> > > > Congratulations!
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Jingsong Li  于2023年7月4日周二 13:47写道：
> > > >
> > > > > Congratulations!
> > > > >
> > > > > Thank you! All of the Flink community!
> > > > >
> > > > > Best,
> > > > > Jingsong
> > > > >
> > > > > On Tue, Jul 4, 2023 at 1:24 PM tison  wrote:
> > > > > >
> > > > > > Congrats and with honor :D
> > > > > >
> > > > > > Best,
> > > > > > tison.
> > > > > >
> > > > > >
> > > > > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > > > > >
> > > > > > > Congratulations!--
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Mang Zhang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > > > > >Congrats everyone
> > > > > > > >
> > > > > > > >Best,
> > > > > > > >Ron
> > > > > > > >
> > > > > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > > > > >
> > > > > > > >> Congrats everyone!
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Jark
> > > > > > > >>
> > > > > > > >> > 2023年7月3日 22:37，Yuval Itzchakov  写道：
> > > > > > > >> >
> > > > > > > >> > Congrats team!
> > > > > > > >> >
> > > > > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > > > > u...@flink.apache.org
> > > > > > > >> > wrote:
> > > > > > > >> >> Congratulations!
> > > > > > > >> >>
> > > > > > > >> >> Best regards,
> > > > > > > >> >> Jing
> > > > > > > >> >>
> > > > > > > >> >>
> > > > > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > > >> > wrote:
> > > > > > > >> >>> Congratulations!
> > > > > > > >> >>>
> > > > > > > >> >>> Best regards,
> > > > > > > >> >>> Yuxia
> > > > > > > >> >>>
> > > > > > > >> >>> 发件人: "Pushpa Ramakrishnan" <
> pushpa.ramakrish...@icloud.com
> > > > >  > > > > > > >> pushpa.ramakrish...@icloud.com>>
> > > > > > > >> >>> 收件人: "Xintong Song"  > > > > > > >> tonysong...@gmail.com>>
> > > > > > > >> >>> 抄送: "dev"  > > dev@flink.apache.org>>,
> > > > > > > >> "User" mailto:u...@flink.apache.org
> >>
> > > > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD
> > > Systems
> > > > > > > Award
> > > > > > > >> >>>
> > > > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > > > >> >>>
> > > > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> > > tonysong...@gmail.com
> > > > > > > >> > wrote:
> > > > > > > >> >>>
> > > > > > > >> >>> 
> > > > > > > >> >>> Dear Community,
> > > > > > > >> >>>
> > > > > > > >> >>> I'm pleased to share this good news with everyone. As
> some
> > > of
> > > > > you
> > > > > > > may
> > > > > > > >> have already heard, Apache Flink has won the 2023 SIGMOD
> Systems
> > > > > Award
> > > > > > > [1].
> > > > > > > >> >>>
> > > > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > > > data-processing."
> > > > > > > --
> > > > > > > >> SIGMOD Awards Committee
> > > > > > > >> >>>
> > > > > > > >> >>> SIGMOD is one of the most influential data management
> > > research
> > > > > > > >> conferences in the world. The Systems Award is awarded to an
> > > > > individual
> > > > > > > or
> > > > > > > >> set of individuals to recognize the development of a
> software or
> > > > > > > hardware
> > > > > > > >> system whose technical contributions have had significant
> > > impact on
> > > > > the
> > > > > > > >> theory or practice of large-scale data management systems.
> > > Winning
> > > > > of
> > > > > > > the
> > > > > > > >> award indicates the high recognition of Flink's
> technological
> > > > > > > advancement
> > > > > > > >> and industry influence from academia.
> > > > > > > >> >>>
> > > > > > > >> >>> As an open-source project, Flink wouldn't have come
> this far
> > > > > without
> > > > > > > >> the wide, active and supportive community behind it. Kudos
> to
> > > all
> > > > > of us
> > > > > > > who
> > > > > > > >> helped make this happen, including the over 1,400
> contributors
> > > and
> > > > > many
> > > > > > > >> others who contributed in ways beyond code.
> > > > > > > >> >>>
> > > > > > > >> >>> Best,
> > > > > > > >> >>> Xintong (on behalf of the Flink PMC)
> > > > > > > >> >>>
> > > > > > > >> >>> [1] https://sigmod.org/2023-sigmod-systems-award/
> > > > > > > >> >>>
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > >
> > >
>

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Yangze Guo]

Congrats everyone!

Best,
Yangze Guo

On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> Congratulations!
>
> Best,
> Rui Fan
>
> On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu  wrote:
>
> > Congratulations everyone!
> >
> > Thanks,
> > Zhu
> >
> > Hang Ruan  于2023年7月4日周二 14:06写道：
> > >
> > > Congratulations!
> > >
> > > Best,
> > > Hang
> > >
> > > Jingsong Li  于2023年7月4日周二 13:47写道：
> > >
> > > > Congratulations!
> > > >
> > > > Thank you! All of the Flink community!
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Tue, Jul 4, 2023 at 1:24 PM tison  wrote:
> > > > >
> > > > > Congrats and with honor :D
> > > > >
> > > > > Best,
> > > > > tison.
> > > > >
> > > > >
> > > > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > > > >
> > > > > > Congratulations!--
> > > > > >
> > > > > > Best regards,
> > > > > > Mang Zhang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > > > >Congrats everyone
> > > > > > >
> > > > > > >Best,
> > > > > > >Ron
> > > > > > >
> > > > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > > > >
> > > > > > >> Congrats everyone!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Jark
> > > > > > >>
> > > > > > >> > 2023年7月3日 22:37，Yuval Itzchakov  写道：
> > > > > > >> >
> > > > > > >> > Congrats team!
> > > > > > >> >
> > > > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > > > u...@flink.apache.org
> > > > > > >> > wrote:
> > > > > > >> >> Congratulations!
> > > > > > >> >>
> > > > > > >> >> Best regards,
> > > > > > >> >> Jing
> > > > > > >> >>
> > > > > > >> >>
> > > > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > > > luoyu...@alumni.sjtu.edu.cn
> > > > > > >> > wrote:
> > > > > > >> >>> Congratulations!
> > > > > > >> >>>
> > > > > > >> >>> Best regards,
> > > > > > >> >>> Yuxia
> > > > > > >> >>>
> > > > > > >> >>> 发件人: "Pushpa Ramakrishnan"  > > >  > > > > > >> pushpa.ramakrish...@icloud.com>>
> > > > > > >> >>> 收件人: "Xintong Song"  > > > > > >> tonysong...@gmail.com>>
> > > > > > >> >>> 抄送: "dev"  > dev@flink.apache.org>>,
> > > > > > >> "User" mailto:u...@flink.apache.org>>
> > > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD
> > Systems
> > > > > > Award
> > > > > > >> >>>
> > > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > > >> >>>
> > > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> > tonysong...@gmail.com
> > > > > > >> > wrote:
> > > > > > >> >>>
> > > > > > >> >>> 
> > > > > > >> >>> Dear Community,
> > > > > > >> >>>
> > > > > > >> >>> I'm pleased to share this good news with everyone. As some
> > of
> > > > you
> > > > > > may
> > > > > > >> have already heard, Apache Flink has won the 2023 SIGMOD Systems
> > > > Award
> > > > > > [1].
> > > > > > >> >>>
> > > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > > data-processing."
> > > > > > --
> > > > > > >> SIGMOD Awards Committee
> > > > > > >> >>>
> > > > > > >> >>> SIGMOD is one of the most influential data management
> > research
> > > > > > >> conferences in the world. The Systems Award is awarded to an
> > > > individual
> > > > > > or
> > > > > > >> set of individuals to recognize the development of a software or
> > > > > > hardware
> > > > > > >> system whose technical contributions have had significant
> > impact on
> > > > the
> > > > > > >> theory or practice of large-scale data management systems.
> > Winning
> > > > of
> > > > > > the
> > > > > > >> award indicates the high recognition of Flink's technological
> > > > > > advancement
> > > > > > >> and industry influence from academia.
> > > > > > >> >>>
> > > > > > >> >>> As an open-source project, Flink wouldn't have come this far
> > > > without
> > > > > > >> the wide, active and supportive community behind it. Kudos to
> > all
> > > > of us
> > > > > > who
> > > > > > >> helped make this happen, including the over 1,400 contributors
> > and
> > > > many
> > > > > > >> others who contributed in ways beyond code.
> > > > > > >> >>>
> > > > > > >> >>> Best,
> > > > > > >> >>> Xintong (on behalf of the Flink PMC)
> > > > > > >> >>>
> > > > > > >> >>> [1] https://sigmod.org/2023-sigmod-systems-award/
> > > > > > >> >>>
> > > > > > >>
> > > > > > >>
> > > > > >
> > > >
> >

Re: Working improving the REST API

[David Morávek]

I've left some comments on the PR.

On Tue, Jul 4, 2023 at 9:20 AM Martijn Visser 
wrote:

> Hi Hong,
>
> Given that this changes the REST API, which is a public interface, I'm
> wondering if this shouldn't first have had a (small) FLIP if I follow the
> guidelines from the overview page [1].
>
> Best regards,
>
> Martijn
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
>
> On Mon, Jul 3, 2023 at 6:04 PM Teoh, Hong 
> wrote:
>
> > Just adding the updates from the Slack thread -
> > @SamratDeb and @JulietLee have expressed interest, and there is a PR
> ready
> > for review!
> > https://github.com/apache/flink/pull/22901
> >
> > Regards,
> > Hong
> >
> >
> > On 3 Jul 2023, at 16:56, Teoh, Hong 
> wrote:
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > Hi all,
> >
> > (This is a cross-post from a Slack message posted in #dev [1]. Posting
> > here for visibility)
> >
> > Recently I’ve been looking into improving the REST API that Flink has, to
> > make it more generally available for programmatic access (e.g. control
> > functions) rather than just for the Flink dashboard.
> >
> > In particular, various APIs seem to consume from the ExecutionGraph
> cache,
> > which is useful when displaying on a Flink dashboard (to guarantee
> > consistent behaviour), but since there is no way to “force” the latest
> > result, it might be not very useful for a control-function that wants to
> > retrieve the latest data. This could be achieved via Cache-Control HTTP
> > headers, as mentioned on this thread [2].
> >
> > Looking for any contributors / committers who are interested in working
> on
> > these items together! (To bounce ideas / deliver this faster)
> >
> >
> > [1]
> https://apache-flink.slack.com/archives/C03GV7L3G2C/p1688051972688149
> > [2] https://lists.apache.org/thread/7o330hfyoqqkkrfhtvz3kp448jcspjrm
> >
> >
> >
> > Regards,
> > Hong
> >
> >
> >
>

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Rui Fan]

Congratulations!

Best,
Rui Fan

On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu  wrote:

> Congratulations everyone!
>
> Thanks,
> Zhu
>
> Hang Ruan  于2023年7月4日周二 14:06写道：
> >
> > Congratulations!
> >
> > Best,
> > Hang
> >
> > Jingsong Li  于2023年7月4日周二 13:47写道：
> >
> > > Congratulations!
> > >
> > > Thank you! All of the Flink community!
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Tue, Jul 4, 2023 at 1:24 PM tison  wrote:
> > > >
> > > > Congrats and with honor :D
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > > >
> > > > > Congratulations!--
> > > > >
> > > > > Best regards,
> > > > > Mang Zhang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > > >Congrats everyone
> > > > > >
> > > > > >Best,
> > > > > >Ron
> > > > > >
> > > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > > >
> > > > > >> Congrats everyone!
> > > > > >>
> > > > > >> Best,
> > > > > >> Jark
> > > > > >>
> > > > > >> > 2023年7月3日 22:37，Yuval Itzchakov  写道：
> > > > > >> >
> > > > > >> > Congrats team!
> > > > > >> >
> > > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > > u...@flink.apache.org
> > > > > >> > wrote:
> > > > > >> >> Congratulations!
> > > > > >> >>
> > > > > >> >> Best regards,
> > > > > >> >> Jing
> > > > > >> >>
> > > > > >> >>
> > > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > > luoyu...@alumni.sjtu.edu.cn
> > > > > >> > wrote:
> > > > > >> >>> Congratulations!
> > > > > >> >>>
> > > > > >> >>> Best regards,
> > > > > >> >>> Yuxia
> > > > > >> >>>
> > > > > >> >>> 发件人: "Pushpa Ramakrishnan"  > >  > > > > >> pushpa.ramakrish...@icloud.com>>
> > > > > >> >>> 收件人: "Xintong Song"  > > > > >> tonysong...@gmail.com>>
> > > > > >> >>> 抄送: "dev"  dev@flink.apache.org>>,
> > > > > >> "User" mailto:u...@flink.apache.org>>
> > > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD
> Systems
> > > > > Award
> > > > > >> >>>
> > > > > >> >>> Congratulations \uD83E\uDD73
> > > > > >> >>>
> > > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song <
> tonysong...@gmail.com
> > > > > >> > wrote:
> > > > > >> >>>
> > > > > >> >>> 
> > > > > >> >>> Dear Community,
> > > > > >> >>>
> > > > > >> >>> I'm pleased to share this good news with everyone. As some
> of
> > > you
> > > > > may
> > > > > >> have already heard, Apache Flink has won the 2023 SIGMOD Systems
> > > Award
> > > > > [1].
> > > > > >> >>>
> > > > > >> >>> "Apache Flink greatly expanded the use of stream
> > > data-processing."
> > > > > --
> > > > > >> SIGMOD Awards Committee
> > > > > >> >>>
> > > > > >> >>> SIGMOD is one of the most influential data management
> research
> > > > > >> conferences in the world. The Systems Award is awarded to an
> > > individual
> > > > > or
> > > > > >> set of individuals to recognize the development of a software or
> > > > > hardware
> > > > > >> system whose technical contributions have had significant
> impact on
> > > the
> > > > > >> theory or practice of large-scale data management systems.
> Winning
> > > of
> > > > > the
> > > > > >> award indicates the high recognition of Flink's technological
> > > > > advancement
> > > > > >> and industry influence from academia.
> > > > > >> >>>
> > > > > >> >>> As an open-source project, Flink wouldn't have come this far
> > > without
> > > > > >> the wide, active and supportive community behind it. Kudos to
> all
> > > of us
> > > > > who
> > > > > >> helped make this happen, including the over 1,400 contributors
> and
> > > many
> > > > > >> others who contributed in ways beyond code.
> > > > > >> >>>
> > > > > >> >>> Best,
> > > > > >> >>> Xintong (on behalf of the Flink PMC)
> > > > > >> >>>
> > > > > >> >>> [1] https://sigmod.org/2023-sigmod-systems-award/
> > > > > >> >>>
> > > > > >>
> > > > > >>
> > > > >
> > >
>

[jira] [Created] (FLINK-32525) Update commons-beanutils to 1.9.4

[Martijn Visser (Jira)]

Martijn Visser created FLINK-32525:
--

 Summary: Update commons-beanutils to 1.9.4
 Key: FLINK-32525
 URL: https://issues.apache.org/jira/browse/FLINK-32525
 Project: Flink
  Issue Type: Technical Debt
  Components: Deployment / YARN
Reporter: Martijn Visser


YARN still tests with commons-beanutils 1.8.3 with a remark that beanutil 1.9+ 
doesn't work with Hadoop, but Hadoop 2.10.2 (which is our minimum supported 
version) uses beanutils 1.9.4 itself, per 
https://github.com/apache/hadoop/blob/rel/release-2.10.2/hadoop-project/pom.xml#L861-L863



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Working improving the REST API

[Martijn Visser]

Hi Hong,

Given that this changes the REST API, which is a public interface, I'm
wondering if this shouldn't first have had a (small) FLIP if I follow the
guidelines from the overview page [1].

Best regards,

Martijn

[1]
https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals

On Mon, Jul 3, 2023 at 6:04 PM Teoh, Hong 
wrote:

> Just adding the updates from the Slack thread -
> @SamratDeb and @JulietLee have expressed interest, and there is a PR ready
> for review!
> https://github.com/apache/flink/pull/22901
>
> Regards,
> Hong
>
>
> On 3 Jul 2023, at 16:56, Teoh, Hong  wrote:
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi all,
>
> (This is a cross-post from a Slack message posted in #dev [1]. Posting
> here for visibility)
>
> Recently I’ve been looking into improving the REST API that Flink has, to
> make it more generally available for programmatic access (e.g. control
> functions) rather than just for the Flink dashboard.
>
> In particular, various APIs seem to consume from the ExecutionGraph cache,
> which is useful when displaying on a Flink dashboard (to guarantee
> consistent behaviour), but since there is no way to “force” the latest
> result, it might be not very useful for a control-function that wants to
> retrieve the latest data. This could be achieved via Cache-Control HTTP
> headers, as mentioned on this thread [2].
>
> Looking for any contributors / committers who are interested in working on
> these items together! (To bounce ideas / deliver this faster)
>
>
> [1] https://apache-flink.slack.com/archives/C03GV7L3G2C/p1688051972688149
> [2] https://lists.apache.org/thread/7o330hfyoqqkkrfhtvz3kp448jcspjrm
>
>
>
> Regards,
> Hong
>
>
>

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

[David Morávek]

> They will struggle if they add new resources and nothing happens for 5
minutes.

The same applies if they start playing with FLIP-291 APIs. I'm wondering if
the cooldown makes sense there since it was the user's deliberate choice to
push new requirements. 樂

Best,
D.

On Tue, Jul 4, 2023 at 9:11 AM David Morávek  wrote:

> The FLIP reads sane to me. I'm unsure about the default values, though; 5
> minutes of wait time between rescales feels rather strict, and we should
> rethink it to provide a better out-of-the-box experience.
>
> I'd focus on newcomers trying AS / Reactive Mode out. They will struggle
> if they add new resources and nothing happens for 5 minutes. I'd suggest
> defaulting to
> *jobmanager.adaptive-scheduler.resource-stabilization-timeout* (which
> defaults to 10s).
>
> I'm still struggling to grasp max internal (force rescale). Ignoring 
> `AdaptiveScheduler#shouldRescale()`
> condition seems rather dangerous. Wouldn't a simple case where you add a
> new TM and remove it before the max interval is reached (so there is
> nothing to do) result in an unnecessary job restart?
>
> Best,
> D.
>
> On Thu, Jun 29, 2023 at 3:43 PM Etienne Chauchot 
> wrote:
>
>> Thanks Chesnay for your feedback. I have updated the FLIP. I'll start a
>> vote thread.
>>
>> Best
>>
>> Etienne
>>
>> Le 28/06/2023 à 11:49, Chesnay Schepler a écrit :
>> > > we should schedule a check that will rescale if
>> > min-parallelism-increase is met. Then, what it the use of
>> > scaling-interval.max timeout in that context ?
>> >
>> > To force a rescale if min-parallelism-increase is not met (but we
>> > could still run above the current parallelism).
>> >
>> > min-parallelism-increase is a trade-off between the cost of rescaling
>> > vs the performance benefit of the parallelism increase. Over time the
>> > balance tips more and more in favor of the parallelism increase, hence
>> > we should eventually rescale anyway even if the minimum isn't met, or
>> > at least give users the option to do so.
>> >
>> > > I meant the opposite: not having only the cooldown but having only
>> > the stabilization time. I must have missed something because what I
>> > wonder is: if every rescale entails a restart of the pipeline and
>> > every restart entails passing in waiting for resources state, then why
>> > introduce a cooldown when there is already at each rescale a stable
>> > resource timeout ?
>> >
>> > It is technically correct that the stable resource timeout can be used
>> > to limit the number of rescale operations per interval, however during
>> > that time the job isn't running, in contrast to the cooldown.
>> >
>> > Having both just gives you a lot more flexibility.
>> > "I want at most 1 rescale operation per hour, and wait at most 1
>> > minute for resource to stabilize when a rescale happens".
>> > You can't express this with only one of the options.
>> >
>> > On 20/06/2023 14:41, Etienne Chauchot wrote:
>> >> Hi Chesnay,
>> >>
>> >> Thanks for your feedback. Comments inline
>> >>
>> >> Le 16/06/2023 à 17:24, Chesnay Schepler a écrit :
>> >>> 1) Options specific to the adaptive scheduler should start with
>> >>> "jobmanager.adaptive-scheduler".
>> >>
>> >>
>> >> ok
>> >>
>> >>
>> >>> 2)
>> >>> There isn't /really /a notion of a "scaling event". The scheduler is
>> >>> informed about new/lost slots and job failures, and reacts
>> >>> accordingly by maybe rescaling the job.
>> >>> (sure, you can think of these as events, but you can think of
>> >>> practically everything as events)
>> >>>
>> >>> There shouldn't be a queue for events. All the scheduler should have
>> >>> to know is that the next rescale check is scheduled for time T,
>> >>> which in practice boils down to a flag and a scheduled action that
>> >>> runs Executing#maybeRescale.
>> >>
>> >>
>> >> Makes total sense, its very simple like this. Thanks for the
>> >> precision and pointer. After the related FLIPs, I'll look at the code
>> >> now.
>> >>
>> >>
>> >>> With that in mind, we also have to look at how we keep this state
>> >>> around. Presumably it is scoped to the current state, such that the
>> >>> cooldown is reset if a job fails.
>> >>> Maybe we should add a separate ExecutingWithCooldown state; not sure
>> >>> yet.
>> >>
>> >>
>> >> Yes loosing cooldown state and cooldown reset upon failure is what I
>> >> suggested in point 3 in previous email. Not sure either for a new
>> >> state, I'll figure it out after experimenting with the code. I'll
>> >> update the FLIP then.
>> >>
>> >>
>> >>>
>> >>> It would be good to clarify whether this FLIP only attempts to cover
>> >>> scale up operations, or also scale downs in case of slot losses.
>> >>
>> >>
>> >> When there are slots loss, most of the time it is due to a TM loss so
>> >> there should be several slots lost at the same time but (hopefully)
>> >> only once. There should not be many scale downs in a row (but still
>> >> cascading failures can happen). I think, we should just protect
>> >> against having 

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

[David Morávek]

The FLIP reads sane to me. I'm unsure about the default values, though; 5
minutes of wait time between rescales feels rather strict, and we should
rethink it to provide a better out-of-the-box experience.

I'd focus on newcomers trying AS / Reactive Mode out. They will struggle if
they add new resources and nothing happens for 5 minutes. I'd suggest
defaulting to *jobmanager.adaptive-scheduler.resource-stabilization-timeout*
(which
defaults to 10s).

I'm still struggling to grasp max internal (force rescale). Ignoring
`AdaptiveScheduler#shouldRescale()`
condition seems rather dangerous. Wouldn't a simple case where you add a
new TM and remove it before the max interval is reached (so there is
nothing to do) result in an unnecessary job restart?

Best,
D.

On Thu, Jun 29, 2023 at 3:43 PM Etienne Chauchot 
wrote:

> Thanks Chesnay for your feedback. I have updated the FLIP. I'll start a
> vote thread.
>
> Best
>
> Etienne
>
> Le 28/06/2023 à 11:49, Chesnay Schepler a écrit :
> > > we should schedule a check that will rescale if
> > min-parallelism-increase is met. Then, what it the use of
> > scaling-interval.max timeout in that context ?
> >
> > To force a rescale if min-parallelism-increase is not met (but we
> > could still run above the current parallelism).
> >
> > min-parallelism-increase is a trade-off between the cost of rescaling
> > vs the performance benefit of the parallelism increase. Over time the
> > balance tips more and more in favor of the parallelism increase, hence
> > we should eventually rescale anyway even if the minimum isn't met, or
> > at least give users the option to do so.
> >
> > > I meant the opposite: not having only the cooldown but having only
> > the stabilization time. I must have missed something because what I
> > wonder is: if every rescale entails a restart of the pipeline and
> > every restart entails passing in waiting for resources state, then why
> > introduce a cooldown when there is already at each rescale a stable
> > resource timeout ?
> >
> > It is technically correct that the stable resource timeout can be used
> > to limit the number of rescale operations per interval, however during
> > that time the job isn't running, in contrast to the cooldown.
> >
> > Having both just gives you a lot more flexibility.
> > "I want at most 1 rescale operation per hour, and wait at most 1
> > minute for resource to stabilize when a rescale happens".
> > You can't express this with only one of the options.
> >
> > On 20/06/2023 14:41, Etienne Chauchot wrote:
> >> Hi Chesnay,
> >>
> >> Thanks for your feedback. Comments inline
> >>
> >> Le 16/06/2023 à 17:24, Chesnay Schepler a écrit :
> >>> 1) Options specific to the adaptive scheduler should start with
> >>> "jobmanager.adaptive-scheduler".
> >>
> >>
> >> ok
> >>
> >>
> >>> 2)
> >>> There isn't /really /a notion of a "scaling event". The scheduler is
> >>> informed about new/lost slots and job failures, and reacts
> >>> accordingly by maybe rescaling the job.
> >>> (sure, you can think of these as events, but you can think of
> >>> practically everything as events)
> >>>
> >>> There shouldn't be a queue for events. All the scheduler should have
> >>> to know is that the next rescale check is scheduled for time T,
> >>> which in practice boils down to a flag and a scheduled action that
> >>> runs Executing#maybeRescale.
> >>
> >>
> >> Makes total sense, its very simple like this. Thanks for the
> >> precision and pointer. After the related FLIPs, I'll look at the code
> >> now.
> >>
> >>
> >>> With that in mind, we also have to look at how we keep this state
> >>> around. Presumably it is scoped to the current state, such that the
> >>> cooldown is reset if a job fails.
> >>> Maybe we should add a separate ExecutingWithCooldown state; not sure
> >>> yet.
> >>
> >>
> >> Yes loosing cooldown state and cooldown reset upon failure is what I
> >> suggested in point 3 in previous email. Not sure either for a new
> >> state, I'll figure it out after experimenting with the code. I'll
> >> update the FLIP then.
> >>
> >>
> >>>
> >>> It would be good to clarify whether this FLIP only attempts to cover
> >>> scale up operations, or also scale downs in case of slot losses.
> >>
> >>
> >> When there are slots loss, most of the time it is due to a TM loss so
> >> there should be several slots lost at the same time but (hopefully)
> >> only once. There should not be many scale downs in a row (but still
> >> cascading failures can happen). I think, we should just protect
> >> against having scale ups immediately following. For that, I think we
> >> could just keep the current behavior of transitioning to Restarting
> >> state and then back to Waiting for Resources state. This state will
> >> protect us against scale ups immediately following failure/restart.
> >>
> >>
> >>>
> >>> We should also think about how it relates to the externalized
> >>> declarative resource management. Should we always rescale
> >>> immediately? Should we wait 

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

[David Morávek]

Hmm, sorry for the confusion; it seems that Google is playing games with me
(I see this chained under the old [DISCUSS] thread), but it seems correct
in the mail archive [1] :/

Just ignore the -1 above.

[1] https://lists.apache.org/thread/22fovrkmzcvzblcohhtsp5t96vd64obj

On Tue, Jul 4, 2023 at 8:49 AM David Morávek  wrote:

> The vote closes within 6 hours and, as for now, there was no vote. This
>> is a very short FLIP, that takes a few minutes to read.
>>
>
> Maybe because there should have been a dedicated voting thread (marked as
> [VOTE]), this one was hidden and hard to notice.
>
> We should restart the vote with proper mechanics to allow everyone to
> participate.
>
> Soft -1 from my side until there is a proper voting thread.
>
> Best,
> D.
>
> On Mon, Jul 3, 2023 at 4:40 PM ConradJam  wrote:
>
>> +1 (no-binding)
>>
>> Etienne Chauchot  于2023年7月3日周一 15:57写道：
>>
>> > Hi all,
>> >
>> > The vote closes within 6 hours and, as for now, there was no vote. This
>> > is a very short FLIP, that takes a few minutes to read.
>> >
>> > Please cast your vote so that the development could start.
>> >
>> > Thanks.
>> >
>> > Best
>> >
>> > Etienne
>> >
>> > Le 29/06/2023 à 15:51, Etienne Chauchot a écrit :
>> > >
>> > > Hi all,
>> > >
>> > > Thanks for all the feedback about the FLIP-322: Cooldown period for
>> > > adaptive scheduler [1].
>> > >
>> > > This FLIP was discussed in [2].
>> > >
>> > > I'd like to start a vote for it. The vote will be open for at least 72
>> > > hours (until July 3rd 14:00 GMT) unless there is an objection or
>> > > insufficient votes.
>> > >
>> > > [1]
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-322+Cooldown+period+for+adaptive+scheduler
>> > > [2] https://lists.apache.org/thread/qvgxzhbp9rhlsqrybxdy51h05zwxfns6
>> > >
>> > > Best,
>> > >
>> > > Etienne
>> > >
>>
>

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

[David Morávek]

>
> The vote closes within 6 hours and, as for now, there was no vote. This
> is a very short FLIP, that takes a few minutes to read.
>

Maybe because there should have been a dedicated voting thread (marked as
[VOTE]), this one was hidden and hard to notice.

We should restart the vote with proper mechanics to allow everyone to
participate.

Soft -1 from my side until there is a proper voting thread.

Best,
D.

On Mon, Jul 3, 2023 at 4:40 PM ConradJam  wrote:

> +1 (no-binding)
>
> Etienne Chauchot  于2023年7月3日周一 15:57写道：
>
> > Hi all,
> >
> > The vote closes within 6 hours and, as for now, there was no vote. This
> > is a very short FLIP, that takes a few minutes to read.
> >
> > Please cast your vote so that the development could start.
> >
> > Thanks.
> >
> > Best
> >
> > Etienne
> >
> > Le 29/06/2023 à 15:51, Etienne Chauchot a écrit :
> > >
> > > Hi all,
> > >
> > > Thanks for all the feedback about the FLIP-322: Cooldown period for
> > > adaptive scheduler [1].
> > >
> > > This FLIP was discussed in [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours (until July 3rd 14:00 GMT) unless there is an objection or
> > > insufficient votes.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-322+Cooldown+period+for+adaptive+scheduler
> > > [2] https://lists.apache.org/thread/qvgxzhbp9rhlsqrybxdy51h05zwxfns6
> > >
> > > Best,
> > >
> > > Etienne
> > >
>

[jira] [Created] (FLINK-32524) Improve the watermark aggregation performance when enabling the watermark alignment

[Rui Fan (Jira)]

Rui Fan created FLINK-32524:
---

 Summary: Improve the watermark aggregation performance when 
enabling the watermark alignment
 Key: FLINK-32524
 URL: https://issues.apache.org/jira/browse/FLINK-32524
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Common
Affects Versions: 1.18.0
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: 1.18.0


Improve the watermark aggregation performance when enabling the watermark 
alignment, and add the related benchmark.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Zhu Zhu]

Congratulations everyone!

Thanks,
Zhu

Hang Ruan  于2023年7月4日周二 14:06写道：
>
> Congratulations!
>
> Best,
> Hang
>
> Jingsong Li  于2023年7月4日周二 13:47写道：
>
> > Congratulations!
> >
> > Thank you! All of the Flink community!
> >
> > Best,
> > Jingsong
> >
> > On Tue, Jul 4, 2023 at 1:24 PM tison  wrote:
> > >
> > > Congrats and with honor :D
> > >
> > > Best,
> > > tison.
> > >
> > >
> > > Mang Zhang  于2023年7月4日周二 11:08写道：
> > >
> > > > Congratulations!--
> > > >
> > > > Best regards,
> > > > Mang Zhang
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > > >Congrats everyone
> > > > >
> > > > >Best,
> > > > >Ron
> > > > >
> > > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > > >
> > > > >> Congrats everyone!
> > > > >>
> > > > >> Best,
> > > > >> Jark
> > > > >>
> > > > >> > 2023年7月3日 22:37，Yuval Itzchakov  写道：
> > > > >> >
> > > > >> > Congrats team!
> > > > >> >
> > > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> > u...@flink.apache.org
> > > > >> > wrote:
> > > > >> >> Congratulations!
> > > > >> >>
> > > > >> >> Best regards,
> > > > >> >> Jing
> > > > >> >>
> > > > >> >>
> > > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> > luoyu...@alumni.sjtu.edu.cn
> > > > >> > wrote:
> > > > >> >>> Congratulations!
> > > > >> >>>
> > > > >> >>> Best regards,
> > > > >> >>> Yuxia
> > > > >> >>>
> > > > >> >>> 发件人: "Pushpa Ramakrishnan"  >  > > > >> pushpa.ramakrish...@icloud.com>>
> > > > >> >>> 收件人: "Xintong Song"  > > > >> tonysong...@gmail.com>>
> > > > >> >>> 抄送: "dev" mailto:dev@flink.apache.org>>,
> > > > >> "User" mailto:u...@flink.apache.org>>
> > > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems
> > > > Award
> > > > >> >>>
> > > > >> >>> Congratulations \uD83E\uDD73
> > > > >> >>>
> > > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song  > > > >> > wrote:
> > > > >> >>>
> > > > >> >>> 
> > > > >> >>> Dear Community,
> > > > >> >>>
> > > > >> >>> I'm pleased to share this good news with everyone. As some of
> > you
> > > > may
> > > > >> have already heard, Apache Flink has won the 2023 SIGMOD Systems
> > Award
> > > > [1].
> > > > >> >>>
> > > > >> >>> "Apache Flink greatly expanded the use of stream
> > data-processing."
> > > > --
> > > > >> SIGMOD Awards Committee
> > > > >> >>>
> > > > >> >>> SIGMOD is one of the most influential data management research
> > > > >> conferences in the world. The Systems Award is awarded to an
> > individual
> > > > or
> > > > >> set of individuals to recognize the development of a software or
> > > > hardware
> > > > >> system whose technical contributions have had significant impact on
> > the
> > > > >> theory or practice of large-scale data management systems. Winning
> > of
> > > > the
> > > > >> award indicates the high recognition of Flink's technological
> > > > advancement
> > > > >> and industry influence from academia.
> > > > >> >>>
> > > > >> >>> As an open-source project, Flink wouldn't have come this far
> > without
> > > > >> the wide, active and supportive community behind it. Kudos to all
> > of us
> > > > who
> > > > >> helped make this happen, including the over 1,400 contributors and
> > many
> > > > >> others who contributed in ways beyond code.
> > > > >> >>>
> > > > >> >>> Best,
> > > > >> >>> Xintong (on behalf of the Flink PMC)
> > > > >> >>>
> > > > >> >>> [1] https://sigmod.org/2023-sigmod-systems-award/
> > > > >> >>>
> > > > >>
> > > > >>
> > > >
> >

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

[Hang Ruan]

Congratulations!

Best,
Hang

Jingsong Li  于2023年7月4日周二 13:47写道：

> Congratulations!
>
> Thank you! All of the Flink community!
>
> Best,
> Jingsong
>
> On Tue, Jul 4, 2023 at 1:24 PM tison  wrote:
> >
> > Congrats and with honor :D
> >
> > Best,
> > tison.
> >
> >
> > Mang Zhang  于2023年7月4日周二 11:08写道：
> >
> > > Congratulations!--
> > >
> > > Best regards,
> > > Mang Zhang
> > >
> > >
> > >
> > >
> > >
> > > 在 2023-07-04 01:53:46，"liu ron"  写道：
> > > >Congrats everyone
> > > >
> > > >Best,
> > > >Ron
> > > >
> > > >Jark Wu  于2023年7月3日周一 22:48写道：
> > > >
> > > >> Congrats everyone!
> > > >>
> > > >> Best,
> > > >> Jark
> > > >>
> > > >> > 2023年7月3日 22:37，Yuval Itzchakov  写道：
> > > >> >
> > > >> > Congrats team!
> > > >> >
> > > >> > On Mon, Jul 3, 2023, 17:28 Jing Ge via user <
> u...@flink.apache.org
> > > >> > wrote:
> > > >> >> Congratulations!
> > > >> >>
> > > >> >> Best regards,
> > > >> >> Jing
> > > >> >>
> > > >> >>
> > > >> >> On Mon, Jul 3, 2023 at 3:21 PM yuxia <
> luoyu...@alumni.sjtu.edu.cn
> > > >> > wrote:
> > > >> >>> Congratulations!
> > > >> >>>
> > > >> >>> Best regards,
> > > >> >>> Yuxia
> > > >> >>>
> > > >> >>> 发件人: "Pushpa Ramakrishnan"   > > >> pushpa.ramakrish...@icloud.com>>
> > > >> >>> 收件人: "Xintong Song"  > > >> tonysong...@gmail.com>>
> > > >> >>> 抄送: "dev" mailto:dev@flink.apache.org>>,
> > > >> "User" mailto:u...@flink.apache.org>>
> > > >> >>> 发送时间: 星期一, 2023年 7 月 03日 下午 8:36:30
> > > >> >>> 主题: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems
> > > Award
> > > >> >>>
> > > >> >>> Congratulations \uD83E\uDD73
> > > >> >>>
> > > >> >>> On 03-Jul-2023, at 3:30 PM, Xintong Song  > > >> > wrote:
> > > >> >>>
> > > >> >>> 
> > > >> >>> Dear Community,
> > > >> >>>
> > > >> >>> I'm pleased to share this good news with everyone. As some of
> you
> > > may
> > > >> have already heard, Apache Flink has won the 2023 SIGMOD Systems
> Award
> > > [1].
> > > >> >>>
> > > >> >>> "Apache Flink greatly expanded the use of stream
> data-processing."
> > > --
> > > >> SIGMOD Awards Committee
> > > >> >>>
> > > >> >>> SIGMOD is one of the most influential data management research
> > > >> conferences in the world. The Systems Award is awarded to an
> individual
> > > or
> > > >> set of individuals to recognize the development of a software or
> > > hardware
> > > >> system whose technical contributions have had significant impact on
> the
> > > >> theory or practice of large-scale data management systems. Winning
> of
> > > the
> > > >> award indicates the high recognition of Flink's technological
> > > advancement
> > > >> and industry influence from academia.
> > > >> >>>
> > > >> >>> As an open-source project, Flink wouldn't have come this far
> without
> > > >> the wide, active and supportive community behind it. Kudos to all
> of us
> > > who
> > > >> helped make this happen, including the over 1,400 contributors and
> many
> > > >> others who contributed in ways beyond code.
> > > >> >>>
> > > >> >>> Best,
> > > >> >>> Xintong (on behalf of the Flink PMC)
> > > >> >>>
> > > >> >>> [1] https://sigmod.org/2023-sigmod-systems-award/
> > > >> >>>
> > > >>
> > > >>
> > >
>