[jira] [Created] (FLINK-34410) Disable nightly trigger in forks

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-34410:
-

 Summary: Disable nightly trigger in forks
 Key: FLINK-34410
 URL: https://issues.apache.org/jira/browse/FLINK-34410
 Project: Flink
  Issue Type: Technical Debt
  Components: Build System / CI
Affects Versions: 1.20.0
Reporter: Matthias Pohl


We can disable the automatic triggering of the nightly trigger workflow in fork 
(see [GHA 
docs|https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions]s:
{code}
if: github.repository == 'octo-org/octo-repo-prod'
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

[Piotr Nowojski]

Hey

> AFAIK, there's no way to set UIDs for a SQL job,

AFAIK you can't set UID manually, but  Flink SQL generates a compiled plan
of a query with embedded UIDs. As I understand it, using a compiled plan is
the preferred (only?) way for Flink SQL if one wants to make any changes to
query later on or support Flink's runtime upgrades, without losing the
state.

If that's the case, what would be the usefulness of this FLIP? Only for
DataStream API for users that didn't know that they should have manually
configured UIDs? But they have the workaround to actually post-factum add
the UIDs anyway, right? So maybe indeed Chesnay is right that this FLIP is
not that helpful/worth the extra effort?

Best,
Piotrek

czw., 8 lut 2024 o 03:55 Zhanghao Chen 
napisał(a):

> Hi Chesnay,
>
> AFAIK, there's no way to set UIDs for a SQL job, it'll be great if you can
> share how you allow UID setting for SQL jobs. We've explored providing a
> visualized DAG editor for SQL jobs that allows UID setting on our internal
> platform, but most users found it too complicated to use. Another
> possible way is to utilize SQL hints, but that's complicated as well. From
> our experience, many SQL users are not familiar with Flink, what they want
> is an experience similar to writing a normal SQL in MySQL, without
> involving much extra concepts like the DAG and the UID. In fact, some
> DataStream and PyFlink users also share the same concern.
>
> On the other hand, some performance-tuning is inevitable for a
> long-running jobs in production, and parallelism tuning is among the most
> common techniques. FLIP-367 [1] and FLIP-146 [2] allow user to tune the
> parallelism of source and sinks, and both are well-received in the
> discussion thread. Users definitely don't want to lost state after a
> parallelism tuning, which is highly risky at present.
>
> Putting these together, I think the FLIP has a high value in production.
> Through offline discussion, I leant that multiple companies have developed
> or trying to develop similar hasher changes in their internal distribution,
> including ByteDance, Xiaohongshu, and Bilibili. It'll be great if we can
> improve the SQL experience for all community users as well, WDYT?
>
> Best,
> Zhanghao Chen
> --
> *From:* Chesnay Schepler 
> *Sent:* Thursday, February 8, 2024 2:01
> *To:* dev@flink.apache.org ; Zhanghao Chen <
> zhanghao.c...@outlook.com>; Piotr Nowojski ; Yu
> Chen 
> *Subject:* Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID
> generation for improved state compatibility on parallelism change
>
> The FLIP is a bit weird to be honest. It only applies in cases where
> users haven't set uids, but that goes against best-practices and as far
> as I'm told SQL also sets UIDs everywhere.
>
> I'm wondering if this is really worth the effort.
>
> On 07/02/2024 10:23, Zhanghao Chen wrote:
> > After offline discussion with @Yu Chen >, I've updated the FLIP [1] to include a design
> that allows for compatible hasher upgrade by adding StreamGraphHasherV2 to
> the legacy hasher list, which is actually a revival of the idea from
> FLIP-5290 [2] when StreamGraphHasherV2 was introduced in Flink 1.2. We're
> targeting to make V3 the default hasher in Flink 1.20 given that
> state-compatibility is no longer an issue. Take a review when you have a
> chance, and I'd like to especially thank @Yu Chen<
> mailto:yuchen.e...@gmail.com > for the through
> offline discussion and code debugging help to make this possible.
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-411%3A+Chaining-agnostic+Operator+ID+generation+for+improved+state+compatibility+on+parallelism+change
> > [2] https://issues.apache.org/jira/browse/FLINK-5290
> >
> > Best,
> > Zhanghao Chen
> > 
> > From: Zhanghao Chen 
> > Sent: Friday, January 12, 2024 10:46
> > To: Piotr Nowojski ; Yu Chen <
> yuchen.e...@gmail.com>
> > Cc: dev@flink.apache.org 
> > Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID
> generation for improved state compatibility on parallelism change
> >
> > Thanks for the input, Piotr. It might still be possible to make it
> compatible with the old snapshots, following the direction of FLINK-5290<
> https://issues.apache.org/jira/browse/FLINK-5290> suggested by Yu. I'll
> discuss with Yu on more details.
> >
> > Best,
> > Zhanghao Chen
> > 
> > From: Piotr Nowojski 
> > Sent: Friday, January 12, 2024 1:55
> > To: Yu Chen 
> > Cc: Zhanghao Chen ; dev@flink.apache.org <
> dev@flink.apache.org>
> > Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID
> generation for improved state compatibility on parallelism change
> >
> > Hi,
> >
> > Using unaligned checkpoints is orthogonal to this FLIP.
> >
> > Yes, unaligned checkpoints are not supported for pointwise connections,
> so most of the cases go away anyway.
> > It is possible to switch from unchained to 

Re: [DISCUSS] Alternative way of posting FLIPs

[Piotr Nowojski]

+1 for the first option as well

Best,
Piotrek

śr., 7 lut 2024 o 16:48 Matthias Pohl 
napisał(a):

> +1 for option 1 since it's a reasonable temporary workaround
>
> Moving to GitHub discussions would either mean moving the current FLIP
> collection or having the FLIPs in two locations. Both options do not seem
> to be optimal. Another concern I had was that GitHub Discussions wouldn't
> allow integrating diagrams that easily. But it looks like they support
> Mermaid [1] for diagrams.
>
> One flaw of the GoogleDocs approach is, though, that we have to rely on
> diagrams being provided as PNG/JPG/SVG rather than draw.io diagrams.
> draw.io
> is more tightly integrated with the Confluence wiki which allows
> editing/updating diagrams in the wiki rather than using some external tool.
> Google Draw is also not that convenient to use in my opinion. Anyway,
> that's a minor issue, I guess.
>
> Matthias
>
> [1]
>
> https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-diagrams
>
> On Wed, Feb 7, 2024 at 3:30 PM Lincoln Lee  wrote:
>
> > Thanks Martijn moving this forward!
> >
> > +1 for the first solution, because as of now it looks like this is a
> > temporary solution and we're still looking forward to the improvement by
> > ASF Infra, when the access is ok for contributors, we can back to the
> > current workflow.
> >
> > For solution 2, one visible downside is that it becomes inconvenient to
> > look for flips (unless we permanently switch to github discussion).
> >
> > Looking forward to hearing more thoughts.
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Martijn Visser  于2024年2月7日周三 21:51写道：
> >
> > > Hi all,
> > >
> > > ASF Infra has confirmed to me that only ASF committers can access the
> > > ASF Confluence site since a recent change. One of the results of this
> > > decision is that users can't signup and access Confluence, so only
> > > committers+ can create FLIPs.
> > >
> > > ASF Infra hopes to improve this situation when they move to the Cloud
> > > shortly (as in: some months), but they haven't committed on an actual
> > > date. The idea would be that we find a temporary solution until anyone
> > > can request access to Confluence.
> > >
> > > There are a couple of ways we could resolve this situation:
> > > 1. Contributors create a Google Doc and make that view-only, and post
> > > that Google Doc to the mailing list for a discussion thread. When the
> > > discussions have been resolved, the contributor ask on the Dev mailing
> > > list to a committer/PMC to copy the contents from the Google Doc, and
> > > create a FLIP number for them. The contributor can then use that FLIP
> > > to actually have a VOTE thread.
> > > 2. We could consider moving FLIPs to "Discussions" on Github, like
> > > Airflow does at https://github.com/apache/airflow/discussions
> > > 3. Perhaps someone else has another good idea.
> > >
> > > Looking forward to your thoughts.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> >
>

Re: FW: RE: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

[Yong Fang]

Hi devs,

According to the online-discussion in FLINK-3127 [1] and offline-discussion
with Maciej Obuchowski and Zhenqiu Huang, we would like to update the
lineage vertex relevant interfaces in FLIP-314 [2] as follows:

1. Introduce `LineageDataset` which represents source and sink in
`LineageVertex`. The fields in `LineageDataset` are as follows:
/* Name for this particular dataset. */
String name;
/* Unique name for this dataset's storage, for example, url for jdbc
connector and location for lakehouse connector. */
String namespace;
/* Facets for the lineage vertex to describe the particular information
of dataset, such as schema and config. */
Map facets;

2. There may be multiple datasets in one `LineageVertex`, for example,
kafka source or hybrid source. So users can get dataset list from
`LineageVertex`:
/** Get datasets from the lineage vertex. */
List datasets();

3. There will be built in facets for config and schema. To describe columns
in table/sql jobs and datastream jobs, we introduce `DatasetSchemaField`.
/** Builtin config facet for dataset. */
@PublicEvolving
public interface DatasetConfigFacet extends LineageDatasetFacet {
Map config();
}

/** Field for schema in dataset. */
public interface DatasetSchemaField {
/** The name of the field. */
String name();
/** The type of the field. */
T type();
}

Thanks for valuable inputs from @Maciej and @Zhenqiu. And looking forward
to your feedback, thanks

Best,
Fang Yong

On Mon, Sep 25, 2023 at 1:18 PM Shammon FY  wrote:

> Hi David,
>
> Do you want the detailed topology for Flink job? You can get
> `JobDetailsInfo` in `RestCusterClient` with the submitted job id, it has
> `String jsonPlan`. You can parse the json plan to get all steps and
> relations between them in a Flink job. Hope this can help you, thanks!
>
> Best,
> Shammon FY
>
> On Tue, Sep 19, 2023 at 11:46 PM David Radley 
> wrote:
>
>> Hi there,
>> I am looking at the interfaces. If I am reading it correctly,there is one
>> relationship between the source and sink and this relationship represents
>> the operational lineage. Lineage is usually represented as asset -> process
>> - > asset – see for example
>> https://egeria-project.org/features/lineage-management/overview/#the-lineage-graph
>>
>> Maybe I am missing it, but it seems to be that it would be useful to
>> store the process in the lineage graph.
>>
>> It is useful to have the top level lineage as source -> Flink job ->
>> sink. Where the Flink job is the process, but also to have this asset ->
>> process -> asset pattern for each of the steps in the job. If this is
>> present, please could you point me to it,
>>
>>   Kind regards, David.
>>
>>
>>
>>
>>
>> From: David Radley 
>> Date: Tuesday, 19 September 2023 at 16:11
>> To: dev@flink.apache.org 
>> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-314: Support Customized Job
>> Lineage Listener
>> Hi,
>> I notice that there is an experimental lineage integration for Flink with
>> OpenLineage https://openlineage.io/docs/integrations/flink  . I think
>> this feature would allow for a superior Flink OpenLineage integration,
>> Kind regards, David.
>>
>> From: XTransfer 
>> Date: Tuesday, 19 September 2023 at 15:47
>> To: dev@flink.apache.org 
>> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-314: Support Customized Job
>> Lineage Listener
>> Thanks Shammon for this proposal.
>>
>> That’s helpful for collecting the lineage of Flink tasks.
>> Looking forward to its implementation.
>>
>> Best,
>> Jiabao
>>
>>
>> > 2023年9月18日 20:56，Leonard Xu  写道：
>> >
>> > Thanks Shammon for the informations, the comment makes the lifecycle
>> clearer.
>> > +1
>> >
>> >
>> > Best,
>> > Leonard
>> >
>> >
>> >> On Sep 18, 2023, at 7:54 PM, Shammon FY  wrote:
>> >>
>> >> Hi devs,
>> >>
>> >> After discussing with @Qingsheng, I fixed a minor issue of the lineage
>> lifecycle in `StreamExecutionEnvironment`. I have added the comment to
>> explain that the lineage information in `StreamExecutionEnvironment` will
>> be consistent with that of transformations. When users clear the existing
>> transformations, the added lineage information will also be deleted.
>> >>
>> >> Please help to review it again, and If there are no more concerns
>> about FLIP-314[1], I would like to start voting later, thanks. cc @
>> <>Leonard
>> >>
>> >> Best,
>> >> Shammon FY
>> >>
>> >> On Mon, Jul 17, 2023 at 3:43 PM Shammon FY > zjur...@gmail.com>> wrote:
>> >> Hi devs,
>> >>
>> >> Thanks for all the valuable feedback. If there are no more concerns
>> about FLIP-314[1], I would like to start voting later, thanks.
>> >>
>> >>
>> >> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
>>  <
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
>> >
>> >>
>> >> Best,
>> >> Shammon FY
>> >>
>> >>
>> >> On Wed, Jul 12, 2023 at 11:18 

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

[Zhanghao Chen]

Hi Chesnay,

AFAIK, there's no way to set UIDs for a SQL job, it'll be great if you can 
share how you allow UID setting for SQL jobs. We've explored providing a 
visualized DAG editor for SQL jobs that allows UID setting on our internal 
platform, but most users found it too complicated to use. Another possible way 
is to utilize SQL hints, but that's complicated as well. From our experience, 
many SQL users are not familiar with Flink, what they want is an experience 
similar to writing a normal SQL in MySQL, without involving much extra concepts 
like the DAG and the UID. In fact, some DataStream and PyFlink users also share 
the same concern.

On the other hand, some performance-tuning is inevitable for a long-running 
jobs in production, and parallelism tuning is among the most common techniques. 
FLIP-367 [1] and FLIP-146 [2] allow user to tune the parallelism of source and 
sinks, and both are well-received in the discussion thread. Users definitely 
don't want to lost state after a parallelism tuning, which is highly risky at 
present.

Putting these together, I think the FLIP has a high value in production. 
Through offline discussion, I leant that multiple companies have developed or 
trying to develop similar hasher changes in their internal distribution, 
including ByteDance, Xiaohongshu, and Bilibili. It'll be great if we can 
improve the SQL experience for all community users as well, WDYT?

Best,
Zhanghao Chen

From: Chesnay Schepler 
Sent: Thursday, February 8, 2024 2:01
To: dev@flink.apache.org ; Zhanghao Chen 
; Piotr Nowojski ; Yu Chen 

Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

The FLIP is a bit weird to be honest. It only applies in cases where
users haven't set uids, but that goes against best-practices and as far
as I'm told SQL also sets UIDs everywhere.

I'm wondering if this is really worth the effort.

On 07/02/2024 10:23, Zhanghao Chen wrote:
> After offline discussion with @Yu Chen, I've 
> updated the FLIP [1] to include a design that allows for compatible hasher 
> upgrade by adding StreamGraphHasherV2 to the legacy hasher list, which is 
> actually a revival of the idea from FLIP-5290 [2] when StreamGraphHasherV2 
> was introduced in Flink 1.2. We're targeting to make V3 the default hasher in 
> Flink 1.20 given that state-compatibility is no longer an issue. Take a 
> review when you have a chance, and I'd like to especially thank @Yu 
> Chen for the through offline discussion and 
> code debugging help to make this possible.
>
> [1] 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-411%3A+Chaining-agnostic+Operator+ID+generation+for+improved+state+compatibility+on+parallelism+change
> [2] https://issues.apache.org/jira/browse/FLINK-5290
>
> Best,
> Zhanghao Chen
> 
> From: Zhanghao Chen 
> Sent: Friday, January 12, 2024 10:46
> To: Piotr Nowojski ; Yu Chen 
> Cc: dev@flink.apache.org 
> Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
> improved state compatibility on parallelism change
>
> Thanks for the input, Piotr. It might still be possible to make it compatible 
> with the old snapshots, following the direction of 
> FLINK-5290 suggested by Yu. 
> I'll discuss with Yu on more details.
>
> Best,
> Zhanghao Chen
> 
> From: Piotr Nowojski 
> Sent: Friday, January 12, 2024 1:55
> To: Yu Chen 
> Cc: Zhanghao Chen ; dev@flink.apache.org 
> 
> Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
> improved state compatibility on parallelism change
>
> Hi,
>
> Using unaligned checkpoints is orthogonal to this FLIP.
>
> Yes, unaligned checkpoints are not supported for pointwise connections, so 
> most of the cases go away anyway.
> It is possible to switch from unchained to chained subtasks by removing a 
> keyBy exchange, and this would be
> a problem, but that's just one of the things that we claim that unaligned 
> checkpoints do not support [1]. But as
> I stated above, this is an orthogonal issue to this FLIP.
>
> Regarding the proposal itself, generally speaking it makes sense to me as 
> well. However I'm quite worried about
> the compatibility and/or migration path. The:
>> (v2.0) Make HasherV3 the default hasher, mark HasherV2 deprecated.
> step would break the compatibility with Flink 1.xx snapshots. But as this is 
> for v2.0, maybe that's not the end of
> the world?
>
> Best,
> Piotrek
>
> [1] 
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/checkpoints_vs_savepoints/#capabilities-and-limitations
>
> czw., 11 sty 2024 o 12:10 Yu Chen 
> mailto:yuchen.e...@gmail.com>> napisał(a):
> Hi Zhanghao,
>
> Actually, Stefan has done similar compatibility work in the early 
> FLINK-5290[1], where he 

Re: Impact of redacting UPDATE_BEFORE fields?

[Yaroslav Tkachenko]

Hey Kevin,

In my experience it mostly depends on the type of your sinks. If all of
your sinks can leverage primary keys and support upsert semantics, you
don't really need UPDATE_BEFOREs altogether (you can even filter them out).
But if you have sinks with append-only semantics (OR if you don't have
primary keys defined) you need UPDATE_BEFOREs to correctly support
retractions (in case of updates and deletes).

Great talk on this topic:
https://www.youtube.com/watch?v=iRlLaY-P6iE_channel=PlainSchwarz (the
middle part is the most relevant).


On Wed, Feb 7, 2024 at 12:13 PM Kevin Lam 
wrote:

> Hi there!
>
> I have a question about Changelog Stream Processing with Flink SQL and the
> Flink Table API. I would like to better understand how UPDATE_BEFORE fields
> are used by Flink.
>
> Our team uses Debezium to extract Change Data Capture events from MySQL
> databases. We currently redact the `before` fields in the envelope [0] so
> that redacted PII doesn't sit in our Kafka topics in the `before` field of
> UPDATE events.
>
> As a result if we were to consume these CDC streams with Flink, there would
> be missing UPDATE_BEFORE fields for UPDATE events. What kind of impact
> would this have on performance and correctness, if any? Any other
> considerations we should be aware of?
>
> Thanks in advance for your help!
>
>
> [0]
> https://debezium.io/documentation/reference/stable/connectors/mysql.html
>

Impact of redacting UPDATE_BEFORE fields?

[Kevin Lam]

Hi there!

I have a question about Changelog Stream Processing with Flink SQL and the
Flink Table API. I would like to better understand how UPDATE_BEFORE fields
are used by Flink.

Our team uses Debezium to extract Change Data Capture events from MySQL
databases. We currently redact the `before` fields in the envelope [0] so
that redacted PII doesn't sit in our Kafka topics in the `before` field of
UPDATE events.

As a result if we were to consume these CDC streams with Flink, there would
be missing UPDATE_BEFORE fields for UPDATE events. What kind of impact
would this have on performance and correctness, if any? Any other
considerations we should be aware of?

Thanks in advance for your help!


[0] https://debezium.io/documentation/reference/stable/connectors/mysql.html

Re: Re: [VOTE] FLIP-417: Expose JobManagerOperatorMetrics via REST API

[Mason Chen]

Hi Voters,

JFYI, I have modified the proposed REST API path and added changes to the
metric scope configuration--you can find the reasoning and discussion in
the `[DISCUSS]` thread and FLIP doc. Please let me know if there are any
concerns.

Best,
Mason

On Mon, Jan 29, 2024 at 5:32 AM Thomas Weise  wrote:

> +1 (binding)
>
>
> On Mon, Jan 29, 2024 at 5:45 AM Maximilian Michels  wrote:
>
> > +1 (binding)
> >
> > On Fri, Jan 26, 2024 at 6:03 AM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > +1(binding)
> > >
> > > Best,
> > > Rui
> > >
> > > On Fri, Jan 26, 2024 at 11:55 AM Xuyang  wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > >
> > > > --
> > > >
> > > > Best！
> > > > Xuyang
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 在 2024-01-26 10:12:34，"Hang Ruan"  写道：
> > > > >Thanks for the FLIP.
> > > > >
> > > > >+1 (non-binding)
> > > > >
> > > > >Best,
> > > > >Hang
> > > > >
> > > > >Mason Chen  于2024年1月26日周五 04:51写道：
> > > > >
> > > > >> Hi Devs,
> > > > >>
> > > > >> I would like to start a vote on FLIP-417: Expose
> > > > JobManagerOperatorMetrics
> > > > >> via REST API [1] which has been discussed in this thread [2].
> > > > >>
> > > > >> The vote will be open for at least 72 hours unless there is an
> > > > objection or
> > > > >> not enough votes.
> > > > >>
> > > > >> [1]
> > > > >>
> > > > >>
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-417%3A+Expose+JobManagerOperatorMetrics+via+REST+API
> > > > >> [2]
> > https://lists.apache.org/thread/tt0hf6kf5lcxd7g62v9dhpn3z978pxw0
> > > > >>
> > > > >> Best,
> > > > >> Mason
> > > > >>
> > > >
> >
>

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

[Chesnay Schepler]

The FLIP is a bit weird to be honest. It only applies in cases where 
users haven't set uids, but that goes against best-practices and as far 
as I'm told SQL also sets UIDs everywhere.


I'm wondering if this is really worth the effort.

On 07/02/2024 10:23, Zhanghao Chen wrote:

After offline discussion with @Yu Chen, I've updated 
the FLIP [1] to include a design that allows for compatible hasher upgrade by adding 
StreamGraphHasherV2 to the legacy hasher list, which is actually a revival of the idea from 
FLIP-5290 [2] when StreamGraphHasherV2 was introduced in Flink 1.2. We're targeting to make 
V3 the default hasher in Flink 1.20 given that state-compatibility is no longer an issue. 
Take a review when you have a chance, and I'd like to especially thank @Yu 
Chen for the through offline discussion and code 
debugging help to make this possible.

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-411%3A+Chaining-agnostic+Operator+ID+generation+for+improved+state+compatibility+on+parallelism+change
[2] https://issues.apache.org/jira/browse/FLINK-5290

Best,
Zhanghao Chen

From: Zhanghao Chen 
Sent: Friday, January 12, 2024 10:46
To: Piotr Nowojski ; Yu Chen 
Cc: dev@flink.apache.org 
Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Thanks for the input, Piotr. It might still be possible to make it compatible with 
the old snapshots, following the direction of 
FLINK-5290 suggested by Yu. 
I'll discuss with Yu on more details.

Best,
Zhanghao Chen

From: Piotr Nowojski 
Sent: Friday, January 12, 2024 1:55
To: Yu Chen 
Cc: Zhanghao Chen ; dev@flink.apache.org 

Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Hi,

Using unaligned checkpoints is orthogonal to this FLIP.

Yes, unaligned checkpoints are not supported for pointwise connections, so most 
of the cases go away anyway.
It is possible to switch from unchained to chained subtasks by removing a keyBy 
exchange, and this would be
a problem, but that's just one of the things that we claim that unaligned 
checkpoints do not support [1]. But as
I stated above, this is an orthogonal issue to this FLIP.

Regarding the proposal itself, generally speaking it makes sense to me as well. 
However I'm quite worried about
the compatibility and/or migration path. The:

(v2.0) Make HasherV3 the default hasher, mark HasherV2 deprecated.

step would break the compatibility with Flink 1.xx snapshots. But as this is 
for v2.0, maybe that's not the end of
the world?

Best,
Piotrek

[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/checkpoints_vs_savepoints/#capabilities-and-limitations

czw., 11 sty 2024 o 12:10 Yu Chen 
mailto:yuchen.e...@gmail.com>> napisał(a):
Hi Zhanghao,

Actually, Stefan has done similar compatibility work in the early 
FLINK-5290[1], where he introduced the legacyStreamGraphHashers list for hasher 
backward compatibility.

We have attempted to implement a similar feature in the internal version of 
FLINK and tried to include the new hasher as part of the 
legacyStreamGraphHashers,
which would ensure that the corresponding Operator State could be found at 
restore while ignoring the chaining condition(without changing the default 
hasher).

However, we have found that such a solution may lead to some unexpected 
situations in some cases. While I have no time to find out the root cause 
recently.

If you're interested, I'd be happy to discuss it with you and try to solve the 
problem.

[1] https://issues.apache.org/jira/browse/FLINK-5290

Best,
Yu Chen



2024年1月11日 15:07，Zhanghao Chen 
mailto:zhanghao.c...@outlook.com>> 写道：

Hi Yu,

I haven't thought too much about the compatibility design before. By the nature 
of the problem, it's impossible to make V3 compatible with V2, what we can do 
is to somewhat better inform users when switching the hasher, but I don't have 
any good idea so far. Do you have any suggestions on this?

Best,
Zhanghao Chen

From: Yu Chen mailto:yuchen.e...@gmail.com>>
Sent: Thursday, January 11, 2024 13:52
To: dev@flink.apache.org 
mailto:dev@flink.apache.org>>
Cc: Piotr Nowojski mailto:piotr.nowoj...@gmail.com>>; 
zhanghao.c...@outlook.com 
mailto:zhanghao.c...@outlook.com>>
Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Hi Zhanghao,

Thanks for driving this, that’s really painful for us when we need to switch 
config `pipeline.operator-chaining`.

But I have a Concern, according to FLIP description, modifying `isChainable` 
related code in `StreamGraphHasherV2` will cause the 

Re: [DISCUSS] Alternative way of posting FLIPs

[Matthias Pohl]

+1 for option 1 since it's a reasonable temporary workaround

Moving to GitHub discussions would either mean moving the current FLIP
collection or having the FLIPs in two locations. Both options do not seem
to be optimal. Another concern I had was that GitHub Discussions wouldn't
allow integrating diagrams that easily. But it looks like they support
Mermaid [1] for diagrams.

One flaw of the GoogleDocs approach is, though, that we have to rely on
diagrams being provided as PNG/JPG/SVG rather than draw.io diagrams. draw.io
is more tightly integrated with the Confluence wiki which allows
editing/updating diagrams in the wiki rather than using some external tool.
Google Draw is also not that convenient to use in my opinion. Anyway,
that's a minor issue, I guess.

Matthias

[1]
https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-diagrams

On Wed, Feb 7, 2024 at 3:30 PM Lincoln Lee  wrote:

> Thanks Martijn moving this forward!
>
> +1 for the first solution, because as of now it looks like this is a
> temporary solution and we're still looking forward to the improvement by
> ASF Infra, when the access is ok for contributors, we can back to the
> current workflow.
>
> For solution 2, one visible downside is that it becomes inconvenient to
> look for flips (unless we permanently switch to github discussion).
>
> Looking forward to hearing more thoughts.
>
> Best,
> Lincoln Lee
>
>
> Martijn Visser  于2024年2月7日周三 21:51写道：
>
> > Hi all,
> >
> > ASF Infra has confirmed to me that only ASF committers can access the
> > ASF Confluence site since a recent change. One of the results of this
> > decision is that users can't signup and access Confluence, so only
> > committers+ can create FLIPs.
> >
> > ASF Infra hopes to improve this situation when they move to the Cloud
> > shortly (as in: some months), but they haven't committed on an actual
> > date. The idea would be that we find a temporary solution until anyone
> > can request access to Confluence.
> >
> > There are a couple of ways we could resolve this situation:
> > 1. Contributors create a Google Doc and make that view-only, and post
> > that Google Doc to the mailing list for a discussion thread. When the
> > discussions have been resolved, the contributor ask on the Dev mailing
> > list to a committer/PMC to copy the contents from the Google Doc, and
> > create a FLIP number for them. The contributor can then use that FLIP
> > to actually have a VOTE thread.
> > 2. We could consider moving FLIPs to "Discussions" on Github, like
> > Airflow does at https://github.com/apache/airflow/discussions
> > 3. Perhaps someone else has another good idea.
> >
> > Looking forward to your thoughts.
> >
> > Best regards,
> >
> > Martijn
> >
>

[jira] [Created] (FLINK-34409) Increase test coverage for AdaptiveScheduler

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-34409:
-

 Summary: Increase test coverage for AdaptiveScheduler
 Key: FLINK-34409
 URL: https://issues.apache.org/jira/browse/FLINK-34409
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Coordination
Affects Versions: 1.18.1, 1.19.0, 1.20.0
Reporter: Matthias Pohl


There are still several tests disabled for the {{AdaptiveScheduler}} which we 
can enable now. All the issues seem to have been fixed.

We can even remove the annotation {{@FailsWithAdaptiveScheduler}} now. It's not 
needed anymore.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Kubernetes Operator 1.8.0 release planning

[Ryan van Huuksloot]

I can volunteer to be a release manager. I haven't done it for Apache/Flink
or the operator before so I may be a good candidate.

Ryan van Huuksloot
Sr. Production Engineer | Streaming Platform
[image: Shopify]



On Wed, Feb 7, 2024 at 6:06 AM Maximilian Michels  wrote:

> It's very considerate that you want to volunteer to be the release
> manager, but given that you have already managed one release, I would
> ideally like somebody else to do it. Personally, I haven't managed an
> operator release, although I've done it for Flink itself in the past.
> Nevertheless, it would be nice to have somebody new to the process.
>
> Anyone reading this who wants to try being a release manager, please
> don't be afraid to volunteer. Of course we'll be able to assist. That
> would also be a good opportunity for us to update the docs regarding
> the release process.
>
> Cheers,
> Max
>
>
> On Wed, Feb 7, 2024 at 10:08 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > If the release is postponed 1-2 more weeks, I could volunteer
> > as the one of the release managers.
> >
> > Best,
> > Rui
> >
> > On Wed, Feb 7, 2024 at 4:54 AM Gyula Fóra  wrote:
> >>
> >> Given the proposed timeline was a bit short / rushed I agree with Max
> that
> >> we could wait 1-2 more weeks to wrap up the current outstanding bigger
> >> features around memory tuning and the JDBC state store.
> >>
> >> In the meantime it would be great to involve 1-2 new committers (or
> other
> >> contributors) in the operator release process so that we have some fresh
> >> eyes on the process.
> >> Would anyone be interested in volunteering to help with the next
> release?
> >>
> >> Cheers,
> >> Gyula
> >>
> >> On Tue, Feb 6, 2024 at 4:35 PM Maximilian Michels 
> wrote:
> >>
> >> > Thanks for starting the discussion Gyula!
> >> >
> >> > It comes down to how important the outstanding changes are for the
> >> > release. Both the memory tuning as well as the JDBC changes probably
> >> > need 1-2 weeks realistically to complete the initial spec. For the
> >> > memory tuning, I would prefer merging it in the current state as an
> >> > experimental feature for the release which comes disabled out of the
> >> > box. The reason is that it can already be useful to users who want to
> >> > try it out; we have seen some interest in it. Then for the next
> >> > release we will offer a richer feature set and might enable it by
> >> > default.
> >> >
> >> > Cheers,
> >> > Max
> >> >
> >> > On Tue, Feb 6, 2024 at 10:53 AM Rui Fan <1996fan...@gmail.com> wrote:
> >> > >
> >> > > Thanks Gyula for driving this release!
> >> > >
> >> > > Release 1.8.0 sounds make sense to me.
> >> > >
> >> > > As you said, I'm developing the JDBC event handler.
> >> > > Since I'm going on vacation starting this Friday, and I have some
> >> > > other work before I go on vacation. After evaluating my time today,
> >> > > I found that I cannot complete the development, testing, and merging
> >> > > of the JDBC event handler this week. So I tend to put the JDBC
> >> > > event handler in the next version.
> >> > >
> >> > > Best,
> >> > > Rui
> >> > >
> >> > > On Mon, Feb 5, 2024 at 11:42 PM Gyula Fóra 
> wrote:
> >> > >
> >> > > > Hi all!
> >> > > >
> >> > > > I would like to kick off the release planning for the operator
> 1.8.0
> >> > > > release. The last operator release was November 22 last year.
> Since
> >> > then we
> >> > > > have added a number of fixes and improvements to both the
> operator and
> >> > the
> >> > > > autoscaler logic.
> >> > > >
> >> > > > There are a few outstanding PRs currently, including some larger
> >> > features
> >> > > > for the Autoscaler (JDBC event handler, Heap tuning), we have to
> make a
> >> > > > decision regarding those as well whether to include in the
> release or
> >> > not. @Maximilian
> >> > > > Michels  , @Rui Fan <1996fan...@gmail.com>
> what's your
> >> > > > take regarding those PRs? I generally like to be a bit more
> >> > conservative
> >> > > > with large new features to avoid introducing last minute
> instabilities.
> >> > > >
> >> > > > My proposal would be to aim for the end of this week as the
> freeze date
> >> > > > (Feb 9) and then we can prepare RC1 on monday.
> >> > > >
> >> > > > I am happy to volunteer as a release manager but I am of course
> open to
> >> > > > working together with someone on this.
> >> > > >
> >> > > > What do you think?
> >> > > >
> >> > > > Cheers,
> >> > > > Gyula
> >> > > >
> >> >
>

Re: jira permission request

[Martijn Visser]

Hi,

In order to contribute to Apache Flink, please ping the author of the
Jira ticket that you would like to work on, so it can be assigned to
you. There are no other permissions required.

Best regards,

Martijn

On Tue, Feb 6, 2024 at 2:25 AM 李游  wrote:
>
> HI,
> I want to contribute to Apache flink.
> Would you please give me the contributor permission?
> My jira id is lxliyou.
> Thanks.

Re: Confluence access request

[Martijn Visser]

Hi,

Sorry for the late reply, but this has been recently been disabled by
the ASF. There's an open discussion thread at
https://lists.apache.org/thread/rkpvlnwj9gv1hvx1dyklx6k88qpnvk2t on
how to deal with this.

Best regards,

Martijn

On Tue, Jan 30, 2024 at 10:16 AM tanjialiang  wrote:
>
> Hi, devs! I want to prepare a FLIP and start a discussion on the dev mailing 
> list, but I find I don't have the access, can someone give me access to 
> confluence?
>
>
> My Confluence username: tanjialiang
>
>
> Best regards,
> tanjialiang

Re: Confluence access

[Martijn Visser]

Hi,

Sorry for the late reply, but this has been recently been disabled by
the ASF. There's an open discussion thread at
https://lists.apache.org/thread/rkpvlnwj9gv1hvx1dyklx6k88qpnvk2t on
how to deal with this.

Best regards,

Martijn

On Tue, Jan 23, 2024 at 4:03 AM jufang he  wrote:
>
> Hi devs! I want to suggest a flip, but I couldn't find the Confluence
> registration portal.
>
>-
>
>Can you help me create an account and add editing permission？

Re: [DISCUSS] Alternative way of posting FLIPs

[Lincoln Lee]

Thanks Martijn moving this forward!

+1 for the first solution, because as of now it looks like this is a
temporary solution and we're still looking forward to the improvement by
ASF Infra, when the access is ok for contributors, we can back to the
current workflow.

For solution 2, one visible downside is that it becomes inconvenient to
look for flips (unless we permanently switch to github discussion).

Looking forward to hearing more thoughts.

Best,
Lincoln Lee


Martijn Visser  于2024年2月7日周三 21:51写道：

> Hi all,
>
> ASF Infra has confirmed to me that only ASF committers can access the
> ASF Confluence site since a recent change. One of the results of this
> decision is that users can't signup and access Confluence, so only
> committers+ can create FLIPs.
>
> ASF Infra hopes to improve this situation when they move to the Cloud
> shortly (as in: some months), but they haven't committed on an actual
> date. The idea would be that we find a temporary solution until anyone
> can request access to Confluence.
>
> There are a couple of ways we could resolve this situation:
> 1. Contributors create a Google Doc and make that view-only, and post
> that Google Doc to the mailing list for a discussion thread. When the
> discussions have been resolved, the contributor ask on the Dev mailing
> list to a committer/PMC to copy the contents from the Google Doc, and
> create a FLIP number for them. The contributor can then use that FLIP
> to actually have a VOTE thread.
> 2. We could consider moving FLIPs to "Discussions" on Github, like
> Airflow does at https://github.com/apache/airflow/discussions
> 3. Perhaps someone else has another good idea.
>
> Looking forward to your thoughts.
>
> Best regards,
>
> Martijn
>

[DISCUSS] Alternative way of posting FLIPs

[Martijn Visser]

Hi all,

ASF Infra has confirmed to me that only ASF committers can access the
ASF Confluence site since a recent change. One of the results of this
decision is that users can't signup and access Confluence, so only
committers+ can create FLIPs.

ASF Infra hopes to improve this situation when they move to the Cloud
shortly (as in: some months), but they haven't committed on an actual
date. The idea would be that we find a temporary solution until anyone
can request access to Confluence.

There are a couple of ways we could resolve this situation:
1. Contributors create a Google Doc and make that view-only, and post
that Google Doc to the mailing list for a discussion thread. When the
discussions have been resolved, the contributor ask on the Dev mailing
list to a committer/PMC to copy the contents from the Google Doc, and
create a FLIP number for them. The contributor can then use that FLIP
to actually have a VOTE thread.
2. We could consider moving FLIPs to "Discussions" on Github, like
Airflow does at https://github.com/apache/airflow/discussions
3. Perhaps someone else has another good idea.

Looking forward to your thoughts.

Best regards,

Martijn

[ANNOUNCE] Apache flink-connector-kafka v3.1.0 released

[Martijn Visser]

The Apache Flink community is very happy to announce the release of
Apache flink-connector-kafka v3.1.0. This release is compatible with
Apache Flink 1.17 and 1.18.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data
streaming applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353135

We would like to thank all contributors of the Apache Flink community
who made this release possible!

Regards,
Release Manager

[RESULT][VOTE] Release flink-connector-kafka v3.1.0, release candidate #1

[Martijn Visser]

I'm happy to announce that we have unanimously approved this release.

There are 5 approving votes, 3 of which are binding:
* Hang Ruan (non-binding)
* Mason Chen (non-binding)
* Qingsheng Ren (binding)
* Maximilian Michels (binding)
* Martijn Visser (binding)

There are no disapproving votes.

I've worked yesterday already on completing the release, and I'll
announce it shortly after this email.

Best regards,

Martijn

[jira] [Created] (FLINK-34408) VeryBigPbProtoToRowTest#testSimple fails with OOM

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-34408:
-

 Summary: VeryBigPbProtoToRowTest#testSimple fails with OOM
 Key: FLINK-34408
 URL: https://issues.apache.org/jira/browse/FLINK-34408
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.20.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57371=logs=fc5181b0-e452-5c8f-68de-1097947f6483=995c650b-6573-581c-9ce6-7ad4cc038461=23861

{code}
Feb 07 09:40:16 09:40:16.314 [ERROR] Tests run: 1, Failures: 0, Errors: 1, 
Skipped: 0, Time elapsed: 29.58 s <<< FAILURE! -- in 
org.apache.flink.formats.protobuf.VeryBigPbProtoToRowTest
Feb 07 09:40:16 09:40:16.314 [ERROR] 
org.apache.flink.formats.protobuf.VeryBigPbProtoToRowTest.testSimple -- Time 
elapsed: 29.57 s <<< ERROR!
Feb 07 09:40:16 org.apache.flink.util.FlinkRuntimeException: Error in 
serialization.
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:327)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:162)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:1007)
Feb 07 09:40:16 at 
org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:56)
Feb 07 09:40:16 at 
org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:45)
Feb 07 09:40:16 at 
org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61)
Feb 07 09:40:16 at 
org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:104)
Feb 07 09:40:16 at 
org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:81)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2440)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.datastream.DataStream.executeAndCollectWithClient(DataStream.java:1495)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1382)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1367)
Feb 07 09:40:16 at 
org.apache.flink.formats.protobuf.ProtobufTestHelper.validateRow(ProtobufTestHelper.java:66)
Feb 07 09:40:16 at 
org.apache.flink.formats.protobuf.ProtobufTestHelper.pbBytesToRow(ProtobufTestHelper.java:121)
Feb 07 09:40:16 at 
org.apache.flink.formats.protobuf.ProtobufTestHelper.pbBytesToRow(ProtobufTestHelper.java:103)
Feb 07 09:40:16 at 
org.apache.flink.formats.protobuf.ProtobufTestHelper.pbBytesToRow(ProtobufTestHelper.java:98)
Feb 07 09:40:16 at 
org.apache.flink.formats.protobuf.VeryBigPbProtoToRowTest.testSimple(VeryBigPbProtoToRowTest.java:36)
Feb 07 09:40:16 at java.lang.reflect.Method.invoke(Method.java:498)
Feb 07 09:40:16 Caused by: java.util.concurrent.ExecutionException: 
java.lang.OutOfMemoryError: Java heap space
Feb 07 09:40:16 at 
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
Feb 07 09:40:16 at 
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:323)
Feb 07 09:40:16 ... 18 more
Feb 07 09:40:16 Caused by: java.lang.OutOfMemoryError: Java heap space
Feb 07 09:40:16 at java.util.Arrays.copyOf(Arrays.java:3236)
Feb 07 09:40:16 at 
java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:191)
Feb 07 09:40:16 at 
org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:555)
Feb 07 09:40:16 at 
org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:486)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.graph.StreamConfig.lambda$triggerSerializationAndReturnFuture$0(StreamConfig.java:182)
Feb 07 09:40:16 at 
org.apache.flink.streaming.api.graph.StreamConfig$$Lambda$1582/1961611609.accept(Unknown
 Source)
Feb 07 09:40:16 at 
java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
Feb 07 09:40:16 at 
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
Feb 07 09:40:16 at 

Re: Flink jdbc connector rc3 for flink 1.18

[Sergey Nuyanzin]

Hi David,

Thanks for testing.

Yes the jars are built from the same sources and same git tag apart from
the Flink version.

as it was mentioned in jdbc connector RC thread [1]

>The complete staging area is available for your review, which includes:
>* all artifacts to be deployed to the Maven Central Repository [2]

which contains jars for three Flink versions (1.16.x, 1.17.x, 1.18.x)

Please let  me know whether this answers your question or not

[1] https://lists.apache.org/thread/rlk5kp2vxgkmbxmq4wnco885q5vv9rtp
[2] https://repository.apache.org/content/repositories/orgapacheflink-1706/

On Wed, Feb 7, 2024 at 12:18 PM David Radley 
wrote:

> Hi ,
> I had a question on Flink jdbc connector new release. I notice for the
> last release we have
>
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-jdbc/3.1.1-1.16
>
> https://mvnrepository.com/artifact/org.apache.flink/flink-connector-jdbc/3.1.1-1.17
> It seems that the 2 jars above are identical apart from the manifest.
>
> I assume for the new Flink JDBC connector, there will be a 1.16 1.17 and
> 1.18 versions in Maven central, each level will be compiled against Flink
> 1.6.0, 1.17.0 and 1.18.0 ( or would it be 1.16.3 , 1.17.2 and 1.18.1?)
> respectively. Either way the 3 jar files should be the same (apart from the
> manifest names) as the dependencies on core Flink are forward compatible.
>
> We are looking to use a JDBC connector that works with Flink 1.18 and
> fixes the lookup join filter issue. So we are planning to build against the
> latest 3.1 branch code against 1.18.0– unless the connector is released
> very soon – and we would pick that up.
>
>Kind regards, David.
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>


-- 
Best regards,
Sergey

Flink jdbc connector rc3 for flink 1.18

[David Radley]

Hi ,
I had a question on Flink jdbc connector new release. I notice for the last 
release we have
https://mvnrepository.com/artifact/org.apache.flink/flink-connector-jdbc/3.1.1-1.16
https://mvnrepository.com/artifact/org.apache.flink/flink-connector-jdbc/3.1.1-1.17
It seems that the 2 jars above are identical apart from the manifest.

I assume for the new Flink JDBC connector, there will be a 1.16 1.17 and 1.18 
versions in Maven central, each level will be compiled against Flink 1.6.0, 
1.17.0 and 1.18.0 ( or would it be 1.16.3 , 1.17.2 and 1.18.1?) respectively. 
Either way the 3 jar files should be the same (apart from the manifest names) 
as the dependencies on core Flink are forward compatible.

We are looking to use a JDBC connector that works with Flink 1.18 and fixes the 
lookup join filter issue. So we are planning to build against the latest 3.1 
branch code against 1.18.0– unless the connector is released very soon – and we 
would pick that up.

   Kind regards, David.

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [DISCUSS] Kubernetes Operator 1.8.0 release planning

[Maximilian Michels]

It's very considerate that you want to volunteer to be the release
manager, but given that you have already managed one release, I would
ideally like somebody else to do it. Personally, I haven't managed an
operator release, although I've done it for Flink itself in the past.
Nevertheless, it would be nice to have somebody new to the process.

Anyone reading this who wants to try being a release manager, please
don't be afraid to volunteer. Of course we'll be able to assist. That
would also be a good opportunity for us to update the docs regarding
the release process.

Cheers,
Max


On Wed, Feb 7, 2024 at 10:08 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> If the release is postponed 1-2 more weeks, I could volunteer
> as the one of the release managers.
>
> Best,
> Rui
>
> On Wed, Feb 7, 2024 at 4:54 AM Gyula Fóra  wrote:
>>
>> Given the proposed timeline was a bit short / rushed I agree with Max that
>> we could wait 1-2 more weeks to wrap up the current outstanding bigger
>> features around memory tuning and the JDBC state store.
>>
>> In the meantime it would be great to involve 1-2 new committers (or other
>> contributors) in the operator release process so that we have some fresh
>> eyes on the process.
>> Would anyone be interested in volunteering to help with the next release?
>>
>> Cheers,
>> Gyula
>>
>> On Tue, Feb 6, 2024 at 4:35 PM Maximilian Michels  wrote:
>>
>> > Thanks for starting the discussion Gyula!
>> >
>> > It comes down to how important the outstanding changes are for the
>> > release. Both the memory tuning as well as the JDBC changes probably
>> > need 1-2 weeks realistically to complete the initial spec. For the
>> > memory tuning, I would prefer merging it in the current state as an
>> > experimental feature for the release which comes disabled out of the
>> > box. The reason is that it can already be useful to users who want to
>> > try it out; we have seen some interest in it. Then for the next
>> > release we will offer a richer feature set and might enable it by
>> > default.
>> >
>> > Cheers,
>> > Max
>> >
>> > On Tue, Feb 6, 2024 at 10:53 AM Rui Fan <1996fan...@gmail.com> wrote:
>> > >
>> > > Thanks Gyula for driving this release!
>> > >
>> > > Release 1.8.0 sounds make sense to me.
>> > >
>> > > As you said, I'm developing the JDBC event handler.
>> > > Since I'm going on vacation starting this Friday, and I have some
>> > > other work before I go on vacation. After evaluating my time today,
>> > > I found that I cannot complete the development, testing, and merging
>> > > of the JDBC event handler this week. So I tend to put the JDBC
>> > > event handler in the next version.
>> > >
>> > > Best,
>> > > Rui
>> > >
>> > > On Mon, Feb 5, 2024 at 11:42 PM Gyula Fóra  wrote:
>> > >
>> > > > Hi all!
>> > > >
>> > > > I would like to kick off the release planning for the operator 1.8.0
>> > > > release. The last operator release was November 22 last year. Since
>> > then we
>> > > > have added a number of fixes and improvements to both the operator and
>> > the
>> > > > autoscaler logic.
>> > > >
>> > > > There are a few outstanding PRs currently, including some larger
>> > features
>> > > > for the Autoscaler (JDBC event handler, Heap tuning), we have to make a
>> > > > decision regarding those as well whether to include in the release or
>> > not. @Maximilian
>> > > > Michels  , @Rui Fan <1996fan...@gmail.com> what's your
>> > > > take regarding those PRs? I generally like to be a bit more
>> > conservative
>> > > > with large new features to avoid introducing last minute instabilities.
>> > > >
>> > > > My proposal would be to aim for the end of this week as the freeze date
>> > > > (Feb 9) and then we can prepare RC1 on monday.
>> > > >
>> > > > I am happy to volunteer as a release manager but I am of course open to
>> > > > working together with someone on this.
>> > > >
>> > > > What do you think?
>> > > >
>> > > > Cheers,
>> > > > Gyula
>> > > >
>> >

[jira] [Created] (FLINK-34407) Flaky tests causing workflow timeout

[Aleksandr Pilipenko (Jira)]

Aleksandr Pilipenko created FLINK-34407:
---

 Summary: Flaky tests causing workflow timeout
 Key: FLINK-34407
 URL: https://issues.apache.org/jira/browse/FLINK-34407
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kinesis
Affects Versions: aws-connector-4.2.0, aws-connector-3.0.0
Reporter: Aleksandr Pilipenko


Example build: 
[https://github.com/apache/flink-connector-aws/actions/runs/7735404733]

Tests are stuck retrying due to the following exception:
{code:java}
797445 [main] WARN  
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher
 [] - Encountered recoverable error TimeoutException. Backing off for 0 millis 
00 (arn)
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber$RecoverableFanOutSubscriberException:
 java.util.concurrent.TimeoutException: Timed out acquiring subscription - 
00 (arn)
at 
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.handleErrorAndRethrow(FanOutShardSubscriber.java:327)
 ~[classes/:?]
at 
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.openSubscriptionToShard(FanOutShardSubscriber.java:283)
 ~[classes/:?]
at 
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutShardSubscriber.subscribeToShardAndConsumeRecords(FanOutShardSubscriber.java:210)
 ~[classes/:?]
at 
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.runWithBackoff(FanOutRecordPublisher.java:177)
 ~[classes/:?]
at 
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisher.run(FanOutRecordPublisher.java:130)
 ~[classes/:?]
at 
org.apache.flink.streaming.connectors.kinesis.internals.publisher.fanout.FanOutRecordPublisherTest.testCancelExitsGracefully(FanOutRecordPublisherTest.java:595)
 ~[test-classes/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
~[?:1.8.0_402]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for improved state compatibility on parallelism change

[Zhanghao Chen]

After offline discussion with @Yu Chen, I've 
updated the FLIP [1] to include a design that allows for compatible hasher 
upgrade by adding StreamGraphHasherV2 to the legacy hasher list, which is 
actually a revival of the idea from FLIP-5290 [2] when StreamGraphHasherV2 was 
introduced in Flink 1.2. We're targeting to make V3 the default hasher in Flink 
1.20 given that state-compatibility is no longer an issue. Take a review when 
you have a chance, and I'd like to especially thank @Yu 
Chen for the through offline discussion and code 
debugging help to make this possible.

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-411%3A+Chaining-agnostic+Operator+ID+generation+for+improved+state+compatibility+on+parallelism+change
[2] https://issues.apache.org/jira/browse/FLINK-5290

Best,
Zhanghao Chen

From: Zhanghao Chen 
Sent: Friday, January 12, 2024 10:46
To: Piotr Nowojski ; Yu Chen 
Cc: dev@flink.apache.org 
Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Thanks for the input, Piotr. It might still be possible to make it compatible 
with the old snapshots, following the direction of 
FLINK-5290 suggested by Yu. 
I'll discuss with Yu on more details.

Best,
Zhanghao Chen

From: Piotr Nowojski 
Sent: Friday, January 12, 2024 1:55
To: Yu Chen 
Cc: Zhanghao Chen ; dev@flink.apache.org 

Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Hi,

Using unaligned checkpoints is orthogonal to this FLIP.

Yes, unaligned checkpoints are not supported for pointwise connections, so most 
of the cases go away anyway.
It is possible to switch from unchained to chained subtasks by removing a keyBy 
exchange, and this would be
a problem, but that's just one of the things that we claim that unaligned 
checkpoints do not support [1]. But as
I stated above, this is an orthogonal issue to this FLIP.

Regarding the proposal itself, generally speaking it makes sense to me as well. 
However I'm quite worried about
the compatibility and/or migration path. The:
> (v2.0) Make HasherV3 the default hasher, mark HasherV2 deprecated.

step would break the compatibility with Flink 1.xx snapshots. But as this is 
for v2.0, maybe that's not the end of
the world?

Best,
Piotrek

[1] 
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/checkpoints_vs_savepoints/#capabilities-and-limitations

czw., 11 sty 2024 o 12:10 Yu Chen 
mailto:yuchen.e...@gmail.com>> napisał(a):
Hi Zhanghao,

Actually, Stefan has done similar compatibility work in the early 
FLINK-5290[1], where he introduced the legacyStreamGraphHashers list for hasher 
backward compatibility.

We have attempted to implement a similar feature in the internal version of 
FLINK and tried to include the new hasher as part of the 
legacyStreamGraphHashers,
which would ensure that the corresponding Operator State could be found at 
restore while ignoring the chaining condition(without changing the default 
hasher).

However, we have found that such a solution may lead to some unexpected 
situations in some cases. While I have no time to find out the root cause 
recently.

If you're interested, I'd be happy to discuss it with you and try to solve the 
problem.

[1] https://issues.apache.org/jira/browse/FLINK-5290

Best,
Yu Chen



2024年1月11日 15:07，Zhanghao Chen 
mailto:zhanghao.c...@outlook.com>> 写道：

Hi Yu,

I haven't thought too much about the compatibility design before. By the nature 
of the problem, it's impossible to make V3 compatible with V2, what we can do 
is to somewhat better inform users when switching the hasher, but I don't have 
any good idea so far. Do you have any suggestions on this?

Best,
Zhanghao Chen

From: Yu Chen mailto:yuchen.e...@gmail.com>>
Sent: Thursday, January 11, 2024 13:52
To: dev@flink.apache.org 
mailto:dev@flink.apache.org>>
Cc: Piotr Nowojski mailto:piotr.nowoj...@gmail.com>>; 
zhanghao.c...@outlook.com 
mailto:zhanghao.c...@outlook.com>>
Subject: Re: [DISCUSS] FLIP 411: Chaining-agnostic Operator ID generation for 
improved state compatibility on parallelism change

Hi Zhanghao,

Thanks for driving this, that’s really painful for us when we need to switch 
config `pipeline.operator-chaining`.

But I have a Concern, according to FLIP description, modifying `isChainable` 
related code in `StreamGraphHasherV2` will cause the generated operator id to 
be changed, which will result in the user unable to recover from the old state 
(old and new Operator IDs can't be mapped).
Therefore switching Hasher strategy (V2->V3 or V3->V2) will lead to an 
incompatibility, is there any relevant compatibility design 

Re: [DISCUSS] Kubernetes Operator 1.8.0 release planning

[Rui Fan]

If the release is postponed 1-2 more weeks, I could volunteer
as the one of the release managers.

Best,
Rui

On Wed, Feb 7, 2024 at 4:54 AM Gyula Fóra  wrote:

> Given the proposed timeline was a bit short / rushed I agree with Max that
> we could wait 1-2 more weeks to wrap up the current outstanding bigger
> features around memory tuning and the JDBC state store.
>
> In the meantime it would be great to involve 1-2 new committers (or other
> contributors) in the operator release process so that we have some fresh
> eyes on the process.
> Would anyone be interested in volunteering to help with the next release?
>
> Cheers,
> Gyula
>
> On Tue, Feb 6, 2024 at 4:35 PM Maximilian Michels  wrote:
>
> > Thanks for starting the discussion Gyula!
> >
> > It comes down to how important the outstanding changes are for the
> > release. Both the memory tuning as well as the JDBC changes probably
> > need 1-2 weeks realistically to complete the initial spec. For the
> > memory tuning, I would prefer merging it in the current state as an
> > experimental feature for the release which comes disabled out of the
> > box. The reason is that it can already be useful to users who want to
> > try it out; we have seen some interest in it. Then for the next
> > release we will offer a richer feature set and might enable it by
> > default.
> >
> > Cheers,
> > Max
> >
> > On Tue, Feb 6, 2024 at 10:53 AM Rui Fan <1996fan...@gmail.com> wrote:
> > >
> > > Thanks Gyula for driving this release!
> > >
> > > Release 1.8.0 sounds make sense to me.
> > >
> > > As you said, I'm developing the JDBC event handler.
> > > Since I'm going on vacation starting this Friday, and I have some
> > > other work before I go on vacation. After evaluating my time today,
> > > I found that I cannot complete the development, testing, and merging
> > > of the JDBC event handler this week. So I tend to put the JDBC
> > > event handler in the next version.
> > >
> > > Best,
> > > Rui
> > >
> > > On Mon, Feb 5, 2024 at 11:42 PM Gyula Fóra 
> wrote:
> > >
> > > > Hi all!
> > > >
> > > > I would like to kick off the release planning for the operator 1.8.0
> > > > release. The last operator release was November 22 last year. Since
> > then we
> > > > have added a number of fixes and improvements to both the operator
> and
> > the
> > > > autoscaler logic.
> > > >
> > > > There are a few outstanding PRs currently, including some larger
> > features
> > > > for the Autoscaler (JDBC event handler, Heap tuning), we have to
> make a
> > > > decision regarding those as well whether to include in the release or
> > not. @Maximilian
> > > > Michels  , @Rui Fan <1996fan...@gmail.com> what's
> your
> > > > take regarding those PRs? I generally like to be a bit more
> > conservative
> > > > with large new features to avoid introducing last minute
> instabilities.
> > > >
> > > > My proposal would be to aim for the end of this week as the freeze
> date
> > > > (Feb 9) and then we can prepare RC1 on monday.
> > > >
> > > > I am happy to volunteer as a release manager but I am of course open
> to
> > > > working together with someone on this.
> > > >
> > > > What do you think?
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> >
>

[jira] [Created] (FLINK-34406) Expose more RuntimeContext functionalities in FunctionContext

[yisha zhou (Jira)]

yisha zhou created FLINK-34406:
--

 Summary: Expose more RuntimeContext functionalities in 
FunctionContext
 Key: FLINK-34406
 URL: https://issues.apache.org/jira/browse/FLINK-34406
 Project: Flink
  Issue Type: New Feature
Reporter: yisha zhou






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34405) RightOuterJoinTaskTest#testCancelOuterJoinTaskWhileSort2 fails

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-34405:
-

 Summary: RightOuterJoinTaskTest#testCancelOuterJoinTaskWhileSort2 
fails
 Key: FLINK-34405
 URL: https://issues.apache.org/jira/browse/FLINK-34405
 Project: Flink
  Issue Type: Bug
  Components: API / Core
Affects Versions: 1.19.0, 1.20.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57357=logs=d89de3df-4600-5585-dadc-9bbc9a5e661c=be5a4b15-4b23-56b1-7582-795f58a645a2=9027

{code}
Feb 07 03:20:16 03:20:16.223 [ERROR] Failures: 
Feb 07 03:20:16 03:20:16.223 [ERROR] 
org.apache.flink.runtime.operators.RightOuterJoinTaskTest.testCancelOuterJoinTaskWhileSort2
Feb 07 03:20:16 03:20:16.223 [ERROR]   Run 1: 
RightOuterJoinTaskTest>AbstractOuterJoinTaskTest.testCancelOuterJoinTaskWhileSort2:435
 
Feb 07 03:20:16 expected: 
Feb 07 03:20:16   null
Feb 07 03:20:16  but was: 
Feb 07 03:20:16   java.lang.Exception: The data preparation caused an error: 
Interrupted
Feb 07 03:20:16 at 
org.apache.flink.runtime.operators.testutils.BinaryOperatorTestBase.testDriverInternal(BinaryOperatorTestBase.java:209)
Feb 07 03:20:16 at 
org.apache.flink.runtime.operators.testutils.BinaryOperatorTestBase.testDriver(BinaryOperatorTestBase.java:189)
Feb 07 03:20:16 at 
org.apache.flink.runtime.operators.AbstractOuterJoinTaskTest.access$100(AbstractOuterJoinTaskTest.java:48)
Feb 07 03:20:16 ...(1 remaining lines not displayed - this can be 
changed with Assertions.setMaxStackTraceElementsDisplayed)
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-418: Show data skew score on Flink Dashboard

[Rui Fan]

Thanks Emre for the feedback!

I still think max/mean is more simple and easy to understand
for users. But I don’t have a strong opinion about it.

This proposal is absolutely useful for flink users! In order to
ensure the value for users, would you mind if we wait for
a while and check if there is more feedback from the community.
Also, would you mind sharing these 2 solutions to the
user[1] & user-zh[2] mail list as well? Flink users may give some
valuable feedback there, thanks~

[1] u...@flink.apache.org
[2] user...@flink.apache.org

Best,
Rui

On Thu, Feb 1, 2024 at 5:52 PM Kartoglu, Emre 
wrote:

> Hi Rui,
>
> Thanks for the useful feedback and caring about the user experience.
> I will update the FLIP based on 1 comment. I consider this a minor update.
>
> Please find my detailed responses below.
>
> "numRecordsInPerSecond sounds make sense to me, and I think
> it's necessary to mention it in the FLIP wiki. It will let other developers
> to easily understand. WDYT?"
>
> I feel like this might be touching implementation details. No objections
> though,
>  I will update the FLIP with this as one of the ways in which we can
> achieve the proposal.
>
>
> "After I detailed read the FLIP and Average_absolute_deviation, we know
> 0% is the best, 100% is worst."
>
> Correct.
>
>
> "I guess it is difficult for users who have not read the documentation to
> know the meaning of 50%. We hope that the designed Data skew will
> be easy for users to understand without reading or learning a series
> of backgrounds."
>
> I think I understand where you're coming from. My thought is that the user
> won't have to
> know exactly how the skew percentage/score is calculated. But this score
> will
> act as a warning sign for them. Upon seeing a skew score of 80% for an
> operator, as a user
> I will go and click on the operator to see many of my subtasks are not
> receiving any data at all.
> So it acts as a metric to get the user's attention to the skewed operator
> and fix issues.
>
>
> "For example, as you mentioned before, flink has a metric:
> numRecordsInPerSecond.
> I believe users know what numRecordsInPerSecond means even if they
> didn't read any documentation."
>
> The FLIP suggests that we will provide an explanation of the data skew
> score
> under the proposed Data Skew tab. I would like the exact wording to be
> left to
> the code review process to prevent these from blocking the implementation
> work/progress.
> This will be a user-friendly explanation with an option for the curious
> user to see the exact formula.
>
>
> Kind regards,
> Emre
>
>
> On 01/02/2024, 03:26, "Rui Fan" <1996fan...@gmail.com  1996fan...@gmail.com>> wrote:
>
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
>
>
>
> > I was thinking about using the existing numRecordsInPerSecond metric
>
>
> numRecordsInPerSecond sounds make sense to me, and I think
> it's necessary to mention it in the FLIP wiki. It will let other developers
> to easily understand. WDYT?
>
>
> BTW, that's why I ask whether the data skew score means total
> receive records.
>
>
> > this would always give you a score higher than 1, with no way to cap the
> score.
>
>
> Yeah, you are right. max/mean is not a score, it's the data skew multiple.
> And I guess max/mean is easier to understand than
> Average_absolute_deviation.
>
>
> > I'm more used to working with percentages. The problem with the max/mean
> metric is I wouldn't immediately know whether a score of 300 is bad for
> instance.
> > Whereas if users saw above 50% as suggested in the FLIP for instance,
> they would consider taking action. I'm tempted to push back on this
> suggestion. Happy to discuss further, there is a chance I'm not seeing the
> downside of the proposed percentage based metric yet. Please let me know.
>
>
> After I detailed read the FLIP and Average_absolute_deviation, we know
> 0% is the best, 100% is worst.
>
>
> I guess it is difficult for users who have not read the documentation to
> know the meaning of 50%. We hope that the designed Data skew will
> be easy for users to understand without reading or learning a series
> of backgrounds.
>
>
> For example, as you mentioned before, flink has a metric:
> numRecordsInPerSecond.
> I believe users know what numRecordsInPerSecond means even if they
> didn't read any documentation.
>
>
> Of course, I'm opening for it. I may have missed something. I'd like to
> hear
> more feedback from the community.
>
>
> Best,
> Rui
>
>
> On Thu, Feb 1, 2024 at 4:13 AM Kartoglu, Emre  lid>
> wrote:
>
>
> > Hi Rui,
> >
> > " and provide the total and current score in the detailed tab. I didn't
> > see the detailed design in the FLIP, would you mind
> > improve the design doc? Thanks".
> >
> > It will essentially be a basic list view similar to the "Checkpoints"
> tab.
> > I only briefly mentioned 

[jira] [Created] (FLINK-34404) RestoreTestBase#testRestore times out

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-34404:
-

 Summary: RestoreTestBase#testRestore times out
 Key: FLINK-34404
 URL: https://issues.apache.org/jira/browse/FLINK-34404
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.19.0, 1.20.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57357=logs=32715a4c-21b8-59a3-4171-744e5ab107eb=ff64056b-5320-5afe-c22c-6fa339e59586=11603

{code}
Feb 07 02:17:40 "ForkJoinPool-74-worker-1" #382 daemon prio=5 os_prio=0 
cpu=282.22ms elapsed=961.78s tid=0x7f880a485c00 nid=0x6745 waiting on 
condition  [0x7f878a6f9000]
Feb 07 02:17:40java.lang.Thread.State: WAITING (parking)
Feb 07 02:17:40 at 
jdk.internal.misc.Unsafe.park(java.base@17.0.7/Native Method)
Feb 07 02:17:40 - parking to wait for  <0xff73d060> (a 
java.util.concurrent.CompletableFuture$Signaller)
Feb 07 02:17:40 at 
java.util.concurrent.locks.LockSupport.park(java.base@17.0.7/LockSupport.java:211)
Feb 07 02:17:40 at 
java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.7/CompletableFuture.java:1864)
Feb 07 02:17:40 at 
java.util.concurrent.ForkJoinPool.compensatedBlock(java.base@17.0.7/ForkJoinPool.java:3449)
Feb 07 02:17:40 at 
java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.7/ForkJoinPool.java:3432)
Feb 07 02:17:40 at 
java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.7/CompletableFuture.java:1898)
Feb 07 02:17:40 at 
java.util.concurrent.CompletableFuture.get(java.base@17.0.7/CompletableFuture.java:2072)
Feb 07 02:17:40 at 
org.apache.flink.table.planner.plan.nodes.exec.testutils.RestoreTestBase.testRestore(RestoreTestBase.java:292)
Feb 07 02:17:40 at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.7/Native 
Method)
Feb 07 02:17:40 at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.7/NativeMethodAccessorImpl.java:77)
Feb 07 02:17:40 at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.7/DelegatingMethodAccessorImpl.java:43)
Feb 07 02:17:40 at 
java.lang.reflect.Method.invoke(java.base@17.0.7/Method.java:568)
Feb 07 02:17:40 at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
[...]
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)