[jira] [Created] (FLINK-35802) Deadlock may happen after adding new tables

[LvYanquan (Jira)]

LvYanquan created FLINK-35802:
-

 Summary: Deadlock may happen after adding new tables
 Key: FLINK-35802
 URL: https://issues.apache.org/jira/browse/FLINK-35802
 Project: Flink
  Issue Type: Bug
  Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: LvYanquan
 Fix For: cdc-3.2.0
 Attachments: image-2024-07-10-13-44-49-972.png, 
image-2024-07-10-13-45-52-450.png, image-2024-07-10-13-47-07-190.png

Problem Description:

1.CDC originally consumed the full incremental data of a table, and currently, 
the snapshot phase has ended, and it is in the binlog consumption phase.

2.Stop the job to add the full incremental data synchronization for a new table.

3.After the full phase of the new table ends, it fails to return to the binlog 
consumption phase.

4. Checking the thread that consumes the binlog, a deadlock situation is 
discovered, and the specific thread stack is as follows.

5. The likely cause is that after the Enumerator issues a 
BinlogSplitUpdateRequestEvent, both the MysqlSplitReader and 
MySqlBinlogSplitReadTask close the binlogClient connection but fail to acquire 
the lock.

6. The lock is held by the consumer thread, but the queue is full, waiting for 
consumers to consume the data out, and yet there are no consumers, thus causing 
a deadlock.

ThreadDump：

1.  MysqlSplitReader.pollSplitRecords method

!image-2024-07-10-13-44-49-972.png!

2. MySqlStreamingChangeEventSource.execute method

!image-2024-07-10-13-45-52-450.png!

3. MySqlBinlogSplitReadTask.handleEvent method
!image-2024-07-10-13-47-07-190.png!

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Kafka Connector Consulting: Why there is no limit size of Kafka producer pool in KafkaWriter

[Rascal Wu]

Hi,

May I consult one question about why there is no limit size of
`producerPool` in KafkaWriter. It seems that the current logical will
create a new producer during create KafkaWriter and start
checkpoint(snapshotState been invoked) if the producer pool is empty, and
then add the producer into pool after KafkaCommitter#commit(), but only
close these producers during closing KakfWriter.

I'm using flink 1.16, I meet a problem that there are too many kafka
connections and many
kafka-producer-network-thread.
And I test it again with a simple demo which consume data from datagen and
write to kafka with EXACTALY_ONCE semantics with parallelism. By checking
ThreadDump and the log of task manager, there are 200+
kafka-producer-network-thread and `create new transactional producer` logs,
and only one `Closing the Kafak producer` log found. The demo as follow:
CREATE TABLE source_table (
id INT,
name STRING,
age INT,
event_time TIMESTAMP(3)
) WITH (
'connector' = 'datagen',
'rows-per-second' = '10',
'fields.id.min' = '1',
'fields.id.max' = '1000',
'fields.name.length' = '10',
'fields.age.min' = '18',
'fields.age.max' = '60'
);

CREATE TABLE kafka_sink (
id INT,
name STRING,
age INT,
event_time TIMESTAMP(3)
) WITH (
'connector' = 'kafka',
'topic' = 'flink-demo',
'properties.bootstrap.servers' = '192.168.0.13:9092,192.168.0.14:9092',
'format' = 'json',
'sink.partitioner' = 'fixed',
'sink.delivery-guarantee' = 'exactly-once',
'sink.transactional-id-prefix' = 'kafka-demo',
'properties.transaction.timeout.ms' = '30'
);

set execution.checkpointing.interval=10s;
INSERT INTO kafka_sink SELECT id, name, age, event_time FROM source_table;

And i found the old version logic is that the KafkaCommitter will close the
producer after commit, but right now we want to recycle the producer by add
the producer back to pool, but without any limitation or other actions to
close the idle producer, like the thread pool logic. Any specific
consideration on this case?

Best Regards

[jira] [Created] (FLINK-35801) testSwitchFromEnablingToDisablingFileMerging failed in AZP

[Weijie Guo (Jira)]

Weijie Guo created FLINK-35801:
--

 Summary: testSwitchFromEnablingToDisablingFileMerging failed in AZP
 Key: FLINK-35801
 URL: https://issues.apache.org/jira/browse/FLINK-35801
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI
Affects Versions: 1.20.0
Reporter: Weijie Guo






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35800) Update chinese documentation for json_(un)quote function

[Anupam Aggarwal (Jira)]

Anupam Aggarwal created FLINK-35800:
---

 Summary: Update chinese documentation for json_(un)quote function
 Key: FLINK-35800
 URL: https://issues.apache.org/jira/browse/FLINK-35800
 Project: Flink
  Issue Type: Improvement
  Components: chinese-translation
Reporter: Anupam Aggarwal


Update chinese documentation corresponding to `json_quote` and `json_unquote` 
function

as per instructions  in 
[https://github.com/apache/flink/blob/master/docs/data/sql_functions_zh.yml] 

 

Changes are in
PR link - [https://github.com/apache/flink/pull/24967] 

[https://github.com/apache/flink/blob/4267018323dc3bfa1d65ee9fcb49b024e03d/docs/data/sql_functions.yml#L380]

 

Changes would be needed in file
[https://github.com/apache/flink/blob/master/docs/data/sql_functions_zh.yml] 

Jira for functions - https://issues.apache.org/jira/browse/FLINK-34111 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Static OpenSSL lib licensing

[Robert Young]

Hi,

I see in flink-shaded main we depend
on 2.0.62.Final and we
currently avoid building/distributing flink-shaded-netty-tcnative-static
due to legal issues

> This module is excluded by default since it is not compliant with the
apache license. https://issues.apache.org/jira/browse/LEGAL-393

netty-tcnative 2.0.62.Final upgraded openssl to 3.1.2 [1]

Openssl 3 introduced a license change to apache 2 [2] so I wondered if it's
now okay to build/deploy flink-shaded-netty-tcnative-static? Then we could
avoid having to build the static lib in the flink core end-to-end tests [3]
and include it in the flink binary distribution for users to access.

1.
https://github.com/netty/netty-tcnative/blob/d70e48a56c4aaa8df86a7de2e275e51aa0cc43a1/pom.xml#L97
2. https://www.openssl.org/docs/man3.0/man7/migration_guide.html
3.
https://github.com/apache/flink/blob/4e86d98437480377973f66600c2d5bda907589d6/flink-end-to-end-tests/test-scripts/common_ssl.sh#L82

Thanks,
Rob Young

[jira] [Created] (FLINK-35799) Add CompiledPlan annotations to BatchExecCalc

[Jim Hughes (Jira)]

Jim Hughes created FLINK-35799:
--

 Summary: Add CompiledPlan annotations to BatchExecCalc
 Key: FLINK-35799
 URL: https://issues.apache.org/jira/browse/FLINK-35799
 Project: Flink
  Issue Type: Sub-task
Reporter: Jim Hughes


In addition to the annotations, implement the BatchCompiledPlan test for this 
operator.

Since this is the first operator, sink and source operators must be annotated 
as well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35798) Implement a BatchCompiledPlanTestBase

[Jim Hughes (Jira)]

Jim Hughes created FLINK-35798:
--

 Summary: Implement a BatchCompiledPlanTestBase
 Key: FLINK-35798
 URL: https://issues.apache.org/jira/browse/FLINK-35798
 Project: Flink
  Issue Type: Sub-task
Reporter: Jim Hughes


The goal of BatchCompiledPlanTestBase has two golas:
1. Take TableTestPrograms and produce compiled plans for the latest version of 
the operator being tested.
2. Load all compiled plans from disk and execute them against the first batch 
of data described by the TableTestProgram.

This will ensure that there are no errors in serialization or deserialization 
for the operator under test.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35797) FLIP-456: CompiledPlan support for Batch Execution Mode

[Jim Hughes (Jira)]

Jim Hughes created FLINK-35797:
--

 Summary: FLIP-456: CompiledPlan support for Batch Execution Mode
 Key: FLINK-35797
 URL: https://issues.apache.org/jira/browse/FLINK-35797
 Project: Flink
  Issue Type: New Feature
Reporter: Jim Hughes
Assignee: Jim Hughes


The CompiledPlan feature, introduced in FLIP-190: Support Version Upgrades for 
Table API & SQL Programs, supports only streaming execution mode. Batch 
ExecNodes were explicitly excluded from the JSON plan (aka CompiledPlan).

This ticket will cover adding CompiledPlan support for the Batch ExecNodes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35796) Ensure that MailboxExecutor.submit is used correctly

[Arvid Heise (Jira)]

Arvid Heise created FLINK-35796:
---

 Summary: Ensure that MailboxExecutor.submit is used correctly
 Key: FLINK-35796
 URL: https://issues.apache.org/jira/browse/FLINK-35796
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 2.0.0
Reporter: Arvid Heise
Assignee: Arvid Heise


https://issues.apache.org/jira/browse/FLINK-34470 showed that 
MailboxExecutor.submit may result in unexpected exception handling: while 
{{execute}} will bubble up the exception in the task thread and result in some 
fault, {{submit}} can hide the exception because the API assumes that the 
returned {{Future}} is checked for it explicitly or implicitly.

We can solve the situation by improving the doc of MailboxExecutor and 
double-check the internal usages.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[SUMMARY] Flink 1.20 Release Sync 07/09/2024

[weijie guo]

Dear devs,


This is the fourth meeting after feature freeze of Flink 1.20.


I'd like to share the information synced in the meeting.


- *Timeline*


Flink 1.20 doesn't have any blocker for now.


We plan to create and vote the release-1.20.0-rc1 next monday if we're
not aware of any new blocker.


- *Cross-team release testing*


Thanks to all the developers involved, we only left two unclosed
release testing ticket for now:

 -  Verify FLINK-26050 Too many small sst files in rocksdb state
backend when using time window created in ascending order
[1]

 -
Release Testing: Verify FLIP-306 Unified File Merging Mechanism for Checkpoints
[2]


We hope all cross-team release testing can be finished this week.


-*Sync meeting[3]*


The next meeting is 07/16/2024 9:30am (UTC+2) and 3:30pm (UTC+8),

please feel free to join us.


Lastly, we encourage attendees to fill out the topics to be discussed at
the bottom of 1.20 wiki page[4] a day in advance, to make it easier for
everyone to understand the background of the topics, thanks!



[1] https://issues.apache.org/jira/browse/FLINK-35738

[2] https://issues.apache.org/jira/browse/FLINK-35624

[3] https://meet.google.com/mtj-huez-apu

[4] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release



Best,

Robert, Rui, Ufuk, Weijie

[jira] [Created] (FLINK-35795) FLIP-466: Introduce ProcessFunction Attribute in DataStream API V2

[Wencong Liu (Jira)]

Wencong Liu created FLINK-35795:
---

 Summary: FLIP-466: Introduce ProcessFunction Attribute in 
DataStream API V2
 Key: FLINK-35795
 URL: https://issues.apache.org/jira/browse/FLINK-35795
 Project: Flink
  Issue Type: Sub-task
Reporter: Wencong Liu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: [DISCUSS] FLIP-XXX Apicurio-avro format

[David Radley]

Hi Kevin,
I have agreed a design with Chesnay and Danny. I am implementing a prototype, 
to prove it works,  then will update the Flip text with the new design. Initial 
testing is showing it working.

Here is a quick history so you can understand our current thinking.

  1.  Initially we passed maps for header information from the kafka connector 
to Flink for deserialization. Similar for serialize. This was not great, 
because maps are not ideal and it was a big change as it needed core Flink 
interface changes
  2.  We then moved the Avro Apicurio format to the Kafka connector and looked 
to discover a new record based de/serialization interface. So we could pass 
down the record (containing the headers) rather than the payload. This did not 
work, because there is a dependence on the Avro connector that is not  aware of 
the new interface.
  3.  We considered using Thread local storage to pass the headers, we did not 
like this as there was a risk of memory leaks if we did not manage the thread 
well, also the contract is hidden.
  4.  We then came up with the current design that augments the deserialization 
in the Kafka connector in a new discovered record based deserialization, it 
then takes the headers out in the schema coder, leaving the message as it was. 
Similar for serialization.


One piece I need to work out the details of, is how to work when there are 2 
implementations that can be discovered, probably using an augmented format name 
as a factory identifier,

I hope to put up a new design in the Flip by the end of next week, for wider 
review,
Kind regards, David.


From: Kevin Lam 
Date: Monday, 8 July 2024 at 21:16
To: dev@flink.apache.org 
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi David,

Any updates on the Kafka Message Header support? I am also interested in
supporting headers with the Flink SQL Formats:
https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6

On Fri, Jun 14, 2024 at 6:10 AM David Radley 
wrote:

> Hi everyone,
> I have talked with Chesnay and Danny offline. Danny and I were not very
> happy with the passing Maps around, and were looking for a neater design.
> Chesnay suggested that we could move the new format to the Kafka connector,
> then pass the Kafka record down to the deserialize logic so it can make use
> of the headers during deserialization and serialisation.
>
> I think this is a neat idea. This would mean:
> - the Kafka connector code would need to be updated to pass down the Kafka
> record
> - there would be the Avro Apicurio format and SQL in the kafka repository.
> We feel it is unlikely to want to use the Apicurio registry with files, as
> the Avro format could be used.
>
> Unfortunately I have found that this as not so straight forward to
> implement as the Avro Apicurio format uses the Avro format, which is tied
> to the DeserializationSchema. We were hoping to have a new decoding
> implementation that would pass down the Kafka record rather than the
> payload. This does not appear possible without a Avro format change.
>
>
> Inspired by this idea, I notice that
> KafkaValueOnlyRecordDeserializerWrapper extends
> KafkaValueOnlyDeserializerWrapper
>
> Does
>
> deserializer.deserialize(record.topic(),record.value())
>
>
>
> I am investigating If I can add a factory/reflection to provide an
> alternative
> Implementation that will pass the record based (the kafka record is not
> serializable so I will pick what we need and deserialize) as a byte array.
>
> I would need to do this 4 times (value ,key for deserialisation and
> serialisation. To do this I would need to convert the record into a byte
> array, so it fits into the existing interface (DeserializationSchema).  I
> think this could be a way through, to avoid using maps and avoid changing
> the existing Avro format and avoid change any core Flink interfaces.
>
> I am going to prototype this idea. WDYT?
>
> My thanks go to Chesnay and Danny for their support and insight around
> this Flip,
>Kind regards, David.
>
>
>
>
>
>
> From: David Radley 
> Date: Wednesday, 29 May 2024 at 11:39
> To: dev@flink.apache.org 
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Danny,
> Thank you for your feedback on this.
>
> I agree that using maps has pros and cons. The maps are flexible, but do
> require the sender and receiver to know what is in the map.
>
> When you say “That sounds like it would fit in better, I assume we cannot
> just take that approach?” The motivation behind this Flip is to support the
> headers which is the usual way that Apicurio runs. We will support the
> “schema id in the payload” as well.
>
> I agree with you when you say “ I am not 100% happy with the solution but I
> cannot offer a better option.” – this is a pragmatic way we have found to
> solve this issue. I am open to any suggestions to improve this as well.
>
> If we are going with the maps design (which is the best we have at the
> moment) ; it would be good to 

[jira] [Created] (FLINK-35794) Ability to skip kubernetes tests in e2e nightly test script

[David Kornel (Jira)]

David Kornel created FLINK-35794:


 Summary: Ability to skip kubernetes tests in e2e nightly test 
script
 Key: FLINK-35794
 URL: https://issues.apache.org/jira/browse/FLINK-35794
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: David Kornel


Currently if environment for testing does not support running minikube which is 
installed by script we can't run nightly test executed by script. So it would 
be nice to add skip mark and then kubernetes tests are skipped.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35793) BatchSQLTest.testBatchSQL failed in hybrid shuffle selective mode

[Weijie Guo (Jira)]

Weijie Guo created FLINK-35793:
--

 Summary: BatchSQLTest.testBatchSQL failed in hybrid shuffle 
selective mode
 Key: FLINK-35793
 URL: https://issues.apache.org/jira/browse/FLINK-35793
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI
Affects Versions: 1.20.0
Reporter: Weijie Guo






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache Flink-shaded 19.0 released

[Dawid Wysakowicz]

The Apache Flink community is very happy to announce the release of Apache
Flink-shaded 19.0.

The flink-shaded project contains a number of shaded dependencies for
Apache Flink.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353853

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Dawid

Re: [ANNOUNCE] Release 1.20.0, release candidate #0

[weijie guo]

Thanks Santwana and Martijn for the testing!

> I've closed my blocker ticket; after force killing all my Flink Java
processes, it did work when I restarted it.

That would be great!

Best regards,

Weijie


Martijn Visser  于2024年7月9日周二 03:04写道：

> Hi all,
>
> I've closed my blocker ticket; after force killing all my Flink Java
> processes, it did work when I restarted it.
>
> Thanks, Martijn
>
> On Mon, Jul 8, 2024 at 6:13 PM Santwana Verma  >
> wrote:
>
> > Hi everyone,
> >
> > Thanks for creating the release candidate. I've successfully validated
> the
> > release candidate locally with the DataStream API.
> >
> > 1. I created a DataStream API based job, which read and deserialized JSON
> > strings from an input Kafka topic using flink-connector-kafka,
> transformed
> > the data, and wrote it in the Avro format to an output Kafka topic.
> > 2. I used Maven dependencies for the job from the repository
> > https://repository.apache.org/content/repositories/orgapacheflink-1742
> > (flink version 1.20.0) to create the job JAR.
> > 3. I ran flink from the binaries within
> >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc0/flink-1.20.0-bin-scala_2.12.tgz
> > .
> > 4. The job ran as expected when I produced to the input topic with ~500k
> > msgs and consumed from the output topic.
> >
> > Best,
> > Santwana
> >
> > On Fri, Jun 28, 2024 at 9:39 PM weijie guo 
> > wrote:
> >
> > > Hi everyone,
> > >
> > >
> > > The release candidate #0(i.e. RC0) for Apache Flink 1.20.0 has been
> > > created.
> > >
> > >
> > > This RC is currently for preview only to facilitate the integrated
> > testing
> > > and
> > >
> > > we don't have to vote on it.
> > >
> > >
> > > RC1 is expected to be released a week later If we find no new blocker
> in
> > > RC0.
> > >
> > > The related voting process will be triggered once the announcement is
> > > ready.
> > >
> > >
> > > The RC0 has all the artifacts that we would typically have for a
> release,
> > > except
> > >
> > > for the release note and the website pull request for the release
> > > announcement.
> > >
> > >
> > > The following contents are available for your review:
> > >
> > >
> > > - The preview source release and binary convenience releases [1], which
> > >
> > > are signed with the key with fingerprint
> > > 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [2].
> > >
> > >
> > > - All artifacts that would normally be deployed to the Maven
> > > Central Repository [3].
> > >
> > >
> > > - Source code tag "release-1.20.0-rc0" [4]
> > >
> > >
> > > Your help testing the release will be greatly appreciated! And we'll
> > >
> > > create the RC1 release and the voting thread as soon as all the efforts
> > are
> > >
> > > finished.
> > >
> > >
> > > [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc0/
> > >
> > > [2] https://dist.apache.org/repos/dist/release/flink/KEYS
> > >
> > > [3]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1742/
> > >
> > > [4] https://github.com/apache/flink/releases/tag/release-1.20.0-rc0
> > >
> > >
> > > Best,
> > >
> > > Robert, Rui, Ufuk, Weijie
> > >
> >
>

[jira] [Created] (FLINK-35792) Sorting by proctime does not work in rank

[xuyang (Jira)]

xuyang created FLINK-35792:
--

 Summary: Sorting by proctime does not work in rank
 Key: FLINK-35792
 URL: https://issues.apache.org/jira/browse/FLINK-35792
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.19.0, 1.20.0
Reporter: xuyang


Take the following sql as an example:
{code:java}
@Test
def test(): Unit = {
  val sql =
"""
  |SELECT *
  |FROM (
  |  SELECT a, b, c,
  |  ROW_NUMBER() OVER (PARTITION BY a ORDER BY b, proctime DESC) as 
rank_num
  |  FROM MyTable)
  |WHERE rank_num = 1
""".stripMargin

  // This rank can't be converted into Deduplicated because it also uses `b`   
  // as order key.    
  util.verifyExecPlan(sql)
} {code}
The rank node will not materialize the `proctime` in 
`RelTimeIndicatorConverter`, thus the order key `proctime` is always null.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] CLI action deprecation process

[Xintong Song]

I think there are three questions to be anwsered, and here are my two cents.

1) How do we define the compatibility guarantees for cli interfaces?

I'd be +1 for reusing the existing @Public and @PublicEvolving annotations,
as suggested by Ferenc. Having multiple sets of rules in the same project
may easily confuse users.

2) What deprecation process is required for cli interfaces?

If we are reusing the existing annotations, I think we'd better to have the
same deprecation process as well, for the same reason not to confuse users.

3) How do we make user aware of the compatibility guarantees and
deprecations of cli interfaces?

I think there are several things that we can do.
- In addition to codes and JavaDocs, we should also display the annotations
(Public / PublicEvolving / Experimental Dprecated) in documentation [1] and
cli helps (things you see when execute `/bin/flink -h`).
- Print a warning message if a deprecated command / argument is used, as
suggested by Muhammet.
- We can also provide a strict / compatible mode. E.g., we use strict mode
by default, which fails if any deprecated interface is used. In the error
message, we hint user that he/she can manually enable the compatible mode,
which allows deprecated interfaces. This should draw users' attention to
the deprecation of the interfaces, while not block the adoption of a new
Flink version with breaking changes. There are probably more details to be
considered, e.g., should we fail immediately once an interface is
deprecated in strict mode, how long do we support a deprecated interface in
compatible mode, how to properly suggest users about the compatible mode
while not encourage them to always stay in that mode, etc.

Best,

Xintong


[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/



On Mon, Jul 8, 2024 at 5:54 PM Muhammet Orazov
 wrote:

> Hey Ferenc,
>
> Yes correct. My thoughts were based on finding tradeoff between
> following fully deprecation process and leaner one for CLIs.
>
> Since cli are not like APIs, I think users would be aware of
> deprecation only when were remove the commands. That is they
> try to run their jobs with upgrade and it fails with action
> not available.
>
> So maybe we don't have to follow fully API `@PublicEvolving`
> process for this.
>
> Another maybe user friendly approach would be to inform with
> warning that the `run-application` cli action will be dropped,
> and suggest new action and migration on the log message.
>
> Best,
> Muhammet
>
> On 2024-07-04 20:17, Ferenc Csaky wrote:
> > Hi Muhammet,
> >
> > Thank you for your thoughts!
> >
> >> After two minor releases, and on next major version bump,
> >> we could drop the `run-application` method as suggested
> >> on discussion by Xintong.
> >
> > Here, you describe the deprecation/removal process of a public
> > API by the definition we have in the project now. So if the same
> > applies to a CLI action, why should we not enforce such behavior
> > for those as well?
> >
> > If following the same process for CLIs make sense, we should also
> > enforce the same compatibility guarantees IMO.
> >
> > Best,
> > Ferenc
> >
> >
> >
> > On Friday, 28 June 2024 at 09:30, Muhammet Orazov
> >  wrote:
> >
> >>
> >>
> >> Hey Ferenc,
> >>
> >> Thanks for starting the discussion!
> >>
> >> I agree that the CLI is user facing, but I think we don't
> >> have to treat it as other public APIs.
> >>
> >> I'd propose to throw user friendly exception for
> >> `run-application` with suggestion to use `run` case instead.
> >> This would make users aware of the change and require them
> >> to migrate their scripts.
> >>
> >> After two minor releases, and on next major version bump,
> >> we could drop the `run-application` method as suggested
> >> on discussion by Xintong.
> >>
> >> Best,
> >> Muhammet
> >>
> >>
> >> On 2024-06-26 15:33, Ferenc Csaky wrote:
> >>
> >> > Hello devs,
> >> > I would like to open a discussion about considerations regarding how
> to
> >> > deprecate CLI
> >> > actions, and what compatibility guarantees should apply to such cases.
> >> > The topic came up in
> >> > a previous discussion [1] about a current FLIP to merge the run and
> >> > run-application
> >> > behavior [2].
> >> >
> >> > According to Xintong's previous inputs, currently the Flink CLI, or
> its
> >> > actions are not handled
> >> > as public APIs by the existing definition (@Public or @PublicEvolving
> >> > annotated). So
> >> > legally it would be possible to change CLIs anytime. I agree with
> >> > Xintong that CLI actions
> >> > should be considered as public APIs, and as such, compatibility
> >> > guarantees should be
> >> > provided.
> >> >
> >> > CLI actions are defined as private constants in CliFrontend [3], so
> >> > IMO the existing rules
> >> > are not perfectly applicable as is. Both @Public and @PublicEvolving
> >> > can be applied to
> >> > fields (although for @Public that is only true as per the javadoc,
> >> > technically it can only be
> 

[jira] [Created] (FLINK-35791) Add database and table infos to Kafka json output.

[LvYanquan (Jira)]

LvYanquan created FLINK-35791:
-

 Summary: Add database and table infos to Kafka json output.
 Key: FLINK-35791
 URL: https://issues.apache.org/jira/browse/FLINK-35791
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: LvYanquan
 Fix For: cdc-3.2.0


Currently, database and table were not passed to canal/debezium json output 
format of Kafka sink.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35790) Update docs for new schema definition in CTAS and RTAS

[Jira]

Sergio Peña created FLINK-35790:
---

 Summary: Update docs for new schema definition in CTAS and RTAS
 Key: FLINK-35790
 URL: https://issues.apache.org/jira/browse/FLINK-35790
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergio Peña






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35789) Allow WATERMARK & PRIMARY KEY in CREATE TABLE AS (CTAS)

[Jira]

Sergio Peña created FLINK-35789:
---

 Summary: Allow WATERMARK & PRIMARY KEY in CREATE TABLE AS (CTAS)
 Key: FLINK-35789
 URL: https://issues.apache.org/jira/browse/FLINK-35789
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergio Peña






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

[Kevin Lam]

Hi David,

Any updates on the Kafka Message Header support? I am also interested in
supporting headers with the Flink SQL Formats:
https://lists.apache.org/thread/spl88o63sjm2dv4l5no0ym632d2yt2o6

On Fri, Jun 14, 2024 at 6:10 AM David Radley 
wrote:

> Hi everyone,
> I have talked with Chesnay and Danny offline. Danny and I were not very
> happy with the passing Maps around, and were looking for a neater design.
> Chesnay suggested that we could move the new format to the Kafka connector,
> then pass the Kafka record down to the deserialize logic so it can make use
> of the headers during deserialization and serialisation.
>
> I think this is a neat idea. This would mean:
> - the Kafka connector code would need to be updated to pass down the Kafka
> record
> - there would be the Avro Apicurio format and SQL in the kafka repository.
> We feel it is unlikely to want to use the Apicurio registry with files, as
> the Avro format could be used.
>
> Unfortunately I have found that this as not so straight forward to
> implement as the Avro Apicurio format uses the Avro format, which is tied
> to the DeserializationSchema. We were hoping to have a new decoding
> implementation that would pass down the Kafka record rather than the
> payload. This does not appear possible without a Avro format change.
>
>
> Inspired by this idea, I notice that
> KafkaValueOnlyRecordDeserializerWrapper extends
> KafkaValueOnlyDeserializerWrapper
>
> Does
>
> deserializer.deserialize(record.topic(),record.value())
>
>
>
> I am investigating If I can add a factory/reflection to provide an
> alternative
> Implementation that will pass the record based (the kafka record is not
> serializable so I will pick what we need and deserialize) as a byte array.
>
> I would need to do this 4 times (value ,key for deserialisation and
> serialisation. To do this I would need to convert the record into a byte
> array, so it fits into the existing interface (DeserializationSchema).  I
> think this could be a way through, to avoid using maps and avoid changing
> the existing Avro format and avoid change any core Flink interfaces.
>
> I am going to prototype this idea. WDYT?
>
> My thanks go to Chesnay and Danny for their support and insight around
> this Flip,
>Kind regards, David.
>
>
>
>
>
>
> From: David Radley 
> Date: Wednesday, 29 May 2024 at 11:39
> To: dev@flink.apache.org 
> Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hi Danny,
> Thank you for your feedback on this.
>
> I agree that using maps has pros and cons. The maps are flexible, but do
> require the sender and receiver to know what is in the map.
>
> When you say “That sounds like it would fit in better, I assume we cannot
> just take that approach?” The motivation behind this Flip is to support the
> headers which is the usual way that Apicurio runs. We will support the
> “schema id in the payload” as well.
>
> I agree with you when you say “ I am not 100% happy with the solution but I
> cannot offer a better option.” – this is a pragmatic way we have found to
> solve this issue. I am open to any suggestions to improve this as well.
>
> If we are going with the maps design (which is the best we have at the
> moment) ; it would be good to have the Flink core changes in base Flink
> version 2.0 as this would mean we do not need to use reflection in a Flink
> Kafka version 2 connector to work out if the runtime Flink has the new
> methods.
>
> At this stage we only have one committer (yourself) backing this. Do you
> know of other 2 committers who would support this Flip?
>
>  Kind regards, David.
>
>
>
> From: Danny Cranmer 
> Date: Friday, 24 May 2024 at 19:32
> To: dev@flink.apache.org 
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format
> Hello,
>
> > I am curious what you mean by abused.
>
> I just meant we will end up adding more and more fields to this map over
> time, and it may be hard to undo.
>
> > For Apicurio it can be sent at the start of the payload like Confluent
> Avro does. Confluent Avro have a magic byte followed by 4 bytes of schema
> id, at the start of the payload. Apicurio clients and SerDe libraries can
> be configured to not put the schema id in the headers in which case there
> is a magic byte followed by an 8 byte schema at the start of the payload.
> In the deserialization case, we would not need to look at the headers –
> though the encoding is also in the headers.
>
> That sounds like it would fit in better, I assume we cannot just take that
> approach?
>
> Thanks for the discussion. I am not 100% happy with the solution but I
> cannot offer a better option. I would be interested to hear if others have
> any suggestions. Playing devil's advocate against myself, we pass maps
> around to configure connectors so it is not too far away from that.
>
> Thanks,
> Danny
>
>
> On Fri, May 24, 2024 at 2:23 PM David Radley 
> wrote:
>
> > Hi Danny,
> > No worries, thanks for replying. I have working prototype code that is
> > 

Re: [VOTE] FLIP-456: CompiledPlan support for Batch Execution Mode

[Alexey Leonov-Vendrovskiy]

Thank you all!
The voting is closed now.
I've posted results [1].

[1] https://lists.apache.org/thread/mx3qd5qgyjtqk66p6sv639g777lm0jc6

Thanks,
Alexey

On Wed, Jul 3, 2024 at 6:39 PM Yuepeng Pan  wrote:

> +1 (non-binding)
>
> Best regards,
>
> Yuepeng Pan
>
>
>
>
>
> At 2024-07-03 01:46:13, "Sergey Nuyanzin"  wrote:
> >Thanks for driving this
> >
> >+1 (binding)
> >
> >On Tue, Jul 2, 2024, 11:21 Martijn Visser 
> wrote:
> >
> >> +1 (binding)
> >>
> >> On Mon, Jul 1, 2024 at 7:00 PM Jim Hughes  >
> >> wrote:
> >>
> >> > Hi Alexey,
> >> >
> >> > +1 (non-binding)
> >> >
> >> > I'm looking forward to parity between streaming and batch bound for
> >> > compiled plans!
> >> >
> >> > Cheers,
> >> >
> >> > Jim
> >> >
> >> > On Mon, Jul 1, 2024 at 12:55 PM Alexey Leonov-Vendrovskiy <
> >> > vendrov...@gmail.com> wrote:
> >> >
> >> > > Hello everyone,
> >> > >
> >> > > We had a good discussion of FLIP-456: CompiledPlan support for Batch
> >> > > Execution Mode [1]. Discussion thread is here: [2].
> >> > >
> >> > > Let's start voting on it. The vote will be open for at least 72
> >> > > hours unless there is an objection or insufficient votes. The FLIP
> will
> >> > be
> >> > > considered accepted if 3 binding votes (from active committers
> >> according
> >> > to
> >> > > the Flink bylaws [3]) are gathered by the community.
> >> > >
> >> > > Thanks,
> >> > > Alexey
> >> > >
> >> > > [1]
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode
> >> > > [2]
> https://lists.apache.org/thread/7gpyqvdnnbjwbh3vbk6b0pj38l91crvv
> >> > > [3]
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals
> >> > > <
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals)
> >> > > >
> >> > >
> >> >
> >>
>

[RESULT][VOTE] FLIP-456: CompiledPlan support for Batch Execution Mode

[Alexey Leonov-Vendrovskiy]

Hello everyone,

The vote [1] for FLIP-456 [2] is over. The number of required binding votes
(3) was reached (total: 5, binding: 3, non-binding: 2). No objections were
raised.

Binding:

   - Martijn Visser
   - Sergey Nuyanzin
   - Timo Walther

Non-binding:

   - Jim Hughes
   - Yuepeng Pan


PRs will be prepared. Thank you everyone who participated!

All the best,
Alexey

[1] https://lists.apache.org/thread/2ycws2zplcyd1k25pn1ljmxvg5lgd8r2
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode

Re: [ANNOUNCE] Release 1.20.0, release candidate #0

[Martijn Visser]

Hi all,

I've closed my blocker ticket; after force killing all my Flink Java
processes, it did work when I restarted it.

Thanks, Martijn

On Mon, Jul 8, 2024 at 6:13 PM Santwana Verma 
wrote:

> Hi everyone,
>
> Thanks for creating the release candidate. I've successfully validated the
> release candidate locally with the DataStream API.
>
> 1. I created a DataStream API based job, which read and deserialized JSON
> strings from an input Kafka topic using flink-connector-kafka, transformed
> the data, and wrote it in the Avro format to an output Kafka topic.
> 2. I used Maven dependencies for the job from the repository
> https://repository.apache.org/content/repositories/orgapacheflink-1742
> (flink version 1.20.0) to create the job JAR.
> 3. I ran flink from the binaries within
>
> https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc0/flink-1.20.0-bin-scala_2.12.tgz
> .
> 4. The job ran as expected when I produced to the input topic with ~500k
> msgs and consumed from the output topic.
>
> Best,
> Santwana
>
> On Fri, Jun 28, 2024 at 9:39 PM weijie guo 
> wrote:
>
> > Hi everyone,
> >
> >
> > The release candidate #0(i.e. RC0) for Apache Flink 1.20.0 has been
> > created.
> >
> >
> > This RC is currently for preview only to facilitate the integrated
> testing
> > and
> >
> > we don't have to vote on it.
> >
> >
> > RC1 is expected to be released a week later If we find no new blocker in
> > RC0.
> >
> > The related voting process will be triggered once the announcement is
> > ready.
> >
> >
> > The RC0 has all the artifacts that we would typically have for a release,
> > except
> >
> > for the release note and the website pull request for the release
> > announcement.
> >
> >
> > The following contents are available for your review:
> >
> >
> > - The preview source release and binary convenience releases [1], which
> >
> > are signed with the key with fingerprint
> > 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [2].
> >
> >
> > - All artifacts that would normally be deployed to the Maven
> > Central Repository [3].
> >
> >
> > - Source code tag "release-1.20.0-rc0" [4]
> >
> >
> > Your help testing the release will be greatly appreciated! And we'll
> >
> > create the RC1 release and the voting thread as soon as all the efforts
> are
> >
> > finished.
> >
> >
> > [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc0/
> >
> > [2] https://dist.apache.org/repos/dist/release/flink/KEYS
> >
> > [3]
> > https://repository.apache.org/content/repositories/orgapacheflink-1742/
> >
> > [4] https://github.com/apache/flink/releases/tag/release-1.20.0-rc0
> >
> >
> > Best,
> >
> > Robert, Rui, Ufuk, Weijie
> >
>

Re: [ANNOUNCE] Release 1.20.0, release candidate #0

[Santwana Verma]

Hi everyone,

Thanks for creating the release candidate. I've successfully validated the
release candidate locally with the DataStream API.

1. I created a DataStream API based job, which read and deserialized JSON
strings from an input Kafka topic using flink-connector-kafka, transformed
the data, and wrote it in the Avro format to an output Kafka topic.
2. I used Maven dependencies for the job from the repository
https://repository.apache.org/content/repositories/orgapacheflink-1742
(flink version 1.20.0) to create the job JAR.
3. I ran flink from the binaries within
https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc0/flink-1.20.0-bin-scala_2.12.tgz
.
4. The job ran as expected when I produced to the input topic with ~500k
msgs and consumed from the output topic.

Best,
Santwana

On Fri, Jun 28, 2024 at 9:39 PM weijie guo 
wrote:

> Hi everyone,
>
>
> The release candidate #0(i.e. RC0) for Apache Flink 1.20.0 has been
> created.
>
>
> This RC is currently for preview only to facilitate the integrated testing
> and
>
> we don't have to vote on it.
>
>
> RC1 is expected to be released a week later If we find no new blocker in
> RC0.
>
> The related voting process will be triggered once the announcement is
> ready.
>
>
> The RC0 has all the artifacts that we would typically have for a release,
> except
>
> for the release note and the website pull request for the release
> announcement.
>
>
> The following contents are available for your review:
>
>
> - The preview source release and binary convenience releases [1], which
>
> are signed with the key with fingerprint
> 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [2].
>
>
> - All artifacts that would normally be deployed to the Maven
> Central Repository [3].
>
>
> - Source code tag "release-1.20.0-rc0" [4]
>
>
> Your help testing the release will be greatly appreciated! And we'll
>
> create the RC1 release and the voting thread as soon as all the efforts are
>
> finished.
>
>
> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc0/
>
> [2] https://dist.apache.org/repos/dist/release/flink/KEYS
>
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1742/
>
> [4] https://github.com/apache/flink/releases/tag/release-1.20.0-rc0
>
>
> Best,
>
> Robert, Rui, Ufuk, Weijie
>

[jira] [Created] (FLINK-35788) Deprecate old InputFormat and SinkFunction

[Jira]

João Boto created FLINK-35788:
-

 Summary: Deprecate old InputFormat and SinkFunction
 Key: FLINK-35788
 URL: https://issues.apache.org/jira/browse/FLINK-35788
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / JDBC
Reporter: João Boto






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Potential Kafka Connector FLIP: Large Message Handling

[Kevin Lam]

Hi Fabian,

Awesome, this project looks great! Thanks for sharing. It would work well
with KafkaSource and the DataStream API as you've mentioned. We have
something similar internally, but where we are encountering difficulty is
integrating it with the Flink SQL Kafka DynamicTable Source and Sinks. Our
Large Message SerDe uses the Kafka Message header to store the URI on
object storage, and currently the Flink SQL Format Interfaces do not allow
passing data to/from the Kafka message headers, which lead me to suggest my
proposal. It's not easy for us to change our Large Message SerDe to use the
value to provide the reference, as it's already widely used and would
require a significant migration.

However, thinking further, maybe we should not bring in any Large message
concerns into Flink, but instead better support reading and writing
headers from Flink Formats.

I'm aware of the existing work in progress on handling headers via FLIP-454

and
this mailing list discussion
.

On Mon, Jul 8, 2024 at 10:08 AM Fabian Paul  wrote:

> Hi Kevin,
>
> I worked on a project [1] in the past that had a similar purpose. You
> should be able to use a similar approach with the existing KafkaSource by
> implementing your own KafkaRecordDeserializationSchema that hides the logic
> of pulling the records from blob storage from the connector. You can even
> use the linked project directly with the KafkaSource using [2] and [3].
>
> I agree there is room for improvements, like propagating Flink's Filesystem
> credentials to the custom deserializer, but the overall idea seems to
> require only very few changes to Flink.
>
> Best,
> Fabian
>
> [1] https://github.com/bakdata/kafka-large-message-serde
> [2]
>
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/deserializer/KafkaRecordDeserializationSchema.java#L107
> [3]
>
> https://github.com/bakdata/kafka-large-message-serde/blob/09eae933afaf8a1970b1b1bebcdffe934c368cb9/large-message-serde/src/main/java/com/bakdata/kafka/LargeMessageDeserializer.java#L50
>
> On Mon, Jul 8, 2024 at 3:49 PM Kevin Lam 
> wrote:
>
> > Hi all,
> >
> > Thanks for the responses.
> >
> > Grace those are indeed both challenges, thanks for flagging them.
> Regarding
> > expiry, we could consider having a Mark and Sweep garbage collection
> > system. A service can consume the topics with large messages, and track
> > references. When there are no references left for large messages, they
> can
> > be removed.
> >
> > Martjin, I will take a look at if there's any prior discussions in the
> > Kafka community and send the proposal to the Kafka Dev mailing list if it
> > makes sense :). It'd be much preferred if this was natively supported by
> > Kafka, since it's not currently I was also exploring making this work in
> > Flink.
> >
> >
> >
> > On Mon, Jul 8, 2024 at 3:23 AM Martijn Visser 
> > wrote:
> >
> > > Hi Kevin,
> > >
> > > I just want to double check, were you planning to send this proposal to
> > the
> > > Kafka Dev mailing list? Because I don't see directly how this affects
> > Flink
> > > :)
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Mon, Jul 8, 2024 at 8:05 AM Grace Grimwood 
> > wrote:
> > >
> > > > Hi Kevin,
> > > >
> > > > Thanks for starting this thread.
> > > >
> > > > This idea is something that was discussed in Kroxylicious (an open
> > source
> > > > Kafka proxy, I'm a maintainer there). In that discussion [1] we came
> to
> > > the
> > > > conclusion that there are a couple of issues with implementing this:
> > > > 1. Doesn't scale - very large messages (>1GiB) or large batch sizes
> > could
> > > > cause extreme memory bloat in clients, as the entire thing would need
> > to
> > > be
> > > > fed into the producer which could very quickly fill its buffers.
> > > Depending
> > > > on how the subsequent deserialization and payload fetch is handled at
> > the
> > > > consumer end, it's likely that the same behaviour would also be seen
> > > there.
> > > > 2. Difficult to sync expiry - when Kafka deletes messages due to
> > > retention
> > > > (or topic compaction), it does so without notifying clients. There is
> > no
> > > > (easy) way to ensure the associated payload is deleted from object
> > > storage
> > > > at the same time.
> > > >
> > > > It's not totally clear how Conduktor solved these issues, but IMO
> they
> > > are
> > > > worth keeping in mind. For Kroxylicious we decided these problems
> meant
> > > it
> > > > wasn't practical for us to implement this, but I'd be curious to know
> > if
> > > > you've got any ideas :)
> > > >
> > > > Regards,
> > > > Grace
> > > >
> > > > [1] https://github.com/kroxylicious/kroxylicious/discussions/1244
> > > >
> > > > On Sat, Jul 6, 2024 at 8:21 AM 

[jira] [Created] (FLINK-35787) DefaultSlotStatusSyncer might bring down JVM (exit code 239 instead of a proper shutdown)

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-35787:
-

 Summary: DefaultSlotStatusSyncer might bring down JVM (exit code 
239 instead of a proper shutdown)
 Key: FLINK-35787
 URL: https://issues.apache.org/jira/browse/FLINK-35787
 Project: Flink
  Issue Type: Bug
Reporter: Roman Khachatryan


In our internal CI, I've encountered the following error:
{code:java}
* 12:02:47,205 [   pool-126-thread-1] ERROR 
org.apache.flink.util.FatalExitExceptionHandler              [] - FATAL: Thread 
'pool-126-thread-1' produced an uncaught exception. Stopping the process...
  java.util.concurrent.CompletionException: 
java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@38ce013a[Not
 completed, task = 
java.util.concurrent.Executors$RunnableAdapter@640a9cf7[Wrapped task = 
java.util.concurrent.>
          at 
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
 ~[?:?]
          at 
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:951)
 ~[?:?]
          at 
java.util.concurrent.CompletableFuture.handleAsync(CompletableFuture.java:2282) 
~[?:?]
          at 
org.apache.flink.runtime.resourcemanager.slotmanager.DefaultSlotStatusSyncer.allocateSlot(DefaultSlotStatusSyncer.java:138)
 ~[classes/:?]
          at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager.allocateSlotsAccordingTo(FineGrainedSlotManager.java:722)
 ~[classes/:?]
          at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager.checkResourceRequirements(FineGrainedSlotManager.java:645)
 ~[classes/:?]
          at 
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager.lambda$checkResourceRequirementsWithDelay$12(FineGrainedSlotManager.java:603)
 ~[classes/:?]
          at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
          at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
          at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
 [?:?]
          at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
[?:?]
          at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
[?:?]
          at java.lang.Thread.run(Thread.java:829) [?:?]
  Caused by: java.util.concurrent.RejectedExecutionException: Task 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@38ce013a[Not
 completed, task = 
java.util.concurrent.Executors$RunnableAdapter@640a9cf7[Wrapped task = 
java.util.concurrent.CompletableFuture$UniHandle@f3d>
          at 
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
 ~[?:?]
          at 
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) 
~[?:?]
          at 
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
 ~[?:?]
          at 
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
 ~[?:?]
          at 
java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:705)
 ~[?:?]
          at 
java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:687)
 ~[?:?]
          at 
java.util.concurrent.CompletableFuture.uniHandleStage(CompletableFuture.java:949)
 ~[?:?]
          ... 11 more{code}
>From the code, it looks like RM main thread executor was shut down, and that 
>triggered JVM exit:
{code:java}
        CompletableFuture requestFuture =
                gateway.requestSlot(
                        SlotID.getDynamicSlotID(resourceId),
                        jobId,
                        allocationId,
                        resourceProfile,
                        targetAddress,
                        resourceManagerId,
                        taskManagerRequestTimeout);        
CompletableFuture returnedFuture = new CompletableFuture<>();        
FutureUtils.assertNoException(
                requestFuture.handleAsync(
                        (Acknowledge acknowledge, Throwable throwable) -> { ... 
},
                        mainThreadExecutor));
 {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Potential Kafka Connector FLIP: Large Message Handling

[Fabian Paul]

Hi Kevin,

I worked on a project [1] in the past that had a similar purpose. You
should be able to use a similar approach with the existing KafkaSource by
implementing your own KafkaRecordDeserializationSchema that hides the logic
of pulling the records from blob storage from the connector. You can even
use the linked project directly with the KafkaSource using [2] and [3].

I agree there is room for improvements, like propagating Flink's Filesystem
credentials to the custom deserializer, but the overall idea seems to
require only very few changes to Flink.

Best,
Fabian

[1] https://github.com/bakdata/kafka-large-message-serde
[2]
https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/deserializer/KafkaRecordDeserializationSchema.java#L107
[3]
https://github.com/bakdata/kafka-large-message-serde/blob/09eae933afaf8a1970b1b1bebcdffe934c368cb9/large-message-serde/src/main/java/com/bakdata/kafka/LargeMessageDeserializer.java#L50

On Mon, Jul 8, 2024 at 3:49 PM Kevin Lam 
wrote:

> Hi all,
>
> Thanks for the responses.
>
> Grace those are indeed both challenges, thanks for flagging them. Regarding
> expiry, we could consider having a Mark and Sweep garbage collection
> system. A service can consume the topics with large messages, and track
> references. When there are no references left for large messages, they can
> be removed.
>
> Martjin, I will take a look at if there's any prior discussions in the
> Kafka community and send the proposal to the Kafka Dev mailing list if it
> makes sense :). It'd be much preferred if this was natively supported by
> Kafka, since it's not currently I was also exploring making this work in
> Flink.
>
>
>
> On Mon, Jul 8, 2024 at 3:23 AM Martijn Visser 
> wrote:
>
> > Hi Kevin,
> >
> > I just want to double check, were you planning to send this proposal to
> the
> > Kafka Dev mailing list? Because I don't see directly how this affects
> Flink
> > :)
> >
> > Best regards,
> >
> > Martijn
> >
> > On Mon, Jul 8, 2024 at 8:05 AM Grace Grimwood 
> wrote:
> >
> > > Hi Kevin,
> > >
> > > Thanks for starting this thread.
> > >
> > > This idea is something that was discussed in Kroxylicious (an open
> source
> > > Kafka proxy, I'm a maintainer there). In that discussion [1] we came to
> > the
> > > conclusion that there are a couple of issues with implementing this:
> > > 1. Doesn't scale - very large messages (>1GiB) or large batch sizes
> could
> > > cause extreme memory bloat in clients, as the entire thing would need
> to
> > be
> > > fed into the producer which could very quickly fill its buffers.
> > Depending
> > > on how the subsequent deserialization and payload fetch is handled at
> the
> > > consumer end, it's likely that the same behaviour would also be seen
> > there.
> > > 2. Difficult to sync expiry - when Kafka deletes messages due to
> > retention
> > > (or topic compaction), it does so without notifying clients. There is
> no
> > > (easy) way to ensure the associated payload is deleted from object
> > storage
> > > at the same time.
> > >
> > > It's not totally clear how Conduktor solved these issues, but IMO they
> > are
> > > worth keeping in mind. For Kroxylicious we decided these problems meant
> > it
> > > wasn't practical for us to implement this, but I'd be curious to know
> if
> > > you've got any ideas :)
> > >
> > > Regards,
> > > Grace
> > >
> > > [1] https://github.com/kroxylicious/kroxylicious/discussions/1244
> > >
> > > On Sat, Jul 6, 2024 at 8:21 AM Kevin Lam  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Writing to see if the community would be open to exploring a FLIP for
> > the
> > > > Kafka Table Connectors. The FLIP would allow for storing Kafka
> Messages
> > > > beyond a Kafka cluster's message limit (1 MB by default) out of band
> in
> > > > cloud object storage or another backend.
> > > >
> > > > During serialization the message would be replaced with a reference,
> > and
> > > > during deserialization the reference would be used to fetch the large
> > > > message and pass it to Flink. Something like Option 1 in this blog
> post
> > > > <
> > > >
> > >
> >
> https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka/#Option-1:-using-an-external-store-(GB-size-messages)-0
> > > > >
> > > > .
> > > >
> > > > What do you think?
> > > >
> > > > We can make it generic by allowing users to implement their own
> > > > LargeMessageSerializer/Deserializer interface for serializing and
> > > > deserializing and handling interactions with object storage or some
> > other
> > > > backend.
> > > >
> > > > The Kafka Connectors can be extended to support ConfigOptions to
> > > > specify the class to load, as well as some user-specified properties.
> > For
> > > > example: `large-record-handling.class` and `
> > > > large-record-handling.properties.*` (where the user can specify any
> > > > properties similar to how 

[jira] [Created] (FLINK-35786) NPE in BlobServer / shutdownHook

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-35786:
-

 Summary: NPE in BlobServer / shutdownHook
 Key: FLINK-35786
 URL: https://issues.apache.org/jira/browse/FLINK-35786
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.19.1
Reporter: Roman Khachatryan
Assignee: Roman Khachatryan
 Fix For: 1.20.0, 1.19.2


In constructor, BlobServer registers a shutdown hook to close the socket.

Later in constructor, BlobServer creates this socket (and makes sure it's not 
null).

 

But if the shutdown hook gets invoked before opening the socket, NPE will be 
thrown:
{code:java}
  12:02:49,983 [PermanentBlobCache shutdown hook] INFO  
org.apache.flink.runtime.blob.PermanentBlobCache             [] - Shutting down 
BLOB cache
  12:02:49,985 [BlobServer shutdown hook] ERROR 
org.apache.flink.runtime.blob.BlobServer                     [] - Error during 
shutdown of BlobServer via JVM shutdown hook.
  java.lang.NullPointerException: null
          at 
org.apache.flink.runtime.blob.BlobServer.close(BlobServer.java:358) 
~[classes/:?]
          at 
org.apache.flink.util.ShutdownHookUtil.lambda$addShutdownHook$0(ShutdownHookUtil.java:39)
 ~[flink-core-1.19-SNAPSHOT.jar:1.19-SNAPSHOT]
          at java.lang.Thread.run(Thread.java:829) [?:?]
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Potential Kafka Connector FLIP: Large Message Handling

[Kevin Lam]

Hi all,

Thanks for the responses.

Grace those are indeed both challenges, thanks for flagging them. Regarding
expiry, we could consider having a Mark and Sweep garbage collection
system. A service can consume the topics with large messages, and track
references. When there are no references left for large messages, they can
be removed.

Martjin, I will take a look at if there's any prior discussions in the
Kafka community and send the proposal to the Kafka Dev mailing list if it
makes sense :). It'd be much preferred if this was natively supported by
Kafka, since it's not currently I was also exploring making this work in
Flink.



On Mon, Jul 8, 2024 at 3:23 AM Martijn Visser 
wrote:

> Hi Kevin,
>
> I just want to double check, were you planning to send this proposal to the
> Kafka Dev mailing list? Because I don't see directly how this affects Flink
> :)
>
> Best regards,
>
> Martijn
>
> On Mon, Jul 8, 2024 at 8:05 AM Grace Grimwood  wrote:
>
> > Hi Kevin,
> >
> > Thanks for starting this thread.
> >
> > This idea is something that was discussed in Kroxylicious (an open source
> > Kafka proxy, I'm a maintainer there). In that discussion [1] we came to
> the
> > conclusion that there are a couple of issues with implementing this:
> > 1. Doesn't scale - very large messages (>1GiB) or large batch sizes could
> > cause extreme memory bloat in clients, as the entire thing would need to
> be
> > fed into the producer which could very quickly fill its buffers.
> Depending
> > on how the subsequent deserialization and payload fetch is handled at the
> > consumer end, it's likely that the same behaviour would also be seen
> there.
> > 2. Difficult to sync expiry - when Kafka deletes messages due to
> retention
> > (or topic compaction), it does so without notifying clients. There is no
> > (easy) way to ensure the associated payload is deleted from object
> storage
> > at the same time.
> >
> > It's not totally clear how Conduktor solved these issues, but IMO they
> are
> > worth keeping in mind. For Kroxylicious we decided these problems meant
> it
> > wasn't practical for us to implement this, but I'd be curious to know if
> > you've got any ideas :)
> >
> > Regards,
> > Grace
> >
> > [1] https://github.com/kroxylicious/kroxylicious/discussions/1244
> >
> > On Sat, Jul 6, 2024 at 8:21 AM Kevin Lam 
> > wrote:
> >
> > > Hi all,
> > >
> > > Writing to see if the community would be open to exploring a FLIP for
> the
> > > Kafka Table Connectors. The FLIP would allow for storing Kafka Messages
> > > beyond a Kafka cluster's message limit (1 MB by default) out of band in
> > > cloud object storage or another backend.
> > >
> > > During serialization the message would be replaced with a reference,
> and
> > > during deserialization the reference would be used to fetch the large
> > > message and pass it to Flink. Something like Option 1 in this blog post
> > > <
> > >
> >
> https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka/#Option-1:-using-an-external-store-(GB-size-messages)-0
> > > >
> > > .
> > >
> > > What do you think?
> > >
> > > We can make it generic by allowing users to implement their own
> > > LargeMessageSerializer/Deserializer interface for serializing and
> > > deserializing and handling interactions with object storage or some
> other
> > > backend.
> > >
> > > The Kafka Connectors can be extended to support ConfigOptions to
> > > specify the class to load, as well as some user-specified properties.
> For
> > > example: `large-record-handling.class` and `
> > > large-record-handling.properties.*` (where the user can specify any
> > > properties similar to how the Kafka Consumer and Producer properties
> are
> > > handled
> > > <
> > >
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicTableFactory.java#L201
> > > >
> > > ).
> > >
> > > In terms of call sites for the LargeMessage handling, I think we can
> > > consider inside of DynamicKafkaDeserializationSchema
> > > <
> > >
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaDeserializationSchema.java
> > > >
> > > and DynamicKafkaRecordSerializationSchema
> > > <
> > >
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaRecordSerializationSchema.java
> > > >,
> > > where the ConsumerRecord
> > > <
> > >
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaDeserializationSchema.java#L108
> > > >
> > > and ProducerRecords
> > > <
> > >
> >
> 

Re: [ANNOUNCE] Release 1.20.0, release candidate #0

[Martijn Visser]

Hi all,

I've found that no query works in the SQL Client for RC0. See
https://issues.apache.org/jira/browse/FLINK-35785

Best regards,

Martijn

On Fri, Jun 28, 2024 at 6:09 PM weijie guo 
wrote:

> Hi everyone,
>
>
> The release candidate #0(i.e. RC0) for Apache Flink 1.20.0 has been
> created.
>
>
> This RC is currently for preview only to facilitate the integrated testing
> and
>
> we don't have to vote on it.
>
>
> RC1 is expected to be released a week later If we find no new blocker in
> RC0.
>
> The related voting process will be triggered once the announcement is
> ready.
>
>
> The RC0 has all the artifacts that we would typically have for a release,
> except
>
> for the release note and the website pull request for the release
> announcement.
>
>
> The following contents are available for your review:
>
>
> - The preview source release and binary convenience releases [1], which
>
> are signed with the key with fingerprint
> 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [2].
>
>
> - All artifacts that would normally be deployed to the Maven
> Central Repository [3].
>
>
> - Source code tag "release-1.20.0-rc0" [4]
>
>
> Your help testing the release will be greatly appreciated! And we'll
>
> create the RC1 release and the voting thread as soon as all the efforts are
>
> finished.
>
>
> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc0/
>
> [2] https://dist.apache.org/repos/dist/release/flink/KEYS
>
> [3]
> https://repository.apache.org/content/repositories/orgapacheflink-1742/
>
> [4] https://github.com/apache/flink/releases/tag/release-1.20.0-rc0
>
>
> Best,
>
> Robert, Rui, Ufuk, Weijie
>

[jira] [Created] (FLINK-35785) Executing query in SQL client results in "java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode"

[Martijn Visser (Jira)]

Martijn Visser created FLINK-35785:
--

 Summary: Executing query in SQL client results in 
"java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode"
 Key: FLINK-35785
 URL: https://issues.apache.org/jira/browse/FLINK-35785
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing, Table SQL / Client
Reporter: Martijn Visser


Tested with Flink 1.20 RC0

Reproducer:

{code:sql}
-- Create the product table
CREATE TABLE `product` (
id INT,
brandId INT,
grade STRING,
PRIMARY KEY (id) NOT ENFORCED
) WITH (
'connector' = 'datagen',
'rows-per-second' = '10',
'fields.id.kind' = 'random',
'fields.brandId.min' = '1',
'fields.brandId.max' = '100',
'fields.grade.length' = '10'
);
{code}

Followed by:
{code:sql}
SELECT * FROM product
{code}

Results in:
[ERROR] Could not execute SQL statement. Reason:
java.lang.ClassNotFoundException: org.apache.flink.core.execution.RestoreMode



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35784) The cp file-merging directory not properly registered in SharedStateRegistry

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-35784:
---

 Summary: The cp file-merging directory not properly registered in 
SharedStateRegistry
 Key: FLINK-35784
 URL: https://issues.apache.org/jira/browse/FLINK-35784
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.20.0
Reporter: Zakelly Lan
Assignee: Zakelly Lan


The {{OperatorSubtaskState}} only make keyed state to register with 
{{SharedStateRegistry}}. However, the file-merging directories's handle are 
wrapped in {{FileMergingOperatorStreamStateHandle}}, which is an 
{{OperatorStreamStateHandle}}. That means the {{#registerSharedStates}} is 
never called, so the registry will never delete the directories.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35783) Flink CDC Could not start the yaml Job

[layhuts (Jira)]

layhuts created FLINK-35783:
---

 Summary: Flink CDC Could not start the yaml Job
 Key: FLINK-35783
 URL: https://issues.apache.org/jira/browse/FLINK-35783
 Project: Flink
  Issue Type: Bug
  Components: Flink CDC
Affects Versions: cdc-3.1.1
Reporter: layhuts


* flink版本1.19.1
 * flink CDC版本3.1.1
 * 在${FLINK_HOME}/lib下增加了 mysql-connector-java-8.0.27.jar 和 
flink-sql-connector-mysql-cdc-3.1.1.jar
 * 在flink-cdc/lib下增加了flink-cdc-pipeline-connector-mysql-3.1.1.jar 和 
flink-cdc-pipeline-connector-doris-3.1.1.jar

第一次使用
{code:java}
bin/flink-cdc.sh ***.yaml {code}
 
提交作业提示java.lang.NoClassDefFoundError:org/apache/flink/cdc/runtime/typeutils/EventTypeInfo
{code:java}
Caused by: java.lang.NoClassDefFoundError: 
org/apache/flink/cdc/runtime/typeutils/EventTypeInfo   at 
java.lang.Class.getDeclaredFields0(Native Method)   at 
java.lang.Class.privateGetDeclaredFields(Class.java:2583)   at 
java.lang.Class.getDeclaredField(Class.java:2068)   at 
java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1872)   at 
java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79)   at 
java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506)   at 
java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494)   at 
java.security.AccessController.doPrivileged(Native Method)   at 
java.io.ObjectStreamClass.(ObjectStreamClass.java:494)   at 
java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391)   at 
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681)   at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2028)   at 
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1875)   at 
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2028)   at 
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1875)   at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2209)   at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1692)   at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2454)   at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2378)   at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2236)   at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1692)   at 
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2454)   at 
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2378)   at 
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2236)   at 
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1692)   at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:508)   at 
java.io.ObjectInputStream.readObject(ObjectInputStream.java:466)   at 
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:539)
   at 
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:527)
   at 
org.apache.flink.util.SerializedValue.deserializeValue(SerializedValue.java:67) 
  at 
org.apache.flink.runtime.operators.coordination.OperatorCoordinatorHolder.create(OperatorCoordinatorHolder.java:496)
   at 
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.createOperatorCoordinatorHolder(ExecutionJobVertex.java:294)
   at 
org.apache.flink.runtime.executiongraph.ExecutionJobVertex.(ExecutionJobVertex.java:173)
   ... 19 more Caused by: java.lang.ClassNotFoundException: 
org.apache.flink.cdc.runtime.typeutils.EventTypeInfo   at 
java.net.URLClassLoader.findClass(URLClassLoader.java:387)   at 
java.lang.ClassLoader.loadClass(ClassLoader.java:418)   at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)   at 
java.lang.ClassLoader.loadClass(ClassLoader.java:351)   ... 52 more {code}
按照提示在${FLINK_HOME}/lib下增加了 flink-cdc-runtime-3.1.1.jar 后再次运行出现如下问题：
{code:java}
Exception in thread "main" org.apache.flink.util.FlinkException: Failed to 
execute job 'Sync mid_cloud Database to Doris'.
    at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2455)
    at 
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
    at 
org.apache.flink.cdc.composer.flink.FlinkPipelineExecution.execute(FlinkPipelineExecution.java:43)
    at org.apache.flink.cdc.cli.CliExecutor.run(CliExecutor.java:74)
    at org.apache.flink.cdc.cli.CliFrontend.main(CliFrontend.java:71)
Caused by: java.lang.RuntimeException: 
org.apache.flink.runtime.client.JobInitializationException: Could not start the 
JobMaster.
    at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
    at 
org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75)
    at 
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
    at 

[jira] [Created] (FLINK-35782) Flink connector jdbc works wrong when using sql gateway

[Yi Cai (Jira)]

Yi Cai created FLINK-35782:
--

 Summary: Flink connector jdbc works wrong when using sql gateway
 Key: FLINK-35782
 URL: https://issues.apache.org/jira/browse/FLINK-35782
 Project: Flink
  Issue Type: Bug
  Components: Connectors / JDBC, Table SQL / Gateway
Affects Versions: 1.19.1, 1.18.1
Reporter: Yi Cai


When using sql clent to submit jobs to sql gateway will cause no suitable 
driver found for x

 

script:
add jar 's3://flink/lib/flink-connector-jdbc-3.1.2-1.19.jar';
add jar 's3://flink/lib/mysql-connector-j-8.0.33.jar';
CREATE CATALOG xxx WITH(
...
);
select ...



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] CLI action deprecation process

[Muhammet Orazov]

Hey Ferenc,

Yes correct. My thoughts were based on finding tradeoff between
following fully deprecation process and leaner one for CLIs.

Since cli are not like APIs, I think users would be aware of
deprecation only when were remove the commands. That is they
try to run their jobs with upgrade and it fails with action
not available.

So maybe we don't have to follow fully API `@PublicEvolving`
process for this.

Another maybe user friendly approach would be to inform with
warning that the `run-application` cli action will be dropped,
and suggest new action and migration on the log message.

Best,
Muhammet

On 2024-07-04 20:17, Ferenc Csaky wrote:

Hi Muhammet,

Thank you for your thoughts!


After two minor releases, and on next major version bump,
we could drop the `run-application` method as suggested
on discussion by Xintong.


Here, you describe the deprecation/removal process of a public
API by the definition we have in the project now. So if the same
applies to a CLI action, why should we not enforce such behavior
for those as well?

If following the same process for CLIs make sense, we should also
enforce the same compatibility guarantees IMO.

Best,
Ferenc



On Friday, 28 June 2024 at 09:30, Muhammet Orazov 
 wrote:





Hey Ferenc,

Thanks for starting the discussion!

I agree that the CLI is user facing, but I think we don't
have to treat it as other public APIs.

I'd propose to throw user friendly exception for
`run-application` with suggestion to use `run` case instead.
This would make users aware of the change and require them
to migrate their scripts.

After two minor releases, and on next major version bump,
we could drop the `run-application` method as suggested
on discussion by Xintong.

Best,
Muhammet


On 2024-06-26 15:33, Ferenc Csaky wrote:

> Hello devs,
> I would like to open a discussion about considerations regarding how to
> deprecate CLI
> actions, and what compatibility guarantees should apply to such cases.
> The topic came up in
> a previous discussion [1] about a current FLIP to merge the run and
> run-application
> behavior [2].
>
> According to Xintong's previous inputs, currently the Flink CLI, or its
> actions are not handled
> as public APIs by the existing definition (@Public or @PublicEvolving
> annotated). So
> legally it would be possible to change CLIs anytime. I agree with
> Xintong that CLI actions
> should be considered as public APIs, and as such, compatibility
> guarantees should be
> provided.
>
> CLI actions are defined as private constants in CliFrontend [3], so
> IMO the existing rules
> are not perfectly applicable as is. Both @Public and @PublicEvolving
> can be applied to
> fields (although for @Public that is only true as per the javadoc,
> technically it can only be
> applied to classes), so I think that could be a good approach that is
> achievable with minimal
> effort and changes.
>
> It would also be possible to define a different process, but IMO the
> more lightweight and
> maintainable the process the better, so it is a good thing if it is not
> necessary to bring in
> new entities and/or rules to check and enforce compatibility.
>
> WDYT?
>
> Best,
> Ferenc
>
> [1] https://lists.apache.org/thread/0vc8v3t7fr6w9hmwf9zbjbyk5c3bcj50
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179
> [3]
> 
https://github.com/apache/flink/blob/27287a105f6585e89795e2a6cbffa8254abb6e57/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L98

Re: Potential Kafka Connector FLIP: Large Message Handling

[Martijn Visser]

Hi Kevin,

I just want to double check, were you planning to send this proposal to the
Kafka Dev mailing list? Because I don't see directly how this affects Flink
:)

Best regards,

Martijn

On Mon, Jul 8, 2024 at 8:05 AM Grace Grimwood  wrote:

> Hi Kevin,
>
> Thanks for starting this thread.
>
> This idea is something that was discussed in Kroxylicious (an open source
> Kafka proxy, I'm a maintainer there). In that discussion [1] we came to the
> conclusion that there are a couple of issues with implementing this:
> 1. Doesn't scale - very large messages (>1GiB) or large batch sizes could
> cause extreme memory bloat in clients, as the entire thing would need to be
> fed into the producer which could very quickly fill its buffers. Depending
> on how the subsequent deserialization and payload fetch is handled at the
> consumer end, it's likely that the same behaviour would also be seen there.
> 2. Difficult to sync expiry - when Kafka deletes messages due to retention
> (or topic compaction), it does so without notifying clients. There is no
> (easy) way to ensure the associated payload is deleted from object storage
> at the same time.
>
> It's not totally clear how Conduktor solved these issues, but IMO they are
> worth keeping in mind. For Kroxylicious we decided these problems meant it
> wasn't practical for us to implement this, but I'd be curious to know if
> you've got any ideas :)
>
> Regards,
> Grace
>
> [1] https://github.com/kroxylicious/kroxylicious/discussions/1244
>
> On Sat, Jul 6, 2024 at 8:21 AM Kevin Lam 
> wrote:
>
> > Hi all,
> >
> > Writing to see if the community would be open to exploring a FLIP for the
> > Kafka Table Connectors. The FLIP would allow for storing Kafka Messages
> > beyond a Kafka cluster's message limit (1 MB by default) out of band in
> > cloud object storage or another backend.
> >
> > During serialization the message would be replaced with a reference, and
> > during deserialization the reference would be used to fetch the large
> > message and pass it to Flink. Something like Option 1 in this blog post
> > <
> >
> https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka/#Option-1:-using-an-external-store-(GB-size-messages)-0
> > >
> > .
> >
> > What do you think?
> >
> > We can make it generic by allowing users to implement their own
> > LargeMessageSerializer/Deserializer interface for serializing and
> > deserializing and handling interactions with object storage or some other
> > backend.
> >
> > The Kafka Connectors can be extended to support ConfigOptions to
> > specify the class to load, as well as some user-specified properties. For
> > example: `large-record-handling.class` and `
> > large-record-handling.properties.*` (where the user can specify any
> > properties similar to how the Kafka Consumer and Producer properties are
> > handled
> > <
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicTableFactory.java#L201
> > >
> > ).
> >
> > In terms of call sites for the LargeMessage handling, I think we can
> > consider inside of DynamicKafkaDeserializationSchema
> > <
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaDeserializationSchema.java
> > >
> > and DynamicKafkaRecordSerializationSchema
> > <
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaRecordSerializationSchema.java
> > >,
> > where the ConsumerRecord
> > <
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaDeserializationSchema.java#L108
> > >
> > and ProducerRecords
> > <
> >
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaRecordSerializationSchema.java#L75
> > >
> > are passed respectively.
> >
> > If there's interest, I would be happy to help flesh out the proposal
> more.
> >
>

Re: Potential Kafka Connector FLIP: Large Message Handling

[Grace Grimwood]

Hi Kevin,

Thanks for starting this thread.

This idea is something that was discussed in Kroxylicious (an open source
Kafka proxy, I'm a maintainer there). In that discussion [1] we came to the
conclusion that there are a couple of issues with implementing this:
1. Doesn't scale - very large messages (>1GiB) or large batch sizes could
cause extreme memory bloat in clients, as the entire thing would need to be
fed into the producer which could very quickly fill its buffers. Depending
on how the subsequent deserialization and payload fetch is handled at the
consumer end, it's likely that the same behaviour would also be seen there.
2. Difficult to sync expiry - when Kafka deletes messages due to retention
(or topic compaction), it does so without notifying clients. There is no
(easy) way to ensure the associated payload is deleted from object storage
at the same time.

It's not totally clear how Conduktor solved these issues, but IMO they are
worth keeping in mind. For Kroxylicious we decided these problems meant it
wasn't practical for us to implement this, but I'd be curious to know if
you've got any ideas :)

Regards,
Grace

[1] https://github.com/kroxylicious/kroxylicious/discussions/1244

On Sat, Jul 6, 2024 at 8:21 AM Kevin Lam 
wrote:

> Hi all,
>
> Writing to see if the community would be open to exploring a FLIP for the
> Kafka Table Connectors. The FLIP would allow for storing Kafka Messages
> beyond a Kafka cluster's message limit (1 MB by default) out of band in
> cloud object storage or another backend.
>
> During serialization the message would be replaced with a reference, and
> during deserialization the reference would be used to fetch the large
> message and pass it to Flink. Something like Option 1 in this blog post
> <
> https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka/#Option-1:-using-an-external-store-(GB-size-messages)-0
> >
> .
>
> What do you think?
>
> We can make it generic by allowing users to implement their own
> LargeMessageSerializer/Deserializer interface for serializing and
> deserializing and handling interactions with object storage or some other
> backend.
>
> The Kafka Connectors can be extended to support ConfigOptions to
> specify the class to load, as well as some user-specified properties. For
> example: `large-record-handling.class` and `
> large-record-handling.properties.*` (where the user can specify any
> properties similar to how the Kafka Consumer and Producer properties are
> handled
> <
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/KafkaDynamicTableFactory.java#L201
> >
> ).
>
> In terms of call sites for the LargeMessage handling, I think we can
> consider inside of DynamicKafkaDeserializationSchema
> <
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaDeserializationSchema.java
> >
> and DynamicKafkaRecordSerializationSchema
> <
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaRecordSerializationSchema.java
> >,
> where the ConsumerRecord
> <
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaDeserializationSchema.java#L108
> >
> and ProducerRecords
> <
> https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/DynamicKafkaRecordSerializationSchema.java#L75
> >
> are passed respectively.
>
> If there's interest, I would be happy to help flesh out the proposal more.
>

[jira] [Created] (FLINK-35781) Make pipeline parallelism config optional

[yux (Jira)]

yux created FLINK-35781:
---

 Summary: Make pipeline parallelism config optional
 Key: FLINK-35781
 URL: https://issues.apache.org/jira/browse/FLINK-35781
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Reporter: yux


Currently, Flink CDC `PIPELINE_PARALLELISM` option is forcefully required in 
pipeline definition, which turns out to be unnecessary since Flink already has 
a fallback parallelism option.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35780) Support state migration from disabling to enabling ttl in RocksDBState

[xiangyu feng (Jira)]

xiangyu feng created FLINK-35780:


 Summary: Support state migration from disabling to enabling ttl in 
RocksDBState
 Key: FLINK-35780
 URL: https://issues.apache.org/jira/browse/FLINK-35780
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / State Backends
Reporter: xiangyu feng






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35779) Update documentation for PubSubSinkV2

[Ahmed Hamdy (Jira)]

Ahmed Hamdy created FLINK-35779:
---

 Summary: Update documentation for PubSubSinkV2
 Key: FLINK-35779
 URL: https://issues.apache.org/jira/browse/FLINK-35779
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Google Cloud PubSub
Reporter: Ahmed Hamdy
 Fix For: gcp-pubsub-3.2.0


Update PubSub documentation with {{PubSubSinkV2}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35778) Escape URI reserved characters when creating file-merging directories

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-35778:
---

 Summary: Escape URI reserved characters when creating file-merging 
directories
 Key: FLINK-35778
 URL: https://issues.apache.org/jira/browse/FLINK-35778
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.20.0
Reporter: Zakelly Lan
Assignee: Zakelly Lan
 Fix For: 1.20.0


Currently, the file-merging manager for checkpoint files will create 
directories based on tm resource id, job id and operator ids. While in some 
cases, these ids include some characters that are reserved in URI scheme. So we 
should do a simple escape for those ids.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35777) Observe session jobs during cleanup

[Gyula Fora (Jira)]

Gyula Fora created FLINK-35777:
--

 Summary: Observe session jobs during cleanup
 Key: FLINK-35777
 URL: https://issues.apache.org/jira/browse/FLINK-35777
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.10.0


In the same way we do for FlinkDeployments, session jobs should be also 
observed in the cleanup logic to let the reconciler work correctly



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35776) Simplify job observe logic

[Gyula Fora (Jira)]

Gyula Fora created FLINK-35776:
--

 Summary: Simplify job observe logic
 Key: FLINK-35776
 URL: https://issues.apache.org/jira/browse/FLINK-35776
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora
Assignee: Gyula Fora
 Fix For: kubernetes-operator-1.10.0


There is a fairly complicated listing / observe logic for jobs currently that 
is no longer necessary as we have a stable logic to always record the jobID in 
the status before submission.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION

[Yubin Li]

+1 (non-binding)

Best,
Yubin

On Fri, Jul 5, 2024 at 4:09 AM Ferenc Csaky  wrote:
>
> +1 (non-binding)
>
> Best,
> Ferenc
>
>
>
>
> On Thursday, 4 July 2024 at 16:20, Feng Jin  wrote:
>
> >
> >
> > +1 (non-binding)
> >
> > Best,
> > Feng Jin
> >
> > On Thu, Jul 4, 2024 at 6:02 PM Martijn Visser martijnvis...@apache.org
> >
> > wrote:
> >
> > > +1 (binding)
> > >
> > > On Thu, Jul 4, 2024 at 5:39 AM Yanquan Lv decq12y...@gmail.com wrote:
> > >
> > > > Hi Natea, thanks for driving it.
> > > > +1 (non-binding).
> > > >
> > > > Jim Hughes jhug...@confluent.io.invalid 于2024年7月4日周四 04:41写道：
> > > >
> > > > > Hi Natea,
> > > > >
> > > > > Looks good to me!
> > > > >
> > > > > +1 (non-binding).
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Jim
> > > > >
> > > > > On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada
> > > > > nbesh...@confluent.io.invalid wrote:
> > > > >
> > > > > > Sorry I forgot to include the FLIP [1] and the mailing thread
> > > > > > discussion
> > > > > > link [2] in my previous email.
> > > > > >
> > > > > > [1]
> > >
> > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION
> > >
> > > > > > [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph
> > > > > >
> > > > > > Thanks,
> > > > > > Natea
> > > > > >
> > > > > > On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada <
> > > > > > nbesh...@confluent.io>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello everyone,
> > > > > > >
> > > > > > > I would like to start a vote on FLIP-465 [1]. It proposes adding
> > > > > > > SQL
> > > > > > > syntax that would allow users to describe the metadata of a given
> > > > > > > function.
> > > > > > >
> > > > > > > The vote will be open for at least 72 hours (Saturday, July 6th of
> > > > > > > July
> > > > > > > 2024,
> > > > > > > 12:30 PST) unless there is an objection or insufficient votes.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Natea

Re: [DISCUSS] FLIP-467: Introduce Generalized Watermarks

[Xintong Song]

Hi Jeyhun,

Thanks for working on this FLIP.

In general, I think it's a good idea to generalize the concept of Watermark
to not only representing the advancing of event time, but general
indicators / events / signals that need to be passed along the data
streams. So +1 for working towards this direction.

However, my major concern after reading this FLIP is that, the current
design might be too complicated. It tries take all possible kinds of events
(timestamp watermark, end-of-data, end-of-partition, internal watermark,
and arbitrary user defined watermarks) into consideration, which
complicates the design when it comes to serialization and propagation.

IMHO, this feature, the ability to send custom events across operators
along the data streams, has the potential to become a major differentiator
of DataStream API V2 comparing to V1. For such a feature, I don't think
it's feasible to design everything properly at the very beginning without
enough user feedbacks. I'd suggest to start with a smaller scope, and build
the feature incrementally as new demands and feedbacks arise.

To be specific, I'd suggest to:
1. Only focus on user facing events, thus Watermarks that are either
generated or handled by user codes (process functions and connectors).
Refactor of existing internal events  does not bring any benefit to users,
and may even unstablize existing mechanisms. We could do that incrementally
after the generalized watermark mechsnism becomes stable.
2. Start with a limited set of supported data types and propagation
strategies. We can add suport for arbitrary types and strategies later, if
proved necessary. By that time, we should be able to better understand the
use cases based on real feedbacks.
3. Try to minimize the set of concepts and apis that users need to
understand and work with, and make them simple and easy to understand. I'm
not saying we should not discuss designs of internal implementations in
this FLIP. Just it would be easier to understand the FLIP if it presents
first how users should understand and use the feature, then the key
internal designs in order to achieve that.

# Some detailed suggestions

## Use cases

Concrete use cases are usually helpful for designing such general
mechanism. You may examine the design by trying to use it to fulfill the
demands from the use cases. In cases you are looking for such use cases in
addition to the event-time watermaks, here are some inputs.
- In FLIP-309/327/329 [1-3], we proposed to detect the data freshness from
source, and use that information for various improvements. In DataStream
API V1, such information is carried by RecordAttributes, which is quite
similar to the genralized watermark except that we do not allow defining
arbitrary custom attributes.
- In Flink CDC, there are two phases, scaning the table at certain time
point, and cosuming the binlog from that time point. In the first phase,
there's only +I but no +U/-U/-D in the changelog, and downstream operators
can do many optimizations based on that information. We haven't bring those
optimizations to the community, because that requires the runtime layer to
understand the concept of table / sql changelogs. If we can send custom
events accross operators, without requiring runtime to understand those
events, the problem would be solved.
- In processing-time temporal join, the join operator does not wait for the
build side to complete before consuming the probe side data. This is
because the build side is contineously updated and the join operator does
not know when the initial build is finished. The result is that, initial
data from the probe side that should be joined with initial data from the
build side are missed. If we can send a signal from the build side source
to the join operator, notifying about the completion of initial build, the
problem would be solved. Similar to the previous case, such information
should not be understood by the runtime layer.

## Watermark Definition

The FLIP defines the new generalized Watermak as "indicators in data
streams", which is a bit too general.

I think we should at least include the following information:
- non-data events / records / indicators
- flow along the data streams
- can be generated / handled by process functions, connectors, and the
framework
- may need to be aligned across parallel data streams

## Types of Watermarks

Requiring users to always implement serializations for custom watermarks
might be a bit too heavy. Alternatively, we may consider only support
primitive types for Watermarks, i.e., BOOLEAN, LONG, etc. If complex types
are proved necessary in future, we can introduce STRING or BYTES so that
users can do custom serde by themselves.

Another benefit of using primitive types is that, it simplifies the
alignment semantics. Currently in this FLIP, users are required to
implement a WatermarkCombiner, which is not trivil. If we only support
limited types, we can (maybe only) provide built-in combiners for users,
e.g., 

[jira] [Created] (FLINK-35775) README.md changes

[911432 (Jira)]

911432 created FLINK-35775:
--

 Summary: README.md changes
 Key: FLINK-35775
 URL: https://issues.apache.org/jira/browse/FLINK-35775
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.20.0
Reporter: 911432


I understand that flink 1.20 becomes java 8, 11, 17, and 21.

Of course, in flink 2.0, the basic versions are said to be 17 and 21.

So I hope you can edit the part in github 
[README.md|https://github.com/apache/flink/pull/24983].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re:Re: Re: [DISCUSS] FLIP-466: Introduce ProcessFunction Attribute in DataStream API V2

[Wencong Liu]

Hi Yuxin,
Thanks for the reply.
> For idempotence annotation, what's the specific behavior?


StreamTask will reduce the frequency of output record, which will
have a default value and can also be set through configuration options.
The specific rules will be described in detail in the subsequent FLIP.


Best,
Wencong













在 2024-07-05 14:19:44，"Yuxin Tan"  写道：
>Hi, Wencong,
>Thanks for driving the FLIP.
>
>+1 for this FLIP.
>
>I believe these hints will improve the performance in many use cases.
>I only have a minor question about the Idempotence annotation. When
>this annotation is added, how does StreamTask optimize the frequency?
>Does it ensure a single output, or does it merely reduce the frequency
>of the outputs?
>
>Best,
>Yuxin
>
>
>Wencong Liu  于2024年7月1日周一 16:39写道：
>
>> Hi, Jeyhun,
>> Thanks for the reply.
>>
>>
>> > Integrate these annotations with the execution plan.
>> I believe DataStream is an Imperative API, which means
>> that the actual execution plan is basically consistent with
>> the computational logic expressed by the user with DataStream,
>> and it is different from SQL, so the significance of supporting
>> getExecutionPlan in the short term may not be great. If it is to
>> be supported later, it is possible to consider the impact of Hints.
>>
>>
>> > Check for misuse of attributes or ignore it.
>> For illegal use (annotated on the inappropriate ProcessFunction),
>> an exception will be thrown. For legal use, the framework can also
>> choose to ignore it.
>>
>>
>> > A framework to include attributes.
>> Yes, we will provide a basic framework in the implementation
>> to help developers for extension.
>>
>>
>> Best,
>> Wencong
>>
>>
>> At 2024-06-28 02:06:37, "Jeyhun Karimov"  wrote:
>> >Hi Wencong,
>> >
>> >Thanks for the FLIP. +1 for it.
>> >
>> >Providing hints to users will enable more optimization potential for DSv2.
>> >I have a few questions.
>> >
>> >I think currently, DSv2 ExecutionEnvironment does not support getting
>> >execution plan (getExecutionPlan()).
>> >Do you plan to integrate these annotations with the execution plan?
>> >
>> >Any plans to check for misuse of attributes? Or any plans for a framework
>> >to implicitly include attributes?
>> >
>> >Also, now that we make analogy with SQL hints, SQL query planners usually
>> >ignore wrong hints and continue with its best plan.
>> >Do we want to consider this approach? Or should we throw exception
>> whenever
>> >the hint (attribute in this case) is wrong?
>> >
>> >
>> >Regards,
>> >Jeyhun
>> >
>> >
>> >On Thu, Jun 27, 2024 at 7:47 AM Xintong Song 
>> wrote:
>> >
>> >> +1 for this FLIP.
>> >>
>> >> I think this is similar to SQL hints, where users can provide optional
>> >> information to help the engine execute the workload more efficiently.
>> >> Having a unified mechanism for such kind of hints should improve
>> usability
>> >> compared to introducing tons of configuration knobs.
>> >>
>> >> Best,
>> >>
>> >> Xintong
>> >>
>> >>
>> >>
>> >> On Wed, Jun 26, 2024 at 8:09 PM Wencong Liu 
>> wrote:
>> >>
>> >> > Hi devs,
>> >> >
>> >> >
>> >> > I'm proposing a new FLIP[1] to introduce the ProcessFunction
>> Attribute in
>> >> > the
>> >> > DataStream API V2. The goal is to optimize job execution by leveraging
>> >> > additional information about users' ProcessFunction logic. The
>> proposal
>> >> > includes
>> >> > the following scenarios where the ProcessFunction Attribute can
>> >> > significantly
>> >> > enhance optimization:
>> >> >
>> >> >
>> >> > Scenario 1: If the framework recognizes that the ProcessFunction
>> outputs
>> >> > data
>> >> > only after all input is received, the downstream operators can be
>> >> > scheduled until
>> >> > the ProcessFunction is finished, which effectively reduces resource
>> >> > consumption.
>> >> > Ignoring this information could lead to premature scheduling of
>> >> downstream
>> >> > operators with no data to process. This scenario is addressed and
>> >> > optimized by FLIP-331[2].
>> >> >
>> >> >
>> >> > Scenario 2: For stream processing, where users are only interested in
>> the
>> >> > latest
>> >> > result per key at the current time, the framework can optimize by
>> >> > adjusting the
>> >> > frequency of ProcessFunction outputs. This reduces shuffle data volume
>> >> and
>> >> > downstream operator workload. If this optimization is ignored, each
>> new
>> >> > input
>> >> > would trigger a new output. This scenario is addressed and
>> >> > optimized by FLIP-365[3].
>> >> >
>> >> >
>> >> > Scenario 3: If a user's ProcessFunction neither caches inputs nor
>> >> outputs,
>> >> > recognizing this can enable object reuse for this data within the
>> >> > OperatorChain,
>> >> > enhancing performance. Without this optimization, data would be copied
>> >> > before
>> >> > being passed to the next operator. This scenario is addressed and
>> >> > optimized by FLIP-329[4].
>> >> >
>> >> >
>> >> > To unify the mechanism for utilizing additional information and
>> >> optimizing

[jira] [Created] (FLINK-35774) The cache of transform is not updated after process schema change event

[Wenkai Qi (Jira)]

Wenkai Qi created FLINK-35774:
-

 Summary: The cache of transform is not updated after process 
schema change event
 Key: FLINK-35774
 URL: https://issues.apache.org/jira/browse/FLINK-35774
 Project: Flink
  Issue Type: Bug
  Components: Flink CDC
Affects Versions: cdc-3.1.1
Reporter: Wenkai Qi


The cache of transform is not updated after process schema change event.

For example, when add column event, tableInfo is not updated in TransformSchema 
and TransformData.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Potential Kafka Connector FLIP: Large Message Handling

[Kevin Lam]

Hi all,

Writing to see if the community would be open to exploring a FLIP for the
Kafka Table Connectors. The FLIP would allow for storing Kafka Messages
beyond a Kafka cluster's message limit (1 MB by default) out of band in
cloud object storage or another backend.

During serialization the message would be replaced with a reference, and
during deserialization the reference would be used to fetch the large
message and pass it to Flink. Something like Option 1 in this blog post

.

What do you think?

We can make it generic by allowing users to implement their own
LargeMessageSerializer/Deserializer interface for serializing and
deserializing and handling interactions with object storage or some other
backend.

The Kafka Connectors can be extended to support ConfigOptions to
specify the class to load, as well as some user-specified properties. For
example: `large-record-handling.class` and `
large-record-handling.properties.*` (where the user can specify any
properties similar to how the Kafka Consumer and Producer properties are
handled

).

In terms of call sites for the LargeMessage handling, I think we can
consider inside of DynamicKafkaDeserializationSchema

and DynamicKafkaRecordSerializationSchema
,
where the ConsumerRecord

and ProducerRecords

are passed respectively.

If there's interest, I would be happy to help flesh out the proposal more.

[jira] [Created] (FLINK-35773) Document s5cmd

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-35773:
--

 Summary: Document s5cmd
 Key: FLINK-35773
 URL: https://issues.apache.org/jira/browse/FLINK-35773
 Project: Flink
  Issue Type: Sub-task
Reporter: Piotr Nowojski






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35772) Deprecate/remove DuplicatingFileSystem

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-35772:
--

 Summary: Deprecate/remove DuplicatingFileSystem
 Key: FLINK-35772
 URL: https://issues.apache.org/jira/browse/FLINK-35772
 Project: Flink
  Issue Type: Sub-task
Reporter: Piotr Nowojski






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35771) Limit s5cmd resource usage

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-35771:
--

 Summary: Limit s5cmd resource usage
 Key: FLINK-35771
 URL: https://issues.apache.org/jira/browse/FLINK-35771
 Project: Flink
  Issue Type: Sub-task
Reporter: Piotr Nowojski






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35770) Interrupt s5cmd call on cancellation

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-35770:
--

 Summary: Interrupt s5cmd call on cancellation 
 Key: FLINK-35770
 URL: https://issues.apache.org/jira/browse/FLINK-35770
 Project: Flink
  Issue Type: Sub-task
Reporter: Piotr Nowojski






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35769) State files might not be deleted on task cancellation

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-35769:
-

 Summary: State files might not be deleted on task cancellation
 Key: FLINK-35769
 URL: https://issues.apache.org/jira/browse/FLINK-35769
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Affects Versions: 1.19.1
Reporter: Roman Khachatryan
Assignee: Roman Khachatryan
 Fix For: 1.20.0


We have a job in an infinite (fast) restart loop, that’s crashing with a 
serialization issue.
The issue here is that each restart seems to leak state files (not cleaning up 
ones from the previous run):

{{/tmp/tm_10.56.9.147:6122-c560c5/tmp $ ls | grep KeyedProcessOperator | wc -l
7990}}
{{/tmp/tm_10.56.9.147:6122-c560c5/tmp $ ls | grep StreamingJoinOperator | wc -l
689}}
Eventually TM will use too much disk space.

 

The problem is in 
[https://github.com/apache/flink/blob/64f745a5b1fc14a2cba1ddd977ab8e8db9cf45a4/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateDownloader.java#L75]
{code:java}
try {
            List> futures =
                    transferAllStateDataToDirectoryAsync(downloadRequests, 
internalCloser)
                            .collect(Collectors.toList());
            // Wait until either all futures completed successfully or one 
failed exceptionally.
            FutureUtils.completeAll(futures).get();
        } catch (Exception e) {
            downloadRequests.stream()
                    .map(StateHandleDownloadSpec::getDownloadDestination)
                    .map(Path::toFile)
                    .forEach(FileUtils::deleteDirectoryQuietly); {code}
Where {{FileUtils::deleteDirectoryQuietly}} will list the files and delete them.
But if {{completeAll}} is interrupted, then download runnable might re-create 
it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [2.0] How to handle on-going feature development in Flink 2.0?

[Matthias Pohl]

Thanks for the feedback. I went ahead and added a list for new features [1]
to the end of the 2.0 release page. That enables us to document such
changes.
I hope that's ok for the 2.0 release manager. Feel free to revert my
changes if you have anything else in mind.

Best,
Matthias

[1]
https://cwiki.apache.org/confluence/display/FLINK/2.0+Release#id-2.0Release-NewFeatures

On Wed, Jun 26, 2024 at 11:09 AM Zakelly Lan  wrote:

> +1 for a preview before the formal release. It would help us find issues in
> advance.
>
>
> Best,
> Zakelly
>
> On Wed, Jun 26, 2024 at 4:44 PM Jingsong Li 
> wrote:
>
> > +1 to release a preview version.
> >
> > Best,
> > Jingsong
> >
> > On Wed, Jun 26, 2024 at 10:12 AM Jark Wu  wrote:
> > >
> > > I also think this should not block new feature development.
> > > Having "nice-to-have" and "must-to-have" tags on the FLIPs is a good
> > idea.
> > >
> > > For the downstream projects, I think we need to release a 2.0 preview
> > > version one or
> > > two months before the formal release. This can leave some time for the
> > > downstream
> > > projects to integrate and provide feedback. So we can fix the problems
> > > (e.g. unexpected
> > > breaking changes, Java versions) before 2.0.
> > >
> > > Best,
> > > Jark
> > >
> > > On Wed, 26 Jun 2024 at 09:39, Xintong Song 
> > wrote:
> > >
> > > > I also don't think we should block new feature development until 2.0.
> > From
> > > > my understanding, the new major release is no different from the
> > regular
> > > > minor releases for new features.
> > > >
> > > > I think tracking new features, either as nice-to-have items or in a
> > > > separate list, is necessary. It helps us understand what's going on
> in
> > the
> > > > release cycle, and what to announce and promote. Maybe we should
> start
> > a
> > > > discussion on updating the 2.0 item list, to 1) collect new items
> that
> > are
> > > > proposed / initiated after the original list being created and 2) to
> > remove
> > > > some items that are no longer suitable. I'll discuss this with the
> > other
> > > > release managers first.
> > > >
> > > > For the connectors and operators, I think it depends on whether they
> > depend
> > > > on any deprecated APIs or internal implementations of Flink. Ideally,
> > > > all @Public APIs and @PublicEvolving APIs that we plan to change /
> > remove
> > > > should have been deprecated in 1.19 and 1.20 respectively. That means
> > if
> > > > the connectors and operators only use non-deprecated @Puclib
> > > > and @PublicEvolving APIs in 1.20, hopefully there should not be any
> > > > problems upgrading to 2.0.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Wed, Jun 26, 2024 at 5:20 AM Becket Qin 
> > wrote:
> > > >
> > > > > Thanks for the question, Matthias.
> > > > >
> > > > > My two cents, I don't think we are blocking new feature
> development.
> > My
> > > > > understanding is that the community will just prioritize removing
> > > > > deprecated APIs in the 2.0 dev cycle. Because of that, it is
> possible
> > > > that
> > > > > some new feature development may slow down a little bit since some
> > > > > contributors may be working on the must-have features for 2.0. But
> > policy
> > > > > wise, I don't see a reason to block the new feature development for
> > the
> > > > 2.0
> > > > > release feature plan[1].
> > > > >
> > > > > Process wise, I like your idea of adding the new features as
> > nice-to-have
> > > > > in the 2.0 feature list.
> > > > >
> > > > > Re: David,
> > > > > Given it is a major version bump. It is possible that some of the
> > > > > downstream projects (e.g. connectors, Paimon, etc) will have to see
> > if a
> > > > > major version bump is also needed there. And it is probably going
> to
> > be
> > > > > decisions made on a per-project basis.
> > > > > Regarding the Java version specifically, this probably worth a
> > separate
> > > > > discussion. According to a recent report[2] on the state of Java,
> it
> > > > might
> > > > > be a little early to drop support for Java 11. We can discuss this
> > > > > separately.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > > > > [2]
> > > > >
> > > > >
> > > >
> >
> https://newrelic.com/sites/default/files/2024-04/new-relic-state-of-the-java-ecosystem-report-2024-04-30.pdf
> > > > >
> > > > > On Tue, Jun 25, 2024 at 4:58 AM David Radley <
> > david_rad...@uk.ibm.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I think this is a great question. I am not sure if this has been
> > > > covered
> > > > > > elsewhere, but it would be good to be clear how this effects the
> > > > > connectors
> > > > > > and operator repos, with potentially v1 and v2 oriented new
> > featuresI
> > > > > > suspect this will be a connector by connector investigation. I am
> > > > > thinking
> > > > > > connectors with Hadoop eco-system 

[jira] [Created] (FLINK-35768) Use native file copy in RocksDBStateDownloader

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-35768:
--

 Summary: Use native file copy in RocksDBStateDownloader
 Key: FLINK-35768
 URL: https://issues.apache.org/jira/browse/FLINK-35768
 Project: Flink
  Issue Type: Sub-task
Reporter: Piotr Nowojski






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35767) Provide native file copy support for S3 using s5cmd

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-35767:
--

 Summary: Provide native file copy support for S3 using s5cmd
 Key: FLINK-35767
 URL: https://issues.apache.org/jira/browse/FLINK-35767
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / FileSystem
Reporter: Piotr Nowojski






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35766) When the job contains many YieldingOperatorFactory instances, compiling the JobGraph hangs

[Junrui Li (Jira)]

Junrui Li created FLINK-35766:
-

 Summary: When the job contains many YieldingOperatorFactory 
instances, compiling the JobGraph hangs
 Key: FLINK-35766
 URL: https://issues.apache.org/jira/browse/FLINK-35766
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Reporter: Junrui Li


When a job contains YieldingOperatorFactory instances, the time complexity of 
compiling the JobGraph is very high (with a complexity of O(N!)). This leads to 
the job compilation hanging on creating chains when there are many 
YieldingOperatorFactory instances (e.g., more than 30).

This is a very rare bug, but we have users who use SQL that contains many 
LookupJoins that use YieldingOperatorFactory in the production environment. A 
simple reproducible case is as follows:
{code:java}
@Test
void test() {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.createLocalEnvironment(1);
env.fromSource(
new NumberSequenceSource(0, 10), 
WatermarkStrategy.noWatermarks(), "input")
.map((x) -> x)
// add 32 YieldingOperatorFactory
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(
"test", BasicTypeInfo.LONG_TYPE_INFO, new 
YieldingTestOperatorFactory<>())
.transform(

[jira] [Created] (FLINK-35765) Support Fine Grained Resource Specifications for Adaptive Scheduler

[RocMarshal (Jira)]

RocMarshal created FLINK-35765:
--

 Summary: Support Fine Grained Resource Specifications for Adaptive 
Scheduler
 Key: FLINK-35765
 URL: https://issues.apache.org/jira/browse/FLINK-35765
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: RocMarshal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35764) TimerGauge is incorrect when update is called during a measurement

[Liu Liu (Jira)]

Liu Liu created FLINK-35764:
---

 Summary: TimerGauge is incorrect when update is called during a 
measurement
 Key: FLINK-35764
 URL: https://issues.apache.org/jira/browse/FLINK-35764
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics
Affects Versions: 1.19.0, 1.18.0, 1.17.0, 1.16.0, 1.15.0
Reporter: Liu Liu


Currently in {{{}TimerGauge{}}}, the {{currentMeasurement}} in {{markEnd}} is 
incorrectly set to the time since the last {{{}markStart{}}}. When calling 
{{{}markStart -> update -> markEnd{}}}, this will result in the time between 
{{markStart}} and {{update}} being counted twice. A piece of test code that 
reflects this scenario:
{code:java}
@Test  
void testUpdateBeforeMarkingEnd() {  
ManualClock clock = new ManualClock(42_000_000);
// time span = 2 intervals
TimerGauge gauge = new TimerGauge(clock, 2 * View.UPDATE_INTERVAL_SECONDS); 
 

// the event spans 2 intervals
// interval 1
gauge.markStart();  
clock.advanceTime(SLEEP, TimeUnit.MILLISECONDS);  
gauge.update();
// interval 2
clock.advanceTime(SLEEP, TimeUnit.MILLISECONDS);  
gauge.markEnd();  
gauge.update();  

// expected: 2, actual: 3
assertThat(gauge.getValue()).isEqualTo(SLEEP / 
View.UPDATE_INTERVAL_SECONDS);  
}
{code}
Proposed changes:
 # Modify {{markEnd}} so that updates to {{currentCount}} and 
{{accumulatedCount}} resembles those in {{{}update{}}}.
 # Add the test case to {{{}TimeGaugeTest{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35763) Bump Java version

[911432 (Jira)]

911432 created FLINK-35763:
--

 Summary: Bump Java version
 Key: FLINK-35763
 URL: https://issues.apache.org/jira/browse/FLINK-35763
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.20.0
Reporter: 911432


java8 is displayed as [document] 
(https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/java_compatibility/)
 is not available in flink1.15).


However, java8 is still available on flink 1.20 at the moment.
[#1] 
(https://github.com/apache/flink/blob/6b08eb15d09d6c8debb609b8b3a2a241ca855155/tools/azure-pipelines/build-apache-repo.yml#L75),
 )
[#2] 
(https://github.com/apache/flink/blob/6b08eb15d09d6c8debb609b8b3a2a241ca855155/tools/azure-pipelines/build-apache-repo.yml#L108),
 )
[#3] 
(https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release) )
I hope you can fix this part.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [DISCUSS] FLIP-466: Introduce ProcessFunction Attribute in DataStream API V2

[Yuxin Tan]

Hi, Wencong,
Thanks for driving the FLIP.

+1 for this FLIP.

I believe these hints will improve the performance in many use cases.
I only have a minor question about the Idempotence annotation. When
this annotation is added, how does StreamTask optimize the frequency?
Does it ensure a single output, or does it merely reduce the frequency
of the outputs?

Best,
Yuxin


Wencong Liu  于2024年7月1日周一 16:39写道：

> Hi, Jeyhun,
> Thanks for the reply.
>
>
> > Integrate these annotations with the execution plan.
> I believe DataStream is an Imperative API, which means
> that the actual execution plan is basically consistent with
> the computational logic expressed by the user with DataStream,
> and it is different from SQL, so the significance of supporting
> getExecutionPlan in the short term may not be great. If it is to
> be supported later, it is possible to consider the impact of Hints.
>
>
> > Check for misuse of attributes or ignore it.
> For illegal use (annotated on the inappropriate ProcessFunction),
> an exception will be thrown. For legal use, the framework can also
> choose to ignore it.
>
>
> > A framework to include attributes.
> Yes, we will provide a basic framework in the implementation
> to help developers for extension.
>
>
> Best,
> Wencong
>
>
> At 2024-06-28 02:06:37, "Jeyhun Karimov"  wrote:
> >Hi Wencong,
> >
> >Thanks for the FLIP. +1 for it.
> >
> >Providing hints to users will enable more optimization potential for DSv2.
> >I have a few questions.
> >
> >I think currently, DSv2 ExecutionEnvironment does not support getting
> >execution plan (getExecutionPlan()).
> >Do you plan to integrate these annotations with the execution plan?
> >
> >Any plans to check for misuse of attributes? Or any plans for a framework
> >to implicitly include attributes?
> >
> >Also, now that we make analogy with SQL hints, SQL query planners usually
> >ignore wrong hints and continue with its best plan.
> >Do we want to consider this approach? Or should we throw exception
> whenever
> >the hint (attribute in this case) is wrong?
> >
> >
> >Regards,
> >Jeyhun
> >
> >
> >On Thu, Jun 27, 2024 at 7:47 AM Xintong Song 
> wrote:
> >
> >> +1 for this FLIP.
> >>
> >> I think this is similar to SQL hints, where users can provide optional
> >> information to help the engine execute the workload more efficiently.
> >> Having a unified mechanism for such kind of hints should improve
> usability
> >> compared to introducing tons of configuration knobs.
> >>
> >> Best,
> >>
> >> Xintong
> >>
> >>
> >>
> >> On Wed, Jun 26, 2024 at 8:09 PM Wencong Liu 
> wrote:
> >>
> >> > Hi devs,
> >> >
> >> >
> >> > I'm proposing a new FLIP[1] to introduce the ProcessFunction
> Attribute in
> >> > the
> >> > DataStream API V2. The goal is to optimize job execution by leveraging
> >> > additional information about users' ProcessFunction logic. The
> proposal
> >> > includes
> >> > the following scenarios where the ProcessFunction Attribute can
> >> > significantly
> >> > enhance optimization:
> >> >
> >> >
> >> > Scenario 1: If the framework recognizes that the ProcessFunction
> outputs
> >> > data
> >> > only after all input is received, the downstream operators can be
> >> > scheduled until
> >> > the ProcessFunction is finished, which effectively reduces resource
> >> > consumption.
> >> > Ignoring this information could lead to premature scheduling of
> >> downstream
> >> > operators with no data to process. This scenario is addressed and
> >> > optimized by FLIP-331[2].
> >> >
> >> >
> >> > Scenario 2: For stream processing, where users are only interested in
> the
> >> > latest
> >> > result per key at the current time, the framework can optimize by
> >> > adjusting the
> >> > frequency of ProcessFunction outputs. This reduces shuffle data volume
> >> and
> >> > downstream operator workload. If this optimization is ignored, each
> new
> >> > input
> >> > would trigger a new output. This scenario is addressed and
> >> > optimized by FLIP-365[3].
> >> >
> >> >
> >> > Scenario 3: If a user's ProcessFunction neither caches inputs nor
> >> outputs,
> >> > recognizing this can enable object reuse for this data within the
> >> > OperatorChain,
> >> > enhancing performance. Without this optimization, data would be copied
> >> > before
> >> > being passed to the next operator. This scenario is addressed and
> >> > optimized by FLIP-329[4].
> >> >
> >> >
> >> > To unify the mechanism for utilizing additional information and
> >> optimizing
> >> > jobs,
> >> > we propose introducing the ProcessFunction Attribute represented by
> >> > Java annotations, which allow users to provide relevant information
> about
> >> > their
> >> > ProcessFunctions. The framework can then use this to optimize job
> >> > execution.
> >> > Importantly, regular job execution remains unaffected whether users
> use
> >> > this
> >> > attribute or not.
> >> >
> >> >
> >> > Looking forward to discussing this in the upcoming FLIP.
> >> >
> >> >
> >> > Best regards,
> >> > 

[jira] [Created] (FLINK-35762) Cache hashCode for immutable map key classes

[yux (Jira)]

yux created FLINK-35762:
---

 Summary: Cache hashCode for immutable map key classes
 Key: FLINK-35762
 URL: https://issues.apache.org/jira/browse/FLINK-35762
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Reporter: yux


As suggested by [~kunni], hash code caching is a common optimization in Java 
world (for example, java.lang.String uses such optimization to reduce duplicate 
hash calculation since String as a hashmap key is quite common). Such 
optimization could be applied in CDC to optimize execution performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35761) Speed up the restore process of unaligned checkpoint

[Rui Fan (Jira)]

Rui Fan created FLINK-35761:
---

 Summary: Speed up the restore process of unaligned checkpoint
 Key: FLINK-35761
 URL: https://issues.apache.org/jira/browse/FLINK-35761
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Affects Versions: 1.19.1, 1.20.0
Reporter: Rui Fan
Assignee: Rui Fan


Currently, the task will transition state from ExecutionState.INITIALIZING to 
ExecutionState.RUNNING after all input buffers are processed.

It will cause the restore time is very long if the performance is not strong 
and unaligned checkpoint snapshotted too many input buffers. From my 
experience, the restore time will excess 30 minutes when job with high 
parallelism.

We hope the job is switched to RUNNING asap. Because the new checkpoint is 
unable to be triggered during INITIALIZING. If the job is switched to RUNNING, 
the new unaligned checkpoint can be made.
h2. Brief Solution:
 # The task is switched to RUNNING after all input buffers are added to 
RecoveredInputChannel.
 ** In general, it's quick unless the network buffer isn't enough.
 # RecoveredInputChannel supports snapshot for network buffers

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35760) [docs] Add scan.newly-added-table.enabled to docs.

[Hongshun Wang (Jira)]

Hongshun Wang created FLINK-35760:
-

 Summary: [docs] Add scan.newly-added-table.enabled to docs.
 Key: FLINK-35760
 URL: https://issues.apache.org/jira/browse/FLINK-35760
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Affects Versions: cdc-3.1.1
Reporter: Hongshun Wang
 Fix For: cdc-3.2.0


scan.newly-added-table.enabled  has been exposed for a long time, but lacks of 
docs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35759) [docs] Add 'scan.incremental.snapshot.backfill.skip' to docs.

[Hongshun Wang (Jira)]

Hongshun Wang created FLINK-35759:
-

 Summary:  [docs] Add 'scan.incremental.snapshot.backfill.skip' to 
docs. 
 Key: FLINK-35759
 URL: https://issues.apache.org/jira/browse/FLINK-35759
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Affects Versions: cdc-3.1.1
Reporter: Hongshun Wang
 Fix For: cdc-3.2.0


scan.incremental.snapshot.backfill.skip has been exposed for a long time, but 
lack docs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] CLI action deprecation process

[Ferenc Csaky]

Hi Muhammet,

Thank you for your thoughts!

> After two minor releases, and on next major version bump,
> we could drop the `run-application` method as suggested
> on discussion by Xintong.

Here, you describe the deprecation/removal process of a public
API by the definition we have in the project now. So if the same
applies to a CLI action, why should we not enforce such behavior
for those as well?

If following the same process for CLIs make sense, we should also
enforce the same compatibility guarantees IMO.

Best,
Ferenc



On Friday, 28 June 2024 at 09:30, Muhammet Orazov 
 wrote:

> 
> 
> Hey Ferenc,
> 
> Thanks for starting the discussion!
> 
> I agree that the CLI is user facing, but I think we don't
> have to treat it as other public APIs.
> 
> I'd propose to throw user friendly exception for
> `run-application` with suggestion to use `run` case instead.
> This would make users aware of the change and require them
> to migrate their scripts.
> 
> After two minor releases, and on next major version bump,
> we could drop the `run-application` method as suggested
> on discussion by Xintong.
> 
> Best,
> Muhammet
> 
> 
> On 2024-06-26 15:33, Ferenc Csaky wrote:
> 
> > Hello devs,
> > I would like to open a discussion about considerations regarding how to
> > deprecate CLI
> > actions, and what compatibility guarantees should apply to such cases.
> > The topic came up in
> > a previous discussion [1] about a current FLIP to merge the run and
> > run-application
> > behavior [2].
> > 
> > According to Xintong's previous inputs, currently the Flink CLI, or its
> > actions are not handled
> > as public APIs by the existing definition (@Public or @PublicEvolving
> > annotated). So
> > legally it would be possible to change CLIs anytime. I agree with
> > Xintong that CLI actions
> > should be considered as public APIs, and as such, compatibility
> > guarantees should be
> > provided.
> > 
> > CLI actions are defined as private constants in CliFrontend [3], so
> > IMO the existing rules
> > are not perfectly applicable as is. Both @Public and @PublicEvolving
> > can be applied to
> > fields (although for @Public that is only true as per the javadoc,
> > technically it can only be
> > applied to classes), so I think that could be a good approach that is
> > achievable with minimal
> > effort and changes.
> > 
> > It would also be possible to define a different process, but IMO the
> > more lightweight and
> > maintainable the process the better, so it is a good thing if it is not
> > necessary to bring in
> > new entities and/or rules to check and enforce compatibility.
> > 
> > WDYT?
> > 
> > Best,
> > Ferenc
> > 
> > [1] https://lists.apache.org/thread/0vc8v3t7fr6w9hmwf9zbjbyk5c3bcj50
> > [2]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179
> > [3]
> > https://github.com/apache/flink/blob/27287a105f6585e89795e2a6cbffa8254abb6e57/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L98

Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION

[Ferenc Csaky]

+1 (non-binding)

Best,
Ferenc




On Thursday, 4 July 2024 at 16:20, Feng Jin  wrote:

> 
> 
> +1 (non-binding)
> 
> Best,
> Feng Jin
> 
> On Thu, Jul 4, 2024 at 6:02 PM Martijn Visser martijnvis...@apache.org
> 
> wrote:
> 
> > +1 (binding)
> > 
> > On Thu, Jul 4, 2024 at 5:39 AM Yanquan Lv decq12y...@gmail.com wrote:
> > 
> > > Hi Natea, thanks for driving it.
> > > +1 (non-binding).
> > > 
> > > Jim Hughes jhug...@confluent.io.invalid 于2024年7月4日周四 04:41写道：
> > > 
> > > > Hi Natea,
> > > > 
> > > > Looks good to me!
> > > > 
> > > > +1 (non-binding).
> > > > 
> > > > Cheers,
> > > > 
> > > > Jim
> > > > 
> > > > On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada
> > > > nbesh...@confluent.io.invalid wrote:
> > > > 
> > > > > Sorry I forgot to include the FLIP [1] and the mailing thread
> > > > > discussion
> > > > > link [2] in my previous email.
> > > > > 
> > > > > [1]
> > 
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION
> > 
> > > > > [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph
> > > > > 
> > > > > Thanks,
> > > > > Natea
> > > > > 
> > > > > On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada <
> > > > > nbesh...@confluent.io>
> > > > > wrote:
> > > > > 
> > > > > > Hello everyone,
> > > > > > 
> > > > > > I would like to start a vote on FLIP-465 [1]. It proposes adding
> > > > > > SQL
> > > > > > syntax that would allow users to describe the metadata of a given
> > > > > > function.
> > > > > > 
> > > > > > The vote will be open for at least 72 hours (Saturday, July 6th of
> > > > > > July
> > > > > > 2024,
> > > > > > 12:30 PST) unless there is an objection or insufficient votes.
> > > > > > 
> > > > > > Thanks,
> > > > > > Natea

Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION

[Feng Jin]

+1 (non-binding)

Best,
Feng Jin

On Thu, Jul 4, 2024 at 6:02 PM Martijn Visser 
wrote:

> +1 (binding)
>
> On Thu, Jul 4, 2024 at 5:39 AM Yanquan Lv  wrote:
>
> > Hi Natea, thanks for driving it.
> > +1 (non-binding).
> >
> > Jim Hughes  于2024年7月4日周四 04:41写道：
> >
> > > Hi Natea,
> > >
> > > Looks good to me!
> > >
> > > +1 (non-binding).
> > >
> > > Cheers,
> > >
> > > Jim
> > >
> > > On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada
> > >  wrote:
> > >
> > > > Sorry I forgot to include the FLIP [1] and the mailing thread
> > discussion
> > > > link [2] in my previous email.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION
> > > > [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph
> > > >
> > > > Thanks,
> > > > Natea
> > > >
> > > > On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada <
> > > > nbesh...@confluent.io>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I would like to start a vote on FLIP-465 [1]. It proposes adding
> SQL
> > > > > syntax that would allow users to describe the metadata of a given
> > > > function.
> > > > >
> > > > > The vote will be open for at least 72 hours (Saturday, July 6th of
> > July
> > > > > 2024,
> > > > > 12:30 PST) unless there is an objection or insufficient votes.
> > > > >
> > > > > Thanks,
> > > > > Natea
> > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-35758) [Doc] add scan.startup.timestamp-millis options to MySQL connector docs.

[LvYanquan (Jira)]

LvYanquan created FLINK-35758:
-

 Summary: [Doc] add scan.startup.timestamp-millis options to MySQL 
connector docs.
 Key: FLINK-35758
 URL: https://issues.apache.org/jira/browse/FLINK-35758
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Affects Versions: cdc-3.1.1
Reporter: LvYanquan
 Fix For: cdc-3.2.0


Startup reading position support to specify timestamp position, but this was 
not included in connector options. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION

[Martijn Visser]

+1 (binding)

On Thu, Jul 4, 2024 at 5:39 AM Yanquan Lv  wrote:

> Hi Natea, thanks for driving it.
> +1 (non-binding).
>
> Jim Hughes  于2024年7月4日周四 04:41写道：
>
> > Hi Natea,
> >
> > Looks good to me!
> >
> > +1 (non-binding).
> >
> > Cheers,
> >
> > Jim
> >
> > On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada
> >  wrote:
> >
> > > Sorry I forgot to include the FLIP [1] and the mailing thread
> discussion
> > > link [2] in my previous email.
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION
> > > [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph
> > >
> > > Thanks,
> > > Natea
> > >
> > > On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada <
> > > nbesh...@confluent.io>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I would like to start a vote on FLIP-465 [1]. It proposes adding SQL
> > > > syntax that would allow users to describe the metadata of a given
> > > function.
> > > >
> > > > The vote will be open for at least 72 hours (Saturday, July 6th of
> July
> > > > 2024,
> > > > 12:30 PST) unless there is an objection or insufficient votes.
> > > >
> > > > Thanks,
> > > > Natea
> > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-35757) support read new tables when scanNewlyTableAdded is enabled

[shikai.wang (Jira)]

shikai.wang created FLINK-35757:
---

 Summary: support read new tables when scanNewlyTableAdded is 
enabled 
 Key: FLINK-35757
 URL: https://issues.apache.org/jira/browse/FLINK-35757
 Project: Flink
  Issue Type: Improvement
  Components: Flink CDC
Reporter: shikai.wang
 Fix For: cdc-3.2.0


currently if we set tableList with regex like db.*, after snapshot finished, we 
can't 

capture tables when create new tables in db. instead,we need to restart to 
snapshot new table. this is very troublesome.

so we should support dynamically capture new table that match the regex when 
enable scanNewlyTableAdded.

https://github.com/apache/flink-cdc/discussions/3338



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35756) FileSystemTableSourceStreamingITCase.testSourceWithRegexPattern produced no output for 900 seconds

[Weijie Guo (Jira)]

Weijie Guo created FLINK-35756:
--

 Summary: 
FileSystemTableSourceStreamingITCase.testSourceWithRegexPattern produced no 
output for 900 seconds
 Key: FLINK-35756
 URL: https://issues.apache.org/jira/browse/FLINK-35756
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI
Affects Versions: 1.18.1
Reporter: Weijie Guo






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION

[Yanquan Lv]

Hi Natea, thanks for driving it.
+1 (non-binding).

Jim Hughes  于2024年7月4日周四 04:41写道：

> Hi Natea,
>
> Looks good to me!
>
> +1 (non-binding).
>
> Cheers,
>
> Jim
>
> On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada
>  wrote:
>
> > Sorry I forgot to include the FLIP [1] and the mailing thread discussion
> > link [2] in my previous email.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION
> > [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph
> >
> > Thanks,
> > Natea
> >
> > On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada <
> > nbesh...@confluent.io>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I would like to start a vote on FLIP-465 [1]. It proposes adding SQL
> > > syntax that would allow users to describe the metadata of a given
> > function.
> > >
> > > The vote will be open for at least 72 hours (Saturday, July 6th of July
> > > 2024,
> > > 12:30 PST) unless there is an objection or insufficient votes.
> > >
> > > Thanks,
> > > Natea
> > >
> > >
> >
>

Re: [VOTE] FLIP-456: CompiledPlan support for Batch Execution Mode

[Yuepeng Pan]

+1 (non-binding)

Best regards,

Yuepeng Pan





At 2024-07-03 01:46:13, "Sergey Nuyanzin"  wrote:
>Thanks for driving this
>
>+1 (binding)
>
>On Tue, Jul 2, 2024, 11:21 Martijn Visser  wrote:
>
>> +1 (binding)
>>
>> On Mon, Jul 1, 2024 at 7:00 PM Jim Hughes 
>> wrote:
>>
>> > Hi Alexey,
>> >
>> > +1 (non-binding)
>> >
>> > I'm looking forward to parity between streaming and batch bound for
>> > compiled plans!
>> >
>> > Cheers,
>> >
>> > Jim
>> >
>> > On Mon, Jul 1, 2024 at 12:55 PM Alexey Leonov-Vendrovskiy <
>> > vendrov...@gmail.com> wrote:
>> >
>> > > Hello everyone,
>> > >
>> > > We had a good discussion of FLIP-456: CompiledPlan support for Batch
>> > > Execution Mode [1]. Discussion thread is here: [2].
>> > >
>> > > Let's start voting on it. The vote will be open for at least 72
>> > > hours unless there is an objection or insufficient votes. The FLIP will
>> > be
>> > > considered accepted if 3 binding votes (from active committers
>> according
>> > to
>> > > the Flink bylaws [3]) are gathered by the community.
>> > >
>> > > Thanks,
>> > > Alexey
>> > >
>> > > [1]
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode
>> > > [2] https://lists.apache.org/thread/7gpyqvdnnbjwbh3vbk6b0pj38l91crvv
>> > > [3]
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals
>> > > <
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals)
>> > > >
>> > >
>> >
>>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

[Peter Huang]

Thanks for the effort, Guyla!


Best Regards
Peter Huang

On Wed, Jul 3, 2024 at 12:48 PM Őrhidi Mátyás 
wrote:

> Thank you, Gyula! 拾
> Cheers
> On Wed, Jul 3, 2024 at 8:00 AM Gyula Fóra  wrote:
>
> > The Apache Flink community is very happy to announce the release of
> Apache
> > Flink Kubernetes Operator 1.9.0.
> >
> > The Flink Kubernetes Operator allows users to manage their Apache Flink
> > applications and their lifecycle through native k8s tooling like kubectl.
> >
> > Release blogpost:
> >
> https://flink.apache.org/2024/07/02/apache-flink-kubernetes-operator-1.9.0-release-announcement/
> >
> > The release is available for download at:
> > https://flink.apache.org/downloads.html
> >
> > Maven artifacts for Flink Kubernetes Operator can be found at:
> >
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
> >
> > Official Docker image for Flink Kubernetes Operator can be found at:
> > https://hub.docker.com/r/apache/flink-kubernetes-operator
> >
> > The full release notes are available in Jira:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354417
> >
> > We would like to thank all contributors of the Apache Flink community who
> > made this release possible!
> >
> > Regards,
> > Gyula Fora
> >
>

Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION

[Jim Hughes]

Hi Natea,

Looks good to me!

+1 (non-binding).

Cheers,

Jim

On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada
 wrote:

> Sorry I forgot to include the FLIP [1] and the mailing thread discussion
> link [2] in my previous email.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION
> [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph
>
> Thanks,
> Natea
>
> On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada <
> nbesh...@confluent.io>
> wrote:
>
> > Hello everyone,
> >
> > I would like to start a vote on FLIP-465 [1]. It proposes adding SQL
> > syntax that would allow users to describe the metadata of a given
> function.
> >
> > The vote will be open for at least 72 hours (Saturday, July 6th of July
> > 2024,
> > 12:30 PST) unless there is an objection or insufficient votes.
> >
> > Thanks,
> > Natea
> >
> >
>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

[Őrhidi Mátyás]

Thank you, Gyula! 拾
Cheers
On Wed, Jul 3, 2024 at 8:00 AM Gyula Fóra  wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink Kubernetes Operator 1.9.0.
>
> The Flink Kubernetes Operator allows users to manage their Apache Flink
> applications and their lifecycle through native k8s tooling like kubectl.
>
> Release blogpost:
> https://flink.apache.org/2024/07/02/apache-flink-kubernetes-operator-1.9.0-release-announcement/
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Kubernetes Operator can be found at:
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>
> Official Docker image for Flink Kubernetes Operator can be found at:
> https://hub.docker.com/r/apache/flink-kubernetes-operator
>
> The full release notes are available in Jira:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354417
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
> Gyula Fora
>

Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION

[Natea Eshetu Beshada]

Sorry I forgot to include the FLIP [1] and the mailing thread discussion
link [2] in my previous email.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION
[2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph

Thanks,
Natea

On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada 
wrote:

> Hello everyone,
>
> I would like to start a vote on FLIP-465 [1]. It proposes adding SQL
> syntax that would allow users to describe the metadata of a given function.
>
> The vote will be open for at least 72 hours (Saturday, July 6th of July
> 2024,
> 12:30 PST) unless there is an objection or insufficient votes.
>
> Thanks,
> Natea
>
>

[VOTE] FLIP-465: Introduce DESCRIBE FUNCTION

[Natea Eshetu Beshada]

Hello everyone,

I would like to start a vote on FLIP-465 [1]. It proposes adding SQL syntax
that would allow users to describe the metadata of a given function.

The vote will be open for at least 72 hours (Saturday, July 6th of July
2024,
12:30 PST) unless there is an objection or insufficient votes.

Thanks,
Natea

[ANNOUNCE] Apache Flink Kubernetes Operator 1.9.0 released

[Gyula Fóra]

The Apache Flink community is very happy to announce the release of Apache
Flink Kubernetes Operator 1.9.0.

The Flink Kubernetes Operator allows users to manage their Apache Flink
applications and their lifecycle through native k8s tooling like kubectl.

Release blogpost:
https://flink.apache.org/2024/07/02/apache-flink-kubernetes-operator-1.9.0-release-announcement/

The release is available for download at:
https://flink.apache.org/downloads.html

Maven artifacts for Flink Kubernetes Operator can be found at:
https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator

Official Docker image for Flink Kubernetes Operator can be found at:
https://hub.docker.com/r/apache/flink-kubernetes-operator

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12354417

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Regards,
Gyula Fora

[jira] [Created] (FLINK-35755) Upgrade flink-shaded to 19.0

[Dawid Wysakowicz (Jira)]

Dawid Wysakowicz created FLINK-35755:


 Summary: Upgrade flink-shaded to 19.0
 Key: FLINK-35755
 URL: https://issues.apache.org/jira/browse/FLINK-35755
 Project: Flink
  Issue Type: Sub-task
Reporter: Dawid Wysakowicz
Assignee: Dawid Wysakowicz
 Fix For: 1.20.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35754) SqlGatewayE2ECase.testMaterializedTableInFullMode failed due to Internal Server Error

[Weijie Guo (Jira)]

Weijie Guo created FLINK-35754:
--

 Summary: SqlGatewayE2ECase.testMaterializedTableInFullMode failed 
due to Internal Server Error
 Key: FLINK-35754
 URL: https://issues.apache.org/jira/browse/FLINK-35754
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.20.0
Reporter: Weijie Guo






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35753) ParquetColumnarRowInputFormatTest.testContinuousRepetition(int) failed due to ArrayIndexOutOfBoundsException

[Weijie Guo (Jira)]

Weijie Guo created FLINK-35753:
--

 Summary: 
ParquetColumnarRowInputFormatTest.testContinuousRepetition(int) failed due to 
ArrayIndexOutOfBoundsException
 Key: FLINK-35753
 URL: https://issues.apache.org/jira/browse/FLINK-35753
 Project: Flink
  Issue Type: Bug
  Components: Build System / CI
Affects Versions: 1.20.0
Reporter: Weijie Guo






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35752) Migrate PythonUtil

[Jacky Lau (Jira)]

Jacky Lau created FLINK-35752:
-

 Summary: Migrate PythonUtil
 Key: FLINK-35752
 URL: https://issues.apache.org/jira/browse/FLINK-35752
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Affects Versions: 2.0.0
Reporter: Jacky Lau
 Fix For: 2.0.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35751) Migrate SplitPythonConditionFromCorrelateRule

[Jacky Lau (Jira)]

Jacky Lau created FLINK-35751:
-

 Summary: Migrate SplitPythonConditionFromCorrelateRule
 Key: FLINK-35751
 URL: https://issues.apache.org/jira/browse/FLINK-35751
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Affects Versions: 2.0.0
Reporter: Jacky Lau
 Fix For: 2.0.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] Release flink-shaded 19.0, release candidate #1

[Dawid Wysakowicz]

I'm happy to announce that we have unanimously approved this release.

There are 6 approving votes, 4 of which are binding:

* Timo Walther (binding)
* Martijn Visser (binding)
* Weijie Guo (binding)
* Dawid Wysakowicz (binding)
* Sergey Nuyanzin (non-binding)
* Yuxin Tan (non-binding)

There are no disapproving votes.

Thanks everyone!
Dawid

Re: [VOTE] Release flink-shaded 19.0, release candidate #1

[Dawid Wysakowicz]

+1 (binding)

Thanks all for your votes. I'll conclude the vote in a separate email

On Tue, 2 Jul 2024 at 10:39, Yuxin Tan  wrote:

> +1 (non-binding)
>
> - build from source
> - verified website pr
> - verified hash and signature
>
> Best,
> Yuxin
>
>
> weijie guo  于2024年7月2日周二 15:27写道：
>
> > +1 (binding)
> >
> > - Verified hash and signature
> > - Build from source
> > - Verified website PR
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Martijn Visser  于2024年7月1日周一 21:30写道：
> >
> > > +1 (binding)
> > >
> > > - Validated hashes
> > > - Verified signature
> > > - Verified that no binaries exist in the source archive
> > > - Build the source with Maven
> > > - Verified licenses
> > > - Verified web PRs
> > >
> > > On Fri, Jun 28, 2024 at 2:02 PM Timo Walther 
> wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Thanks for fixing the JSON functions!
> > > >
> > > > Timo
> > > >
> > > > On 28.06.24 12:54, Dawid Wysakowicz wrote:
> > > > > Hi everyone,
> > > > > Please review and vote on the release candidate 1 for the version
> > 19.0,
> > > > as
> > > > > follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > >
> > > > > The complete staging area is available for your review, which
> > includes:
> > > > > * JIRA release notes [1],
> > > > > * the official Apache source release to be deployed to
> > dist.apache.org
> > > > [2],
> > > > > which are signed with the key with fingerprint
> > > > > EA93A435B4E2C9B4C9F533F631D2DD10BFC15A2D [3],
> > > > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > > > * source code tag release-19.0-rc1 [5],
> > > > > * website pull request listing the new release [6].
> > > > >
> > > > > The vote will be open for at least 72 hours. It is adopted by
> > majority
> > > > > approval, with at least 3 PMC affirmative votes.
> > > > >
> > > > > Thanks,
> > > > > Dawid
> > > > >
> > > > > [1]
> https://issues.apache.org/jira/projects/FLINK/versions/12353853
> > > > > [2]
> > https://dist.apache.org/repos/dist/dev/flink/flink-shaded-19.0-rc1
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > [4]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1743/
> > > > > [5]
> > > https://github.com/apache/flink-shaded/releases/tag/release-19.0-rc1
> > > > > [6] https://github.com/apache/flink-web/pull/749
> > > > >
> > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-35750) The latency marker metrics aren't updated after failover

[RocMarshal (Jira)]

RocMarshal created FLINK-35750:
--

 Summary: The latency marker metrics aren't updated after failover
 Key: FLINK-35750
 URL: https://issues.apache.org/jira/browse/FLINK-35750
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Metrics
Affects Versions: 1.19.1, 1.18.1, 1.20.0
Reporter: RocMarshal


Described in 
https://docs.google.com/document/d/1WbAgdj8NdrSVChUuEywPgWZo3bAc3JKeVsN27AWDiL4/edit?usp=sharing






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-456: CompiledPlan support for Batch Execution Mode

[Timo Walther]

+1 (binding)

Thanks,
Timo


On 02.07.24 19:46, Sergey Nuyanzin wrote:

Thanks for driving this

+1 (binding)

On Tue, Jul 2, 2024, 11:21 Martijn Visser  wrote:


+1 (binding)

On Mon, Jul 1, 2024 at 7:00 PM Jim Hughes 
wrote:


Hi Alexey,

+1 (non-binding)

I'm looking forward to parity between streaming and batch bound for
compiled plans!

Cheers,

Jim

On Mon, Jul 1, 2024 at 12:55 PM Alexey Leonov-Vendrovskiy <
vendrov...@gmail.com> wrote:


Hello everyone,

We had a good discussion of FLIP-456: CompiledPlan support for Batch
Execution Mode [1]. Discussion thread is here: [2].

Let's start voting on it. The vote will be open for at least 72
hours unless there is an objection or insufficient votes. The FLIP will

be

considered accepted if 3 binding votes (from active committers

according

to

the Flink bylaws [3]) are gathered by the community.

Thanks,
Alexey

[1]





https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode

[2] https://lists.apache.org/thread/7gpyqvdnnbjwbh3vbk6b0pj38l91crvv
[3]





https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals

<




https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals)

[jira] [Created] (FLINK-35749) Kafka sink component will lose data when kafka cluster is unavailable for a while

[Jimmy Zhao (Jira)]

Jimmy Zhao created FLINK-35749:
--

 Summary: Kafka sink component will lose data when kafka cluster is 
unavailable for a while
 Key: FLINK-35749
 URL: https://issues.apache.org/jira/browse/FLINK-35749
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.17.1, 1.16.2, 1.20.0
Reporter: Jimmy Zhao


As the title described, here is the procedure to reproduce the problem:
1. develop a simple flink stream job to consume from one kafka topic and sink 
to anthoer kafka sever and topic
2. make amount of kafka message and produce to the source kafka topic, record 
the message number
3. start the flink stream job, and config to cosume from earliest source topic 
offset
4. during the job cosuming the source topic, restart the kafka cluster(we use 
aws MSK)
5. the flink job will not throw any Exception like nothing happened, but only 
print error log like : [kafka-producer-network-thread | producer-2] INFO  
org.apache.kafka.clients.NetworkClient  - [Producer clientId=producer-2] Node 2 
disconnected.
6. wait for the kafka cluster finished restarting and all the source kafka 
message consumed
7. count the target kafka topic message number, compare to the source, there is 
a high probability of data loss（more than 50%）



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35748) DeduplicateITCase.testLastRowWithoutAllChangelogOnRowtime with MiniBatch mode and RocksDB backend enabled

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-35748:
-

 Summary: DeduplicateITCase.testLastRowWithoutAllChangelogOnRowtime 
with MiniBatch mode and RocksDB backend enabled
 Key: FLINK-35748
 URL: https://issues.apache.org/jira/browse/FLINK-35748
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.20.0
Reporter: Matthias Pohl


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60613=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12259

{code}
Jul 02 14:44:36 14:44:36.737 [ERROR] Tests run: 40, Failures: 1, Errors: 0, 
Skipped: 4, Time elapsed: 18.45 s <<< FAILURE! -- in 
org.apache.flink.table.planner.runtime.stream.sql.DeduplicateITCase
Jul 02 14:44:36 14:44:36.737 [ERROR] 
org.apache.flink.table.planner.runtime.stream.sql.DeduplicateITCase.testLastRowWithoutAllChangelogOnRowtime
 -- Time elapsed: 0.860 s <<< FAILURE!
Jul 02 14:44:36 org.opentest4j.AssertionFailedError: 
Jul 02 14:44:36 
Jul 02 14:44:36 expected: List(+I(1,1,Hi,1970-01-01T00:00:00.001), +I(1,2,Hello 
world,1970-01-01T00:00:00.002), +I(2,3,I am fine.,1970-01-01T00:00:00.003), 
+I(2,6,Comment#1,1970-01-01T00:00:00.006), 
+I(3,5,Comment#2,1970-01-01T00:00:00.005), 
+I(4,4,Comment#3,1970-01-01T00:00:00.004))
Jul 02 14:44:36  but was: ArrayBuffer(+I(1,1,Hi,1970-01-01T00:00:00.001), 
+I(1,2,Hello world,1970-01-01T00:00:00.002), 
+I(1,3,Hello,1970-01-01T00:00:00.003), 
+I(2,6,Comment#1,1970-01-01T00:00:00.006), 
+I(3,5,Comment#2,1970-01-01T00:00:00.005), 
+I(4,4,Comment#3,1970-01-01T00:00:00.004), +U(2,3,I am 
fine.,1970-01-01T00:00:00.003), -U(1,3,Hello,1970-01-01T00:00:00.003))
Jul 02 14:44:36 at 
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jul 02 14:44:36 at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Jul 02 14:44:36 at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jul 02 14:44:36 at 
org.apache.flink.table.planner.runtime.stream.sql.DeduplicateITCase.testLastRowWithoutAllChangelogOnRowtime(DeduplicateITCase.scala:364)
Jul 02 14:44:36 at java.lang.reflect.Method.invoke(Method.java:498)
[...]
{code}

The test failure appeared in a CI run for FLINK-35553. Which does some changes 
to how checkpointing is triggered. I checked the logs and couldn't find any 
evidence that the test run included the FLINK-35553 change (no restoring from 
checkpoint happens in the failed and successful of the test; see attached logs).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35747) customer ‘rest.bind-address' config overwrite by code

[dncba (Jira)]

dncba created FLINK-35747:
-

 Summary:  customer ‘rest.bind-address' config overwrite by code
 Key: FLINK-35747
 URL: https://issues.apache.org/jira/browse/FLINK-35747
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.19.1
Reporter: dncba


When I want flink on Yarn webui  bind on 0.0.0.0 to listen Ipv4 & Ipv6 double 
stack, I found the   ‘rest.bind-address'  config will auto overwrite by here
{code:java}
package org.apache.flink.yarn.entrypoint;



public class YarnEntrypointUtils {


public static Configuration loadConfiguration(

final Configuration configuration =
GlobalConfiguration.loadConfiguration(workingDirectory, 
dynamicParameters);

        final String hostname 
=env.get(ApplicationConstants.Environment.NM_HOST.key());
       configuration.set(JobManagerOptions.ADDRESS, hostname);
        configuration.set(RestOptions.ADDRESS, hostname);

   # overwrite hostname by code
        configuration.set(RestOptions.BIND_ADDRESS, hostname);

`
}
}
{code}
In most case the are right.  when user want config the ‘rest.bind-address' by 
slef , the customer config will be auto overwirte.

 

the best way is check the user config before the ovewrite. like this

 
{code:java}


public class YarnEntrypointUtils {


public static Configuration loadConfiguration(

final Configuration configuration =
GlobalConfiguration.loadConfiguration(workingDirectory, 
dynamicParameters);


        final String hostname 
=env.get(ApplicationConstants.Environment.NM_HOST.key());
       configuration.set(JobManagerOptions.ADDRESS, hostname);
        configuration.set(RestOptions.ADDRESS, hostname);

   # check before the overwrite
String bindAddress = configuration.getString(RestOptions.BIND_ADDRESS);
if (StringUtils.isBlank(bindAddress)) {
configuration.setString(RestOptions.BIND_ADDRESS, hostname);
}
`
}
}

{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35746) Add logic to override observed config based on settings observed through REST API

[Gyula Fora (Jira)]

Gyula Fora created FLINK-35746:
--

 Summary: Add logic to override observed config based on settings 
observed through REST API
 Key: FLINK-35746
 URL: https://issues.apache.org/jira/browse/FLINK-35746
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora
 Fix For: kubernetes-operator-1.10.0


In many cases the Flink operator relies on the user configuration to infer 
running application/cluster settings. While this is mostly fine, many of these 
configs can be programmatically changed by the user in their app main method 
that will ultimately take precedence and can lead to inconsistent behaviour 
from the operators side.

To alleviate this we need to add logic to override parts of the observed config 
based on what we see from the cluster. Such as checkpointing enabled / 
disabled, checkpoint intervals etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35745) Add namespace convention doc for Flink lineage

[Zhenqiu Huang (Jira)]

Zhenqiu Huang created FLINK-35745:
-

 Summary: Add namespace convention doc for Flink lineage 
 Key: FLINK-35745
 URL: https://issues.apache.org/jira/browse/FLINK-35745
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Zhenqiu Huang


We will recommend to follow the convention from openlineage.
https://openlineage.io/docs/spec/naming/



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] FLIP-467: Introduce Generalized Watermarks

[Jeyhun Karimov]

Hi devs,

I'd like to start a discussion about FLIP-467: Introduce Generalized
Watermarks [1] .
This is another sub-FLIP of DataStream API V2 [2].


After this FLIP one can declare generalized (custom) watermarks and define
their custom propagation and alignment process. This FLIP opens new
prospects to simplify "signal"ing mechanisms inside the Flink runtime and
at the same time reveals new use-cases.

You can find more details in the FLIP [1].
Looking forward to hearing your comments, thanks!


Best regards,
Jeyhun

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-467%3A+Introduce+Generalized+Watermarks
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-408%3A+%5BUmbrella%5D+Introduce+DataStream+API+V2

Re: [VOTE] FLIP-456: CompiledPlan support for Batch Execution Mode

[Sergey Nuyanzin]

Thanks for driving this

+1 (binding)

On Tue, Jul 2, 2024, 11:21 Martijn Visser  wrote:

> +1 (binding)
>
> On Mon, Jul 1, 2024 at 7:00 PM Jim Hughes 
> wrote:
>
> > Hi Alexey,
> >
> > +1 (non-binding)
> >
> > I'm looking forward to parity between streaming and batch bound for
> > compiled plans!
> >
> > Cheers,
> >
> > Jim
> >
> > On Mon, Jul 1, 2024 at 12:55 PM Alexey Leonov-Vendrovskiy <
> > vendrov...@gmail.com> wrote:
> >
> > > Hello everyone,
> > >
> > > We had a good discussion of FLIP-456: CompiledPlan support for Batch
> > > Execution Mode [1]. Discussion thread is here: [2].
> > >
> > > Let's start voting on it. The vote will be open for at least 72
> > > hours unless there is an objection or insufficient votes. The FLIP will
> > be
> > > considered accepted if 3 binding votes (from active committers
> according
> > to
> > > the Flink bylaws [3]) are gathered by the community.
> > >
> > > Thanks,
> > > Alexey
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode
> > > [2] https://lists.apache.org/thread/7gpyqvdnnbjwbh3vbk6b0pj38l91crvv
> > > [3]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals)
> > > >
> > >
> >
>