Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink
+1 (non-binding) On 2024/01/14 02:26:13 Yun Gao wrote: > +1 (binding) > > On Sat, Jan 13, 2024 at 10:08 AM Rodrigo Meneses wrote: > > > > +1 (non binding) > > > > On Fri, Jan 12, 2024 at 5:45 PM Dong Lin wrote: > > > > > +1 (binding) > > > > > > On Sat, Jan 13, 2024 at 6:04 AM Austin Bennett wrote: > > > > > > > +1 (non-binding) > > > > > > > > On Fri, Jan 12, 2024 at 5:44 PM Becket Qin wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > Thanks, > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > On Fri, Jan 12, 2024 at 5:58 AM Zhijiang > > > > .invalid> > > > > > wrote: > > > > > > > > > > > +1 (binding) > > > > > > Best, > > > > > > Zhijiang > > > > > > -- > > > > > > From:Kurt Yang > > > > > > Send Time:2024年1月12日(星期五) 15:31 > > > > > > To:dev > > > > > > Subject:Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink > > > > > > +1 (binding) > > > > > > Best, > > > > > > Kurt > > > > > > On Fri, Jan 12, 2024 at 2:21 PM Hequn Cheng > > > wrote: > > > > > > > +1 (binding) > > > > > > > > > > > > > > Thanks, > > > > > > > Hequn > > > > > > > > > > > > > > On Fri, Jan 12, 2024 at 2:19 PM godfrey he > > > > > wrote: > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Godfrey > > > > > > > > > > > > > > > > Zhu Zhu 于2024年1月12日周五 14:10写道： > > > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Zhu > > > > > > > > > > > > > > > > > > Hangxiang Yu 于2024年1月11日周四 14:26写道： > > > > > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > > > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su < > > > > > suxuanna...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Xuannan > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang < > > > xyzhong...@163.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > +1 (non-binding)-- > > > > > > > > > > > > > > > > > > > > > > > > Best！ > > > > > > > > > > > > Xuyang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 在 2024-01-11 10:00:11，"Yang Wang" < > > > wangyang0...@apache.org > > > > > > > > > > > 写道： > > > > > > > > > > > > >+1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Best, > > > > > > > > > > > > >Yang > > > > > > > > > > > > > > > > > > > > > > > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron < > > > > ron9@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> +1 non-binding > > > > > > > > > > > > >> > > > > > > > > > > > > >> Best > > > > > > > > > > > > >> Ron > > > > > > > > > > > > >> > > > > > > > > > > > > >> Matthias Pohl > > > > > > 于2024年1月10日周三 > > > > > > > > > > 23:05写道： > > > > > > > > > > > > >> > > > > > > > > > > > > >> > +1 (binding) > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam < > > > > > > > > jam.gz...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > +1 non-binding > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Dawid Wysakowicz > > > > > 于2024年1月10日周三 > > > > > > > > > > 21:06写道： > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > +1 (binding) > > > > > > > > > > > > >> > > > Best, > > > > > > > > > > > > >> > > > Dawid > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr Nowojski < > > > > > > > > > > > pnowoj...@apache.org> > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > +1 (binding) > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser < > > > > > > > > > > > martijnvis...@apache.org> > > > > > > > > > > > > >> > > > > napisał(a): > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > +1 (binding) > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo > > > Huang < > > > > > > > > > > > hxbks...@gmail.com > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > +1 (binding) > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > Best, > > > > > > > > > > > > >> > > > > > > Xingbo > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > Dian Fu > > > > 于2024年1月10日周三 > > > > > > > > 11:35写道： > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > +1 (binding) > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > Regards, > > > > > > > > >
[jira] [Created] (FLINK-34072) Use JAVA_RUN in shell scripts
Yun Tang created FLINK-34072:
Summary: Use JAVA_RUN in shell scripts
Key: FLINK-34072
URL: https://issues.apache.org/jira/browse/FLINK-34072
Project: Flink
Issue Type: Improvement
Components: Deployment / Scripts
Reporter: Yun Tang
Fix For: 1.19.0
We should call {{JAVA_RUN}} in all cases when we launch {{java}} command,
otherwise we might be able to run the {{java}} if JAVA_HOME is not set.
such as:
{code:java}
flink-1.19-SNAPSHOT-bin/flink-1.19-SNAPSHOT/bin/config.sh: line 339: > 17 :
syntax error: operand expected (error token is "> 17 ")
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34073) Remove old release candidates from dist.apache.org
Jing Ge created FLINK-34073:
---
Summary: Remove old release candidates from dist.apache.org
Key: FLINK-34073
URL: https://issues.apache.org/jira/browse/FLINK-34073
Project: Flink
Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Jing Ge
h3. Remove old release candidates from [dist.apache.org|http://dist.apache.org/]
Remove the old release candidates from
[https://dist.apache.org/repos/dist/dev/flink] using Subversion.
{code:java}
$ svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates
$ cd flink
$ svn remove flink-${RELEASE_VERSION}-rc*
$ svn commit -m "Remove old release candidates for Apache Flink
${RELEASE_VERSION}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34074) Verify artifacts related expectations
Jing Ge created FLINK-34074: --- Summary: Verify artifacts related expectations Key: FLINK-34074 URL: https://issues.apache.org/jira/browse/FLINK-34074 Project: Flink Issue Type: Sub-task Reporter: Jing Ge h3. Expectations * Maven artifacts released and indexed in the [Maven Central Repository|https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.flink%22] (usually takes about a day to show up) * Source & binary distributions available in the release repository of [https://dist.apache.org/repos/dist/release/flink/] * Dev repository [https://dist.apache.org/repos/dist/dev/flink/] is empty * Website contains links to new release binaries and sources in download page * (for minor version updates) the front page references the correct new major release version and directs to the correct link -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34075) Mark version as released in Jira (need PMC role)
Jing Ge created FLINK-34075: --- Summary: Mark version as released in Jira (need PMC role) Key: FLINK-34075 URL: https://issues.apache.org/jira/browse/FLINK-34075 Project: Flink Issue Type: Sub-task Reporter: Jing Ge In JIRA, inside [version management|https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions], hover over the current release and a settings menu will appear. Click Release, and select today’s date. (Note: Only PMC members have access to the project administration. If you do not have access, ask on the mailing list for assistance.) h3. Expectations * Release tagged in the source code repository * Release version finalized in JIRA. (Note: Not all committers have administrator access to JIRA. If you end up getting permissions errors ask on the mailing list for assistance) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
Khanh Vu created FLINK-34076:
Summary: flink-connector-base missing fails kinesis table sink to
create
Key: FLINK-34076
URL: https://issues.apache.org/jira/browse/FLINK-34076
Project: Flink
Issue Type: Bug
Components: Connectors / Kinesis
Affects Versions: aws-connector-4.2.0
Reporter: Khanh Vu
The
[commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
which stops bundling `flink-connector-base` with `flink-connector-kinesis` has
caused kinesis sink failing to create when using Table API as required classes
from `flink-connector-base` are not loaded in execution.
E.g. with following depenency only in pom.xml
```
{{Caused by: org.apache.flink.table.api.ValidationException: Connector
'kinesis' can only be used as a source. It cannot be used as a sink.
at
org.apache.flink.table.factories.FactoryUtil.enrichNoMatchingConnectorError(FactoryUtil.java:756)
at
org.apache.flink.table.factories.FactoryUtil.discoverTableFactory(FactoryUtil.java:710)
at
org.apache.flink.table.factories.FactoryUtil.createDynamicTableSink(FactoryUtil.java:265)
... 20 more}}
```
following exception will be thrown:
```
{{Caused by: org.apache.flink.table.api.ValidationException: Connector
'kinesis' can only be used as a source. It cannot be used as a sink.
at
org.apache.flink.table.factories.FactoryUtil.enrichNoMatchingConnectorError(FactoryUtil.java:756)
at
org.apache.flink.table.factories.FactoryUtil.discoverTableFactory(FactoryUtil.java:710)
at
org.apache.flink.table.factories.FactoryUtil.createDynamicTableSink(FactoryUtil.java:265)
... 20 more}}
```
Workaround is to explicitly specify `flink-connector-base` as dependency of the
project:
```
{{
org.apache.flink
flink-connector-kinesis
${flink.connector.kinesis.version}
org.apache.flink
flink-connector-base
${flink.version}
provided
}}
```
In general, `flink-connector-base` should be pulled in by default when pulling
in the connector, the current separation adds unnecessary hassle to use the
connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY
+1 to removing LEGACY mode in Flink 2.0. Thanks for driving. Danny, On Sat, 13 Jan 2024, 08:20 Yanfei Lei, wrote: > Thanks Zakelly for starting this discussion. > > Regardless of whether it is for users or developers, deprecating > RestoreMode#LEGACY makes the semantics clearer and lower maintenance > costs, and Flink 2.0 is a good time point to do this. > So +1 for the overall idea. > > Best, > Yanfei > > Zakelly Lan 于2024年1月11日周四 14:57写道： > > > > > Hi devs, > > > > I'd like to start a discussion on FLIP-416: Deprecate and remove the > > RestoreMode#LEGACY[1]. > > > > The FLIP-193[2] introduced two modes of state file ownership during > > checkpoint restoration: RestoreMode#CLAIM and RestoreMode#NO_CLAIM. The > > LEGACY mode, which was how Flink worked until 1.15, has been superseded > by > > NO_CLAIM as the default mode. The main drawback of LEGACY mode is that > the > > new job relies on artifacts from the old job without cleaning them up, > > leaving users uncertain about when it is safe to delete the old > checkpoint > > directories. This leads to the accumulation of unnecessary checkpoint > files > > that are never cleaned up. Considering cluster availability and job > > maintenance, it is not recommended to use LEGACY mode. Users could choose > > the other two modes to get a clear semantic for the state file ownership. > > > > This FLIP proposes to deprecate the LEGACY mode and remove it completely > in > > the upcoming Flink 2.0. This will make the semantic clear as well as > > eliminate many bugs caused by mode transitions involving LEGACY mode > (e.g. > > FLINK-27114 [3]) and enhance code maintainability. > > > > Looking forward to hearing from you! > > > > [1] https://cwiki.apache.org/confluence/x/ookkEQ > > [2] https://cwiki.apache.org/confluence/x/bIyqCw > > [3] https://issues.apache.org/jira/browse/FLINK-27114 > > > > Best, > > Zakelly >
Permission to add FLIP document
Dear Flink community, I would like to create a FLIP document. I assume, for that I need to have some permissions (e.g., edit permission for Apache Flink cwiki space). Could you please grant me the required permissions? My cwiki confluence username is jeyhun.karimov Regards, Jeyhun
[jira] [Created] (FLINK-34077) Sphinx version needs upgrade
Yunfeng Zhou created FLINK-34077:
Summary: Sphinx version needs upgrade
Key: FLINK-34077
URL: https://issues.apache.org/jira/browse/FLINK-34077
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.19.0
Reporter: Yunfeng Zhou
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56357&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=c67e71ed-6451-5d26-8920-5a8cf9651901]
{code:java}
Jan 14 15:49:17 /__w/2/s/flink-python/dev/.conda/bin/sphinx-build -b html -d
_build/doctrees -a -W . _build/htmlJan 14 15:49:17 Running Sphinx v4.5.0
Jan 14 15:49:17Jan 14 15:49:17 Sphinx version error:Jan 14 15:49:17 The
sphinxcontrib.applehelp extension used by this project needs at least Sphinx
v5.0; it therefore cannot be built with this version.Jan 14 15:49:17
Makefile:76: recipe for target 'html' failedJan 14 15:49:17 make: ***
[html] Error 2Jan 14 15:49:18 ==sphinx checks...
[FAILED]=== {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Permission to add FLIP document
Hey, Jeyhun I’ve added the permissions for your account. Best, Leonard > 2024年1月15日 上午3:42，Jeyhun Karimov 写道： > > Dear Flink community, > > I would like to create a FLIP document. I assume, for that I need to have > some permissions (e.g., edit permission for Apache Flink cwiki space). > Could you please grant me the required permissions? > My cwiki confluence username is jeyhun.karimov > > Regards, > Jeyhun
Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY
Hi, Zakelly. Thanks for the quick feedback and driving this. +1 for removing LEGACY mode in Flink 2.0. On Mon, Jan 15, 2024 at 3:23 AM Danny Cranmer wrote: > +1 to removing LEGACY mode in Flink 2.0. Thanks for driving. > > Danny, > > On Sat, 13 Jan 2024, 08:20 Yanfei Lei, wrote: > > > Thanks Zakelly for starting this discussion. > > > > Regardless of whether it is for users or developers, deprecating > > RestoreMode#LEGACY makes the semantics clearer and lower maintenance > > costs, and Flink 2.0 is a good time point to do this. > > So +1 for the overall idea. > > > > Best, > > Yanfei > > > > Zakelly Lan 于2024年1月11日周四 14:57写道： > > > > > > > > Hi devs, > > > > > > I'd like to start a discussion on FLIP-416: Deprecate and remove the > > > RestoreMode#LEGACY[1]. > > > > > > The FLIP-193[2] introduced two modes of state file ownership during > > > checkpoint restoration: RestoreMode#CLAIM and RestoreMode#NO_CLAIM. The > > > LEGACY mode, which was how Flink worked until 1.15, has been superseded > > by > > > NO_CLAIM as the default mode. The main drawback of LEGACY mode is that > > the > > > new job relies on artifacts from the old job without cleaning them up, > > > leaving users uncertain about when it is safe to delete the old > > checkpoint > > > directories. This leads to the accumulation of unnecessary checkpoint > > files > > > that are never cleaned up. Considering cluster availability and job > > > maintenance, it is not recommended to use LEGACY mode. Users could > choose > > > the other two modes to get a clear semantic for the state file > ownership. > > > > > > This FLIP proposes to deprecate the LEGACY mode and remove it > completely > > in > > > the upcoming Flink 2.0. This will make the semantic clear as well as > > > eliminate many bugs caused by mode transitions involving LEGACY mode > > (e.g. > > > FLINK-27114 [3]) and enhance code maintainability. > > > > > > Looking forward to hearing from you! > > > > > > [1] https://cwiki.apache.org/confluence/x/ookkEQ > > > [2] https://cwiki.apache.org/confluence/x/bIyqCw > > > [3] https://issues.apache.org/jira/browse/FLINK-27114 > > > > > > Best, > > > Zakelly > > > -- Best, Hangxiang.
Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY
Hi Zakelly, Thanks for driving this. +1 to removing the LEGACY mode. Best regards, Xuannan On Mon, Jan 15, 2024 at 3:22 AM Danny Cranmer wrote: > > +1 to removing LEGACY mode in Flink 2.0. Thanks for driving. > > Danny, > > On Sat, 13 Jan 2024, 08:20 Yanfei Lei, wrote: > > > Thanks Zakelly for starting this discussion. > > > > Regardless of whether it is for users or developers, deprecating > > RestoreMode#LEGACY makes the semantics clearer and lower maintenance > > costs, and Flink 2.0 is a good time point to do this. > > So +1 for the overall idea. > > > > Best, > > Yanfei > > > > Zakelly Lan 于2024年1月11日周四 14:57写道： > > > > > > > > Hi devs, > > > > > > I'd like to start a discussion on FLIP-416: Deprecate and remove the > > > RestoreMode#LEGACY[1]. > > > > > > The FLIP-193[2] introduced two modes of state file ownership during > > > checkpoint restoration: RestoreMode#CLAIM and RestoreMode#NO_CLAIM. The > > > LEGACY mode, which was how Flink worked until 1.15, has been superseded > > by > > > NO_CLAIM as the default mode. The main drawback of LEGACY mode is that > > the > > > new job relies on artifacts from the old job without cleaning them up, > > > leaving users uncertain about when it is safe to delete the old > > checkpoint > > > directories. This leads to the accumulation of unnecessary checkpoint > > files > > > that are never cleaned up. Considering cluster availability and job > > > maintenance, it is not recommended to use LEGACY mode. Users could choose > > > the other two modes to get a clear semantic for the state file ownership. > > > > > > This FLIP proposes to deprecate the LEGACY mode and remove it completely > > in > > > the upcoming Flink 2.0. This will make the semantic clear as well as > > > eliminate many bugs caused by mode transitions involving LEGACY mode > > (e.g. > > > FLINK-27114 [3]) and enhance code maintainability. > > > > > > Looking forward to hearing from you! > > > > > > [1] https://cwiki.apache.org/confluence/x/ookkEQ > > > [2] https://cwiki.apache.org/confluence/x/bIyqCw > > > [3] https://issues.apache.org/jira/browse/FLINK-27114 > > > > > > Best, > > > Zakelly > >
Re: [VOTE] FLIP-407: Improve Flink Client performance in interactive scenarios
+1 binding Best, Yangze Guo On Thu, Jan 11, 2024 at 8:36 PM Rui Fan <1996fan...@gmail.com> wrote: > > +1 binding > > Best, > Rui > > > On Thu, 11 Jan 2024 at 19:45, xiangyu feng wrote: > > > Hi all, > > > > I would like to start the vote for FLIP-407: Improve Flink Client > > performance in interactive scenarios[1]. > > This FLIP was discussed in this thread [2]. > > > > The vote will be open for at least 72 hours unless there is an objection or > > insufficient votes. > > > > Regards, > > Xiangyu > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-407%3A+Improve+Flink+Client+performance+in+interactive+scenarios > > [2] https://lists.apache.org/thread/ccsv66ygffgqbv956bnknbpllj4t24kj > >
Re: [VOTE] FLIP-407: Improve Flink Client performance in interactive scenarios
+1 binding Best, Weihua On Mon, Jan 15, 2024 at 11:06 AM Yangze Guo wrote: > +1 binding > > Best, > Yangze Guo > > On Thu, Jan 11, 2024 at 8:36 PM Rui Fan <1996fan...@gmail.com> wrote: > > > > +1 binding > > > > Best, > > Rui > > > > > > On Thu, 11 Jan 2024 at 19:45, xiangyu feng wrote: > > > > > Hi all, > > > > > > I would like to start the vote for FLIP-407: Improve Flink Client > > > performance in interactive scenarios[1]. > > > This FLIP was discussed in this thread [2]. > > > > > > The vote will be open for at least 72 hours unless there is an > objection or > > > insufficient votes. > > > > > > Regards, > > > Xiangyu > > > > > > [1] > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-407%3A+Improve+Flink+Client+performance+in+interactive+scenarios > > > [2] https://lists.apache.org/thread/ccsv66ygffgqbv956bnknbpllj4t24kj > > > >
Re: [VOTE] FLIP-405: Migrate string configuration key to ConfigOption
Hi all, Thank you all! Closing the vote. The result will be announced in a separate email. Best regards, Xuannan On Fri, Jan 12, 2024 at 2:44 PM Zhu Zhu wrote: > > +1 (binding) > > Thanks, > Zhu > > Xuannan Su 于2024年1月12日周五 14:24写道： > > > Hi all, > > > > I would like to clarify the statement regarding the first improvement > > from the previous email, as it was incomplete. To be more specific, we > > will also deprecate the getClass(String key, Class > > defaultValue, ClassLoader classLoader) and setClass(String key, > > Class klazz), as they are intended for internal use only and are no > > longer in use. > > > > To summarize, the first improvement should be as follows: > > > > We will mark the getBytes(String key, byte[] defaultValue) and > > setBytes(String key, byte[] bytes) methods as @Internal, as they are > > intended for internal use only. Additionally, we will > > deprecatesgetClass(String key, Class defaultValue, > > ClassLoader classLoader) and setClass(String key, Class klazz) > > methods, as they are intended for internal use only and are no longer > > in use. > > > > I apologize for any confusion caused by the incomplete information. > > > > Best regards, > > Xuannan > > > > On Fri, Jan 12, 2024 at 1:36 PM Xuannan Su wrote: > > > > > > Hi all, > > > > > > During voting, we identified two improvements we'd like to make to the > > FLIP: > > > > > > - We will mark the getBytes(String key, byte[] defaultValue) and > > > setBytes(String key, byte[] bytes) methods as @Internal, as they are > > > intended for internal use only. > > > - In addition to marking all getXxx(ConfigOption configOption) methods > > > as @Deprecated, we will also mark the getXxx(ConfigOption > > > configOption, Xxx overrideDefault) methods as @Deprecated. These will > > > be replaced with T get(ConfigOption configOption, T overrideDefault). > > > > > > We had an offline discussion with all the voters and reached a > > > consensus on the above changes. Therefore, we will not initiate > > > another voting thread unless there are objections. > > > > > > Best regards, > > > Xuannan > > > > > > > > > On Tue, Jan 9, 2024 at 11:43 AM Xintong Song > > wrote: > > > > > > > > +1 (binding) > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Mon, Jan 8, 2024 at 1:48 PM Hang Ruan > > wrote: > > > > > > > > > +1(non-binding) > > > > > > > > > > Best, > > > > > Hang > > > > > > > > > > Rui Fan <1996fan...@gmail.com> 于2024年1月8日周一 13:04写道： > > > > > > > > > > > +1(binding) > > > > > > > > > > > > Best, > > > > > > Rui > > > > > > > > > > > > On Mon, Jan 8, 2024 at 1:00 PM Xuannan Su > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > Thanks for all the feedback about the FLIP-405: Migrate string > > > > > > > configuration key to ConfigOption [1] [2]. > > > > > > > > > > > > > > I'd like to start a vote for it. The vote will be open for at > > least 72 > > > > > > > hours(excluding weekends,until Jan 11, 12:00AM GMT) unless there > > is an > > > > > > > objection or an insufficient number of votes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-405%3A+Migrate+string+configuration+key+to+ConfigOption > > > > > > > [2] > > https://lists.apache.org/thread/zfw1b1g3679yn0ppjbsokfrsx9k7ybg0 > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Xuannan > > > > > > > > > > > > > > > > > > > >
[RESULT][VOTE] FLIP-405: Migrate string configuration key to ConfigOption
Hi devs, I'm happy to announce that FLIP-405: Migrate string configuration key to ConfigOption [1] has been accepted with 4 approving votes (3 binding) [2]: - Rui Fan (binding) - Hang Ruan (non-binding) - Xintong Song (binding) - Zhu Zhu (binding) There are no disapproving votes. Thanks again to everyone who participated in the discussion and voting. Best regards, Xuannan [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-405%3A+Migrate+string+configuration+key+to+ConfigOption [2] https://lists.apache.org/thread/joyr7bxpo0lcj1zfzdj5nv0lrhb303rx
Re: [DISCUSS] FLIP-416: Deprecate and remove the RestoreMode#LEGACY
Hi Zakelly, Thanks for driving the discussion. It makes sense to remove LEGACY mode in Flink 2.0. Best, Jinzhong Li On Mon, Jan 15, 2024 at 10:34 AM Xuannan Su wrote: > Hi Zakelly, > > Thanks for driving this. +1 to removing the LEGACY mode. > > Best regards, > Xuannan > > On Mon, Jan 15, 2024 at 3:22 AM Danny Cranmer > wrote: > > > > +1 to removing LEGACY mode in Flink 2.0. Thanks for driving. > > > > Danny, > > > > On Sat, 13 Jan 2024, 08:20 Yanfei Lei, wrote: > > > > > Thanks Zakelly for starting this discussion. > > > > > > Regardless of whether it is for users or developers, deprecating > > > RestoreMode#LEGACY makes the semantics clearer and lower maintenance > > > costs, and Flink 2.0 is a good time point to do this. > > > So +1 for the overall idea. > > > > > > Best, > > > Yanfei > > > > > > Zakelly Lan 于2024年1月11日周四 14:57写道： > > > > > > > > > > > Hi devs, > > > > > > > > I'd like to start a discussion on FLIP-416: Deprecate and remove the > > > > RestoreMode#LEGACY[1]. > > > > > > > > The FLIP-193[2] introduced two modes of state file ownership during > > > > checkpoint restoration: RestoreMode#CLAIM and RestoreMode#NO_CLAIM. > The > > > > LEGACY mode, which was how Flink worked until 1.15, has been > superseded > > > by > > > > NO_CLAIM as the default mode. The main drawback of LEGACY mode is > that > > > the > > > > new job relies on artifacts from the old job without cleaning them > up, > > > > leaving users uncertain about when it is safe to delete the old > > > checkpoint > > > > directories. This leads to the accumulation of unnecessary checkpoint > > > files > > > > that are never cleaned up. Considering cluster availability and job > > > > maintenance, it is not recommended to use LEGACY mode. Users could > choose > > > > the other two modes to get a clear semantic for the state file > ownership. > > > > > > > > This FLIP proposes to deprecate the LEGACY mode and remove it > completely > > > in > > > > the upcoming Flink 2.0. This will make the semantic clear as well as > > > > eliminate many bugs caused by mode transitions involving LEGACY mode > > > (e.g. > > > > FLINK-27114 [3]) and enhance code maintainability. > > > > > > > > Looking forward to hearing from you! > > > > > > > > [1] https://cwiki.apache.org/confluence/x/ookkEQ > > > > [2] https://cwiki.apache.org/confluence/x/bIyqCw > > > > [3] https://issues.apache.org/jira/browse/FLINK-27114 > > > > > > > > Best, > > > > Zakelly > > > >
Re: [VOTE] Release flink-connector-rabbitmq, v3.0.2 release candidate #1
+1 (non-binding) - Build with JDK 11 on ubuntu 22.04 - Gpg sign is correct - No binary files in the source release - Reviewed web pr (need to rebase) Best, Zhongqiang Gong On 2024/01/12 12:50:53 Martijn Visser wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version > 3.0.2, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > This version is compatible with Flink 1.16.x, 1.17.x and 1.18.x. > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org > [2], which are signed with the key with fingerprint > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v3.0.2-rc1 [5], > * website pull request listing the new release [6]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Release Manager > > [1] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353145 > [2] > https://dist.apache.org/repos/dist/dev/flink/flink-connector-rabbitmq-3.0.2-rc1 > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1697 > [5] https://github.com/apache/flink-connector-rabbitmq/releases/tag/v3.0.2-rc1 > [6] https://github.com/apache/flink-web/pull/712 >
[jira] [Created] (FLINK-34078) Move InternalKeyContext classes from o.a.f.runtime.state.heap to o.a.f.runtime.state package
Jinzhong Li created FLINK-34078: --- Summary: Move InternalKeyContext classes from o.a.f.runtime.state.heap to o.a.f.runtime.state package Key: FLINK-34078 URL: https://issues.apache.org/jira/browse/FLINK-34078 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Reporter: Jinzhong Li Attachments: image-2024-01-15-12-57-12-667.png h3. Motication: When Rocksdb statebackend throws a keyGroup check illegal exception, the exception stack contains the heap stateBackend scoped class, which looks so strange to user. !image-2024-01-15-12-57-12-667.png|width=555,height=68! h3. Proposed changes: InternalKeyContext and InternalKeyContextImpl are commonly used by all state backends (heap/rocksdb/changelog), they should be moved from org.apache.flink.runtime.state.heap package to org.apache.flink.runtime.state package. h3. Compatibility: InternalKeyContext is annotated with @Internal, so this change has no compatibility issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34079) FLIP-405: Migrate string configuration key to ConfigOption
Rui Fan created FLINK-34079: --- Summary: FLIP-405: Migrate string configuration key to ConfigOption Key: FLINK-34079 URL: https://issues.apache.org/jira/browse/FLINK-34079 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Rui Fan Assignee: Xuannan Su Fix For: 1.19.0 This is an umbrella Jira of [FLIP-405|https://cwiki.apache.org/confluence/x/6Yr5E] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34080) Simplify the Configuration
Rui Fan created FLINK-34080: --- Summary: Simplify the Configuration Key: FLINK-34080 URL: https://issues.apache.org/jira/browse/FLINK-34080 Project: Flink Issue Type: Sub-task Components: Runtime / Configuration Reporter: Rui Fan Assignee: Rui Fan Fix For: 1.19.0 This Jira is 2.2 part of FLIP-405: * 2.2.1 Update Configuration to encourage the usage of ConfigOption over string configuration key * 2.2.2 Introduce public T get(ConfigOption configOption, T overrideDefault) * 2.2.3 Deprecate some unnecessary setXxx and getXxx methods in Configuration -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34081) Refactor all callers that using the public Xxx getXxx(ConfigOption configOption) and public void setXxx(ConfigOption key, Xxx value)
Rui Fan created FLINK-34081: --- Summary: Refactor all callers that using the public Xxx getXxx(ConfigOption configOption) and public void setXxx(ConfigOption key, Xxx value) Key: FLINK-34081 URL: https://issues.apache.org/jira/browse/FLINK-34081 Project: Flink Issue Type: Sub-task Components: Runtime / Configuration Reporter: Rui Fan Assignee: Rui Fan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34082) Remove deprecated methods of Configuration in 2.0
Rui Fan created FLINK-34082: --- Summary: Remove deprecated methods of Configuration in 2.0 Key: FLINK-34082 URL: https://issues.apache.org/jira/browse/FLINK-34082 Project: Flink Issue Type: Sub-task Components: Runtime / Configuration Reporter: Rui Fan Assignee: Rui Fan Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re:Re: [DISCUSS] FLIP-415: Introduce a new join operator to support minibatch
Hi, shuai. Thanks for this explaination. This scenario sounds reasonable to me. I agree that we need to split the behavior in minibatch into two types of options: 1. Whether to open minibatch to save batch data; 2. Whether to compress the changelog data while saving the batch, and merge the data with the same upsert key. For the latter the new option ''table.exec.mini-batch.compact-changes-enabled'' looks pretty good. -- Best！ Xuyang At 2024-01-12 18:13:12, "shuai xu" wrote: >Hi all. > >The point I want to highlight is that minibatch join could potentially yield >incomplete changelog which existing jobs are not supposed to be. For example, >the scenario that joins two CDC sources after de-duplicating them and the >output would be used for audit analysis could not accept incomplete changelog. >While the minibatch processing itself would not introduce any problem. > >The internal behavior of minibatch processing is not well-defined now. I don't >think reusing the minibatch option for minibatch join is problematic, but >precise control is necessary to mitigate the risk of generating incomplete >changelog within minibatch. > >Controlling the behavior on changelog within minibatch should be a global >option. Therefore, I propose introducing a new option >'table.exec.mini-batch.compact-changes-enabled' to precisely control changelog >compaction within minibatch. Then we deprecate the option >'table.exec.deduplicate.mini-batch.compact-changes-enabled' . The deduplicate >operator would fall back to follow the newly introduced option and the >minibatch join would follow it as well. > > >> 2024年1月12日 16:30，Jane Chan 写道： >> >> Hi shuai, >> >> Thanks for the update! Regarding the newly introduced configuration, I hold >> the same concern with Benchao and Xuyang. >> >> First of all, in most cases, the fact that users choose to enable >> mini-batch configuration indicates they are aware of the trade-off between >> throughput and completeness of the changelog. >> And if we finally adopt this configuration solely to avoid state >> incompatibility, does it mean that we will need to introduce a new >> configuration for every future operator's mini-batch optimization, similar >> to what we did today? >> >> Best, >> Jane >> >> On Fri, Jan 12, 2024 at 1:45 PM Xuyang wrote: >> >>> Hi, Xu Shuai. Thanks for driving this flip. >>> >>> >>> The CDC message amplification of cascade join has always been a problem >>> for users. Judging from the >>> nexmark results, this optimization is very meaningful. I just have the >>> same doubts as Benchao, why can't we >>> use minibatch join as the default behavior when the user turns on >>> minibatch? >>> >>> Although the semantic of changelog emitted by the Join operator is >>> eventual consistency, the change might >>> not be supposed for the downstream of the job which requires details of >>> changelog. >>> >>> >>> I think if the user adds the minibatch options to his job to enable >>> minibatch, he should know that flink will reduce >>> the amount of data sent to downstream by folding CDC messages as much as >>> possible. In scenarios where all >>> details of CDC records need to be retained, such as just synchronizing >>> data with jobs from one db to another db, >>> users have no reason to enable minibatch. >>> >>> >>> The only scenario I can think of that requires adding this independent >>> minibatch join option is to ensure that the state >>> is compatible between multiple versions, but we have not promised users >>> state compatibility during cross-version upgrades. >>> >>> >>> Maybe we need to figure it out why does the >>> 'table.exec.deduplicate.mini-batch.compact-changes-enabled' option need to >>> be added to deduplicate operator? I think this is the same reason as >>> adding a separate parameter to join to control CDC message folding. >>> >>> >>> >>> >>> -- >>> >>>Best！ >>>Xuyang >>> >>> >>> >>> >>> >>> 在 2024-01-11 16:19:30，"Benchao Li" 写道： > the change might not be supposed for the downstream of the job which >>> requires details of changelog Could you elaborate on this a bit? I've never met such kinds of requirements before, I'm curious what is the scenario that requires this. shuai xu 于2024年1月11日周四 13:08写道： > > Thanks for your response, Benchao. > > Here is my thought on the newly added option. > Users' current jobs are running on a version without minibatch join. If >>> the existing option to enable minibatch join is utilized, then when users' >>> jobs are migrated to the new version, the internal behavior of the join >>> operation within the jobs will change. Although the semantic of changelog >>> emitted by the Join operator is eventual consistency, the change might not >>> be supposed for the downstream of the job which requires details of >>> changelog. This newly added option also refers to >>> 'table.exec.deduplicate.mini-batch.co
[jira] [Created] (FLINK-34083) Deprecate string configuration keys and unused constants in ConfigConstants
Xuannan Su created FLINK-34083: -- Summary: Deprecate string configuration keys and unused constants in ConfigConstants Key: FLINK-34083 URL: https://issues.apache.org/jira/browse/FLINK-34083 Project: Flink Issue Type: Sub-task Components: Runtime / Configuration Reporter: Xuannan Su Fix For: 1.19.0 * Update ConfigConstants.java to deprecate and replace string configuration keys * Mark unused constants in ConfigConstants.java as deprecated -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34084) Deprecate unused configuration in BinaryInput/OutputFormat and FileInput/OutputFormat
Xuannan Su created FLINK-34084: -- Summary: Deprecate unused configuration in BinaryInput/OutputFormat and FileInput/OutputFormat Key: FLINK-34084 URL: https://issues.apache.org/jira/browse/FLINK-34084 Project: Flink Issue Type: Sub-task Components: Runtime / Configuration Reporter: Xuannan Su Fix For: 1.19.0 Update FileInputFormat.java, FileOutputFormat.java, BinaryInputFormat.java, and BinaryOutputFormat.java to deprecate unused string configuration keys. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34085) Remove deprecated string configuration keys in Flink 2.0
Xuannan Su created FLINK-34085: -- Summary: Remove deprecated string configuration keys in Flink 2.0 Key: FLINK-34085 URL: https://issues.apache.org/jira/browse/FLINK-34085 Project: Flink Issue Type: Sub-task Components: Runtime / Configuration Reporter: Xuannan Su Fix For: 2.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink
+1 (binding) On Sun, Jan 14, 2024 at 9:01 AM Yun Tang wrote: > +1 (non-binding) > > On 2024/01/14 02:26:13 Yun Gao wrote: > > +1 (binding) > > > > On Sat, Jan 13, 2024 at 10:08 AM Rodrigo Meneses > wrote: > > > > > > +1 (non binding) > > > > > > On Fri, Jan 12, 2024 at 5:45 PM Dong Lin wrote: > > > > > > > +1 (binding) > > > > > > > > On Sat, Jan 13, 2024 at 6:04 AM Austin Bennett > wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > On Fri, Jan 12, 2024 at 5:44 PM Becket Qin > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > On Fri, Jan 12, 2024 at 5:58 AM Zhijiang < > wangzhijiang...@aliyun.com > > > > > > .invalid> > > > > > > wrote: > > > > > > > > > > > > > +1 (binding) > > > > > > > Best, > > > > > > > Zhijiang > > > > > > > > -- > > > > > > > From:Kurt Yang > > > > > > > Send Time:2024年1月12日(星期五) 15:31 > > > > > > > To:dev > > > > > > > Subject:Re: Re: Re: [VOTE] Accept Flink CDC into Apache Flink > > > > > > > +1 (binding) > > > > > > > Best, > > > > > > > Kurt > > > > > > > On Fri, Jan 12, 2024 at 2:21 PM Hequn Cheng > > > > wrote: > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Hequn > > > > > > > > > > > > > > > > On Fri, Jan 12, 2024 at 2:19 PM godfrey he < > godfre...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Godfrey > > > > > > > > > > > > > > > > > > Zhu Zhu 于2024年1月12日周五 14:10写道： > > > > > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Zhu > > > > > > > > > > > > > > > > > > > > Hangxiang Yu 于2024年1月11日周四 > 14:26写道： > > > > > > > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 11, 2024 at 11:19 AM Xuannan Su < > > > > > > suxuanna...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Xuannan > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 11, 2024 at 10:28 AM Xuyang < > > > > xyzhong...@163.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > +1 (non-binding)-- > > > > > > > > > > > > > > > > > > > > > > > > > > Best！ > > > > > > > > > > > > > Xuyang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 在 2024-01-11 10:00:11，"Yang Wang" < > > > > wangyang0...@apache.org > > > > > > > > > > > > > 写道： > > > > > > > > > > > > > >+1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >Best, > > > > > > > > > > > > > >Yang > > > > > > > > > > > > > > > > > > > > > > > > > > > >On Thu, Jan 11, 2024 at 9:53 AM liu ron < > > > > > ron9@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> +1 non-binding > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Best > > > > > > > > > > > > > >> Ron > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> Matthias Pohl > > > > > > > 于2024年1月10日周三 > > > > > > > > > > > 23:05写道： > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > +1 (binding) > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > On Wed, Jan 10, 2024 at 3:35 PM ConradJam < > > > > > > > > > jam.gz...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > +1 non-binding > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > Dawid Wysakowicz > > > > > > 于2024年1月10日周三 > > > > > > > > > > > 21:06写道： > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > +1 (binding) > > > > > > > > > > > > > >> > > > Best, > > > > > > > > > > > > > >> > > > Dawid > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > On Wed, 10 Jan 2024 at 11:54, Piotr > Nowojski < > > > > > > > > > > > > pnowoj...@apache.org> > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > +1 (binding) > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > śr., 10 sty 2024 o 11:25 Martijn Visser > < > > > > > > > > > > > > martijnvis...@apache.org> > > > > > > > > > > > > > >> > > > > napisał(a): > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > +1 (binding) > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > On Wed, Jan 10, 2024 at 4:43 AM Xingbo > > > > Huang < > > > > > > > > > > > > hxbks...@gmail.com > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > +1 (binding) > > > > > > > > > > > > > >> > > > >
