[jira] [Created] (FLINK-29278) BINARY type is not supported in table store

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-29278:


 Summary: BINARY type is not supported in table store
 Key: FLINK-29278
 URL: https://issues.apache.org/jira/browse/FLINK-29278
 Project: Flink
  Issue Type: New Feature
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.3.0, table-store-0.2.1
 Attachments: image-2022-09-13-15-21-55-116.png

 !image-2022-09-13-15-21-55-116.png! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Sink V2 interface replacement for GlobalCommitter

[Yun Gao]

Hi,
Very sorry for the late reply for being in the holiday. 
And also very thanks for the discussion, it also reminds me 
one more background on the change of the GlobalCommitter:
When we are refactoring the job finish process in FLIP-147 to
ensures all the records could be committed at the end of bounded
streaming job, we have to desert the support for the cascade commits, 
which makes the cascade commit of `committer -> global committer` not work
in all cases. 
For the current issues, one possible alternative option from my side is that we
may support setting the committer parallelism to 1. Could this option solves
the issue in the current scenarios? I'll also have a double check with if
it could be implemented and the failed tests Krzysztof met. 
Best,
Yun
--
From:Steven Wu 
Send Time:2022 Sep. 10 (Sat.) 11:31
To:dev 
Cc:Yun Gao ; hililiwei 
Subject:Re: Sink V2 interface replacement for GlobalCommitter
Martjin, thanks a lot for chiming in!
Here are my concerns with adding GlobalCommitter in the PostCommitTopology 
1. when we use TwoPhaseCommittingSink. We would need to create a noop/dummy 
committer. Actual Iceberg/DeltaLake commits happen in the PostCommit stage. The 
PostCommit stage should be doing some work after the commit (not for the 
commit).
2. GlobalCommitter is marked as @deprecated. It will be removed at a certain 
point. What then?
Thanks,
Steven
On Fri, Sep 9, 2022 at 1:20 PM Krzysztof Chmielewski 
mailto:krzysiek.chmielew...@gmail.com >> wrote:
Thanks Martijn,
 I'm actually trying to run our V1 Delta connector on Flink 1.15 using
 SinkV1Adapter with GlobalCommitterOperator.
 Having said that, I might have found a potential issue with
 GlobalCommitterOperator, checkpoitining and failover recovery [1].
 For "normal" scenarios it does look good though.
 Regards,
 Krzysztof Chmielewski
 [1] https://lists.apache.org/thread/otscy199g1l9t3llvo8s2slntyn2r1jc 

 pt., 9 wrz 2022 o 20:49 Martijn Visser mailto:martijnvis...@apache.org >>
 napisał(a):
 > Hi all,
 >
 > A couple of bits from when work was being done on the new sink: V1 is
 > completely simulated as V2 [1]. V2 is strictly more expressive.
 >
 > If there's desire to stick to the `GlobalCommitter` interface, have a
 > look at the StandardSinkTopologies. Or you can just add your own more
 > fitting PostCommitTopology. The important part to remember is that this
 > topology is lagging one checkpoint behind in terms of fault-tolerance: it
 > only receives data once the committer committed
 > on notifyCheckpointComplete. Thus, the global committer needs to be
 > idempotent and able to restore the actual state on recovery. That
 > limitation is coming in from Flink's checkpointing behaviour and applies to
 > both V1 and V2. GlobalCommitterOperator is abstracting these issues along
 > with handling retries (so commits that happen much later). So it's probably
 > a good place to start just with the standard topology.
 >
 > Best regards,
 >
 > Martijn
 >
 > [1]
 >
 > https://github.com/apache/flink/blob/955e5ff34082ff8a4a46bb74889612235458eb76/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L359-L370
 >  
 >   >
 >
 > Op do 8 sep. 2022 om 20:51 schreef Krzysztof Chmielewski <
 > krzysiek.chmielew...@gmail.com >:
 >
 > > Hi,
 > > Krzysztof Chmielewski [1] from Delta-Flink connector open source
 > community
 > > here [2].
 > >
 > > I'm totally agree with Steven on this. Sink's V1 GlobalCommitter is
 > > something exactly what Flink-Delta Sink needs since it is the place where
 > > we do an actual commit to the Delta Log which should be done from a one
 > > place/instance.
 > >
 > > Currently I'm evaluating V2 for our connector and having, how Steven
 > > described it a "more natural, built-in concept/support of GlobalCommitter
 > > in the sink v2 interface" would be greatly appreciated.
 > >
 > > Cheers,
 > > Krzysztof Chmielewski
 > >
 > > [1] https://github.com/kristoffSC 
 > > [2] https://github.com/delta-io/connectors/tree/master/flink 
 > > 
 > >
 > > czw., 8 wrz 2022 o 19:51 Steven Wu  > > napisał(a):
 > >
 > > > Hi Yun,
 > > >
 > > > Thanks a lot for the reply!
 > > >
 > > > While we can add the global committer in the WithPostCommitTopology,
 > the
 > > > semantics are weird. The Commit stage actually didn't commit anything
 > to
 > > > the Iceberg table, and the PostCommit stage is where the Iceberg commit
 > > > happens.
 > > >
 > > > I just took a quick look at DeltaLake Flink sink. It still uses the V1
 > > sin

Re: [VOTE] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

[Teoh, Hong]

+1 (non-binding)

Thanks for driving this!

Regards,
Hong

On 11/09/2022, 19:23, "Gyula Fóra"  wrote:

CAUTION: This email originated from outside of the organization. Do not 
click links or open attachments unless you can confirm the sender and know the 
content is safe.



+1 (binding)

Gyula

On Sun, 11 Sep 2022 at 08:12, Danny Cranmer  wrote:

> Thanks for driving this FLIP.
>
> +1 (binding)
>
>
> On Sun, 11 Sept 2022, 13:03 Zheng Yu Chen,  wrote:
>
> > Since no one objected in the discuss thread, let's vote! FLIP-256 :
> >
> >
> 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-256+Support+Job+Dynamic+Parameter+With+Flink+Rest+Api
> >
> >
> > The vote will be open for at least 72h.
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Martijn Visser

[Danny Cranmer]

Congrats Martijn, a well deserved placement. Thankyou for all your efforts
in the Flink community!

Danny

On Tue, Sep 13, 2022 at 4:04 AM Qingsheng Ren  wrote:

> Congratulations Martijn!
>
> Best regards,
> Qingsheng
>
> > On Sep 9, 2022, at 23:08, Timo Walther  wrote:
> >
> > Hi everyone,
> >
> > I'm very happy to announce that Martijn Visser has joined the Flink PMC!
> >
> > Martijn has helped the community in many different ways over the past
> months. Externalizing the connectors from the Flink repo to their own
> repository, continously updating dependencies, and performing other
> project-wide refactorings. He is constantly coordinating contributions,
> connecting stakeholders, finding committers for contributions, driving
> release syncs, and helping in making the ASF a better place (e.g. by using
> Matomo an ASF-compliant tracking solution for all projects).
> >
> > Congratulations and welcome, Martijn!
> >
> > Cheers,
> > Timo Walther
> > (On behalf of the Apache Flink PMC)
>
>

[jira] [Created] (FLINK-29279) ElasticsearchWriterITCase fails without logging enabled

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-29279:


 Summary: ElasticsearchWriterITCase fails without logging enabled
 Key: FLINK-29279
 URL: https://issues.apache.org/jira/browse/FLINK-29279
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / ElasticSearch, Tests
Affects Versions: elasticsearch-3.0.0
Reporter: Chesnay Schepler
 Fix For: elasticsearch-3.0.0


The test relies on certain messages being logged, but by default logging is 
disabled so the test fails locally as-is.
We need to find a solution for this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[SUMMARY] Flink 1.16 release sync of 2022-09-13

[Xingbo Huang]

I would like to give you a brief update of the Flink 1.16 release sync
meating of 2022-09-13.

*We have finished x-team testing and all blocker issues have been fixed, so
we will start to prepare rc0*

*We still have some critical test stabilities[1] need to be resolved*

For more information about Flink release 1.16, you can refer to
https://cwiki.apache.org/confluence/display/FLINK/1.16+Release

The next Flink release sync will be on Tuesday the 20th of September at 9am
CEST/ 3pm China Standard Time / 7am UTC. The link could be found on the
following page
https://cwiki.apache.org/confluence/display/FLINK/1.16+Release#id-1.16Release-Syncmeeting
.

On behalf of all the release managers,

best regards,
Xingbo

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20issuetype%20%3D%20Bug%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20fixVersion%20%3D%201.16.0%20ORDER%20BY%20summary%20ASC%2C%20priority%20DESC

[jira] [Created] (FLINK-29280) Hint will not be propagated in subquery

[xuyang (Jira)]

xuyang created FLINK-29280:
--

 Summary: Hint will not be propagated in subquery
 Key: FLINK-29280
 URL: https://issues.apache.org/jira/browse/FLINK-29280
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.16.0
Reporter: xuyang


Add the following code in JoinHintTestBase to re-produce this bug.
{code:java}

@Test
public void testJoinHintWithJoinHintInSubQuery() {
String sql =
"select * from T1 WHERE a1 IN (select /*+ %s(T2) */ a2 from T2 join 
T3 on T2.a2 = T3.a3)";

verifyRelPlanByCustom(String.format(sql, 
buildCaseSensitiveStr(getTestSingleJoinHint(;
} {code}
This is because that calcite will not propagate the hint in subquery and flink 
also doesn't resolve it in FlinkSubQueryRemoveRule



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29281) Replace Akka

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-29281:


 Summary: Replace Akka
 Key: FLINK-29281
 URL: https://issues.apache.org/jira/browse/FLINK-29281
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / RPC
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler


Following the license change I propose to eventually replace Akka.

Based on LEGAL-619 an exemption is not feasible, and while a fork _may_ be 
created it's long-term future is up in the air and I'd be uncomfortable with 
relying on it.

I've been experimenting with a new RPC implementation based on gRPC and so far 
I'm quite optimistic. It's also based on Netty while not requiring as much of a 
tight coupling as Akka did.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29282) Decouple Quickstart E2E test from Elasticsearch

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-29282:


 Summary: Decouple Quickstart E2E test from Elasticsearch
 Key: FLINK-29282
 URL: https://issues.apache.org/jira/browse/FLINK-29282
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / ElasticSearch, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.17.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29283) Remove hardcoded apiVersion from operator unit test

[Tony Garrard (Jira)]

Tony Garrard created FLINK-29283:


 Summary: Remove hardcoded apiVersion from operator unit test
 Key: FLINK-29283
 URL: https://issues.apache.org/jira/browse/FLINK-29283
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.1.0
Reporter: Tony Garrard
 Fix For: kubernetes-operator-1.2.0


The unit test 
flink-kubernetes-operator/src/test/java/org/apache/flink/kubernetes/operator/utils/ReconciliationUtilsTest.java
 has a hardcoded apiVersion. To facilitate modifications, it should be using 
the constants provided in the class CrdConstants i.e. 

assertEquals(API_GROUP + "/" + API_VERSION, 
internalMeta.get("apiVersion").asText());

instead of "flink.apache.org/v1beta1"



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29284) Bump derby in flink-connector-hive to v10.14.2.0

[Martijn Visser (Jira)]

Martijn Visser created FLINK-29284:
--

 Summary: Bump derby in flink-connector-hive to v10.14.2.0
 Key: FLINK-29284
 URL: https://issues.apache.org/jira/browse/FLINK-29284
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Hive
Reporter: Martijn Visser
Assignee: Martijn Visser






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29285) Public ParameterProperty and TestUtils

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-29285:


 Summary: Public ParameterProperty and TestUtils
 Key: FLINK-29285
 URL: https://issues.apache.org/jira/browse/FLINK-29285
 Project: Flink
  Issue Type: Technical Debt
  Components: Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.16.0


The elasticsearch e2e tests need some utils from flink-end-to-end-test-common, 
that we should move to the flink test utils.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29286) Potential compatibility risk between snapshot binary and jars

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-29286:


 Summary: Potential compatibility risk between snapshot binary and 
jars
 Key: FLINK-29286
 URL: https://issues.apache.org/jira/browse/FLINK-29286
 Project: Flink
  Issue Type: Technical Debt
  Components: Tests
Affects Versions: elasticsearch-3.0.0
Reporter: Chesnay Schepler


E2E tests generally require a binary, where external repos only real choice is 
downloading the snapshot binaries from S3.

There is however no guarantee that these are actually compatible with the Maven 
artifacts that were downloaded.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29287) Add PackagingTestUtils

[Chesnay Schepler (Jira)]

Chesnay Schepler created FLINK-29287:


 Summary: Add PackagingTestUtils
 Key: FLINK-29287
 URL: https://issues.apache.org/jira/browse/FLINK-29287
 Project: Flink
  Issue Type: Technical Debt
  Components: Connectors / Common, Tests
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.16.0


We currently have test for the packaging of various connectors, scattered 
around in various bash e2e tests.

Add a java utility for writing such tests and make it available to external 
connector repos.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Sink V2 interface replacement for GlobalCommitter

[Krzysztof Chmielewski]

Hi  Martijn
Could you clarify a little bit what do you mean by:

"The important part to remember is that this
topology is lagging one checkpoint behind in terms of fault-tolerance: it
only receives data once the committer committed"

What are the implications?

Thanks,
Krzysztof Chmielewski

wt., 13 wrz 2022 o 09:57 Yun Gao  napisał(a):

> Hi,
> Very sorry for the late reply for being in the holiday.
> And also very thanks for the discussion, it also reminds me
> one more background on the change of the GlobalCommitter:
> When we are refactoring the job finish process in FLIP-147 to
> ensures all the records could be committed at the end of bounded
> streaming job, we have to desert the support for the cascade commits,
> which makes the cascade commit of `committer -> global committer` not work
> in all cases.
> For the current issues, one possible alternative option from my side is
> that we
> may support setting the committer parallelism to 1. Could this option
> solves
> the issue in the current scenarios? I'll also have a double check with if
> it could be implemented and the failed tests Krzysztof met.
> Best,
> Yun
> --
> From:Steven Wu 
> Send Time:2022 Sep. 10 (Sat.) 11:31
> To:dev 
> Cc:Yun Gao ; hililiwei 
> Subject:Re: Sink V2 interface replacement for GlobalCommitter
> Martjin, thanks a lot for chiming in!
> Here are my concerns with adding GlobalCommitter in the PostCommitTopology
> 1. when we use TwoPhaseCommittingSink. We would need to create a
> noop/dummy committer. Actual Iceberg/DeltaLake commits happen in the
> PostCommit stage. The PostCommit stage should be doing some work after the
> commit (not for the commit).
> 2. GlobalCommitter is marked as @deprecated. It will be removed at a
> certain point. What then?
> Thanks,
> Steven
> On Fri, Sep 9, 2022 at 1:20 PM Krzysztof Chmielewski <
> krzysiek.chmielew...@gmail.com >
> wrote:
> Thanks Martijn,
>  I'm actually trying to run our V1 Delta connector on Flink 1.15 using
>  SinkV1Adapter with GlobalCommitterOperator.
>  Having said that, I might have found a potential issue with
>  GlobalCommitterOperator, checkpoitining and failover recovery [1].
>  For "normal" scenarios it does look good though.
>  Regards,
>  Krzysztof Chmielewski
>  [1] https://lists.apache.org/thread/otscy199g1l9t3llvo8s2slntyn2r1jc <
> https://lists.apache.org/thread/otscy199g1l9t3llvo8s2slntyn2r1jc >
>  pt., 9 wrz 2022 o 20:49 Martijn Visser  martijnvis...@apache.org >>
>  napisał(a):
>  > Hi all,
>  >
>  > A couple of bits from when work was being done on the new sink: V1 is
>  > completely simulated as V2 [1]. V2 is strictly more expressive.
>  >
>  > If there's desire to stick to the `GlobalCommitter` interface, have a
>  > look at the StandardSinkTopologies. Or you can just add your own more
>  > fitting PostCommitTopology. The important part to remember is that this
>  > topology is lagging one checkpoint behind in terms of fault-tolerance:
> it
>  > only receives data once the committer committed
>  > on notifyCheckpointComplete. Thus, the global committer needs to be
>  > idempotent and able to restore the actual state on recovery. That
>  > limitation is coming in from Flink's checkpointing behaviour and
> applies to
>  > both V1 and V2. GlobalCommitterOperator is abstracting these issues
> along
>  > with handling retries (so commits that happen much later). So it's
> probably
>  > a good place to start just with the standard topology.
>  >
>  > Best regards,
>  >
>  > Martijn
>  >
>  > [1]
>  >
>  >
> https://github.com/apache/flink/blob/955e5ff34082ff8a4a46bb74889612235458eb76/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L359-L370
> <
> https://github.com/apache/flink/blob/955e5ff34082ff8a4a46bb74889612235458eb76/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L359-L370
> >
>  >
>  > Op do 8 sep. 2022 om 20:51 schreef Krzysztof Chmielewski <
>  > krzysiek.chmielew...@gmail.com  >>:
>  >
>  > > Hi,
>  > > Krzysztof Chmielewski [1] from Delta-Flink connector open source
>  > community
>  > > here [2].
>  > >
>  > > I'm totally agree with Steven on this. Sink's V1 GlobalCommitter is
>  > > something exactly what Flink-Delta Sink needs since it is the place
> where
>  > > we do an actual commit to the Delta Log which should be done from a
> one
>  > > place/instance.
>  > >
>  > > Currently I'm evaluating V2 for our connector and having, how Steven
>  > > described it a "more natural, built-in concept/support of
> GlobalCommitter
>  > > in the sink v2 interface" would be greatly appreciated.
>  > >
>  > > Cheers,
>  > > Krzysztof Chmielewski
>  > >
>  > > [1] https://github.com/kristoffSC 
>  > > [2] https://github.com/delta-io/connectors/tree/master/flink <
> https://github.co

[jira] [Created] (FLINK-29288) Can't start a job with a jar in the system classpath

[Yaroslav Tkachenko (Jira)]

Yaroslav Tkachenko created FLINK-29288:
--

 Summary: Can't start a job with a jar in the system classpath
 Key: FLINK-29288
 URL: https://issues.apache.org/jira/browse/FLINK-29288
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Affects Versions: kubernetes-operator-1.1.0
Reporter: Yaroslav Tkachenko


I'm using the latest (unreleased) version of the Kubernetes operator.

It looks like currently, it's impossible to use it with a job jar file in the 
system classpath (/opt/flink/lib). *jarURI* is required and it's always passed 
as a *pipeline.jars* parameter to the Flink process. In practice, it means that 
the same class is loaded twice: once by the system classloader and another time 
by the user classloader. This leads to exceptions like this:
{quote}java.lang.LinkageError: loader constraint violation: when resolving 
method 'XXX' the class loader org.apache.flink.util.ChildFirstClassLoader 
@47a5b70d of the current class, YYY, and the class loader 'app' for the 
method's defining class, ZZZ, have different Class objects for the type AAA 
used in the signature
{quote}
In my opinion, jarURI must be made optional even for the application mode. In 
this case, it's assumed that it's already available in the system classpath.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29289) SequencefileInputFormat based on the new Source API

[Jing Ge (Jira)]

Jing Ge created FLINK-29289:
---

 Summary: SequencefileInputFormat based on the new Source API
 Key: FLINK-29289
 URL: https://issues.apache.org/jira/browse/FLINK-29289
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / FileSystem
Affects Versions: 1.16.0
Reporter: Jing Ge






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29290) Produce changelog during full compaction in Table Store

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29290:
---

 Summary: Produce changelog during full compaction in Table Store
 Key: FLINK-29290
 URL: https://issues.apache.org/jira/browse/FLINK-29290
 Project: Flink
  Issue Type: New Feature
  Components: Table Store
Affects Versions: table-store-0.3.0
Reporter: Caizhi Weng


Currently Table Store only produces changelog directly from input. Some 
downstream systems, however, require complete changelogs including both 
UPDATE_BEFORE and UPDATE_AFTER messages.

We can only get these information during full compaction, so we should add a 
feature to produce changelog during full compaction.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29291) Change DataFileWriter into a factory to create writers

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29291:
---

 Summary: Change DataFileWriter into a factory to create writers
 Key: FLINK-29291
 URL: https://issues.apache.org/jira/browse/FLINK-29291
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Caizhi Weng
 Fix For: table-store-0.3.0


Currently {{DataFileWriter}} exposes {{write}} method for data files and extra 
files.

However, as the number of patterns to write files is increasing (for example, 
we'd like to write some records into a data file, then write some other records 
into an extra files when producing changelogs from full compaction) we'll have 
to keep adding methods to {{DataFileWriter}} if we keep the current 
implementation.

We'd like to refactor {{DataFileWriter}} into a factory to create writers, so 
that the users of writers can write however they like.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29292) Change MergeFunction to produce not only KeyValues

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29292:
---

 Summary: Change MergeFunction to produce not only KeyValues
 Key: FLINK-29292
 URL: https://issues.apache.org/jira/browse/FLINK-29292
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Caizhi Weng
 Fix For: table-store-0.3.0


{{MergeFunction}} of full compaction need to produce changelogs instead of 
single {{KeyValue}}. We need to modify {{MergeFunction}} into a generic class.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29293) Introduce a MergeFunction for full compaction

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29293:
---

 Summary: Introduce a MergeFunction for full compaction
 Key: FLINK-29293
 URL: https://issues.apache.org/jira/browse/FLINK-29293
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Caizhi Weng
 Fix For: table-store-0.3.0


We need to introduce a special {{MergeFunction}} to produce changelogs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29294) Introduce a CompactTask for full compaction

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29294:
---

 Summary: Introduce a CompactTask for full compaction
 Key: FLINK-29294
 URL: https://issues.apache.org/jira/browse/FLINK-29294
 Project: Flink
  Issue Type: Sub-task
Reporter: Caizhi Weng


We need to introduce a special {{CompactTask}} for full compaction to write 
changelog files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-256 Support Job Dynamic Parameter With Flink Rest Api

[Peng Kang]

+1

On Sun, Sep 11, 2022 at 8:04 PM Zheng Yu Chen  wrote:

> Now FLIP-256 is start vote in this thread url:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread_89f7kwnqgs8mfwos0h6b27lt3xn7383t&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=SECfrK1ARoa03VLYzzznEE30H5w_zne5CCcjBFv0xIM&e=
>
>
> zhengyu chen  于2022年8月15日周一 18:06写道：
>
> >
> > Hi all,
> >
> >
> > We would like to start a discussion thread on FLIP-256 Support Job
> > Dynamic Parameter With Flink Rest Api
> > <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D256-2BSupport-2BJob-2BDynamic-2BParameter-2BWith-2BFlink-2BRest-2BApi&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=CBpiVI1TfwkD5hnDB-m6LlzobLn5_AFwUtru-_65xdA&e=
> >
> > [1]
> >
> > After the user submits the jar package, running a job through restapi
> > (/jars/:jarid/run) [2] can only pass in (allowNonRestoredState,
> > savepointPath, programArg, entry-class, parallelism) parameters, which is
> > obvious with the diversification of job parameters  (eg Checkpoint
> > address)
> >
> > This solves the problem that the user can pass in other parameters when
> > submitting a job, avoiding the user to define these job parameters in the
> > code, resulting in the need to repackage the job for each modification
> >
> > There was some interest from users [3] from a meetup and the mailing
> list.
> > Looking forward to comments and feedback, thanks!
> >
> >
> > [1]
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_FLIP-2D256-2BSupport-2BJob-2BDynamic-2BParameter-2BWith-2BFlink-2BRest-2BApi&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=CBpiVI1TfwkD5hnDB-m6LlzobLn5_AFwUtru-_65xdA&e=
>
> >
> >
> > [2]
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache.org_flink_flink-2Ddocs-2Dmaster_docs_ops_rest-5Fapi_-23jars-2Djarid-2Drun&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=fgG0oof5F1cCG5a4wsrFhfBOpmdcso9qBGSxTjBwyxM&e=
>
> >
> > [3]
> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_FLINK-2D27060&d=DwIFaQ&c=R1GFtfTqKXCFH-lgEPXWwic6stQkW4U7uVq33mt-crw&r=ul9DxrYp20WQ-VRpcUO6Qze195Xs29UT94mvR4LySW8&m=gKGPfxowBcKR-vBgIUW5Te0HBs5jPM8brRhEEv3GnGiWxBgyhxw6owbmuP7J0G7G&s=VAQa0d5RxLORKT_DhZkfRCf1M6YYbA_Yzl4Ut_vIyeE&e=
>
> >
> >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
>
> --
> Best
>
> ConradJam
>

[jira] [Created] (FLINK-29295) Clear RecordWriter slower to avoid causing frequent compaction conflicts

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-29295:


 Summary: Clear RecordWriter slower to avoid causing frequent 
compaction conflicts
 Key: FLINK-29295
 URL: https://issues.apache.org/jira/browse/FLINK-29295
 Project: Flink
  Issue Type: Bug
  Components: Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: table-store-0.3.0, table-store-0.2.1


In AbstractTableWrite, the Writer is cleaned up as soon as no new files are 
generated, which may lead to the changes generated after the compaction have 
not been committed, but the new data from the next checkpoint comes to create a 
new writer, which conflicts with the changes generated in the next round of 
checkpoint and the previous round, resulting in an exception.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29296) OperatorCoordinatorHolder.create throws NPE

[Xingbo Huang (Jira)]

Xingbo Huang created FLINK-29296:


 Summary: OperatorCoordinatorHolder.create throws NPE
 Key: FLINK-29296
 URL: https://issues.apache.org/jira/browse/FLINK-29296
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.16.0
Reporter: Xingbo Huang
 Fix For: 1.16.0


{code:java}
2022-09-13T15:22:42.3864318Z Sep 13 15:22:42 [ERROR] Tests run: 8, Failures: 0, 
Errors: 1, Skipped: 0, Time elapsed: 5.633 s <<< FAILURE! - in 
org.apache.flink.test.streaming.runtime.SourceNAryInputChainingITCase
2022-09-13T15:22:42.3865377Z Sep 13 15:22:42 [ERROR] 
org.apache.flink.test.streaming.runtime.SourceNAryInputChainingITCase.testDirectSourcesOnlyExecution
  Time elapsed: 0.165 s  <<< ERROR!
2022-09-13T15:22:42.3867571Z Sep 13 15:22:42 java.lang.RuntimeException: Failed 
to fetch next result
2022-09-13T15:22:42.3919112Z Sep 13 15:22:42at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:109)
2022-09-13T15:22:42.3920935Z Sep 13 15:22:42at 
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
2022-09-13T15:22:42.3922442Z Sep 13 15:22:42at 
org.apache.flink.streaming.api.datastream.DataStreamUtils.collectBoundedStream(DataStreamUtils.java:106)
2022-09-13T15:22:42.3924085Z Sep 13 15:22:42at 
org.apache.flink.test.streaming.runtime.SourceNAryInputChainingITCase.testDirectSourcesOnlyExecution(SourceNAryInputChainingITCase.java:89)
2022-09-13T15:22:42.3925493Z Sep 13 15:22:42at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-09-13T15:22:42.3926635Z Sep 13 15:22:42at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-09-13T15:22:42.3928378Z Sep 13 15:22:42at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-09-13T15:22:42.3964273Z Sep 13 15:22:42at 
java.lang.reflect.Method.invoke(Method.java:498)
2022-09-13T15:22:42.3965054Z Sep 13 15:22:42at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
2022-09-13T15:22:42.3965788Z Sep 13 15:22:42at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2022-09-13T15:22:42.3966508Z Sep 13 15:22:42at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
2022-09-13T15:22:42.3967476Z Sep 13 15:22:42at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2022-09-13T15:22:42.3968432Z Sep 13 15:22:42at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
2022-09-13T15:22:42.3969233Z Sep 13 15:22:42at 
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
2022-09-13T15:22:42.3969871Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2022-09-13T15:22:42.3970534Z Sep 13 15:22:42at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
2022-09-13T15:22:42.3971453Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
2022-09-13T15:22:42.3972453Z Sep 13 15:22:42at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
2022-09-13T15:22:42.3973193Z Sep 13 15:22:42at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
2022-09-13T15:22:42.3973857Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
2022-09-13T15:22:42.3974634Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
2022-09-13T15:22:42.3975420Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
2022-09-13T15:22:42.3976060Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
2022-09-13T15:22:42.3976689Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
2022-09-13T15:22:42.3977555Z Sep 13 15:22:42at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2022-09-13T15:22:42.3978248Z Sep 13 15:22:42at 
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
2022-09-13T15:22:42.3978856Z Sep 13 15:22:42at 
org.junit.rules.RunRules.evaluate(RunRules.java:20)
2022-09-13T15:22:42.3979696Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
2022-09-13T15:22:42.3980716Z Sep 13 15:22:42at 
org.junit.runners.ParentRunner.run(ParentRunner.java:413)
2022-09-13T15:22:42.3981785Z Sep 13 15:22:42at 
org.junit.runner.JUnitCore.run(JUnitCore.java:137)
2022-09-13T15:22:42.3982352Z Sep 13 15:22:42at 
org.junit.runner.JUnitCore.run(JUnitCore.java:115)
2022-09-13T15:22:42.3982989Z Sep 13 15:22:42at 
org.junit.vintage.engine.execution.Runner

[jira] [Created] (FLINK-29297) Group Table Store file writers into SingleFileWriter and RollingFileWriter

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-29297:
---

 Summary: Group Table Store file writers into SingleFileWriter and 
RollingFileWriter
 Key: FLINK-29297
 URL: https://issues.apache.org/jira/browse/FLINK-29297
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Reporter: Caizhi Weng
 Fix For: table-store-0.3.0


Currently we have two types of files to write:
* Data files (LSM tree files), where a level 0 data file is a single file and a 
level >= 1 data file is a set of rolling files. Statistics for these files are 
needed for pruning when scanning.
* Extra files (changelog files), just a list of records. No statistics are 
needed.

However, current writers are all based on {{MetricFileWriter}}, which always 
produces statistics.

We'd like to refactor the writers and group them into {{SingleFileWriter}} and 
{{RollingFileWriter}}. {{StatsCollectingSingleFileWriter}} should be a subclass 
of {{SingleFileWriter}} which additionally produces statistics, and data file 
writers should be a subclass of {{StatsCollectingSingleFileWriter}} or 
{{RollingFileWriter}} based on their level. For extra file writers, extending 
from {{SingleFileWriter}} is enough.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29298) LocalBufferPool request buffer from NetworkBufferPool hanging

[Weijie Guo (Jira)]

Weijie Guo created FLINK-29298:
--

 Summary: LocalBufferPool request buffer from NetworkBufferPool 
hanging
 Key: FLINK-29298
 URL: https://issues.apache.org/jira/browse/FLINK-29298
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.16.0
Reporter: Weijie Guo
 Attachments: image-2022-09-14-10-52-15-259.png, 
image-2022-09-14-10-58-45-987.png, image-2022-09-14-11-00-47-309.png

In the scenario where the buffer contention is fierce, sometimes the task hang 
can be observed. Through the thread dump information, we can found that the 
task thread is blocked by requestMemorySegmentBlocking forever. After 
investigating the dumped heap information, I found that the NetworkBufferPool 
actually has many buffers, but the LocalBufferPool is still unavailable and no 
buffer has been obtained.

By looking at the code, I am sure that this is a bug in thread race: when the 
task thread polled out the last buffer in LocalBufferPool and triggered the 
onGlobalPoolAvailable callback itself, it will skip this notification  (as 
currently the LocalBufferPool is available), which will cause the BufferPool to 
eventually become unavailable and will never register a callback to the 
NetworkBufferPool.

The conditions for triggering the problem are relatively strict, but I have 
found a stable way to reproduce it, I will try to fix and verify this problem.

!image-2022-09-14-10-52-15-259.png|width=1021,height=219!

!image-2022-09-14-10-58-45-987.png|width=997,height=315!

!image-2022-09-14-11-00-47-309.png|width=453,height=121!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29299) Fix the network memory size calculation issue in fine-grained resource mode

[Yingjie Cao (Jira)]

Yingjie Cao created FLINK-29299:
---

 Summary: Fix the network memory size calculation issue in 
fine-grained resource mode
 Key: FLINK-29299
 URL: https://issues.apache.org/jira/browse/FLINK-29299
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.16.0
Reporter: Yingjie Cao
 Fix For: 1.16.0


After FLINK-28663, one intermediate dataset can be consumed by multiple 
consumers, there is a case where one vertex can consume one intermediate 
dataset multiple times. However, currently in fine-grained resource mode, when 
computing the required network buffer size, the intermediate dataset is used as 
key to record the size of network buffer per input gate, which means it may 
allocate less network buffers than needed if two input gate of the same vertex 
consumes the same intermediate dataset.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Sink V2 interface replacement for GlobalCommitter

[Steven Wu]

> setting the committer parallelism to 1.

Yun, setting the parallelism to 1 is essentially a global committer. That
would work. not sure about the implications to other parts of the v2 sink
interface.

On Tue, Sep 13, 2022 at 2:17 PM Krzysztof Chmielewski <
krzysiek.chmielew...@gmail.com> wrote:

> Hi  Martijn
> Could you clarify a little bit what do you mean by:
>
> "The important part to remember is that this
> topology is lagging one checkpoint behind in terms of fault-tolerance: it
> only receives data once the committer committed"
>
> What are the implications?
>
> Thanks,
> Krzysztof Chmielewski
>
> wt., 13 wrz 2022 o 09:57 Yun Gao 
> napisał(a):
>
>> Hi,
>> Very sorry for the late reply for being in the holiday.
>> And also very thanks for the discussion, it also reminds me
>> one more background on the change of the GlobalCommitter:
>> When we are refactoring the job finish process in FLIP-147 to
>> ensures all the records could be committed at the end of bounded
>> streaming job, we have to desert the support for the cascade commits,
>> which makes the cascade commit of `committer -> global committer` not work
>> in all cases.
>> For the current issues, one possible alternative option from my side is
>> that we
>> may support setting the committer parallelism to 1. Could this option
>> solves
>> the issue in the current scenarios? I'll also have a double check with if
>> it could be implemented and the failed tests Krzysztof met.
>> Best,
>> Yun
>> --
>> From:Steven Wu 
>> Send Time:2022 Sep. 10 (Sat.) 11:31
>> To:dev 
>> Cc:Yun Gao ; hililiwei 
>> Subject:Re: Sink V2 interface replacement for GlobalCommitter
>> Martjin, thanks a lot for chiming in!
>> Here are my concerns with adding GlobalCommitter in the
>> PostCommitTopology
>> 1. when we use TwoPhaseCommittingSink. We would need to create a
>> noop/dummy committer. Actual Iceberg/DeltaLake commits happen in the
>> PostCommit stage. The PostCommit stage should be doing some work after the
>> commit (not for the commit).
>> 2. GlobalCommitter is marked as @deprecated. It will be removed at a
>> certain point. What then?
>> Thanks,
>> Steven
>> On Fri, Sep 9, 2022 at 1:20 PM Krzysztof Chmielewski <
>> krzysiek.chmielew...@gmail.com >
>> wrote:
>> Thanks Martijn,
>>  I'm actually trying to run our V1 Delta connector on Flink 1.15 using
>>  SinkV1Adapter with GlobalCommitterOperator.
>>  Having said that, I might have found a potential issue with
>>  GlobalCommitterOperator, checkpoitining and failover recovery [1].
>>  For "normal" scenarios it does look good though.
>>  Regards,
>>  Krzysztof Chmielewski
>>  [1] https://lists.apache.org/thread/otscy199g1l9t3llvo8s2slntyn2r1jc <
>> https://lists.apache.org/thread/otscy199g1l9t3llvo8s2slntyn2r1jc >
>>  pt., 9 wrz 2022 o 20:49 Martijn Visser > >
>>  napisał(a):
>>  > Hi all,
>>  >
>>  > A couple of bits from when work was being done on the new sink: V1 is
>>  > completely simulated as V2 [1]. V2 is strictly more expressive.
>>  >
>>  > If there's desire to stick to the `GlobalCommitter` interface, have a
>>  > look at the StandardSinkTopologies. Or you can just add your own more
>>  > fitting PostCommitTopology. The important part to remember is that this
>>  > topology is lagging one checkpoint behind in terms of fault-tolerance:
>> it
>>  > only receives data once the committer committed
>>  > on notifyCheckpointComplete. Thus, the global committer needs to be
>>  > idempotent and able to restore the actual state on recovery. That
>>  > limitation is coming in from Flink's checkpointing behaviour and
>> applies to
>>  > both V1 and V2. GlobalCommitterOperator is abstracting these issues
>> along
>>  > with handling retries (so commits that happen much later). So it's
>> probably
>>  > a good place to start just with the standard topology.
>>  >
>>  > Best regards,
>>  >
>>  > Martijn
>>  >
>>  > [1]
>>  >
>>  >
>> https://github.com/apache/flink/blob/955e5ff34082ff8a4a46bb74889612235458eb76/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L359-L370
>> <
>> https://github.com/apache/flink/blob/955e5ff34082ff8a4a46bb74889612235458eb76/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/SinkV1Adapter.java#L359-L370
>> >
>>  >
>>  > Op do 8 sep. 2022 om 20:51 schreef Krzysztof Chmielewski <
>>  > krzysiek.chmielew...@gmail.com > >>:
>>  >
>>  > > Hi,
>>  > > Krzysztof Chmielewski [1] from Delta-Flink connector open source
>>  > community
>>  > > here [2].
>>  > >
>>  > > I'm totally agree with Steven on this. Sink's V1 GlobalCommitter is
>>  > > something exactly what Flink-Delta Sink needs since it is the place
>> where
>>  > > we do an actual commit to the Delta Log which should be done from a
>> one
>>  > > place/instance.
>>  > >
>>  > > Cu

[ANNOUNCE] Release 1.16.0, release candidate #0

[Xingbo Huang]

Hi everyone,

The RC0 for Apache Flink 1.16.0 has been created. This is still a
preview-only release candidate to facilitate the integrated testing since
there are still some ongoing efforts, thus no official votes will take
place. It has all the artifacts that we would typically have for a release,
except for the release note and the website pull request for the release
announcement.

The following contents are available for your review:
- the preview source release and binary convenience releases [1], which are
signed with the key with fingerprint 3C2C9FFB59DF9F3E [2].
- all artifacts that would normally be deployed to the Maven Central
Repository [3].
- source code tag "release-1.16.0-rc0" [4]

Your help testing the release will be greatly appreciated! And we'll create
the votable RC1 as soon as all the efforts are finished.

Thank you~
Chesnay, Martijn, Godfrey & Xingbo

[1] https://dist.apache.org/repos/dist/dev/flink/flink-1.16.0-rc0/
[2] https://dist.apache.org/repos/dist/release/flink/KEYS
[3] https://repository.apache.org/content/repositories/orgapacheflink-1532/
[4] https://github.com/apache/flink/releases/tag/release-1.16.0-rc0

[DISCUSS] FLIP-260: Expose Finish Method For UserDefinedFunction

[Lincoln Lee]

Hello everyone,

  I’d like to open a discussion on FLIP-260[1]: expose finish method for
UserDefinedFunction, this makes a chance for users who rely on finish logic
in the legacy close() method (< 1.14) to migrate to the new finish() method.

  The task lifecycle was changed in FLINK-22972[2]: a new finish() phase
was introduced (extracted the ‘finish’ part out of the ‘close’) and removed
the dispose() method. This change was also done in table module (e.g.,
`AbstractMapBundleOperator` for mini-batch operation ) but not covered the
UserDefinedFunction which only exposes open() and close() api for custom
usage, those customers who rely on the legacy close() api may encounter
wrong result or suffer runtime errors after upgrading to the new version.
Strictly speaking, it is a bug caused by the breaking change, but due to
the public api change, we propose this flip.

  Looking forward to your comments or feedback.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-260%3A+Expose+Finish+Method+For+UserDefinedFunction
[2] https://issues.apache.org/jira/browse/FLINK-22972

Best,
Lincoln Lee

Re: [ANNOUNCE] New Apache Flink PMC Member - Martijn Visser

[Lincoln Lee]

Congrats, Martijn!

Best,
Lincoln Lee


Danny Cranmer  于2022年9月13日周二 16:50写道：

> Congrats Martijn, a well deserved placement. Thankyou for all your efforts
> in the Flink community!
>
> Danny
>
> On Tue, Sep 13, 2022 at 4:04 AM Qingsheng Ren  wrote:
>
> > Congratulations Martijn!
> >
> > Best regards,
> > Qingsheng
> >
> > > On Sep 9, 2022, at 23:08, Timo Walther  wrote:
> > >
> > > Hi everyone,
> > >
> > > I'm very happy to announce that Martijn Visser has joined the Flink
> PMC!
> > >
> > > Martijn has helped the community in many different ways over the past
> > months. Externalizing the connectors from the Flink repo to their own
> > repository, continously updating dependencies, and performing other
> > project-wide refactorings. He is constantly coordinating contributions,
> > connecting stakeholders, finding committers for contributions, driving
> > release syncs, and helping in making the ASF a better place (e.g. by
> using
> > Matomo an ASF-compliant tracking solution for all projects).
> > >
> > > Congratulations and welcome, Martijn!
> > >
> > > Cheers,
> > > Timo Walther
> > > (On behalf of the Apache Flink PMC)
> >
> >
>