Re: [VOTE] FLIP-198: Working directory for Flink processes

+1 (binding) Best Yun Tang From: Yang Wang Sent: Friday, December 17, 2021 12:42 To: dev Cc: Till Rohrmann Subject: Re: [VOTE] FLIP-198: Working directory for Flink processes +1 (binding) Best, Yang David Morávek 于2021年12月17日周五 05:39写道： > +1 (non-binding) >

[jira] [Created] (FLINK-25359) flink1.13.2 心跳超时

mindezhi created FLINK-25359: Summary: flink1.13.2 心跳超时 Key: FLINK-25359 URL: https://issues.apache.org/jira/browse/FLINK-25359 Project: Flink Issue Type: Bug Reporter: mindezhi

Re: [ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

I guess this is related to publishers everywhere are updating their artifacts in response to the log4shell vulnerability[1]. All we can do and need to do is wait. ☕️ Best, Leonard [1] https://issues.sonatype.org/browse/OSSRH-76300 > 2021年12月17

Re: [ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

Not found in https://repo1.maven.org/maven2/org/apache/flink/flink-table-api-java/ I guess too many people sent versions, resulting in maven central repository synchronization being slower. Best, Jingsong On Fri, Dec 17, 2021 at 2:00 PM casel.chen wrote: > > I can NOT find flink 1.13.5 rel

Re: [ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

Hi, @Chesnay Schepler I want to forward a feedback from the user mailing list, the jars of the new patch releases are not available at the central maven repo. example: https://mvnrepository.com/artifact/org.apache.flink/flink-streaming-java Best, Wenlong On Fri, 17 Dec 2021 at 01:27, Chesnay S

[jira] [Created] (FLINK-25358) New File Sink end-to-end test failed due to TM could not connect to RM

Yun Gao created FLINK-25358: --- Summary: New File Sink end-to-end test failed due to TM could not connect to RM Key: FLINK-25358 URL: https://issues.apache.org/jira/browse/FLINK-25358 Project: Flink

Re: [VOTE] FLIP-198: Working directory for Flink processes

+1 (binding) Best, Yang David Morávek 于2021年12月17日周五 05:39写道： > +1 (non-binding) > > Best, > D. > > On Thu, Dec 16, 2021 at 4:30 PM Chesnay Schepler > wrote: > > > +1 > > > > On 16/12/2021 14:42, Till Rohrmann wrote: > > > Hi everyone, > > > > > > I'd like to start a vote on FLIP-198: Working

[jira] [Created] (FLINK-25357) SQL planner incorrectly changes a streaming join with FLOOR(rowtime) into interval join

Caizhi Weng created FLINK-25357: --- Summary: SQL planner incorrectly changes a streaming join with FLOOR(rowtime) into interval join Key: FLINK-25357 URL: https://issues.apache.org/jira/browse/FLINK-25357

[jira] [Created] (FLINK-25356) Add benchmarks for performance in OLAP scenarios

Xintong Song created FLINK-25356: Summary: Add benchmarks for performance in OLAP scenarios Key: FLINK-25356 URL: https://issues.apache.org/jira/browse/FLINK-25356 Project: Flink Issue Type:

?????? [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

--  -- ??: "dev" https://cwiki.apache.

Re: [VOTE] FLIP-198: Working directory for Flink processes

+1 (non-binding) Best, D. On Thu, Dec 16, 2021 at 4:30 PM Chesnay Schepler wrote: > +1 > > On 16/12/2021 14:42, Till Rohrmann wrote: > > Hi everyone, > > > > I'd like to start a vote on FLIP-198: Working directory for Flink > processes > > [1] which has been discussed in this thread [2]. > > >

[jira] [Created] (FLINK-25355) Upgrade Flink dependency to 1.13.3

Igal Shilman created FLINK-25355: Summary: Upgrade Flink dependency to 1.13.3 Key: FLINK-25355 URL: https://issues.apache.org/jira/browse/FLINK-25355 Project: Flink Issue Type: Improvement

Re: [DISCUSS] Immediate dedicated StateFun releases for log4j vulnerability

+1 On 16/12/2021 19:30, Igal Shilman wrote: Hi All, Following the recent Apache Flink releases due to the log4j vulnerability, I'd like to propose an immediate StateFun release - 3.1.1. This release is basically the same as 3.1 but updates the Flink dependency to 1.13.3. Please raise your conc

Re: [DISCUSS] Immediate dedicated StateFun releases for log4j vulnerability

And I'm happy to help with the release. On Thu, Dec 16, 2021 at 12:55 PM Seth Wiesman wrote: > +1 > > On Thu, Dec 16, 2021 at 12:37 PM Igal Shilman wrote: > >> Hi All, >> >> Following the recent Apache Flink releases due to the log4j vulnerability, >> I'd like to propose an immediate StateFun r

Re: [DISCUSS] Immediate dedicated StateFun releases for log4j vulnerability

+1 On Thu, Dec 16, 2021 at 12:37 PM Igal Shilman wrote: > Hi All, > > Following the recent Apache Flink releases due to the log4j vulnerability, > I'd like to propose an immediate StateFun release - 3.1.1. > This release is basically the same as 3.1 but updates the Flink dependency > to 1.13.3.

Re: [DISCUSS] Immediate dedicated StateFun releases for log4j vulnerability

+1 On Thu, Dec 16, 2021, 10:30 Igal Shilman wrote: > Hi All, > > Following the recent Apache Flink releases due to the log4j vulnerability, > I'd like to propose an immediate StateFun release - 3.1.1. > This release is basically the same as 3.1 but updates the Flink dependency > to 1.13.3. > > P

[DISCUSS] Immediate dedicated StateFun releases for log4j vulnerability

Hi All, Following the recent Apache Flink releases due to the log4j vulnerability, I'd like to propose an immediate StateFun release - 3.1.1. This release is basically the same as 3.1 but updates the Flink dependency to 1.13.3. Please raise your concerns if any, otherwise we'll proceed with the r

[ANNOUNCE] Apache Flink 1.14.2 / 1.13.5 / 1.12.7 / 1.11.6 released

The Apache Flink community has released emergency bugfix versions of Apache Flink for the 1.11, 1.12, 1.13 and 1.14 series. These releases include a version upgrade for Log4j to address [CVE-2021-44228](https://nvd.nist.gov/vuln/detail/CVE-2021-44228) and [CVE-2021-45046](https://nvd.nist.gov/

Re: [DISCUSS] GHA migration roadmap

Thanks for drafting this proposal Nico. I hope that we can improve our development processes and build system stability in the long run with the move to GHA. Hence +1 for this proposal and the timeline. The plan looks thoroughly planned. Cheers, Till On Thu, Dec 16, 2021 at 4:29 PM Chesnay Schep

Re: [VOTE] FLIP-198: Working directory for Flink processes

+1 On 16/12/2021 14:42, Till Rohrmann wrote: Hi everyone, I'd like to start a vote on FLIP-198: Working directory for Flink processes [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwik

Re: [DISCUSS] GHA migration roadmap

We will not use Apache resources, but install self-hosted runners on our current CI machines, similar to what we have done with Azure. On 16/12/2021 16:07, Fabian Paul wrote: Hi Nico, Thanks a lot for drafting the proposal. I really like the fully-fledged phasing model. All in all, I am +1 to

Re: [DISCUSS] GHA migration roadmap

Hi Nico, Thanks a lot for the proposal. I applaud the move to GHA and the planning seems great. +1 from my side. I'm assuming with the converting one CI machine, you mean that we switch convert one of the CI machines to run the Github Selfhosted Runner? Best regards, Martijn On Thu, 16 Dec 202

Re: [DISCUSS] GHA migration roadmap

Hi Nico, Thanks a lot for drafting the proposal. I really like the fully-fledged phasing model. All in all, I am +1 to move away from azure and can only second all the points you have mentioned. I only want to clarify one point. So far my understanding was that the GHA resources are managed on a

[DISCUSS] GHA migration roadmap

Hi all, as several people know by now, we are planning to move from Azure CI to Github Actions. This is motivated by (not an exhaustive list): - Not needing to mirror the repo anymore for CI - Improving the contributor experience, especially for new contributors - GHA development being more active

[jira] [Created] (FLINK-25354) Document API graduation process

Till Rohrmann created FLINK-25354: - Summary: Document API graduation process Key: FLINK-25354 URL: https://issues.apache.org/jira/browse/FLINK-25354 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25353) Add tests to ensure API graduation process is honoured

Till Rohrmann created FLINK-25353: - Summary: Add tests to ensure API graduation process is honoured Key: FLINK-25353 URL: https://issues.apache.org/jira/browse/FLINK-25353 Project: Flink Issu

[jira] [Created] (FLINK-25352) Update stability annotations to include the since and missedGraduations fields

Till Rohrmann created FLINK-25352: - Summary: Update stability annotations to include the since and missedGraduations fields Key: FLINK-25352 URL: https://issues.apache.org/jira/browse/FLINK-25352 Proj

[jira] [Created] (FLINK-25351) Add FlinkVersion

Till Rohrmann created FLINK-25351: - Summary: Add FlinkVersion Key: FLINK-25351 URL: https://issues.apache.org/jira/browse/FLINK-25351 Project: Flink Issue Type: Sub-task Affects Versions:

[jira] [Created] (FLINK-25350) Verify stability guarantees of annotated classes

Till Rohrmann created FLINK-25350: - Summary: Verify stability guarantees of annotated classes Key: FLINK-25350 URL: https://issues.apache.org/jira/browse/FLINK-25350 Project: Flink Issue Type

[jira] [Created] (FLINK-25349) Add test that ensures that parameter/return types have the same or stronger stability guarantee than method

Till Rohrmann created FLINK-25349: - Summary: Add test that ensures that parameter/return types have the same or stronger stability guarantee than method Key: FLINK-25349 URL: https://issues.apache.org/jira/browse/

[jira] [Created] (FLINK-25348) Update release guide to reset japicmp exceptions for every release

Till Rohrmann created FLINK-25348: - Summary: Update release guide to reset japicmp exceptions for every release Key: FLINK-25348 URL: https://issues.apache.org/jira/browse/FLINK-25348 Project: Flink

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

I just noticed offline with Yun that I have some misconception on how how blocking data exchange work in Flink. Apparently, a subtask of the stage 2 is already started when all connected subtasks of stage 1 are finished. I was assuming that it works similarly to Hadoop where all tasks of stage 1 ha

[VOTE] FLIP-198: Working directory for Flink processes

Hi everyone, I'd like to start a vote on FLIP-198: Working directory for Flink processes [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/x/ZZiqCw [2] https://lis

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

I've updated the FLIP and excluded the blob storage from the initial scope. I hope that I could answer all your questions. I will now start a vote. If there are still things that we want to discuss, then please post to this thread. Cheers, Till On Thu, Dec 16, 2021 at 12:37 PM Till Rohrmann wrot

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

Hi Yun, the basic idea is to use different regions for the different stages. So you have your stages: Stage 1: -> pre-writer topology -> writer -> committables (materialized) Stage 2: committables (materialized) -> pre-commit topology -> committer -> succesful committables (materialized) Stage 3

[jira] [Created] (FLINK-25347) Replace deprecated TableSchema with Schema in MergeTableLikeUtil

RocMarshal created FLINK-25347: -- Summary: Replace deprecated TableSchema with Schema in MergeTableLikeUtil Key: FLINK-25347 URL: https://issues.apache.org/jira/browse/FLINK-25347 Project: Flink

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

Hi, Very thanks Fabian for the explanation and it solves most of the issues. There is one left issue I want to have a double confirmation is that for the edges between writer and committer and in the post-committer topology, perhaps the result partition with HA storage is not enough solve all

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

Just a quick amend: There will be no blocking exchange in the pre-writer exchange for performance reasons. After the writer, we have tiny data volume and are free to add as many as we see necessary. On Thu, Dec 16, 2021 at 11:18 AM Fabian Paul wrote: > Hi Till, > > Good point, in the scenario wi

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

Maybe it is a good idea to remove storing the blobs in the working directory from this FLIP and to address the problems with doing this as a follow up. This will keep the FLIP narrowly scoped and faster to implement. I will update the FLIP and move the blob storage part to the follow-up section. C

[jira] [Created] (FLINK-25346) FLIP-197: API stability graduation process

Till Rohrmann created FLINK-25346: - Summary: FLIP-197: API stability graduation process Key: FLINK-25346 URL: https://issues.apache.org/jira/browse/FLINK-25346 Project: Flink Issue Type: Impr

Re: [VOTE] FLIP-190: Support Version Upgrades for Table API & SQL Programs

+1 (non-binding) On Wed, Dec 15, 2021 at 11:22 AM godfrey he wrote: > +1 (binding) > > Best, > Godfrey > > Ingo Bürk 于2021年12月15日周三 16:19写道： > > > > +1 (binding) > > > > Thanks for driving this much needed feature! > > > > On 14.12.21 17:45, Timo Walther wrote: > > > Hi everyone, > > > > > > I'

[jira] [Created] (FLINK-25345) FLIP-196: Source API stability guarantees

Till Rohrmann created FLINK-25345: - Summary: FLIP-196: Source API stability guarantees Key: FLINK-25345 URL: https://issues.apache.org/jira/browse/FLINK-25345 Project: Flink Issue Type: Impro

[RESULT][VOTE] FLIP-197: API stability graduation process

Hi everyone, I am happy to announce that FLIP-197 [1] has been accepted [2]. Binding votes: - Ingo Bürk - Chesnay Schepler - Till Rohrmann [1] https://cwiki.apache.org/confluence/x/J5eqCw [2] https://lists.apache.org/thread/vsn21hmmv1km2ybl3hq00p7nhrp1dnsw Cheers, Till

[RESULT][VOTE] FLIP-196: Source API stability guarantees

Hi everyone, I am happy to announce that FLIP-196 [1] has been accepted [2]. Binding votes: - Ingo Bürk - Chesnay Schepler - Till Rohrmann [1] https://cwiki.apache.org/confluence/x/IJeqCw [2] https://lists.apache.org/thread/716674y8xbpvb5doxzw8tkbl2jhccjg8 Cheers, Till

Re: [VOTE] FLIP-196: Source API stability guarantees

Thanks everyone for voting. I am closing this vote now and will post the results in a separate email. Cheers, Till On Mon, Dec 13, 2021 at 9:28 AM Chesnay Schepler wrote: > +1 > > On 10/12/2021 18:07, Till Rohrmann wrote: > > Hi everyone, > > > > I'd like to start a vote on FLIP-196: Source API

Re: [VOTE] FLIP-197: API stability graduation process

Thanks everyone for the votes. I am closing the vote now and post the result in a separate email. Cheers, Till On Mon, Dec 13, 2021 at 9:28 AM Chesnay Schepler wrote: > +1 > > On 10/12/2021 18:09, Till Rohrmann wrote: > > Hi everyone, > > > > I'd like to start a vote on FLIP-197: API stability

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

The tag has been fixed. The problem did not affect the releases in any way. On 16/12/2021 11:39, Nicolaus Weidner wrote: Thanks for doing the releases, Chesnay! It looks like the release-1.11.6 tag incorrectly points to the 1.12.7 release commit: $ git show release-1.11.6 tag release-1.11.6

[jira] [Created] (FLINK-25344) flink kafka connector Property group.id is required when using committed offset for offsets initializer

jingzi created FLINK-25344: -- Summary: flink kafka connector Property group.id is required when using committed offset for offsets initializer Key: FLINK-25344 URL: https://issues.apache.org/jira/browse/FLINK-25344

Re: [VOTE] Release 1.11.6/1.12.7/1.13.5/1.14.2, release candidate #1

Thanks for doing the releases, Chesnay! It looks like the release-1.11.6 tag incorrectly points to the 1.12.7 release commit: $ git show release-1.11.6 tag release-1.11.6 Tagger: Chesnay Schepler Date: Thu Dec 16 04:12:44 2021 +0100 Release Flink 1.11.6 <...> commit 88d9950bb2c912772c46c676

[jira] [Created] (FLINK-25343) HBaseConnectorITCase.testTableSourceSinkWithDDL fail on azure

Yun Gao created FLINK-25343: --- Summary: HBaseConnectorITCase.testTableSourceSinkWithDDL fail on azure Key: FLINK-25343 URL: https://issues.apache.org/jira/browse/FLINK-25343 Project: Flink Issue Ty

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

Hi Till, Good point, in the scenario with the blocking keyed exchange between the writer and committer my idea is to make the committer effectively the global committer. With Sink V2 there is no real difference anymore between the committer and global committer. You are right that everything after

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

Hi Fabian, quick question on your comment 3. If there is a pipelined data exchange with a keyBy between the writers/committers and the component that does the global commit, then there will only be a single failover region. So is it correct that you assumed blocking data exchanges for the scenario

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

Hi David, I think such an utility can be helpful. I would suggest adding something like this once it is needed by a component. Currently, I think only the BlobServer might be susceptible to this problem because we don't fsync the written bytes and then don't use an atomic rename operation. If we

[jira] [Created] (FLINK-25342) DataStream.sinkTo will not add sink to the sinks field in the StreamGraph

Hang Ruan created FLINK-25342: - Summary: DataStream.sinkTo will not add sink to the sinks field in the StreamGraph Key: FLINK-25342 URL: https://issues.apache.org/jira/browse/FLINK-25342 Project: Flink

[jira] [Created] (FLINK-25341) Improve casting structured type to STRING

Timo Walther created FLINK-25341: Summary: Improve casting structured type to STRING Key: FLINK-25341 URL: https://issues.apache.org/jira/browse/FLINK-25341 Project: Flink Issue Type: Sub-tas

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

Hi Yun, Thanks for your fast feedback. Let me clarify your points. 1. We solve it by using StreamExchangeMode.BATCH before any exchange. That obviously doesn’t help with lost TM but we would need to employ HA storage for that. Same issue as now and orthogonal. 2. Extending V1 with V2 or vice ver

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

Hi Till, thanks for drafting this FLIP, I think it's really a valuable improvement. Agreed with Yang, that YARN / k8s implementation should be out of scope of this FLIP. Just few notes on the possible integrations: For k8s, I think we can also benefit from this FLIP without StatefulSet. If the p

[jira] [Created] (FLINK-25340) Cleanup temporary directories from JM container before tearing down FlinkContainers

Qingsheng Ren created FLINK-25340: - Summary: Cleanup temporary directories from JM container before tearing down FlinkContainers Key: FLINK-25340 URL: https://issues.apache.org/jira/browse/FLINK-25340