[SUMMARY] Flink 1.16 release sync of 2022-10-04

2022-10-04 Thread Xingbo Huang
I would like to give you a brief update of the Flink 1.16 release sync meating of 2022-10-04. *We have prepared rc1 last Friday. Looking forward to your testing.* *This week is Chinese National Day holiday, so the release time maybe a week later than expected* *We still have some critical test s

Re: [DISCUSS] FLIP-264 Extract BaseCoordinatorContext

2022-10-04 Thread Jing Ge
Thanks for bringing this up. It looks overall good! One small thing, you might want to write all content on the wiki page instead of linking to a google doc. The reason is that some people might not be able to access the google doc. Best regards, Jing On Tue, Oct 4, 2022 at 3:57 AM gang ye wrote

[jira] [Created] (FLINK-29499) DispatcherOperationCaches should implement AutoCloseableAsync

2022-10-04 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-29499: Summary: DispatcherOperationCaches should implement AutoCloseableAsync Key: FLINK-29499 URL: https://issues.apache.org/jira/browse/FLINK-29499 Project: Flink

[jira] [Created] (FLINK-29500) InitializeOnMaster uses wrong parallelism with AdaptiveScheduler

2022-10-04 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-29500: Summary: InitializeOnMaster uses wrong parallelism with AdaptiveScheduler Key: FLINK-29500 URL: https://issues.apache.org/jira/browse/FLINK-29500 Project: Fli

Re: [VOTE] Release 1.16.0, release candidate #1

2022-10-04 Thread Matthias Pohl
+1 (non-binding) * Downloaded artifacts * Verified checksums/GPG signatures * Compared checkout with provided sources * Verified pom file versions * Went over NOTICE file/pom files changes without finding anything suspicious * Build Flink from sources * Deployed standalone session cluster and ran

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Martijn Visser
Hi Marton, You're making a good point, I originally wanted to include already the User mailing list to get their feedback but forgot to do so. I'll do some more outreach via other channels as well. @Users of Flink, I've made a proposal to deprecate and remove Scala API support in a future version

[jira] [Created] (FLINK-29501) Allow overriding JobVertex parallelisms during job submission

2022-10-04 Thread Maximilian Michels (Jira)
Maximilian Michels created FLINK-29501: -- Summary: Allow overriding JobVertex parallelisms during job submission Key: FLINK-29501 URL: https://issues.apache.org/jira/browse/FLINK-29501 Project: Fl

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Jing Ge
Hi Martijn, Thanks for bringing this up. It is generally a great idea, so +1. Since both scala extension projects mentioned in the FLIP are still very young and I don't think they will attract more scala developers as Flink could just because they are external projects. It will be a big issue for

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Yaroslav Tkachenko
Hi Martijn, As a Scala user, this change would affect me a lot and I'm not looking forward to rewriting my codebase, and it's not even a very large one :) I'd like to suggest supporting Java 17 as a prerequisite ( https://issues.apache.org/jira/browse/FLINK-15736). Things like switch expressions

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Martijn Visser
Hi Yaroslav, Thanks for the feedback, that's much appreciated! Regarding Java 17 as a prerequisite, we would have to break compatibility already since Scala 2.12.7 doesn't compile on Java 17 [1]. Given that we can only remove Scala APIs with the next major Flink (2.0) version, would that still im

Re: [DISCUSS] FLIP-264 Extract BaseCoordinatorContext

2022-10-04 Thread Steven Wu
Jing, thanks a lot for your reply. The linked google doc is not for this FLIP, which is fully documented in the wiki page. The linked google doc is the design doc to introduce shuffling in Flink Iceberg sink, which motivated this FLIP proposal so that the shuffle coordinator can leverage the introd

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Yaroslav Tkachenko
Hi Martijn, The 2.0 argument makes sense (I agree it's easier to introduce more breaking changes in one major release), but I think my comment about Java 17 also makes sense in this case: 1) easier to introduce because breaking changes are possible 2) you'd need to give some syntax sugar as an alt

[jira] [Created] (FLINK-29502) Update the Hadoop implementation for filesystems to 3.3.4

2022-10-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-29502: -- Summary: Update the Hadoop implementation for filesystems to 3.3.4 Key: FLINK-29502 URL: https://issues.apache.org/jira/browse/FLINK-29502 Project: Flink

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Martijn Visser
Hi Yaroslav, If I could summarize your suggestion, would it mean that you would only be in favour of dropping Scala API support if we introduce Java 17 support exactly at the same time (say Flink 2.0). I was first thinking that an alternative would be to have a Flink 2.0 which supports Java 17 whi

Support avro-confluent format for FileSystem SQL connector

2022-10-04 Thread liuxiangcao
Hi Flink developer community, According to flink doc, avro-confluent([1]) is only supported for kafka sql connector and upsert kafka sql connector. I'm wondering if there is any reason this format is not supported for Filesystem sql connector ([2]) ? We are looking to use FileSystem sink to wri

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.2.0, release candidate #2

2022-10-04 Thread Gyula Fóra
+1 (binding) - Verified Helm repo works as expected, points to correct image tag, build, version - Verified examples + checked operator logs everything looks as expected - Verified hashes, signatures and source release contains no binaries - Ran built-in tests, built jars + docker image from sourc

Re: Support avro-confluent format for FileSystem SQL connector

2022-10-04 Thread Martijn Visser
Hi, I'm wondering if the documentation is correct, because the documentation also mentions the option to use 'avro-confluent.subject' which refers to using it with filesystem [1]. Have you tried to use this already? Did you get an error message? Best regards, Martijn [1] https://nightlies.apac

[jira] [Created] (FLINK-29503) [OpenAPI] enum `backpressure-level` should be backpressurelevel so code generator recognizes it

2022-10-04 Thread Tiger (Apache) Wang (Jira)
Tiger (Apache) Wang created FLINK-29503: --- Summary: [OpenAPI] enum `backpressure-level` should be backpressurelevel so code generator recognizes it Key: FLINK-29503 URL: https://issues.apache.org/jira/browse/

[jira] [Created] (FLINK-29504) [OpenAPI] `application/x-java-archive` should be followed by a structure instead of `{}`

2022-10-04 Thread Tiger (Apache) Wang (Jira)
Tiger (Apache) Wang created FLINK-29504: --- Summary: [OpenAPI] `application/x-java-archive` should be followed by a structure instead of `{}` Key: FLINK-29504 URL: https://issues.apache.org/jira/browse/FLINK-2

[jira] [Created] (FLINK-29505) [OpenAPI] Need `operationId` on certain operations

2022-10-04 Thread Tiger (Apache) Wang (Jira)
Tiger (Apache) Wang created FLINK-29505: --- Summary: [OpenAPI] Need `operationId` on certain operations Key: FLINK-29505 URL: https://issues.apache.org/jira/browse/FLINK-29505 Project: Flink

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

2022-10-04 Thread Yaroslav Tkachenko
Hey Martijn, Well, as a user I think that Scala API still adds a tremendous value, with all its issues. But I'm not a committer and I don't know what effort it takes to keep maintaining it... so I prepare for the worst :) Regarding the proposed timeline, I don't know all the specifics around brea

[DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-04 Thread Martijn Visser
Hi everyone, Little over a year ago a discussion thread was opened on changing the minimal supported version of Hadoop and bringing that to 2.8.5. [1] In this discussion thread, I would like to propose to bring that minimal supported version of Hadoop to 2.10.2. Hadoop 2.8.5 is vulnerable for mul