[jira] [Created] (FLINK-30800) Add doc entry how to re-generate java doc

2023-01-27 Thread Gabor Somogyi (Jira)
Gabor Somogyi created FLINK-30800: - Summary: Add doc entry how to re-generate java doc Key: FLINK-30800 URL: https://issues.apache.org/jira/browse/FLINK-30800 Project: Flink Issue Type: New F

Re: Reworking the Rescale API

2023-01-27 Thread Chesnay Schepler
The adaptive scheduler only supports streaming jobs. That's the biggest limitation that probably won't be fixed anytime soon. The goal was though to make the adaptive scheduler the default for streaming jobs eventually. it was very much meant as a better version of the default scheduler for stre

Re: Reworking the Rescale API

2023-01-27 Thread David Morávek
> > The adaptive scheduler only supports streaming jobs. That's the biggest > limitation that probably won't be fixed anytime soon. Since FLIP-283 [1] has been accepted, I think this limitation might have already been addressed to a certain extent. I'd be completely fine with having a separate sc

Re: Reworking the Rescale API

2023-01-27 Thread Gyula Fóra
Thank you @Chesnay Schepler @David Morávek I think in that case our primary goal should be to make sure that streaming jobs always use the adaptive scheduler. Also then it makes perfect sense to build the rescale api improvements for that specifically. However we should have a clear plan to mak

Re: Reworking the Rescale API

2023-01-27 Thread Gyula Fóra
Also @David Morávek @Chesnay Schepler It would be great if you could update the respective docs page before publishing your improvement FLIPS about the adaptive scheduler: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/elastic_scaling/ I think many of the confusion/

Re: Reworking the Rescale API

2023-01-27 Thread Chesnay Schepler
It is certainly true that the messaging around the AS/reactive mode wasn't good. In part this happened because initially we only intended to advertise reactive mode (at the time), and only later figured that the AS on it's own could already be useful too. That being said, I'm not sure how to

[jira] [Created] (FLINK-30801) Migrate LeaderElection-related unit tests to JUnit5

2023-01-27 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-30801: - Summary: Migrate LeaderElection-related unit tests to JUnit5 Key: FLINK-30801 URL: https://issues.apache.org/jira/browse/FLINK-30801 Project: Flink Issue T

Re: Confluent Avro and Debezium formats - default schema name can be incompatible with registered schema name

2023-01-27 Thread Dawid Wysakowicz
Hi Fruzsina, I think this is a valid issue we should try to solve. A different approach I am thinking about is that we could actually add an option to provide an entire avro schema to use. Something like: `avro-confluent.schema` which we would validate it maps properly to the schema of the ta

Re: [VOTE] Externalize AWS formats to flink-connector-aws

2023-01-27 Thread Danny Cranmer
Hello Hong, thanks for driving this. +1 on the proposal and the voting schema looks good. Thanks On Thu, Jan 26, 2023 at 5:15 PM Teoh, Hong wrote: > Hi all, > > As discussed in the discussion thread [1], I would like to propose we > externalize flink-avro-glue-schema-registry and > flink-json-

Re: [VOTE] Release flink-connector-gcp-pubsub v3.0.0, release candidate #1

2023-01-27 Thread Danny Cranmer
Hey Martijn, thanks for driving yet another connector release. Before I cast my vote I have a couple of queries: * Contents of dist. Why are we publishing the e2e tests jar [1]? We do not typically do this, but I see the old GCP e2e tests are published [2], however not to maven central. * NOTICE a

[jira] [Created] (FLINK-30802) Improve SplitReader#fetch() documentation

2023-01-27 Thread Etienne Chauchot (Jira)
Etienne Chauchot created FLINK-30802: Summary: Improve SplitReader#fetch() documentation Key: FLINK-30802 URL: https://issues.apache.org/jira/browse/FLINK-30802 Project: Flink Issue Type:

Re: [VOTE] Release 1.16.1, release candidate #1

2023-01-27 Thread Teoh, Hong
+1 (non-binding) * Hashes and Signatures look good * All required files on dist.apache.org * Source archive builds using maven * Started packaged example WordCountSQLExample job. Cheers, Hong On 27/01/2023, 07:52, "Gabor Somogyi" wrote: CAUTION: This email originated from outside of the

[jira] [Created] (FLINK-30803) PyFlink mishandles script dependencies

2023-01-27 Thread Nuno Afonso (Jira)
Nuno Afonso created FLINK-30803: --- Summary: PyFlink mishandles script dependencies Key: FLINK-30803 URL: https://issues.apache.org/jira/browse/FLINK-30803 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-30804) flink-kubernetes-operator helm chart - allow specifying operator pod resources requests/limits

2023-01-27 Thread Vincent Chenal (Jira)
Vincent Chenal created FLINK-30804: -- Summary: flink-kubernetes-operator helm chart - allow specifying operator pod resources requests/limits Key: FLINK-30804 URL: https://issues.apache.org/jira/browse/FLINK-30804

Re: [VOTE] Release 1.16.1, release candidate #1

2023-01-27 Thread Gyula Fóra
+1 (binding) * Verified signatures and hashes * Built from source, ran some examples on Kubernetes * Verified NOTICE and LICENSE files Gyula On Fri, Jan 27, 2023 at 3:51 PM Teoh, Hong wrote: > +1 (non-binding) > > * Hashes and Signatures look good > * All required files on dist.apache.org > *

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2023-01-27 Thread John Roesler
Hello Shammon and all, Thanks for this FLIP! I've been working toward this kind of global consistency across large scale data infrastructure for a long time, and it's fantastic to see a high-profile effort like this come into play. I have been lurking in the discussion for a while and delaying

[jira] [Created] (FLINK-30805) SplitEnumerator#handleSplitRequest() should be called automatically

2023-01-27 Thread Etienne Chauchot (Jira)
Etienne Chauchot created FLINK-30805: Summary: SplitEnumerator#handleSplitRequest() should be called automatically Key: FLINK-30805 URL: https://issues.apache.org/jira/browse/FLINK-30805 Project:

[jira] [Created] (FLINK-30806) Bump protobuf-java dependency version for aws connectors

2023-01-27 Thread Aleksandr Pilipenko (Jira)
Aleksandr Pilipenko created FLINK-30806: --- Summary: Bump protobuf-java dependency version for aws connectors Key: FLINK-30806 URL: https://issues.apache.org/jira/browse/FLINK-30806 Project: Flink

Re: [VOTE] Externalize AWS formats to flink-connector-aws

2023-01-27 Thread Jing Ge
+1 Thanks for driving it! Best regards, Jing On Fri, Jan 27, 2023 at 2:36 PM Danny Cranmer wrote: > Hello Hong, thanks for driving this. > > +1 on the proposal and the voting schema looks good. > > Thanks > > On Thu, Jan 26, 2023 at 5:15 PM Teoh, Hong > wrote: > > > Hi all, > > > > As discusse

[jira] [Created] (FLINK-30807) State Processor API - Overwrite Support for Savepoints

2023-01-27 Thread Rion Williams (Jira)
Rion Williams created FLINK-30807: - Summary: State Processor API - Overwrite Support for Savepoints Key: FLINK-30807 URL: https://issues.apache.org/jira/browse/FLINK-30807 Project: Flink Issu

Re: [DISCUSS] FLIP-276: Data Consistency of Streaming and Batch ETL in Flink and Table Store

2023-01-27 Thread Vasiliki Papavasileiou
Hi Shammon, Thank you for opening this FLIP which is very interesting and such an important feature to add to the Flink ecosystem. I have a couple of suggestions/questions: - Consistency is a very broad term with different meanings. There are many variations between the two extremes

Re: Reworking the Rescale API

2023-01-27 Thread ConradJam
Sorry I'm late to join discuss, I've gleaned a lot of useful information from you guys *@max* - when user repartition, we still need to restart the job, can we try to do this part of the work internally instead of externally, as *@konstantin* said only trigger rescaling when the checkpoi

[jira] [Created] (FLINK-30808) MultipleInputITCase failed with AdaptiveBatch Scheduler

2023-01-27 Thread Junrui Li (Jira)
Junrui Li created FLINK-30808: - Summary: MultipleInputITCase failed with AdaptiveBatch Scheduler Key: FLINK-30808 URL: https://issues.apache.org/jira/browse/FLINK-30808 Project: Flink Issue Type:

[Discuss] Conventions on driving FLIPs towards consensus

2023-01-27 Thread Xintong Song
Hi devs, Recently, a discussion about how to drive FLIPs towards consensus has taken place on the private@ mailing list. While many PMC members have already shared their opinions, we believe for this topic the opinions of other committers / contributors are equally important. Therefore, we are mov

Re: [VOTE] FLIP-290: Operator state compression (FLINK-30113)

2023-01-27 Thread Yuan Mei
+1 binding Best Yuan On Sat, Jan 21, 2023 at 12:49 AM Rui Fan <1996fan...@gmail.com> wrote: > +1 (no-binding) > > Best > Rui Fan > > On Fri, Jan 20, 2023 at 10:46 PM ConradJam wrote: > > > +1 (no-binding) > > thanks driving it > > > > Martijn Visser 于2023年1月20日周五 21:16写道: > > > > > +1 (binding

[jira] [Created] (FLINK-30809) flink-connector-elasticsearch7 updates data pipeline does not work

2023-01-27 Thread iduanyingjie (Jira)
iduanyingjie created FLINK-30809: Summary: flink-connector-elasticsearch7 updates data pipeline does not work Key: FLINK-30809 URL: https://issues.apache.org/jira/browse/FLINK-30809 Project: Flink

[jira] [Created] (FLINK-30810) Rework the CliClientITCase to extend AbstractSqlGatewayStatementITCase

2023-01-27 Thread Shengkai Fang (Jira)
Shengkai Fang created FLINK-30810: - Summary: Rework the CliClientITCase to extend AbstractSqlGatewayStatementITCase Key: FLINK-30810 URL: https://issues.apache.org/jira/browse/FLINK-30810 Project: Fli

Re: [Discuss] Conventions on driving FLIPs towards consensus

2023-01-27 Thread Jark Wu
Thank Xintong for starting the discussion in PMC and bringing the summary here. The summary looks good to me. Unresponsive reviewing has happened numerous times in the previous FLIPs. I think the conventions are very valuable for the community to move forward efficiently and avoid potential confli

Re: [DISCUSS] FLIP-264 Extract BaseCoordinatorContext

2023-01-27 Thread Jark Wu
Thank Steven for the explanation. It sounds good to me to implement the shuffle operator in the Iceberg project first. We can contribute it to Flink DataStream in the future if other projects/connectors also need it. Best, Jark On Wed, 18 Jan 2023 at 02:11, Steven Wu wrote: > Jark, > > We wer