[jira] [Created] (FLINK-25205) Optimize SinkUpsertMaterializer

Jingsong Lee created FLINK-25205: Summary: Optimize SinkUpsertMaterializer Key: FLINK-25205 URL: https://issues.apache.org/jira/browse/FLINK-25205 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-25204) Spell one more as in the DataStream interval join docs

tartarus created FLINK-25204: Summary: Spell one more as in the DataStream interval join docs Key: FLINK-25204 URL: https://issues.apache.org/jira/browse/FLINK-25204 Project: Flink Issue Type: Im

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

Maybe we can add support for this case : when an exec node is changed in 1.16, but is compatible with 1.15, we can use the node of 1.16 to deserialize the plan of 1.15. By this way, we don't need to fork the code if the change is compatible, and can avoid fork code frequently. Best, Wenlo

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

hi, Timo, I would prefer to update the version every time we change the state layer too. It could be possible that we change the exec node in 2 steps: First, we add a newStateLayout because of some improvement in state, in order to keep compatibility we may still keep the old state for the first

Re: [DISCUSS] Drop Zookeeper 3.4

FYI: We(Alibaba) are widely using ZooKeeper 3.5.5 for all the YARN and some K8s Flink high-available applications. Best, Yang Chesnay Schepler 于2021年12月7日周二 上午2:22写道： > Current users of ZK 3.4 and below would need to upgrade their Zookeeper > installation that is used by Flink to 3.5+. > > Wh

Re: [VOTE] FLIP-194: Introduce the JobResultStore

+1 (binding) Thanks, Zhu Yang Wang 于2021年12月7日周二 上午9:52写道： > +1 (non-binding) > > Best, > Yang > > Xintong Song 于2021年12月7日周二 上午9:38写道： > > > +1 (binding) > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Mon, Dec 6, 2021 at 5:02 PM Till Rohrmann > wrote: > > > > > +1 (binding) > > >

Re: [VOTE] FLIP-194: Introduce the JobResultStore

+1 (non-binding) Best, Yang Xintong Song 于2021年12月7日周二 上午9:38写道： > +1 (binding) > > Thank you~ > > Xintong Song > > > > On Mon, Dec 6, 2021 at 5:02 PM Till Rohrmann wrote: > > > +1 (binding) > > > > Cheers, > > Till > > > > On Mon, Dec 6, 2021 at 10:00 AM David Morávek wrote: > > > > > Hi eve

Re: [VOTE] FLIP-194: Introduce the JobResultStore

+1 (binding) Thank you~ Xintong Song On Mon, Dec 6, 2021 at 5:02 PM Till Rohrmann wrote: > +1 (binding) > > Cheers, > Till > > On Mon, Dec 6, 2021 at 10:00 AM David Morávek wrote: > > > Hi everyone, > > > > I'd like to open a vote on FLIP-194: Introduce the JobResultStore [1] > which > > ha

Re: [VOTE] Deprecate Java 8 support

+1 (non-binding) Op ma 6 dec. 2021 om 19:58 schreef Ingo Bürk > Before more people let me know, let me update my vote to > > +1 (binding) > > > (In all seriousness, thanks for the reminders!) > > On Mon, Dec 6, 2021, 16:54 Ingo Bürk wrote: > > > +1 (non-binding) > > > > > > Ingo > > > > On Mon,

Re: [VOTE] Deprecate Java 8 support

Before more people let me know, let me update my vote to +1 (binding) (In all seriousness, thanks for the reminders!) On Mon, Dec 6, 2021, 16:54 Ingo Bürk wrote: > +1 (non-binding) > > > Ingo > > On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler > wrote: > >> Hello, >> >> after recent discussi

[jira] [Created] (FLINK-25203) Implement duplicating for aliyun

Dawid Wysakowicz created FLINK-25203: Summary: Implement duplicating for aliyun Key: FLINK-25203 URL: https://issues.apache.org/jira/browse/FLINK-25203 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25202) Implement duplicating for azure

Dawid Wysakowicz created FLINK-25202: Summary: Implement duplicating for azure Key: FLINK-25202 URL: https://issues.apache.org/jira/browse/FLINK-25202 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25201) Implement duplicating for gcs

Dawid Wysakowicz created FLINK-25201: Summary: Implement duplicating for gcs Key: FLINK-25201 URL: https://issues.apache.org/jira/browse/FLINK-25201 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25200) Implement duplicating for s3 filesystem

Dawid Wysakowicz created FLINK-25200: Summary: Implement duplicating for s3 filesystem Key: FLINK-25200 URL: https://issues.apache.org/jira/browse/FLINK-25200 Project: Flink Issue Type: S

Re: [DISCUSS] Drop Zookeeper 3.4

Current users of ZK 3.4 and below would need to upgrade their Zookeeper installation that is used by Flink to 3.5+. Whether K8s users are affected depends on whether they use ZK or not. If they do, see above, otherwise they are not affected at all. On 06/12/2021 18:49, Arvid Heise wrote: Cou

Re: [DISCUSS] Drop Zookeeper 3.4

Could someone please help me understand the implications of the upgrade? As far as I understood this upgrade would only affect users that have a zookeeper shared across multiple services, some of which require ZK 3.4-? A workaround for those users would be to run two ZKs with different versions, e

Re: [VOTE] Deprecate Java 8 support

+1 (binding) On Mon, Dec 6, 2021 at 5:43 PM Timo Walther wrote: > +1 (binding) > > Thanks, > Timo > > On 06.12.21 17:28, David Morávek wrote: > > +1 (non-binding) > > > > On Mon, Dec 6, 2021 at 4:55 PM Ingo Bürk wrote: > > > >> +1 (non-binding) > >> > >> > >> Ingo > >> > >> On Mon, Dec 6, 2021

[jira] [Created] (FLINK-25199) fromValues does not emit final MAX watermark

Timo Walther created FLINK-25199: Summary: fromValues does not emit final MAX watermark Key: FLINK-25199 URL: https://issues.apache.org/jira/browse/FLINK-25199 Project: Flink Issue Type: Bug

Re: [VOTE] Deprecate Java 8 support

+1 (binding) Thanks, Timo On 06.12.21 17:28, David Morávek wrote: +1 (non-binding) On Mon, Dec 6, 2021 at 4:55 PM Ingo Bürk wrote: +1 (non-binding) Ingo On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler wrote: Hello, after recent discussions on the dev

[jira] [Created] (FLINK-25198) add document about how to debug with the name and description

Wenlong Lyu created FLINK-25198: --- Summary: add document about how to debug with the name and description Key: FLINK-25198 URL: https://issues.apache.org/jira/browse/FLINK-25198 Project: Flink

Re: [VOTE] Deprecate Java 8 support

+1 (non-binding) On Mon, Dec 6, 2021 at 4:55 PM Ingo Bürk wrote: > +1 (non-binding) > > > Ingo > > On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler > wrote: > > > Hello, > > > > after recent discussions on the dev > > and > > use

Re: [VOTE] Deprecate Java 8 support

+1 (non-binding) Ingo On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler wrote: > Hello, > > after recent discussions on the dev > and > user > mailing list to dep

Re: [VOTE] Deprecate Java 8 support

+1 On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler wrote: > Hello, > > after recent discussions on the dev > and > user > mailing list to deprecate Java 8 support

[VOTE] Deprecate Java 8 support

Hello, after recent discussions on the dev and user mailing list to deprecate Java 8 support, with a general consensus in favor of it, I would now like tod o a

[jira] [Created] (FLINK-25197) Using Statefun RequestReplyFunctionBuilder fails with Java 8 date/time type `java.time.Duration` not supported by default: add Module "org.apache.flink.shaded.jackson2.c

Galen Warren created FLINK-25197: Summary: Using Statefun RequestReplyFunctionBuilder fails with Java 8 date/time type `java.time.Duration` not supported by default: add Module "org.apache.flink.shaded.jackson2.com.fasterxml.jackson.datatype:jack

[jira] [Created] (FLINK-25196) Add Documentation for Data Sink

ZhuoYu Chen created FLINK-25196: --- Summary: Add Documentation for Data Sink Key: FLINK-25196 URL: https://issues.apache.org/jira/browse/FLINK-25196 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-25195) Use duplicating API for shared artefacts in RocksDB snapshots

Dawid Wysakowicz created FLINK-25195: Summary: Use duplicating API for shared artefacts in RocksDB snapshots Key: FLINK-25195 URL: https://issues.apache.org/jira/browse/FLINK-25195 Project: Flink

[jira] [Created] (FLINK-25194) Implement an API for duplicating artefacts

Dawid Wysakowicz created FLINK-25194: Summary: Implement an API for duplicating artefacts Key: FLINK-25194 URL: https://issues.apache.org/jira/browse/FLINK-25194 Project: Flink Issue Type

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

Hi Godfrey, > design makes thing more complex. Yes, the design might be a bit more complex. But operator migration is way easier than ExecNode migration at a later point in time for code maintenance. We know that ExecNodes can become pretty complex. Even though we have put a lot of code into

[jira] [Created] (FLINK-25193) Document claim & no-claim mode

Dawid Wysakowicz created FLINK-25193: Summary: Document claim & no-claim mode Key: FLINK-25193 URL: https://issues.apache.org/jira/browse/FLINK-25193 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25192) Implement proper no-claim mode support

Dawid Wysakowicz created FLINK-25192: Summary: Implement proper no-claim mode support Key: FLINK-25192 URL: https://issues.apache.org/jira/browse/FLINK-25192 Project: Flink Issue Type: Su

Re: [DISCUSS] Drop Zookeeper 3.4

ping @users; any input on how this would affect you is highly appreciated. On 25/11/2021 22:39, Chesnay Schepler wrote: I included the user ML in the thread. @users Are you still using Zookeeper 3.4? If so, were you planning to upgrade Zookeeper in the near future? I'm not sure about ZK comp

Re: [DISCUSS] Deprecate Java 8 support

> @Piotr I guess that could be investigated along with an update of flink-benchmarks to run on Java 11 as well... I would second Nico's idea to run our benchmarks on Java 11 to see what has changed compared to Java 8. Best, Piotrek śr., 1 gru 2021 o 11:23 wenlong.lwl napisał(a): > Thanks for t

[jira] [Created] (FLINK-25191) Skip savepoints for recovery

Dawid Wysakowicz created FLINK-25191: Summary: Skip savepoints for recovery Key: FLINK-25191 URL: https://issues.apache.org/jira/browse/FLINK-25191 Project: Flink Issue Type: Sub-task

Re: [VOTE] FLIP-193: Snapshots ownership

Dear devs, Thank you all for the participation in all of the comments in the discussion! I'm happy to announce that we have unanimously approved this FLIP. There are 7 approving votes of which 7 are binding: * Konstantin Knauf (binding) * Till Rohrmann (binding) * Yun Gao (binding) * Ro

[DISCUSS] Strong read-after-write consistency of Flink FileSystems

Hi Everyone, as outlined in FLIP-194 discussion [1], for the future directions of Flink HA services, I'd like to verify my thoughts around guarantees of the distributed filesystems used with Flink. Currently some of the services (*JobGraphStore*, *CompletedCheckpointStore*) are implemented using

[jira] [Created] (FLINK-25190) The number of TaskManagers whose status is Pending should be reported

john created FLINK-25190: Summary: The number of TaskManagers whose status is Pending should be reported Key: FLINK-25190 URL: https://issues.apache.org/jira/browse/FLINK-25190 Project: Flink Issue

Re: [DISCUSS] FLIP-194: Introduce the JobResultStore

> > I also hope that we could remove all the pointers in the HA store(ZK, > ConfigMap) in the future. I'll open a new thread with {user,dev}@f.a.o to verify the thoughts around strong-read-after consistency for FileSystems. If that goes well I can see it as one of the possible topics for 1.16 ;)

Re: [DISCUSS] FLIP-197: API stability graduation process

Hi Till, thanks for starting this discussion. I think this topic should have been discussed way earlier. I have two questions: 1) It might be an implementation detail but where do you expect this `FlinkVersion` to be located? This is actually a quite important class that also needs to be mad

Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk

Congratulations, Ingo! Regards, Roman On Mon, Dec 6, 2021 at 11:05 AM Yang Wang wrote: > > Congratulations, Ingo! > > Best, > Yang > > Sergey Nuyanzin 于2021年12月6日周一 下午3:35写道： > > > Congratulations, Ingo! > > > > On Mon, Dec 6, 2021 at 7:32 AM Leonard Xu wrote: > > > > > Congratulations, Ingo!

Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl

Congratulations, Matthias! Regards, Roman On Mon, Dec 6, 2021 at 11:04 AM Yang Wang wrote: > > Congratulations, Matthias! > > Best, > Yang > > Sergey Nuyanzin 于2021年12月6日周一 下午3:35写道： > > > Congratulations, Matthias! > > > > On Mon, Dec 6, 2021 at 7:33 AM Leonard Xu wrote: > > > > > Congratula

Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk

Congratulations, Ingo! Best, Yang Sergey Nuyanzin 于2021年12月6日周一 下午3:35写道： > Congratulations, Ingo! > > On Mon, Dec 6, 2021 at 7:32 AM Leonard Xu wrote: > > > Congratulations, Ingo! Well Deserved. > > > > Best, > > Leonard > > > > > 2021年12月3日 下午11:24，Ingo Bürk 写道： > > > > > > Thank you everyo

Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl

Congratulations, Matthias! Best, Yang Sergey Nuyanzin 于2021年12月6日周一 下午3:35写道： > Congratulations, Matthias! > > On Mon, Dec 6, 2021 at 7:33 AM Leonard Xu wrote: > > > Congratulations Matthias! > > > > Best, > > Leonard > > > 2021年12月3日 下午11:23，Matthias Pohl 写道： > > > > > > Thank you! I'm looki

Re: [DISCUSS] FLIP-194: Introduce the JobResultStore

Thanks for the fruitful discussion. I also hope that we could remove all the pointers in the HA store(ZK, ConfigMap) in the future. After then, we only rely on the ZK/ConfigMap for leader election/retrieval. Best, Yang David Morávek 于2021年12月6日周一 下午4:57写道： > as all of the concerns seems to be

[jira] [Created] (FLINK-25189) Update Elasticsearch Sinks to latest minor versions

Alexander Preuss created FLINK-25189: Summary: Update Elasticsearch Sinks to latest minor versions Key: FLINK-25189 URL: https://issues.apache.org/jira/browse/FLINK-25189 Project: Flink I

Re: [DISCUSS] Change some default config values of blocking shuffle

Hi Till, Thanks for your feedback. >>> How will our tests be affected by these changes? Will Flink require more resources and, thus, will it risk destabilizing our testing infrastructure? There are some tests that need to be adjusted, for example, BlockingShuffleITCase. For other tests, theoreti

[jira] [Created] (FLINK-25188) Cannot install PyFlink in M1 CPU

Ada Wong created FLINK-25188: Summary: Cannot install PyFlink in M1 CPU Key: FLINK-25188 URL: https://issues.apache.org/jira/browse/FLINK-25188 Project: Flink Issue Type: Bug Components

Re: [DISCUSS] FLIP-197: API stability graduation process

Ok, then lets increase the graduation period to 2 releases. If we see that this is super easy for us to do, then we can shorten it in the future. Cheers, Till On Mon, Dec 6, 2021 at 9:54 AM Chesnay Schepler wrote: > Given that you can delay the graduation if there is a good reason for > it, we

Re: [DISCUSS] FLIP-196: Source API stability guarantees

Hi Till, seems I misunderstood it then; thanks for the clarification! And yes, with that I would fully agree. Ingo On Mon, Dec 6, 2021 at 9:59 AM Till Rohrmann wrote: > Hi Ingo, > > No, the added method can have a weaker stability guarantee as long as the > user does not have to implement it.

Re: [VOTE] FLIP-194: Introduce the JobResultStore

+1 (binding) Cheers, Till On Mon, Dec 6, 2021 at 10:00 AM David Morávek wrote: > Hi everyone, > > I'd like to open a vote on FLIP-194: Introduce the JobResultStore [1] which > has been > discussed in this thread [2]. > The vote will be open for at least 72 hours unless there is an objection or

[VOTE] FLIP-194: Introduce the JobResultStore

Hi everyone, I'd like to open a vote on FLIP-194: Introduce the JobResultStore [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726

Re: [DISCUSS] FLIP-196: Source API stability guarantees

Hi Ingo, No, the added method can have a weaker stability guarantee as long as the user does not have to implement it. In order to give an example the following extension would be ok imo: @Public interface Foobar { @Public int foo(); @Experimental default ExperimentalResult bar()

Re: [DISCUSS] FLIP-190: Support Version Upgrades for Table API & SQL Programs

Hi, Timo, Thanks for the detailed explanation. > We change an operator state of B in Flink 1.16. We perform the change in the > operator of B in a way to support both state layouts. Thus, no need for a new > ExecNode version. I think this design makes thing more complex. 1. If there are multip

Re: [DISCUSS] FLIP-194: Introduce the JobResultStore

as all of the concerns seems to be addressed, I'd like to proceed with the vote to move things forward. Thanks everyone for the feedback, it was really helpful! Best, D. On Wed, Dec 1, 2021 at 6:39 AM Zhu Zhu wrote: > Thanks for the explanation Matthias. The solution sounds good to me. > I hav

Re: [DISCUSS] FLIP-197: API stability graduation process

Given that you can delay the graduation if there is a good reason for it, we should be able to cover that case even if the graduation would happen by default after 1 month. That said, personally I would also be in favor of 2 releases; we see plenty of users not upgrading to every single Flink

[jira] [Created] (FLINK-25187) Apply padding for BINARY()

Marios Trivyzas created FLINK-25187: --- Summary: Apply padding for BINARY() Key: FLINK-25187 URL: https://issues.apache.org/jira/browse/FLINK-25187 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-25186) KafkaDynamicTableFactoryTest and UpsertKafkaDynamicTableFactoryTest fails on AZP

Till Rohrmann created FLINK-25186: - Summary: KafkaDynamicTableFactoryTest and UpsertKafkaDynamicTableFactoryTest fails on AZP Key: FLINK-25186 URL: https://issues.apache.org/jira/browse/FLINK-25186 P

[jira] [Created] (FLINK-25185) StreamFaultToleranceTestBase hangs on AZP

Till Rohrmann created FLINK-25185: - Summary: StreamFaultToleranceTestBase hangs on AZP Key: FLINK-25185 URL: https://issues.apache.org/jira/browse/FLINK-25185 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-25184) Introduce setDeliveryCallback to KafkaWriter

Jingsong Lee created FLINK-25184: Summary: Introduce setDeliveryCallback to KafkaWriter Key: FLINK-25184 URL: https://issues.apache.org/jira/browse/FLINK-25184 Project: Flink Issue Type: Sub-

[jira] [Created] (FLINK-25183) Optimize changelog normalize for the managed table upsert mode

Jingsong Lee created FLINK-25183: Summary: Optimize changelog normalize for the managed table upsert mode Key: FLINK-25183 URL: https://issues.apache.org/jira/browse/FLINK-25183 Project: Flink

Re: [DISCUSS] FLIP-197: API stability graduation process

Hi Till, from my (admittedly limited) experience with how far projects lag behind in terms of Flink versions – yes, the combined time it would take to mature then seems reasonable enough for a sufficient adoption, IMO. Another reason why I think two releases as a default for the last step makes s

[jira] [Created] (FLINK-25182) NoClassDefFoundError of PulsarAdminImpl by using flink-connector-pulsar:1.14 on k8s flink cluster

HeYe created FLINK-25182: Summary: NoClassDefFoundError of PulsarAdminImpl by using flink-connector-pulsar:1.14 on k8s flink cluster Key: FLINK-25182 URL: https://issues.apache.org/jira/browse/FLINK-25182 Pro

[jira] [Created] (FLINK-25181) KafkaSourceITCase.testValueOnlyDeserializer fails on AZP

Till Rohrmann created FLINK-25181: - Summary: KafkaSourceITCase.testValueOnlyDeserializer fails on AZP Key: FLINK-25181 URL: https://issues.apache.org/jira/browse/FLINK-25181 Project: Flink Is