Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #3
+1 (non-binding) - Validated checksum hash - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven and jdk8 - Verified web PR - Check that the jar is built by jdk8 - Check synchronizing schemas and data from mysql to starrocks following the quickstart Best, Hang Qingsheng Ren 于2024年5月11日周六 10:10写道： > Hi everyone, > > Please review and vote on the release candidate #3 for the version 3.1.0 of > Apache Flink CDC, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > **Release Overview** > > As an overview, the release consists of the following: > a) Flink CDC source release to be deployed to dist.apache.org > b) Maven artifacts to be deployed to the Maven Central Repository > > **Staging Areas to Review** > > The staging areas containing the above mentioned artifacts are as follows, > for your review: > * All artifacts for a) can be found in the corresponding dev repository at > dist.apache.org [1], which are signed with the key with fingerprint > A1BD477F79D036D2C30CA7DBCA8AEEC2F6EB040B [2] > * All artifacts for b) can be found at the Apache Nexus Repository [3] > > Other links for your review: > * JIRA release notes [4] > * Source code tag "release-3.1.0-rc3" with commit hash > 5452f30b704942d0ede64ff3d4c8699d39c63863 [5] > * PR for release announcement blog post of Flink CDC 3.1.0 in flink-web [6] > > **Vote Duration** > > The voting time will run for at least 72 hours, adopted by majority > approval with at least 3 PMC affirmative votes. > > Thanks, > Qingsheng Ren > > [1] https://dist.apache.org/repos/dist/dev/flink/flink-cdc-3.1.0-rc3/ > [2] https://dist.apache.org/repos/dist/release/flink/KEYS > [3] https://repository.apache.org/content/repositories/orgapacheflink-1733 > [4] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354387 > [5] https://github.com/apache/flink-cdc/releases/tag/release-3.1.0-rc3 > [6] https://github.com/apache/flink-web/pull/739 >
Re: [DISCUSS] FLIP-444: Native file copy support
> > Note that for both recovery and checkpoints, there are no retring > mechanisms. If any part of downloading or > uploading fails, the job fails over, so actually using such interface > extension would be out of scope of this FLIP. In > that case, maybe if this could be extended in the future without breaking > compatibility we could leave it as a > future improvement? > Thanks for the reply. It makes sense to consider as a future optimization. On Fri, May 10, 2024 at 4:31 PM Piotr Nowojski wrote: > Hi! > > Thanks for your suggestions! > > > I'd prefer a unified one interface > > I have updated the FLIP to take that into account. In this case, I would > also propose to completely drop `DuplicatingFileSystem` in favour of a > basically renamed version of it `PathsCopyingFileSystem`. > `DuplicatingFileSystem` was not marked as PublicEvolving/Experimental > (probably by mistake), so technically we can do it. Even if not for that > mistake, I would still vote to replace it to simplify the code, as any > migration would be very easy. At the same time to the best of my knowledge, > no one has ever implemented it. > > > The proposal mentions that s5cmd utilises 100% of CPU similar to Flink > > 1.18. However, this will be a native process outside of the JVM. Are > there > > risk of large/long state download starving the TM of CPU cycle causing > > issues such as heartbeat or ask timeout? > > > > Do you know if there is a way to limit the CPU utilisation of s5cmd? I > see > > worker and concurrency configuration but these do not map directly to cap > > in CPU usage. The experience for feature user in this case will be one of > > trial and error. > > Those are good points. As a matter of fact, shortly after publishing this > FLIP, we started experimenting with using `cpulimit` to achieve just that. > If everything will work out fine, we are planning to expose this as a > configuration option for the S3 file system. I've added that to the FLIP. > > > 2. copyFiles is not an atomic operation, How could we handle the > situation > > when some partial files fail ? > > Could we return the list of successful files then the caller could decide > > to retry or just know them ? > > Do you have some suggestions on how that should be implemented in the > interface and how should it be used? > > Note that for both recovery and checkpoints, there are no retring > mechanisms. If any part of downloading or > uploading fails, the job fails over, so actually using such interface > extension would be out of scope of this FLIP. In > that case, maybe if this could be extended in the future without breaking > compatibility we could leave it as a > future improvement? > > Best, > Piotrek > > > pt., 10 maj 2024 o 07:40 Hangxiang Yu napisał(a): > > > Hi Piotr. > > Thanks for your proposal. > > > > I have some comments, PTAL: > > 1. +1 about unifying the interface with DuplicatingFileSystem. > > IIUC, DuplicatingFileSystem also covers the logic from/to both local and > > remote paths. > > The implementations could define their own logic about how to fast > > copy/duplicate files, e.g. hard link or transfer manager. > > > > 2. copyFiles is not an atomic operation, How could we handle the > situation > > when some partial files fail ? > > Could we return the list of successful files then the caller could decide > > to retry or just know them ? > > > > On Thu, May 9, 2024 at 3:46 PM Keith Lee > > wrote: > > > > > Hi Piotr, > > > > > > Thank you for the proposal. Looks great. > > > > > > Along similar line of Aleks' question on memory usage. > > > > > > The proposal mentions that s5cmd utilises 100% of CPU similar to Flink > > > 1.18. However, this will be a native process outside of the JVM. Are > > there > > > risk of large/long state download starving the TM of CPU cycle causing > > > issues such as heartbeat or ask timeout? > > > > > > Do you know if there is a way to limit the CPU utilisation of s5cmd? I > > see > > > worker and concurrency configuration but these do not map directly to > cap > > > in CPU usage. The experience for feature user in this case will be one > of > > > trial and error. > > > > > > Thanks > > > Keith > > > > > > On Wed, May 8, 2024 at 12:47 PM Ahmed Hamdy > > wrote: > > > > > > > Hi Piotr > > > > +1 for the proposal, it seems to have a lot of gains. > > > > > > > > Best Regards > > > > Ahmed Hamdy > > > > > > > > > > > > On Mon, 6 May 2024 at 12:06, Zakelly Lan > > wrote: > > > > > > > > > Hi Piotrek, > > > > > > > > > > Thanks for your answers! > > > > > > > > > > Good question. The intention and use case behind > > > `DuplicatingFileSystem` > > > > is > > > > > > different. It marks if `FileSystem` can quickly copy/duplicate > > files > > > > > > in the remote `FileSystem`. For example an equivalent of a hard > > link > > > or > > > > > > bumping a reference count in the remote system. That's a bit > > > different > > > > > > to copy paths between remote and local file systems. > > > > > > > > > > > >
Re: Best Practices? Fault Isolation for Processing Large Number of Same-Shaped Input Kafka Topics in a Big Flink Job
Hi Kevin, I think the question is valuable and It looks like this question can be posted at the user email list to receive more feedback. As for the question, I just want to share some observations: 1. When there are hundreds of data pipelines, it is nearly impossible to make all of them work properly for a long time. So we may need to consider the failover strategy more carefully. The Exponential Delay Restart Strategy may be a good choice or you can check the doc for more details: https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/task_failure_recovery/ . Another point is if in your sql job, all of these topic are processed parallely and there is no operator would try to connect them, make sure that the config 'jobmanager.execution.failover-strategy' is set to 'region' so that the failure in one pipeline would not lead other pipelines to be restarted. 2. Sometimes logic isolation may not be enough, and I would say that for some mission-critical pipelines, it is better to keep them in a dedicated flink job and sometimes a backup job is also required. 3. When we use 'hundreds of kafka topics', there is also one point worth mentioning: make sure the topics are distributed or replicated correctly. When upstream kafka brokers fail due to some common reasons, the downstream flink job can do nothing. Hope these points help! Best, Biao Geng Kevin Lam 于2024年5月9日周四 03:52写道： > Hi everyone, > > I'm currently prototyping on a project where we need to process a large > number of Kafka input topics (say, a couple of hundred), all of which share > the same DataType/Schema. > > Our objective is to run the same Flink SQL on all of the input topics, but > I am concerned about doing this in a single large Flink SQL application for > fault isolation purposes. We'd like to limit the "blast radius" in cases of > data issues or "poison pills" in any particular Kafka topic — meaning, if > one topic runs into a problem, it shouldn’t compromise or halt the > processing of the others. > > At the same time, we are concerned about the operational toil associated > with managing hundreds of Flink jobs that are really one logical > application. > > Has anyone here tackled a similar challenge? If so: > >1. How did you design your solution to handle a vast number of topics >without creating a heavy management burden? >2. What strategies or patterns have you found effective in isolating >issues within a specific topic so that they do not affect the > processing of >others? >3. Are there specific configurations or tools within the Flink ecosystem >that you'd recommend to efficiently manage this scenario? > > Any examples, suggestions, or references to relevant documentation would be > helpful. Thank you in advance for your time and help! >
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Jeyhun, Thanks for the comment! Yes, we intended to remove the StreamPiplineOptions in 2.0. I updated the FLIP to include the information. Best regards, Xuannan On Sun, May 12, 2024 at 9:16 PM Jeyhun Karimov wrote: > > Hi Xuannan, > > Thanks for driving this FLIP! > I have a minor comment. Do we plan to remove StreamPipelineOptions in 2.0, > as it only contains deprecated options? > > Regards, > Jeyhun > > On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > > > Thanks Xuannan for the update! > > > > LGTM, +1 for this proposal. > > > > Best, > > Rui > > > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su wrote: > > > > > Hi Rui, > > > > > > Thanks for the suggestion! > > > > > > I updated the description of > > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > > hard-coded it to 20. > > > > > > Best regards, > > > Xuannan > > > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > > > removed > > > > and hard-coded to either 10 or 20. > > > > > > > > Currently, it's a public option. Could we determine the value of > > > > the overdraft buffer in the current FLIP? > > > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > > - Removing this option means users cannot change it, it might be better > > > to > > > > turn it up. > > > > - Most of tasks don't use the overdraft buffer, so increasing it > > doesn't > > > > introduce more risk. > > > > > > > > Best, > > > > Rui > > > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > > wrote: > > > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > > > +1 for the proposal. > > > > > > > > > > Best, > > > > > Yuxin > > > > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > > solely > > > > > > changing the configuration options, but are bound to changing / > > > removal > > > > > of > > > > > > certain features. E.g., the removal of hash-blocking shuffle and > > > legacy > > > > > > hybrid shuffle mode, and the behavior change of overdraft network > > > > > buffers. > > > > > > Therefore, it might be nicer to provide an implementation plan > > with a > > > > > list > > > > > > of related tasks in the FLIP. This should not block the FLIP > > though. > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we > > have > > > > > > > revisited all runtime configurations and identified several > > > > > > > improvements to enhance user-friendliness and maintainability. In > > > this > > > > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank > > you! > > > > > > > > > > > > > > Best regards, > > > > > > > Xuannan > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] Connector releases for Flink 1.19
Hello Danny, Thanks for pushing this forward. I am available to assist with the CDC connector[1]. [1] https://github.com/apache/flink-cdc Best Hongshun On Sun, May 12, 2024 at 8:48 PM Sergey Nuyanzin wrote: > I'm in a process of preparation of RC for OpenSearch connector > > however it seems I need PMC help: need to create opensearch-2.0.0 on jira > since as it was proposed in another ML[1] to have 1.x for OpenSearch > v1 and 2.x for OpenSearch v2 > > would be great if someone from PMC could help here > > [1] https://lists.apache.org/thread/3w1rnjp5y612xy5k9yv44hy37zm9ph15 > > On Wed, Apr 17, 2024 at 12:42 PM Ferenc Csaky > wrote: > > > > Thank you Danny and Sergey for pushing this! > > > > I can help with the HBase connector if necessary, will comment the > > details to the relevant Jira ticket. > > > > Best, > > Ferenc > > > > > > > > > > On Wednesday, April 17th, 2024 at 11:17, Danny Cranmer < > dannycran...@apache.org> wrote: > > > > > > > > > > > Hello all, > > > > > > I have created a parent Jira to cover the releases [1]. I have > assigned AWS > > > and MongoDB to myself and OpenSearch to Sergey. Please assign the > > > relevant issue to yourself as you pick up the tasks. > > > > > > Thanks! > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-35131 > > > > > > On Tue, Apr 16, 2024 at 2:41 PM Muhammet Orazov > > > mor+fl...@morazow.com.invalid wrote: > > > > > > > Thanks Sergey and Danny for clarifying, indeed it > > > > requires committer to go through the process. > > > > > > > > Anyway, please let me know if I can be any help. > > > > > > > > Best, > > > > Muhammet > > > > > > > > On 2024-04-16 11:19, Danny Cranmer wrote: > > > > > > > > > Hello, > > > > > > > > > > I have opened the VOTE thread for the AWS connectors release [1]. > > > > > > > > > > > If I'm not mistaking (please correct me if I'm wrong) this > request is > > > > > > not > > > > > > about version update it is about new releases for connectors > > > > > > > > > > Yes, correct. If there are any other code changes required then > help > > > > > would be appreciated. > > > > > > > > > > > Are you going to create an umbrella issue for it? > > > > > > > > > > We do not usually create JIRA issues for releases. That being said > it > > > > > sounds like a good idea to have one place to track the status of > the > > > > > connector releases and pre-requisite code changes. > > > > > > > > > > > I would like to work on this task, thanks for initiating it! > > > > > > > > > > The actual release needs to be performed by a committer. However, > help > > > > > getting the connectors building against Flink 1.19 and testing the > RC > > > > > is > > > > > appreciated. > > > > > > > > > > Thanks, > > > > > Danny > > > > > > > > > > [1] > https://lists.apache.org/thread/0nw9smt23crx4gwkf6p1dd4jwvp1g5s0 > > > > > > > > > > On Tue, Apr 16, 2024 at 6:34 AM Sergey Nuyanzin > snuyan...@gmail.com > > > > > wrote: > > > > > > > > > > > Thanks for volunteering Muhammet! > > > > > > And thanks Danny for starting the activity. > > > > > > > > > > > > If I'm not mistaking (please correct me if I'm wrong) > > > > > > > > > > > > this request is not about version update it is about new > releases for > > > > > > connectors > > > > > > btw for jdbc connector support of 1.19 and 1.20-SNAPSHOT is > already > > > > > > done > > > > > > > > > > > > I would volunteer for Opensearch connector since currently I'm > working > > > > > > on > > > > > > support of Opensearch v2 > > > > > > and I think it would make sense to have a release after it is > done > > > > > > > > > > > > On Tue, Apr 16, 2024 at 4:29 AM Muhammet Orazov > > > > > > mor+fl...@morazow.com.invalid wrote: > > > > > > > > > > > > > Hello Danny, > > > > > > > > > > > > > > I would like to work on this task, thanks for initiating it! > > > > > > > > > > > > > > I could update the versions on JDBC and Pulsar connectors. > > > > > > > > > > > > > > Are you going to create an umbrella issue for it? > > > > > > > > > > > > > > Best, > > > > > > > Muhammet > > > > > > > > > > > > > > On 2024-04-15 13:44, Danny Cranmer wrote: > > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > > > Flink 1.19 was released on 2024-03-18 [1] and the connectors > have not > > > > > > > > yet > > > > > > > > caught up. I propose we start releasing the connectors with > support > > > > > > > > for > > > > > > > > Flink 1.19 as per the connector support guidelines [2]. > > > > > > > > > > > > > > > > I will make a start on flink-connector-aws, then pickup > others in the > > > > > > > > coming days. Please respond to the thread if you are/want to > work on > > > > > > > > a > > > > > > > > particular connector to avoid duplicate work. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Danny > > > > > > > > > > > > > > > > [1] > > > > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > > > > > > > > > [2] > > > > > > > > > https://cwiki.apache.org/confl
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Thanks Martijn I believe you missed my question, Should this change take place in 1.20, what are the planned release steps > for connectors that only offer a deprecated interface in this case (i.e. > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from releases > till the blockers are implemented? > Could you please clarify? Best Regards Ahmed Hamdy On Sun, 12 May 2024 at 14:07, Martijn Visser wrote: > Hi all, > > If there are no more considerations, I'll open up a vote in the next couple > of days. > > Best regards, > > Martijn > > On Wed, May 8, 2024 at 4:08 AM Hongshun Wang > wrote: > > > Hi Martijn, Thanks for the proposal +1 from me.Some sinks still use > > sinkfunction; it's time to take a step forward. > > > > Best, > > Hongshun > > > > On Mon, May 6, 2024 at 5:44 PM Leonard Xu wrote: > > > > > +1 from my side, thanks Martijn for the effort. > > > > > > Best, > > > Leonard > > > > > > > 2024年5月4日 下午7:41，Ahmed Hamdy 写道： > > > > > > > > Hi Martijn > > > > Thanks for the proposal +1 from me. > > > > Should this change take place in 1.20, what are the planned release > > steps > > > > for connectors that only offer a deprecated interface in this case > > (i.e. > > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from > > > releases > > > > that support 1.20+ till the blockers are implemented? > > > > Best Regards > > > > Ahmed Hamdy > > > > > > > > > > > > On Fri, 3 May 2024 at 14:32, Péter Váry > > > > wrote: > > > > > > > >>> With regards to FLINK-35149, the fix version indicates a change at > > > Flink > > > >> CDC; is that indeed correct, or does it require a change in the > SinkV2 > > > >> interface? > > > >> > > > >> The fix doesn't need change in SinkV2, so we are good there. > > > >> The issue is that the new SinkV2 > > > SupportsCommitter/SupportsPreWriteTopology > > > >> doesn't work with the CDC yet. > > > >> > > > >> Martijn Visser ezt írta (időpont: 2024. > > máj. > > > >> 3., > > > >> P, 14:06): > > > >> > > > >>> Hi Ferenc, > > > >>> > > > >>> You're right, 1.20 it is :) > > > >>> > > > >>> I've assigned the HBase one to you! > > > >>> > > > >>> Thanks, > > > >>> > > > >>> Martijn > > > >>> > > > >>> On Fri, May 3, 2024 at 1:55 PM Ferenc Csaky > > > > > > > > >>> wrote: > > > >>> > > > Hi Martijn, > > > > > > +1 for the proposal. > > > > > > > targeted for Flink 1.19 > > > > > > I guess you meant Flink 1.20 here. > > > > > > Also, I volunteer to take updating the HBase sink, feel free to > > assign > > > that task to me. > > > > > > Best, > > > Ferenc > > > > > > > > > > > > > > > On Friday, May 3rd, 2024 at 10:20, Martijn Visser < > > > martijnvis...@apache.org> wrote: > > > > > > > > > > > > > > > Hi Peter, > > > > > > > > I'll add it for completeness, thanks! > > > > With regards to FLINK-35149, the fix version indicates a change > at > > > >>> Flink > > > > CDC; is that indeed correct, or does it require a change in the > > > >> SinkV2 > > > > interface? > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > > > > > On Fri, May 3, 2024 at 7:47 AM Péter Váry > > > >> peter.vary.apa...@gmail.com > > > > > > > > wrote: > > > > > > > >> Hi Martijn, > > > >> > > > >> We might want to add FLIP-371 [1] to the list. (Or we aim only > for > > > higher > > > >> level FLIPs?) > > > >> > > > >> We are in the process of using the new API in Iceberg connector > > > >> [2] - > > > so > > > >> far, so good. > > > >> > > > >> I know of one minor known issue about the sink [3], which should > > be > > > ready > > > >> for the release. > > > >> > > > >> All-in-all, I think we are in good shape, and we could move > > forward > > > with > > > >> the promotion. > > > >> > > > >> Thanks, > > > >> Peter > > > >> > > > >> [1] - > > > >> > > > >> > > > > > > >>> > > > >> > > > > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=263430387 > > > >> [2] - https://github.com/apache/iceberg/pull/10179 > > > >> [3] - https://issues.apache.org/jira/browse/FLINK-35149 > > > >> > > > >> On Thu, May 2, 2024, 09:47 Muhammet Orazov > > > mor+fl...@morazow.com.invalid > > > >> wrote: > > > >> > > > >>> Got it, thanks! > > > >>> > > > >>> On 2024-05-02 06:53, Martijn Visser wrote: > > > >>> > > > Hi Muhammet, > > > > > > Thanks for joining the discussion! The changes in this FLIP > > > >> would > > > be > > > targeted for Flink 1.19, since it's only a matter of changing > > > >> the > > > annotation. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Thu, May 2, 2024 at 7:26 AM Muhammet Orazov > > > mor+fl...@mora
Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0
Hi Xuannan, Thanks for driving this FLIP! I have a minor comment. Do we plan to remove StreamPipelineOptions in 2.0, as it only contains deprecated options? Regards, Jeyhun On Sat, May 11, 2024 at 4:40 AM Rui Fan <1996fan...@gmail.com> wrote: > Thanks Xuannan for the update! > > LGTM, +1 for this proposal. > > Best, > Rui > > On Sat, May 11, 2024 at 10:20 AM Xuannan Su wrote: > > > Hi Rui, > > > > Thanks for the suggestion! > > > > I updated the description of > > taskmanager.network.memory.max-overdraft-buffers-per-gate and > > hard-coded it to 20. > > > > Best regards, > > Xuannan > > > > On Mon, May 6, 2024 at 11:28 AM Rui Fan <1996fan...@gmail.com> wrote: > > > > > > Thanks Xuannan for driving this proposal! > > > > > > > taskmanager.network.memory.max-overdraft-buffers-per-gate will be > > removed > > > and hard-coded to either 10 or 20. > > > > > > Currently, it's a public option. Could we determine the value of > > > the overdraft buffer in the current FLIP? > > > > > > I vote 20 as the hard code value due to 2 reasons: > > > - Removing this option means users cannot change it, it might be better > > to > > > turn it up. > > > - Most of tasks don't use the overdraft buffer, so increasing it > doesn't > > > introduce more risk. > > > > > > Best, > > > Rui > > > > > > On Mon, May 6, 2024 at 10:47 AM Yuxin Tan > > wrote: > > > > > > > Thanks for the effort, Xuannan. > > > > > > > > +1 for the proposal. > > > > > > > > Best, > > > > Yuxin > > > > > > > > > > > > Xintong Song 于2024年4月29日周一 15:40写道： > > > > > > > > > Thanks for driving this effort, Xuannan. > > > > > > > > > > +1 for the proposed changes. > > > > > > > > > > Just one suggestion: Some of the proposed changes involve not > solely > > > > > changing the configuration options, but are bound to changing / > > removal > > > > of > > > > > certain features. E.g., the removal of hash-blocking shuffle and > > legacy > > > > > hybrid shuffle mode, and the behavior change of overdraft network > > > > buffers. > > > > > Therefore, it might be nicer to provide an implementation plan > with a > > > > list > > > > > of related tasks in the FLIP. This should not block the FLIP > though. > > > > > > > > > > Best, > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su > > > > wrote: > > > > > > > > > > > Hi all, > > > > > > > > > > > > I'd like to start a discussion on FLIP-450: Improve Runtime > > > > > > Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we > have > > > > > > revisited all runtime configurations and identified several > > > > > > improvements to enhance user-friendliness and maintainability. In > > this > > > > > > FLIP, we aim to refine the runtime configuration. > > > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank > you! > > > > > > > > > > > > Best regards, > > > > > > Xuannan > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0 > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Hi all, If there are no more considerations, I'll open up a vote in the next couple of days. Best regards, Martijn On Wed, May 8, 2024 at 4:08 AM Hongshun Wang wrote: > Hi Martijn, Thanks for the proposal +1 from me.Some sinks still use > sinkfunction; it's time to take a step forward. > > Best, > Hongshun > > On Mon, May 6, 2024 at 5:44 PM Leonard Xu wrote: > > > +1 from my side, thanks Martijn for the effort. > > > > Best, > > Leonard > > > > > 2024年5月4日 下午7:41，Ahmed Hamdy 写道： > > > > > > Hi Martijn > > > Thanks for the proposal +1 from me. > > > Should this change take place in 1.20, what are the planned release > steps > > > for connectors that only offer a deprecated interface in this case > (i.e. > > > RabbitMQ, Cassandra, pusbub, Hbase)? Are we going to refrain from > > releases > > > that support 1.20+ till the blockers are implemented? > > > Best Regards > > > Ahmed Hamdy > > > > > > > > > On Fri, 3 May 2024 at 14:32, Péter Váry > > wrote: > > > > > >>> With regards to FLINK-35149, the fix version indicates a change at > > Flink > > >> CDC; is that indeed correct, or does it require a change in the SinkV2 > > >> interface? > > >> > > >> The fix doesn't need change in SinkV2, so we are good there. > > >> The issue is that the new SinkV2 > > SupportsCommitter/SupportsPreWriteTopology > > >> doesn't work with the CDC yet. > > >> > > >> Martijn Visser ezt írta (időpont: 2024. > máj. > > >> 3., > > >> P, 14:06): > > >> > > >>> Hi Ferenc, > > >>> > > >>> You're right, 1.20 it is :) > > >>> > > >>> I've assigned the HBase one to you! > > >>> > > >>> Thanks, > > >>> > > >>> Martijn > > >>> > > >>> On Fri, May 3, 2024 at 1:55 PM Ferenc Csaky > > > > > >>> wrote: > > >>> > > Hi Martijn, > > > > +1 for the proposal. > > > > > targeted for Flink 1.19 > > > > I guess you meant Flink 1.20 here. > > > > Also, I volunteer to take updating the HBase sink, feel free to > assign > > that task to me. > > > > Best, > > Ferenc > > > > > > > > > > On Friday, May 3rd, 2024 at 10:20, Martijn Visser < > > martijnvis...@apache.org> wrote: > > > > > > > > > > > Hi Peter, > > > > > > I'll add it for completeness, thanks! > > > With regards to FLINK-35149, the fix version indicates a change at > > >>> Flink > > > CDC; is that indeed correct, or does it require a change in the > > >> SinkV2 > > > interface? > > > > > > Best regards, > > > > > > Martijn > > > > > > > > > On Fri, May 3, 2024 at 7:47 AM Péter Váry > > >> peter.vary.apa...@gmail.com > > > > > > wrote: > > > > > >> Hi Martijn, > > >> > > >> We might want to add FLIP-371 [1] to the list. (Or we aim only for > > higher > > >> level FLIPs?) > > >> > > >> We are in the process of using the new API in Iceberg connector > > >> [2] - > > so > > >> far, so good. > > >> > > >> I know of one minor known issue about the sink [3], which should > be > > ready > > >> for the release. > > >> > > >> All-in-all, I think we are in good shape, and we could move > forward > > with > > >> the promotion. > > >> > > >> Thanks, > > >> Peter > > >> > > >> [1] - > > >> > > >> > > > > >>> > > >> > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=263430387 > > >> [2] - https://github.com/apache/iceberg/pull/10179 > > >> [3] - https://issues.apache.org/jira/browse/FLINK-35149 > > >> > > >> On Thu, May 2, 2024, 09:47 Muhammet Orazov > > mor+fl...@morazow.com.invalid > > >> wrote: > > >> > > >>> Got it, thanks! > > >>> > > >>> On 2024-05-02 06:53, Martijn Visser wrote: > > >>> > > Hi Muhammet, > > > > Thanks for joining the discussion! The changes in this FLIP > > >> would > > be > > targeted for Flink 1.19, since it's only a matter of changing > > >> the > > annotation. > > > > Best regards, > > > > Martijn > > > > On Thu, May 2, 2024 at 7:26 AM Muhammet Orazov > > mor+fl...@morazow.com > > wrote: > > > > > Hello Martijn, > > > > > > Thanks for the FLIP and detailed history of changes, +1. > > > > > > Would FLIP changes target for 2.0? I think it would be good > > > to have clear APIs on 2.0 release. > > > > > > Best, > > > Muhammet > > > > > > On 2024-05-01 15:30, Martijn Visser wrote: > > > > > >> Hi everyone, > > >> > > >> I would like to start a discussion on FLIP-453: Promote > > Unified Sink > > >> API V2 > > >> to Public and Deprecate SinkFunction > > >> https://cwiki.apache.org/confluence/x/rIobEg > > >> > > >> This FLIP proposes to promote the Unified
Re: [DISCUSS] Connector releases for Flink 1.19
I'm in a process of preparation of RC for OpenSearch connector however it seems I need PMC help: need to create opensearch-2.0.0 on jira since as it was proposed in another ML[1] to have 1.x for OpenSearch v1 and 2.x for OpenSearch v2 would be great if someone from PMC could help here [1] https://lists.apache.org/thread/3w1rnjp5y612xy5k9yv44hy37zm9ph15 On Wed, Apr 17, 2024 at 12:42 PM Ferenc Csaky wrote: > > Thank you Danny and Sergey for pushing this! > > I can help with the HBase connector if necessary, will comment the > details to the relevant Jira ticket. > > Best, > Ferenc > > > > > On Wednesday, April 17th, 2024 at 11:17, Danny Cranmer > wrote: > > > > > > > Hello all, > > > > I have created a parent Jira to cover the releases [1]. I have assigned AWS > > and MongoDB to myself and OpenSearch to Sergey. Please assign the > > relevant issue to yourself as you pick up the tasks. > > > > Thanks! > > > > [1] https://issues.apache.org/jira/browse/FLINK-35131 > > > > On Tue, Apr 16, 2024 at 2:41 PM Muhammet Orazov > > mor+fl...@morazow.com.invalid wrote: > > > > > Thanks Sergey and Danny for clarifying, indeed it > > > requires committer to go through the process. > > > > > > Anyway, please let me know if I can be any help. > > > > > > Best, > > > Muhammet > > > > > > On 2024-04-16 11:19, Danny Cranmer wrote: > > > > > > > Hello, > > > > > > > > I have opened the VOTE thread for the AWS connectors release [1]. > > > > > > > > > If I'm not mistaking (please correct me if I'm wrong) this request is > > > > > not > > > > > about version update it is about new releases for connectors > > > > > > > > Yes, correct. If there are any other code changes required then help > > > > would be appreciated. > > > > > > > > > Are you going to create an umbrella issue for it? > > > > > > > > We do not usually create JIRA issues for releases. That being said it > > > > sounds like a good idea to have one place to track the status of the > > > > connector releases and pre-requisite code changes. > > > > > > > > > I would like to work on this task, thanks for initiating it! > > > > > > > > The actual release needs to be performed by a committer. However, help > > > > getting the connectors building against Flink 1.19 and testing the RC > > > > is > > > > appreciated. > > > > > > > > Thanks, > > > > Danny > > > > > > > > [1] https://lists.apache.org/thread/0nw9smt23crx4gwkf6p1dd4jwvp1g5s0 > > > > > > > > On Tue, Apr 16, 2024 at 6:34 AM Sergey Nuyanzin snuyan...@gmail.com > > > > wrote: > > > > > > > > > Thanks for volunteering Muhammet! > > > > > And thanks Danny for starting the activity. > > > > > > > > > > If I'm not mistaking (please correct me if I'm wrong) > > > > > > > > > > this request is not about version update it is about new releases for > > > > > connectors > > > > > btw for jdbc connector support of 1.19 and 1.20-SNAPSHOT is already > > > > > done > > > > > > > > > > I would volunteer for Opensearch connector since currently I'm working > > > > > on > > > > > support of Opensearch v2 > > > > > and I think it would make sense to have a release after it is done > > > > > > > > > > On Tue, Apr 16, 2024 at 4:29 AM Muhammet Orazov > > > > > mor+fl...@morazow.com.invalid wrote: > > > > > > > > > > > Hello Danny, > > > > > > > > > > > > I would like to work on this task, thanks for initiating it! > > > > > > > > > > > > I could update the versions on JDBC and Pulsar connectors. > > > > > > > > > > > > Are you going to create an umbrella issue for it? > > > > > > > > > > > > Best, > > > > > > Muhammet > > > > > > > > > > > > On 2024-04-15 13:44, Danny Cranmer wrote: > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > Flink 1.19 was released on 2024-03-18 [1] and the connectors have > > > > > > > not > > > > > > > yet > > > > > > > caught up. I propose we start releasing the connectors with > > > > > > > support > > > > > > > for > > > > > > > Flink 1.19 as per the connector support guidelines [2]. > > > > > > > > > > > > > > I will make a start on flink-connector-aws, then pickup others in > > > > > > > the > > > > > > > coming days. Please respond to the thread if you are/want to work > > > > > > > on > > > > > > > a > > > > > > > particular connector to avoid duplicate work. > > > > > > > > > > > > > > Thanks, > > > > > > > Danny > > > > > > > > > > > > > > [1] > > > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > > > > > > > [2] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Flinkcompatibility > > > > > > > > > > [3] https://github.com/apache/flink-connector-aws > > > > > > > > > > -- > > > > > Best regards, > > > > > Sergey -- Best regards, Sergey
Re: [DISCUSS][QUESTION] Drop jdk 8 support for Flink connector Opensearch
yep, it seems you are right we need 2 versions otherwise there will be clash between base artifact from both v1 and v2 just faced this while preparations On Tue, Apr 23, 2024 at 4:13 PM Sergey Nuyanzin wrote: > > thanks for your response, Danny > > yeah, I think we could have 2 releases: one for v1 (jdk8) and one for v2 > (jdk11) > I guess we could even have same version for both since anyway artifact names > are different > > On Thu, Apr 18, 2024 at 4:01 PM Danny Cranmer wrote: >> >> Hey Sergey, >> >> Given the circumstances I think it is ok to drop JDK 8 support with the >> opensearch v2.0.0 connector. However, in parallel we should still support >> the 1.x line for Flink 1.x series with JDK 8. This would mean two releases: >> 1/ flink-connector-opensearch v2.0.0 for Flink 1.18/1.19, opensearch >> 1.x/2.x and JDK 11 >> 2/ flink-connector-opensearch v1.2.0 (or maybe just 1.1.0-1.19) for Flink >> 1.18/1.19, opensearch 1.x and JDK 8 >> >> What do you think? >> >> Thanks, >> Danny >> >> On Wed, Apr 17, 2024 at 10:07 AM Sergey Nuyanzin >> wrote: >> >> > Hi everyone >> > >> > I'm working on support for Opensearch v2.x for Flink connector >> > Opensearch[1]. >> > Unfortunately after several breaking changes (e.g. [2], [3]) on Opensearch >> > side it is not possible >> > anymore to use the same connector built for both Opensearch v1 and v2. >> > This makes us to go in a similar way as for Elasticsearch 6/7 and build a >> > dedicated Opensearch v2 module. >> > >> > However the main pain point here is that Opensearch 2.x is built with jdk11 >> > and requires jdk11 to build and use Flink connector as well. >> > Also in README[4] of most of the connectors it is mentioned explicitly that >> > jdk11 is required to build connectors. >> > >> > At the same time it looks like we need to release a connector for >> > Opensearch v1 with jdk8 and for Opensearch v2 with jdk11. >> > >> > The suggestion is to drop support of jdk8 for the Opensearch connector to >> > make the release/testing for both modules (for Opensearch v1 and Openseach >> > v2) easier. >> > >> > Other opinions are welcome >> > >> > [1] https://github.com/apache/flink-connector-opensearch/pull/38 >> > [2] opensearch-project/OpenSearch#9082 >> > [3] opensearch-project/OpenSearch#5902 >> > [4] >> > >> > https://github.com/apache/flink-connector-opensearch/blob/main/README.md?plain=1#L18 >> > >> > -- >> > Best regards, >> > Sergey >> > > > > > -- > Best regards, > Sergey -- Best regards, Sergey
