Re: [DISCUSS] CLI action deprecation process
Hi Muhammet, Thank you for your thoughts! > After two minor releases, and on next major version bump, > we could drop the `run-application` method as suggested > on discussion by Xintong. Here, you describe the deprecation/removal process of a public API by the definition we have in the project now. So if the same applies to a CLI action, why should we not enforce such behavior for those as well? If following the same process for CLIs make sense, we should also enforce the same compatibility guarantees IMO. Best, Ferenc On Friday, 28 June 2024 at 09:30, Muhammet Orazov wrote: > > > Hey Ferenc, > > Thanks for starting the discussion! > > I agree that the CLI is user facing, but I think we don't > have to treat it as other public APIs. > > I'd propose to throw user friendly exception for > `run-application` with suggestion to use `run` case instead. > This would make users aware of the change and require them > to migrate their scripts. > > After two minor releases, and on next major version bump, > we could drop the `run-application` method as suggested > on discussion by Xintong. > > Best, > Muhammet > > > On 2024-06-26 15:33, Ferenc Csaky wrote: > > > Hello devs, > > I would like to open a discussion about considerations regarding how to > > deprecate CLI > > actions, and what compatibility guarantees should apply to such cases. > > The topic came up in > > a previous discussion [1] about a current FLIP to merge the run and > > run-application > > behavior [2]. > > > > According to Xintong's previous inputs, currently the Flink CLI, or its > > actions are not handled > > as public APIs by the existing definition (@Public or @PublicEvolving > > annotated). So > > legally it would be possible to change CLIs anytime. I agree with > > Xintong that CLI actions > > should be considered as public APIs, and as such, compatibility > > guarantees should be > > provided. > > > > CLI actions are defined as private constants in CliFrontend [3], so > > IMO the existing rules > > are not perfectly applicable as is. Both @Public and @PublicEvolving > > can be applied to > > fields (although for @Public that is only true as per the javadoc, > > technically it can only be > > applied to classes), so I think that could be a good approach that is > > achievable with minimal > > effort and changes. > > > > It would also be possible to define a different process, but IMO the > > more lightweight and > > maintainable the process the better, so it is a good thing if it is not > > necessary to bring in > > new entities and/or rules to check and enforce compatibility. > > > > WDYT? > > > > Best, > > Ferenc > > > > [1] https://lists.apache.org/thread/0vc8v3t7fr6w9hmwf9zbjbyk5c3bcj50 > > [2] > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 > > [3] > > https://github.com/apache/flink/blob/27287a105f6585e89795e2a6cbffa8254abb6e57/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L98
Re: [VOTE] FLIP-465: Introduce DESCRIBE FUNCTION
+1 (non-binding) Best, Ferenc On Thursday, 4 July 2024 at 16:20, Feng Jin wrote: > > > +1 (non-binding) > > Best, > Feng Jin > > On Thu, Jul 4, 2024 at 6:02 PM Martijn Visser martijnvis...@apache.org > > wrote: > > > +1 (binding) > > > > On Thu, Jul 4, 2024 at 5:39 AM Yanquan Lv decq12y...@gmail.com wrote: > > > > > Hi Natea, thanks for driving it. > > > +1 (non-binding). > > > > > > Jim Hughes jhug...@confluent.io.invalid 于2024年7月4日周四 04:41写道: > > > > > > > Hi Natea, > > > > > > > > Looks good to me! > > > > > > > > +1 (non-binding). > > > > > > > > Cheers, > > > > > > > > Jim > > > > > > > > On Wed, Jul 3, 2024 at 3:16 PM Natea Eshetu Beshada > > > > nbesh...@confluent.io.invalid wrote: > > > > > > > > > Sorry I forgot to include the FLIP [1] and the mailing thread > > > > > discussion > > > > > link [2] in my previous email. > > > > > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-465%3A+Introduce+DESCRIBE+FUNCTION > > > > > > > [2] https://lists.apache.org/thread/s46ftnmz4ggmmssgyx6vfhqjttsk9lph > > > > > > > > > > Thanks, > > > > > Natea > > > > > > > > > > On Wed, Jul 3, 2024 at 12:06 PM Natea Eshetu Beshada < > > > > > nbesh...@confluent.io> > > > > > wrote: > > > > > > > > > > > Hello everyone, > > > > > > > > > > > > I would like to start a vote on FLIP-465 [1]. It proposes adding > > > > > > SQL > > > > > > syntax that would allow users to describe the metadata of a given > > > > > > function. > > > > > > > > > > > > The vote will be open for at least 72 hours (Saturday, July 6th of > > > > > > July > > > > > > 2024, > > > > > > 12:30 PST) unless there is an objection or insufficient votes. > > > > > > > > > > > > Thanks, > > > > > > Natea
Re:[VOTE] FLIP-444: Native file copy support
+1 (non-binding) Best, Ferenc On Friday, 28 June 2024 at 06:27, Feifan Wang wrote: > > > +1 (non binding) > > > > > —— > > Best regards, > > Feifan Wang > > > > > At 2024-06-25 16:58:22, "Piotr Nowojski" pnowoj...@apache.org wrote: > > > Hi all, > > > > I would like to start a vote for the FLIP-444 [1]. The discussion thread is > > here [2]. > > > > The vote will be open for at least 72. > > > > Best, > > Piotrek > > > > [1] https://cwiki.apache.org/confluence/x/rAn9EQ > > [2] https://lists.apache.org/thread/lkwmyjt2bnmvgx4qpp82rldwmtd4516c
[DISCUSS] CLI action deprecation process
Hello devs, I would like to open a discussion about considerations regarding how to deprecate CLI actions, and what compatibility guarantees should apply to such cases. The topic came up in a previous discussion [1] about a current FLIP to merge the run and run-application behavior [2]. According to Xintong's previous inputs, currently the Flink CLI, or its actions are not handled as public APIs by the existing definition (@Public or @PublicEvolving annotated). So legally it would be possible to change CLIs anytime. I agree with Xintong that CLI actions should be considered as public APIs, and as such, compatibility guarantees should be provided. CLI actions are defined as private constants in CliFrontend [3], so IMO the existing rules are not perfectly applicable as is. Both @Public and @PublicEvolving can be applied to fields (although for @Public that is only true as per the javadoc, technically it can only be applied to classes), so I think that could be a good approach that is achievable with minimal effort and changes. It would also be possible to define a different process, but IMO the more lightweight and maintainable the process the better, so it is a good thing if it is not necessary to bring in new entities and/or rules to check and enforce compatibility. WDYT? Best, Ferenc [1] https://lists.apache.org/thread/0vc8v3t7fr6w9hmwf9zbjbyk5c3bcj50 [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 [3] https://github.com/apache/flink/blob/27287a105f6585e89795e2a6cbffa8254abb6e57/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L98
[jira] [Created] (FLINK-35699) The flink-kubernetes artifact shades Jackson 2.15.3 from fabric8
Ferenc Csaky created FLINK-35699: Summary: The flink-kubernetes artifact shades Jackson 2.15.3 from fabric8 Key: FLINK-35699 URL: https://issues.apache.org/jira/browse/FLINK-35699 Project: Flink Issue Type: Bug Components: Deployment / Kubernetes Affects Versions: 1.19.1 Reporter: Ferenc Csaky Fix For: 1.20.0, 1.19.2 The {{flink-kubernetes}} artifact shades Jackson classes coming through fabric8, but since Jackson 2.15, Jackson is a [multi-release JAR|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15#jar-changes], which requires some additional relocations for correct shading. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35695) Release Testing: Verify FLINK-32315: Support local file upload in K8s mode
Ferenc Csaky created FLINK-35695: Summary: Release Testing: Verify FLINK-32315: Support local file upload in K8s mode Key: FLINK-35695 URL: https://issues.apache.org/jira/browse/FLINK-35695 Project: Flink Issue Type: Sub-task Components: Runtime / Network Reporter: Ferenc Csaky Fix For: 1.20.0 Follow up the test for https://issues.apache.org/jira/browse/FLINK-35533 In Flink 1.20, we proposed integrating Flink's Hybrid Shuffle with Apache Celeborn through a pluggable remote tier interface. To verify this feature, you should reference these main two steps. 1. Implement Celeborn tier. * Implement a new tier factory and tier for Celeborn, including these APIs, including TierFactory/TierMasterAgent/TierProducerAgent/TierConsumerAgent. * The implementations should support granular data management at the Segment level for both client and server sides. 2. Use the implemented tier to shuffle data. * Compile Flink and Celeborn. * Deploy Celeborn service ** Deploy a new Celeborn service with the new compiled packages. You can reference the doc ([https://celeborn.apache.org/docs/latest/]) to deploy the cluster. * Add the compiled flink plugin jar (celeborn-client-flink-xxx.jar) to Flink classpath. * Configure the options to enable the feature. ** Configure the option taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class to the new Celeborn tier classes. Except for this option, the following options should also be added. {code:java} execution.batch-shuffle-mode: ALL_EXCHANGES_HYBRID_FULL celeborn.master.endpoints: celeborn.client.shuffle.partition.type: MAP{code} * Run some test examples(e.g., WordCount) to verify the feature. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-463: Schema Definition in CREATE TABLE AS Statement
+1 (non-binding) Best, Ferenc On Sunday, June 23rd, 2024 at 05:13, Yanquan Lv wrote: > > > Thnaks Sergio, +1 (non-binding) > > gongzhongqiang gongzhongqi...@apache.org 于2024年6月23日周日 10:06写道: > > > +1 (non-binding) > > > > Best, > > Zhongqiang Gong > > > > Sergio Pena ser...@confluent.io.invalid 于2024年6月21日周五 22:18写道: > > > > > Hi everyone, > > > > > > Thanks for all the feedback about FLIP-463: Schema Definition in CREATE > > > TABLE AS Statement [1]. The discussion thread is here [2]. > > > > > > I'd like to start a vote for it. The vote will be open for at least 72 > > > hours unless there is an objection or insufficient votes. The FLIP will > > > be > > > considered accepted if 3 binding votes (from active committers according > > > to > > > the Flink bylaws [3]) are gathered by the community. > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-463%3A+Schema+Definition+in+CREATE+TABLE+AS+Statement > > > > > [2] https://lists.apache.org/thread/1ryxxyyg3h9v4rbosc80zryvjk6c8k83 > > > [3] [ > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals) > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals) > > > > > < > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals) > > > > > > < > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals](https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws%23FlinkBylaws-Approvals) > > > > > Thanks, > > > Sergio Peña
Re: [ANNOUNCE] Flink 1.20 feature freeze
Hi Xintong, Weijie, Thank you both for your answers! Regarding the CLI deprecation process, I agree that it is tricky business, hence I opened this discussion. The proposal makes sense IMO, I will open a discussion thread. With the present timeline, I think it is reasonable to hold it from 1.20, as this is more like a nice-to-have, just wanted to make sure what is possible or what kind of policies we need to apply when we remove a CLI action. Best, Ferenc On Thursday, 20 June 2024 at 04:05, weijie guo wrote: > > > +1 for Xintong's point. > > Hi Ferenc, > > After discussing this with all the other RM's, we decided not to merge the > PR you mentioned into 1.20. > The main reason is that FLIP-464 was voted in three days ago, which is > already after feature freeze date. It doesn't make sense to us to put it in > the 1.20 release cycle. > > > Best regards, > > Weijie > > > Xintong Song tonysong...@gmail.com 于2024年6月19日周三 17:25写道: > > > Hi Ferenc, > > > > About the deprecation process, removing a @Public API requires the API to > > be deprecated for at least 2 minor releases AND the removal should be in a > > major version bump. That means you cannot remove a @Public API in 2.x when > > x is not 0. The tricky part is, run-application as a command-line interface > > does not have > > any @Public/@PublicEvolving/@Experimental/@Deprecated annotations (nor > > CliFrontend, the class it is defined in). Command-line interfaces are > > definitely public APIs IMO, but there's no explicit deprecation process for > > them, which also means we never committed that command-line interfaces will > > stay compatible. > > > > My suggestion would be to start a separate discussion thread on whether and > > how to add explicit compatibility guarantees for command-line interfaces. > > Without that, "legally" we can change command-line interfaces anytime, > > though I'd be negative doing so. > > > > As for the PR, I'd be in favor of not merging it for 1.20. Because we have > > passed the feature freeze date for half a week and the PR is still not yet > > reviewed, and the necessity for making it into 1.20 is unclear due to > > absence of explicit process. > > > > Best, > > > > Xintong > > > > On Wed, Jun 19, 2024 at 1:33 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > > > Hi Robert, Rui, Ufuk, and Weijie, > > > > > > I would like to raise the PR about merging `flink run` and > > > `flink run-application` functionality [1] to get considered as > > > part of the 1.20 release. > > > > > > The reason IMO is that the `run-application` CLI command should be > > > removed in the same release when Per-Job mode gets removed. AFAIK > > > when we deprecate a public API, it has to stay for 2 minor > > > releases to give time for users to adapt. According to that, if > > > `run-application` is deprecated in Flink 2.0, it can get removed > > > in Flink 2.3. Currently the drop of per-job mode is blocked [2] > > > and probably it will not be resolved for a while, but I could > > > imagine it would be possible in 2.1 or 2.2. > > > > > > The change itself is rather small and concise, and Marton Balassi > > > volunteered to review it ASAP. > > > > > > Pls. correct me if I am wrong about the deprecation process. > > > > > > Looking forward to your opinion! > > > > > > Thanks, > > > Ferenc > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-35625 > > > [2] https://issues.apache.org/jira/browse/FLINK-26000 > > > > > > On Tuesday, 18 June 2024 at 11:27, weijie guo > > > > > wrote: > > > > > > > Hi Zakelly, > > > > > > > > Thank you for informing us! > > > > > > > > After discussion, all RMs agreed that this was an important fix that > > > > should > > > > be merged into 1.20. > > > > > > > > So feel free to merge it. > > > > > > > > Best regards, > > > > > > > > Weijie > > > > > > > > Zakelly Lan zakelly@gmail.com 于2024年6月15日周六 16:29写道: > > > > > > > > > Hi Robert, Rui, Ufuk and Weijie, > > > > > > > > > > Thanks for the update! > > > > > > > > > > FYI: This PR[1] fixes & cleanup the left-over checkpoint directories > > > > > for > > > > > file-merging on TM exit
[jira] [Created] (FLINK-35662) Use maven batch mode in k8s-operator CI
Ferenc Csaky created FLINK-35662: Summary: Use maven batch mode in k8s-operator CI Key: FLINK-35662 URL: https://issues.apache.org/jira/browse/FLINK-35662 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Ferenc Csaky Fix For: kubernetes-operator-1.10.0 Currently, the GitHub workflows do not use batch mode in the k8s-operator repo, so there are a lot of lines in the log like this: {code} Progress (1): 4.1/14 kB Progress (1): 8.2/14 kB Progress (1): 12/14 kB Progress (1): 14 kB {code} To produce logs that are for more easy to navigate, all {{mvn}} calls should apply the batch-mode option {{-B}}. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35649) Bump Flink version to 1.19.1 in k8s-operator
Ferenc Csaky created FLINK-35649: Summary: Bump Flink version to 1.19.1 in k8s-operator Key: FLINK-35649 URL: https://issues.apache.org/jira/browse/FLINK-35649 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Ferenc Csaky Fix For: kubernetes-operator-1.10.0 In FLINK-28915 it came up the the operator is not able to utilize the artifact fetching capabilities that was introduced in Flink 1.19 until it is not built on that version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] Flink 1.20 feature freeze
Hi Robert, Rui, Ufuk, and Weijie, I would like to raise the PR about merging `flink run` and `flink run-application` functionality [1] to get considered as part of the 1.20 release. The reason IMO is that the `run-application` CLI command should be removed in the same release when Per-Job mode gets removed. AFAIK when we deprecate a public API, it has to stay for 2 minor releases to give time for users to adapt. According to that, if `run-application` is deprecated in Flink 2.0, it can get removed in Flink 2.3. Currently the drop of per-job mode is blocked [2] and probably it will not be resolved for a while, but I could imagine it would be possible in 2.1 or 2.2. The change itself is rather small and concise, and Marton Balassi volunteered to review it ASAP. Pls. correct me if I am wrong about the deprecation process. Looking forward to your opinion! Thanks, Ferenc [1] https://issues.apache.org/jira/browse/FLINK-35625 [2] https://issues.apache.org/jira/browse/FLINK-26000 On Tuesday, 18 June 2024 at 11:27, weijie guo wrote: > > > Hi Zakelly, > > Thank you for informing us! > > After discussion, all RMs agreed that this was an important fix that should > be merged into 1.20. > > So feel free to merge it. > > Best regards, > > Weijie > > > Zakelly Lan zakelly@gmail.com 于2024年6月15日周六 16:29写道: > > > Hi Robert, Rui, Ufuk and Weijie, > > > > Thanks for the update! > > > > FYI: This PR[1] fixes & cleanup the left-over checkpoint directories for > > file-merging on TM exit. And the second commit fixes the wrong state handle > > usage. We encountered several unexpected CI fails, so we missed the feature > > freeze time. It is better to have this PR in 1.20 so I will merge this if > > you agree. Thanks. > > > > [1] https://github.com/apache/flink/pull/24933 > > > > Best, > > Zakelly > > > > On Sat, Jun 15, 2024 at 6:00 AM weijie guo guoweijieres...@gmail.com > > wrote: > > > > > Hi everyone, > > > > > > The feature freeze of 1.20 has started now. That means that no new > > > features > > > > > > or improvements should now be merged into the master branch unless you > > > ask > > > > > > the release managers first, which has already been done for PRs, or > > > pending > > > > > > on CI to pass. Bug fixes and documentation PRs can still be merged. > > > > > > - Cutting release branch > > > > > > Currently we have no blocker issues(beside tickets that used for > > > release-testing). > > > > > > We are planning to cut the release branch on next Friday (June 21) if > > > no new test instabilities, and we'll make another announcement in the > > > dev mailing list then. > > > > > > - Cross-team testing > > > > > > The release testing will start right after we cut the release branch, > > > which > > > > > > is expected to come in the next week. As a prerequisite of it, we have > > > created > > > > > > the corresponding instruction ticket in FLINK-35602 [1], please check > > > and complete the > > > > > > documentation and test instruction of your new feature and mark the > > > related JIRA > > > > > > issue in the 1.20 release wiki page [2] before we start testing, which > > > > > > would be quite helpful for other developers to validate your features. > > > > > > Best regards, > > > > > > Robert, Rui, Ufuk and Weijie > > > > > > [1]https://issues.apache.org/jira/browse/FLINK-35602 > > > > > > [2] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
[jira] [Created] (FLINK-35625) FLIP-464: Merge "flink run" and "flink run-application"
Ferenc Csaky created FLINK-35625: Summary: FLIP-464: Merge "flink run" and "flink run-application" Key: FLINK-35625 URL: https://issues.apache.org/jira/browse/FLINK-35625 Project: Flink Issue Type: Improvement Components: Client / Job Submission, Command Line Client Reporter: Ferenc Csaky Fix For: 1.20.0 Ticket to track [FLIP-464|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler
+1 (non-binding) Best, Ferenc On Monday, 17 June 2024 at 15:29, Zdenek Tison wrote: > > > +1 (non-binding) > > Best, > Zdenek > > On Mon, Jun 17, 2024 at 10:24 AM Matthias Pohl map...@apache.org wrote: > > > Hi everyone, > > the discussion in [1] about FLIP-461 [2] is kind of concluded. I am > > starting a vote on this one here. > > > > The vote will be open for at least 72 hours (i.e. until June 20, 2024; > > 8:30am UTC) unless there are any objections. The FLIP will be considered > > accepted if 3 binding votes (from active committers according to the Flink > > bylaws [3]) are gathered by the community. > > > > Best, > > Matthias > > > > [1] https://lists.apache.org/thread/nnkonmsv8xlk0go2sgtwnphkhrr5oc3y > > [2] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing+for+the+AdaptiveScheduler > > [3] > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals
[RESULT][VOTE] FLIP-464: Merge "flink run" and "flink run-application"
Hello devs, I'm happy to announce that FLIP-464: Merge "flink run" and "flink run-application" [1] has been accepted with 13 approving votes, 6 of which are binding: [2]: - Mate Czagany (non-binding) - Mátyás Őrhidi (binding) - Marton Balassi (binding) - Xintong Song (binding)- Weijie Gue (binding)- Biao Geng (non-binding)- Junrui Lee (non-binding)- Zakelly Lan (binding)- Yuepeng Pan (non-binding) - Gabor Somogyi (binding) - Jeyhun Karimov (non-binding)- Venkatakrishnan Sowrirajan (non-binding) - Hang Ruan (non-binding) There are no disapproving votes. Thanks to everyone who participated in the discussion and voting. Best, Ferenc [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 [2] https://lists.apache.org/thread/r2ny7dqomlwlr2f7shtm1zl670w6qnkr
[VOTE] FLIP-464: Merge "flink run" and "flink run-application"
Hello devs, I would like to start a vote about FLIP-464 [1]. The FLIP is about to merge back the "flink run-application" functionality to "flink run", so the latter will be capable to deploy jobs in all deployment modes. More details in the FLIP. Discussion thread [2]. The vote will be open for at least 72 hours (until 2024 March 23 14:03 UTC) unless there are any objections or insufficient votes. Thanks,Ferenc [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179 [2] https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hi, Thank you everyone for the valuable comments, if there are no new messages by then, I will start a vote on Monday. Thanks, Ferenc On Monday, 3 June 2024 at 17:27, Jeyhun Karimov wrote: > > > Hi Ferenc, > > Thanks for the proposal. +1 for it! This FLIP will improve the user > experience. > > Regards, > Jeyhun > > > > > > On Mon, Jun 3, 2024 at 1:50 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi Hang, > > > > Thank you for your inputs, both points make sense, updated the > > FLIP according to them. > > > > Best, > > Ferenc > > > > On Friday, 31 May 2024 at 04:31, Hang Ruan ruanhang1...@gmail.com wrote: > > > > > Hi, Ferenc. > > > > > > +1 for this proposal. This FLIP will help to make the CLI clearer for > > > users. > > > > > > I think we should better add an example in the FLIP about how to use the > > > application mode with the new CLI. > > > Besides that, we need to add some new tests for this change instead of > > > only > > > using the existed tests. > > > > > > Best, > > > Hang > > > > > > Mate Czagany czmat...@gmail.com 于2024年5月29日周三 19:57写道: > > > > > > > Hi Ferenc, > > > > > > > > Thanks for the FLIP, +1 from me for the proposal. I think these changes > > > > would be a great solution to all the confusion that comes from these > > > > two > > > > action parameters. > > > > > > > > Best regards, > > > > Mate > > > > > > > > Ferenc Csaky ferenc.cs...@pm.me.invalid ezt írta (időpont: 2024. máj. > > > > 28., K, 16:13): > > > > > > > > > Thank you Xintong for your input. > > > > > > > > > > I prepared a FLIP for this change [1], looking forward for any > > > > > other opinions. > > > > > > > > > > Thanks, > > > > > Ferenc > > > > > > > > > > [1] > > > > https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing > > > > > > > On Friday, 17 May 2024 at 07:04, Xintong Song tonysong...@gmail.com > > > > > wrote: > > > > > > > > > > > AFAIK, the main purpose of having `run-application` was to make > > > > > > sure > > > > > > the user is aware that application mode is used, which executes the > > > > > > main > > > > > > method of the user program in JM rather than in client. This was > > > > > > important > > > > > > at the time application mode was first introduced, but maybe not > > > > > > that > > > > > > important anymore, given that per-job mode is deprecated and likely > > > > > > removed > > > > > > in 2.0. Therefore, +1 for the proposal. > > > > > > > > > > > > Best, > > > > > > > > > > > > Xintong > > > > > > > > > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky > > > > > > ferenc.cs...@pm.me.invalid > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hello devs, > > > > > > > > > > > > > > I saw quite some examples when customers were confused about > > > > > > > run, and > > > > > > > run- > > > > > > > application in the Flink CLI and I was wondering about the > > > > > > > necessity > > > > > > > of > > > > > > > deploying > > > > > > > Application Mode (AM) jobs with a different command, than > > > > > > > Session and > > > > > > > Per-Job mode jobs. > > > > > > > > > > > > > > I can see a point that YarnDeploymentTarget [1] and > > > > > > > KubernetesDeploymentTarget > > > > > > > [2] are part of their own maven modules and not known in > > > > > > > flink-clients, > > > > > > > so the > > > > > > > deployment mode validations are happening during cluster > > > > > > > deployment > > > > > > > in > > > > > > > their specific > > > > > >
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hi Hang, Thank you for your inputs, both points make sense, updated the FLIP according to them. Best, Ferenc On Friday, 31 May 2024 at 04:31, Hang Ruan wrote: > > > Hi, Ferenc. > > +1 for this proposal. This FLIP will help to make the CLI clearer for users. > > I think we should better add an example in the FLIP about how to use the > application mode with the new CLI. > Besides that, we need to add some new tests for this change instead of only > using the existed tests. > > Best, > Hang > > Mate Czagany czmat...@gmail.com 于2024年5月29日周三 19:57写道: > > > Hi Ferenc, > > > > Thanks for the FLIP, +1 from me for the proposal. I think these changes > > would be a great solution to all the confusion that comes from these two > > action parameters. > > > > Best regards, > > Mate > > > > Ferenc Csaky ferenc.cs...@pm.me.invalid ezt írta (időpont: 2024. máj. > > 28., K, 16:13): > > > > > Thank you Xintong for your input. > > > > > > I prepared a FLIP for this change [1], looking forward for any > > > other opinions. > > > > > > Thanks, > > > Ferenc > > > > > > [1] > > > > https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing > > > > > On Friday, 17 May 2024 at 07:04, Xintong Song tonysong...@gmail.com > > > wrote: > > > > > > > AFAIK, the main purpose of having `run-application` was to make sure > > > > the user is aware that application mode is used, which executes the > > > > main > > > > method of the user program in JM rather than in client. This was > > > > important > > > > at the time application mode was first introduced, but maybe not that > > > > important anymore, given that per-job mode is deprecated and likely > > > > removed > > > > in 2.0. Therefore, +1 for the proposal. > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky > > > > ferenc.cs...@pm.me.invalid > > > > > > > > wrote: > > > > > > > > > Hello devs, > > > > > > > > > > I saw quite some examples when customers were confused about run, and > > > > > run- > > > > > application in the Flink CLI and I was wondering about the necessity > > > > > of > > > > > deploying > > > > > Application Mode (AM) jobs with a different command, than Session and > > > > > Per-Job mode jobs. > > > > > > > > > > I can see a point that YarnDeploymentTarget [1] and > > > > > KubernetesDeploymentTarget > > > > > [2] are part of their own maven modules and not known in > > > > > flink-clients, > > > > > so the > > > > > deployment mode validations are happening during cluster deployment > > > > > in > > > > > their specific > > > > > ClusterDescriptor implementation [3]. Although these are > > > > > implementation > > > > > details that > > > > > IMO should not define user-facing APIs. > > > > > > > > > > The command line setup is the same for both run and run-application, > > > > > so > > > > > I think there > > > > > is a quite simple way to achieve a unified flink run experience, but > > > > > I > > > > > might missed > > > > > something so I would appreciate any inputs on this topic. > > > > > > > > > > Based on my assumptions I think it would be possible to deprecate the > > > > > run- > > > > > application in Flink 1.20 and remove it completely in Flink 2.0. I > > > > > already put together a > > > > > PoC [4], and I was able to deploy AM jobs like this: > > > > > > > > > > flink run --target kubernetes-application ... > > > > > > > > > > If others also agree with this, I would be happy to open a FLIP. > > > > > WDYT? > > > > > > > > > > Thanks, > > > > > Ferenc > > > > > > > > > > [1] > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > > > > > > [2] > > > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > > > > > > > [3] > > > > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > > > > > > > [4] > > > > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0
Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Thank you Xintong for your input. I prepared a FLIP for this change [1], looking forward for any other opinions. Thanks, Ferenc [1] https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing On Friday, 17 May 2024 at 07:04, Xintong Song wrote: > > > AFAIK, the main purpose of having `run-application` was to make sure > the user is aware that application mode is used, which executes the main > method of the user program in JM rather than in client. This was important > at the time application mode was first introduced, but maybe not that > important anymore, given that per-job mode is deprecated and likely removed > in 2.0. Therefore, +1 for the proposal. > > Best, > > Xintong > > > > On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hello devs, > > > > I saw quite some examples when customers were confused about run, and run- > > application in the Flink CLI and I was wondering about the necessity of > > deploying > > Application Mode (AM) jobs with a different command, than Session and > > Per-Job mode jobs. > > > > I can see a point that YarnDeploymentTarget [1] and > > KubernetesDeploymentTarget > > [2] are part of their own maven modules and not known in flink-clients, > > so the > > deployment mode validations are happening during cluster deployment in > > their specific > > ClusterDescriptor implementation [3]. Although these are implementation > > details that > > IMO should not define user-facing APIs. > > > > The command line setup is the same for both run and run-application, so > > I think there > > is a quite simple way to achieve a unified flink run experience, but I > > might missed > > something so I would appreciate any inputs on this topic. > > > > Based on my assumptions I think it would be possible to deprecate the run- > > application in Flink 1.20 and remove it completely in Flink 2.0. I > > already put together a > > PoC [4], and I was able to deploy AM jobs like this: > > > > flink run --target kubernetes-application ... > > > > If others also agree with this, I would be happy to open a FLIP. WDYT? > > > > Thanks, > > Ferenc > > > > [1] > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java > > [2] > > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java > > [3] > > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 > > [4] > > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0
[DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0
Hello devs, I saw quite some examples when customers were confused about run, and run- application in the Flink CLI and I was wondering about the necessity of deploying Application Mode (AM) jobs with a different command, than Session and Per-Job mode jobs. I can see a point that YarnDeploymentTarget [1] and KubernetesDeploymentTarget [2] are part of their own maven modules and not known in flink-clients, so the deployment mode validations are happening during cluster deployment in their specific ClusterDescriptor implementation [3]. Although these are implementation details that IMO should not define user-facing APIs. The command line setup is the same for both run and run-application, so I think there is a quite simple way to achieve a unified flink run experience, but I might missed something so I would appreciate any inputs on this topic. Based on my assumptions I think it would be possible to deprecate the run- application in Flink 1.20 and remove it completely in Flink 2.0. I already put together a PoC [4], and I was able to deploy AM jobs like this: flink run --target kubernetes-application ... If others also agree with this, I would be happy to open a FLIP. WDYT? Thanks, Ferenc [1] https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java [2] https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java [3] https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206 [4] https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0
Re: [VOTE] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
+1 (non-binding) Thanks, Ferenc On Tuesday, 14 May 2024 at 08:51, weijie guo wrote: > > > Thanks Martijn for the effort! > > +1(binding) > > Best regards, > > Weijie > > > Martijn Visser martijnvis...@apache.org 于2024年5月14日周二 14:45写道: > > > Hi everyone, > > > > With no more discussions being open in the thread [1] I would like to start > > a vote on FLIP-453: Promote Unified Sink API V2 to Public and Deprecate > > SinkFunction [2] > > > > The vote will be open for at least 72 hours unless there is an objection or > > insufficient votes. > > > > Best regards, > > > > Martijn > > > > [1] https://lists.apache.org/thread/hod6bg421bzwhbfv60lwsck7r81dvo59 > > [2] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-453%3A+Promote+Unified+Sink+API+V2+to+Public+and+Deprecate+SinkFunction
Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction
Hi Martijn, +1 for the proposal. > targeted for Flink 1.19 I guess you meant Flink 1.20 here. Also, I volunteer to take updating the HBase sink, feel free to assign that task to me. Best, Ferenc On Friday, May 3rd, 2024 at 10:20, Martijn Visser wrote: > > > Hi Peter, > > I'll add it for completeness, thanks! > With regards to FLINK-35149, the fix version indicates a change at Flink > CDC; is that indeed correct, or does it require a change in the SinkV2 > interface? > > Best regards, > > Martijn > > > On Fri, May 3, 2024 at 7:47 AM Péter Váry peter.vary.apa...@gmail.com > > wrote: > > > Hi Martijn, > > > > We might want to add FLIP-371 [1] to the list. (Or we aim only for higher > > level FLIPs?) > > > > We are in the process of using the new API in Iceberg connector [2] - so > > far, so good. > > > > I know of one minor known issue about the sink [3], which should be ready > > for the release. > > > > All-in-all, I think we are in good shape, and we could move forward with > > the promotion. > > > > Thanks, > > Peter > > > > [1] - > > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=263430387 > > [2] - https://github.com/apache/iceberg/pull/10179 > > [3] - https://issues.apache.org/jira/browse/FLINK-35149 > > > > On Thu, May 2, 2024, 09:47 Muhammet Orazov mor+fl...@morazow.com.invalid > > wrote: > > > > > Got it, thanks! > > > > > > On 2024-05-02 06:53, Martijn Visser wrote: > > > > > > > Hi Muhammet, > > > > > > > > Thanks for joining the discussion! The changes in this FLIP would be > > > > targeted for Flink 1.19, since it's only a matter of changing the > > > > annotation. > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > On Thu, May 2, 2024 at 7:26 AM Muhammet Orazov mor+fl...@morazow.com > > > > wrote: > > > > > > > > > Hello Martijn, > > > > > > > > > > Thanks for the FLIP and detailed history of changes, +1. > > > > > > > > > > Would FLIP changes target for 2.0? I think it would be good > > > > > to have clear APIs on 2.0 release. > > > > > > > > > > Best, > > > > > Muhammet > > > > > > > > > > On 2024-05-01 15:30, Martijn Visser wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > I would like to start a discussion on FLIP-453: Promote Unified Sink > > > > > > API V2 > > > > > > to Public and Deprecate SinkFunction > > > > > > https://cwiki.apache.org/confluence/x/rIobEg > > > > > > > > > > > > This FLIP proposes to promote the Unified Sink API V2 from > > > > > > PublicEvolving > > > > > > to Public and to mark the SinkFunction as Deprecated. > > > > > > > > > > > > I'm looking forward to your thoughts. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn
Re: [VOTE] FLIP-446: Kubernetes Operator State Snapshot CRD
+1 (non-binding), looking forward to this! Best, Ferenc On Wednesday, April 24th, 2024 at 10:03, Mate Czagany wrote: > > > Hi everyone, > > I'd like to start a vote on the FLIP-446: Kubernetes Operator State > Snapshot CRD [1]. The discussion thread is here [2]. > > The vote will be open for at least 72 hours unless there is an objection or > insufficient votes. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-446%3A+Kubernetes+Operator+State+Snapshot+CRD > [2] https://lists.apache.org/thread/q5dzjwj0qk34rbg2sczyypfhokxoc3q7 > > Regards, > Mate
Re: [DISCUSS] FLIP-438: Make Flink's Hadoop and YARN configuration probing consistent
Hi Venkata krishnan, My general point was that personally I do not think that the current implementation is wrong or confusing. And the main thing here is that how we define consistent in this case is subjective. >From the proposed point of view, consistent mean we use the same prefix. But we can consistently use the least required prefix groups to identify a subsystem property, which is how it works right now. The prop naming conventions of these dependent systems are different, so do their prefixes in Flink. It is very possible I am in minority with my view, but I do not think duplicating the `yarn` prefix to make it conform with Hadoop props would be a better UX as it is now, just different. One thing that less opinionated maybe is if the proposed solution simplifies the property load logic. Currently, Hadoop props from the Flink conf are parsed in `HadoopUtils` (flink-hadoop-fs module), while Yarn props in `Utils` (flink-yarn module). Maybe `org.apache.flink.configuration.Configuration` could have a helper to extract all prefixed properties to a `Map`, or another `Configuration` object (the latter could be easily added as a resource to the dependent system config objects). That simplification could make the overall prefixed load logic more clean IMO and that is something that would be useful. WDYT? Best, Ferenc On Monday, April 15th, 2024 at 20:55, Venkatakrishnan Sowrirajan wrote: > > > Sorry for the late reply, Ferenc. > > I understand the rationale behind the current implementation as the problem > is slightly different b/w yarn (always prefixed with `yarn`) and hadoop (it > is not guaranteed all `hadoop` configs will be prefixed by `hadoop`) > configs. > > From the dev UX perspective, it is confusing and only if you really pay > close attention to the docs it is evident. I understand your point on added > complexity till Flink-3.0 but if we agree it should be made consistent, it > has to be done at some point of time right? > > Regards > Venkata krishnan > > > On Wed, Apr 3, 2024 at 4:51 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi Venkata, > > > > Thank you for opening the discussion about this! > > > > After taking a look at the YARN and Hadoop configurations, the > > reason why it was implemented this way is that, in case of YARN, > > every YARN-specific property is prefixed with "yarn.", so to get > > the final, YARN-side property it is enough to remove the "flink." > > prefix. > > > > In case of Hadoop, there are properties that not prefixed with > > "hadoop.", e.g. "dfs.replication" so to identify and get the > > Hadoop-side property it is necessary to duplicate the "hadoop" part > > in the properties. > > > > Taking this into consideration I would personally say -0 to this > > change. IMO the current behavior can be justified as giving > > slightly different solutions to slightly different problems, which > > are well documented. Handling both prefixes would complicate the > > parsing logic until the APIs can be removed, which as it looks at > > the moment would only be possible in Flink 3.0, which probably will > > not happen in the foreseeable future, so I do not see the benefit > > of the added complexity. > > > > Regarding the FLIP, in the "YARN configuration override example" > > part, I think you should present an example that works correctly > > at the moment: "flink.yarn.application.classpath" -> > > "yarn.application.classpath". > > > > Best, > > Ferenc > > > > On Friday, March 29th, 2024 at 23:45, Venkatakrishnan Sowrirajan < > > vsowr...@asu.edu> wrote: > > > > > Hi Flink devs, > > > > > > I would like to start a discussion on FLIP-XXX: Make Flink's Hadoop and > > > YARN configuration probing consistent > > > > https://urldefense.com/v3/__https://docs.google.com/document/d/1I2jBFI0eVkofAVCAEeajNQRfOqKGJsRfZd54h79AIYc/edit?usp=sharing__;!!IKRxdwAv5BmarQ!d0XJO_mzLCJZNkrjJDMyRGP95zPLW8Cuym88l7CoAUG8aD_KRYJbll3K-q1Ypplyqe6-jcsWq3S8YJqrDMCpK4IhpT4cZPXy$ > > . > > > > > This stems from an earlier discussion thread here > > > > https://urldefense.com/v3/__https://lists.apache.org/thread/l2fh5shbf59fjgbt1h73pmmsqj038ppv__;!!IKRxdwAv5BmarQ!d0XJO_mzLCJZNkrjJDMyRGP95zPLW8Cuym88l7CoAUG8aD_KRYJbll3K-q1Ypplyqe6-jcsWq3S8YJqrDMCpK4IhpW60A99X$ > > . > > > > > This FLIP is proposing to make the configuration probing behavior between > > > Hadoop and YARN configuration to be consistent. > > > > > > Regards > > > Venkata krishnan
Re: [VOTE] FLIP-435: Introduce a New Materialized Table for Simplifying Data Pipelines
+1 (non-binding) Best, Ferenc On Wednesday, April 17th, 2024 at 10:26, Ahmed Hamdy wrote: > > > + 1 (non-binding) > > Best Regards > Ahmed Hamdy > > > On Wed, 17 Apr 2024 at 08:28, Yuepeng Pan panyuep...@apache.org wrote: > > > +1(non-binding). > > > > Best, > > Yuepeng Pan > > > > At 2024-04-17 14:27:27, "Ron liu" ron9@gmail.com wrote: > > > > > Hi Dev, > > > > > > Thank you to everyone for the feedback on FLIP-435: Introduce a New > > > Materialized Table for Simplifying Data Pipelines[1][2]. > > > > > > I'd like to start a vote for it. The vote will be open for at least 72 > > > hours unless there is an objection or not enough votes. > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-435%3A+Introduce+a+New+Materialized+Table+for+Simplifying+Data+Pipelines > > > > > [2] https://lists.apache.org/thread/c1gnn3bvbfs8v1trlf975t327s4rsffs > > > > > > Best, > > > Ron
Re: [DISCUSS] Connector releases for Flink 1.19
Thank you Danny and Sergey for pushing this! I can help with the HBase connector if necessary, will comment the details to the relevant Jira ticket. Best, Ferenc On Wednesday, April 17th, 2024 at 11:17, Danny Cranmer wrote: > > > Hello all, > > I have created a parent Jira to cover the releases [1]. I have assigned AWS > and MongoDB to myself and OpenSearch to Sergey. Please assign the > relevant issue to yourself as you pick up the tasks. > > Thanks! > > [1] https://issues.apache.org/jira/browse/FLINK-35131 > > On Tue, Apr 16, 2024 at 2:41 PM Muhammet Orazov > mor+fl...@morazow.com.invalid wrote: > > > Thanks Sergey and Danny for clarifying, indeed it > > requires committer to go through the process. > > > > Anyway, please let me know if I can be any help. > > > > Best, > > Muhammet > > > > On 2024-04-16 11:19, Danny Cranmer wrote: > > > > > Hello, > > > > > > I have opened the VOTE thread for the AWS connectors release [1]. > > > > > > > If I'm not mistaking (please correct me if I'm wrong) this request is > > > > not > > > > about version update it is about new releases for connectors > > > > > > Yes, correct. If there are any other code changes required then help > > > would be appreciated. > > > > > > > Are you going to create an umbrella issue for it? > > > > > > We do not usually create JIRA issues for releases. That being said it > > > sounds like a good idea to have one place to track the status of the > > > connector releases and pre-requisite code changes. > > > > > > > I would like to work on this task, thanks for initiating it! > > > > > > The actual release needs to be performed by a committer. However, help > > > getting the connectors building against Flink 1.19 and testing the RC > > > is > > > appreciated. > > > > > > Thanks, > > > Danny > > > > > > [1] https://lists.apache.org/thread/0nw9smt23crx4gwkf6p1dd4jwvp1g5s0 > > > > > > On Tue, Apr 16, 2024 at 6:34 AM Sergey Nuyanzin snuyan...@gmail.com > > > wrote: > > > > > > > Thanks for volunteering Muhammet! > > > > And thanks Danny for starting the activity. > > > > > > > > If I'm not mistaking (please correct me if I'm wrong) > > > > > > > > this request is not about version update it is about new releases for > > > > connectors > > > > btw for jdbc connector support of 1.19 and 1.20-SNAPSHOT is already > > > > done > > > > > > > > I would volunteer for Opensearch connector since currently I'm working > > > > on > > > > support of Opensearch v2 > > > > and I think it would make sense to have a release after it is done > > > > > > > > On Tue, Apr 16, 2024 at 4:29 AM Muhammet Orazov > > > > mor+fl...@morazow.com.invalid wrote: > > > > > > > > > Hello Danny, > > > > > > > > > > I would like to work on this task, thanks for initiating it! > > > > > > > > > > I could update the versions on JDBC and Pulsar connectors. > > > > > > > > > > Are you going to create an umbrella issue for it? > > > > > > > > > > Best, > > > > > Muhammet > > > > > > > > > > On 2024-04-15 13:44, Danny Cranmer wrote: > > > > > > > > > > > Hello all, > > > > > > > > > > > > Flink 1.19 was released on 2024-03-18 [1] and the connectors have > > > > > > not > > > > > > yet > > > > > > caught up. I propose we start releasing the connectors with support > > > > > > for > > > > > > Flink 1.19 as per the connector support guidelines [2]. > > > > > > > > > > > > I will make a start on flink-connector-aws, then pickup others in > > > > > > the > > > > > > coming days. Please respond to the thread if you are/want to work on > > > > > > a > > > > > > particular connector to avoid duplicate work. > > > > > > > > > > > > Thanks, > > > > > > Danny > > > > > > > > > > > > [1] > > > > https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/ > > > > > > > > [2] > > > > https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development#ExternalizedConnectordevelopment-Flinkcompatibility > > > > > > > > [3] https://github.com/apache/flink-connector-aws > > > > > > > > -- > > > > Best regards, > > > > Sergey
Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan
Congratulations! Best, Ferenc On Tuesday, April 16th, 2024 at 16:28, Jeyhun Karimov wrote: > > > Congratulations Zakelly! > > Regards, > Jeyhun > > On Tue, Apr 16, 2024 at 6:35 AM Feifan Wang zoltar9...@163.com wrote: > > > Congratulations, Zakelly!—— > > > > Best regards, > > > > Feifan Wang > > > > At 2024-04-15 10:50:06, "Yuan Mei" yuanmei.w...@gmail.com wrote: > > > > > Hi everyone, > > > > > > On behalf of the PMC, I'm happy to let you know that Zakelly Lan has > > > become > > > a new Flink Committer! > > > > > > Zakelly has been continuously contributing to the Flink project since > > > 2020, > > > with a focus area on Checkpointing, State as well as frocksdb (the default > > > on-disk state db). > > > > > > He leads several FLIPs to improve checkpoints and state APIs, including > > > File Merging for Checkpoints and configuration/API reorganizations. He is > > > also one of the main contributors to the recent efforts of "disaggregated > > > state management for Flink 2.0" and drives the entire discussion in the > > > mailing thread, demonstrating outstanding technical depth and breadth of > > > knowledge. > > > > > > Beyond his technical contributions, Zakelly is passionate about helping > > > the > > > community in numerous ways. He spent quite some time setting up the Flink > > > Speed Center and rebuilding the benchmark pipeline after the original one > > > was out of lease. He helps build frocksdb and tests for the upcoming > > > frocksdb release (bump rocksdb from 6.20.3->8.10). > > > > > > Please join me in congratulating Zakelly for becoming an Apache Flink > > > committer! > > > > > > Best, > > > Yuan (on behalf of the Flink PMC)
Re: [DISCUSS] FLIP-446: Kubernetes Operator State Snapshot CRD
Thank you Mate for initiating this discussion. +1 for this idea. Some Qs: Can you specify the newly introduced configurations in more details? Currently, it is not fully clear to me what are the possible values of `kubernetes.operator.periodic.savepoint.mode`, is it optional, has a default value? I see that in `SavepointSpec.formatType` has a default, although `CheckppointSpec.checkpointType` not. Are we inferring that from the config? My point is, in general I think it would be good to handle the two snapshot types in a similar way when it makes sense to minimize any kind of confusion. Best, Ferenc On Tuesday, April 16th, 2024 at 11:34, Mate Czagany wrote: > > > Hi Everyone, > > I would like to start a discussion on FLIP-446: Kubernetes Operator State > Snapshot CRD. > > This FLIP adds a new custom resource for Operator users to create and > manage their savepoints and checkpoints. I have also developed an initial > POC to prove that this approach is feasible, you can find the link for that > in the FLIP. > > There is a Confluence page [1] and a Google Docs page [2] as I do not have > a Confluence account yet. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-446%3A+Kubernetes+Operator+State+Snapshot+CRD > [2] > https://docs.google.com/document/d/1VdfLFaE4i6ESbCQ38CH7TKOiPQVvXeOxNV2FeSMnOTg > > > Regards, > Mate
[jira] [Created] (FLINK-35114) Remove old Table API implementations
Ferenc Csaky created FLINK-35114: Summary: Remove old Table API implementations Key: FLINK-35114 URL: https://issues.apache.org/jira/browse/FLINK-35114 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky At the moment, the connector has both the old Table sink/source/catalog implementations and the matching Dynamic... implementations as well. Going forward, the deprecated old implementation should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee
Congratulations, Lincoln! Best, Ferenc On Friday, April 12th, 2024 at 15:54, lorenzo.affe...@ververica.com.INVALID wrote: > > > Huge congrats! Well done! > On Apr 12, 2024 at 13:56 +0200, Ron liu ron9@gmail.com, wrote: > > > Congratulations, Lincoln! > > > > Best, > > Ron > > > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道: > > > > > Congratulations, Lincoln! > > > > > > Best, > > > Junrui > > > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:29写道: > > > > > > > > Congratulations, Lincoln! > > > > > > > > > > Best Regards > > > > > Aleksandr
Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge
Congratulations, Jing! Best, Ferenc On Friday, April 12th, 2024 at 13:54, Ron liu wrote: > > > Congratulations, Jing! > > Best, > Ron > > Junrui Lee jrlee@gmail.com 于2024年4月12日周五 18:54写道: > > > Congratulations, Jing! > > > > Best, > > Junrui > > > > Aleksandr Pilipenko z3d...@gmail.com 于2024年4月12日周五 18:28写道: > > > > > Congratulations, Jing! > > > > > > Best Regards, > > > Aleksandr
Re: [VOTE] FLIP-399: Flink Connector Doris
+1 (non-binding) Best, Ferenc On Tuesday, April 9th, 2024 at 10:32, Ahmed Hamdy wrote: > > > Hi Wudi, > > +1 (non-binding). > > Best Regards > Ahmed Hamdy > > > On Tue, 9 Apr 2024 at 09:21, Yuepeng Pan panyuep...@apache.org wrote: > > > Hi, Di. > > > > Thank you for driving it ! > > > > +1 (non-binding). > > > > Best, > > Yuepeng Pan > > > > On 2024/04/09 02:47:55 wudi wrote: > > > > > Hi devs, > > > > > > I would like to start a vote about FLIP-399 [1]. The FLIP is about > > > contributing the Flink Doris Connector[2] to the Flink community. > > > Discussion thread [3]. > > > > > > The vote will be open for at least 72 hours unless there is an objection > > > or > > > insufficient votes. > > > > > > Thanks, > > > Di.Wu > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-399%3A+Flink+Connector+Doris > > > [2] https://github.com/apache/doris-flink-connector > > > [3] https://lists.apache.org/thread/p3z4wsw3ftdyfs9p2wd7bbr2gfyl3xnh
Re: [DISCUSS] FLIP-438: Make Flink's Hadoop and YARN configuration probing consistent
Hi Venkata, Thank you for opening the discussion about this! After taking a look at the YARN and Hadoop configurations, the reason why it was implemented this way is that, in case of YARN, every YARN-specific property is prefixed with "yarn.", so to get the final, YARN-side property it is enough to remove the "flink." prefix. In case of Hadoop, there are properties that not prefixed with "hadoop.", e.g. "dfs.replication" so to identify and get the Hadoop-side property it is necessary to duplicate the "hadoop" part in the properties. Taking this into consideration I would personally say -0 to this change. IMO the current behavior can be justified as giving slightly different solutions to slightly different problems, which are well documented. Handling both prefixes would complicate the parsing logic until the APIs can be removed, which as it looks at the moment would only be possible in Flink 3.0, which probably will not happen in the foreseeable future, so I do not see the benefit of the added complexity. Regarding the FLIP, in the "YARN configuration override example" part, I think you should present an example that works correctly at the moment: "flink.yarn.application.classpath" -> "yarn.application.classpath". Best, Ferenc On Friday, March 29th, 2024 at 23:45, Venkatakrishnan Sowrirajan wrote: > > > Hi Flink devs, > > I would like to start a discussion on FLIP-XXX: Make Flink's Hadoop and > YARN configuration probing consistent > https://docs.google.com/document/d/1I2jBFI0eVkofAVCAEeajNQRfOqKGJsRfZd54h79AIYc/edit?usp=sharing. > > This stems from an earlier discussion thread here > https://lists.apache.org/thread/l2fh5shbf59fjgbt1h73pmmsqj038ppv. > > > This FLIP is proposing to make the configuration probing behavior between > Hadoop and YARN configuration to be consistent. > > Regards > Venkata krishnan
Re: [DISCUSS] FLIP-XXX: Introduce Flink SQL variables
Hi Jeyhun, Thank you for your questions, please see my answers below. > What is its impact on query optimization because resolving > variables at the parsing stage might affect query optimization. The approach I mentioned in the FLIP would not affect query optimization, as it restricts variables to be literals, hence do not support calculated variables. This means that the substitution would be a simple string replace for the variables before the actual parse happens on the already resolved statement. Although this may not be the way to go, according to Yanfei's previous comments I started to check on possible solutions for calculated variables, which will probably change this answer, but I will report back when I have something regarding this topic. > What is the scope of variables? I mean when and how they override > each other and when get out of their scopes? This is a good question, I did not mention this in the FLIP. My thinking on this topic is that a VariableStore is tied to a SQL session, so variables are session scoped. Having a system-wide scope might make sense. In that case, the system-wide variable should be shadowed by the same session-wide variable IMO, as a general rule regarding variable shadowing [1]. Although I did not include system-wide scope in my PoC, but this would basically mean to maintain a specific system-wide VariableStore. > Does the proposal support dynamic assignment of the variables or > the value of variables should be known at query compile time? Covered this in my answer to the first Q. > Can we somehow benefit from/leverage Calcite's parameterization > feature in this proposal? I am not super familiar with Calcite capabilities regarding this topic and the Calcite docs were not really helpful either. But I might looked over something, so can you elaborate more / point me towards what you mean? Best, Ferenc [1] https://en.wikipedia.org/wiki/Variable_shadowing On Monday, April 1st, 2024 at 15:24, Jeyhun Karimov wrote: > > > Hi Ferenc, > > Thanks for the proposal. Sounds like a good idea! > I have a few questions on that: > > - What is its impact on query optimization because resolving variables at > the parsing stage might affect query optimization. > > - What is the scope of variables? I mean when and how they override each > other and when get out of their scopes? > > - Does the proposal support dynamic assignment of the variables or the > value of variables should be known at query compile time? > > - Can we somehow benefit from/leverage Calcite's parameterization feature > in this proposal? > > Regards, > Jeyhun > > On Thu, Mar 28, 2024 at 6:21 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi, Jim, Yanfei, > > > > Thanks for your comments! Let me reply in the order of the > > messages. > > > > > I'd prefer sticking to the SQL standard if possible. Would > > > it be possible / sensible to allow for each syntax, perhaps > > > managed by a config setting? > > > > Correct me if I am wrong, but AFAIK variables are not part of > > the ANSI SQL standard. The '@' prefix is used by some widely > > used DB mgr, e.g. MySQL. > > > > Regarding having multiple resolution syntax, it would be possible, > > if we agree it adds value. Personally I do not have a strong > > opinion on that. > > > > > I'm new to Flink SQL and I'm curious if these variables can be > > > calculated from statements or expression [1]? > > > > Good point! The proposed solution would lack this functionality. > > On our platform, we have a working solution of this that was > > sufficient to solve the main problem we had to carry SQL between > > environments without change. > > > > At this point, variable values can only be literals, and they are > > automatically escaped during resolution. Except if they are > > resolved as a DDL statement property value. > > > > But if the community agrees that it would be useful to have the > > ability of calculated variables I would happily spend some time > > on possible solutions that makes sense in Flink. > > > > WDYT? > > > > Best, > > Ferenc > > > > On Thursday, March 28th, 2024 at 03:58, Yanfei Lei fredia...@gmail.com > > wrote: > > > > > Hi Ferenc, > > > > > > Thanks for the proposal, using SQLvariables to exclude > > > environment-specific configuration from code sounds like a good idea. > > > > > > I'm new to Flink SQL and I'm curious if these variables can be > > > calculated from statements or expression [1]? In FLIP, it seems that > > > the values are in the form of Strin
Re: [DISCUSS] FLIP-XXX: Introduce Flink SQL variables
Hi, Jim, Yanfei, Thanks for your comments! Let me reply in the order of the messages. > I'd prefer sticking to the SQL standard if possible. Would > it be possible / sensible to allow for each syntax, perhaps > managed by a config setting? Correct me if I am wrong, but AFAIK variables are not part of the ANSI SQL standard. The '@' prefix is used by some widely used DB mgr, e.g. MySQL. Regarding having multiple resolution syntax, it would be possible, if we agree it adds value. Personally I do not have a strong opinion on that. > I'm new to Flink SQL and I'm curious if these variables can be > calculated from statements or expression [1]? Good point! The proposed solution would lack this functionality. On our platform, we have a working solution of this that was sufficient to solve the main problem we had to carry SQL between environments without change. At this point, variable values can only be literals, and they are automatically escaped during resolution. Except if they are resolved as a DDL statement property value. But if the community agrees that it would be useful to have the ability of calculated variables I would happily spend some time on possible solutions that makes sense in Flink. WDYT? Best, Ferenc On Thursday, March 28th, 2024 at 03:58, Yanfei Lei wrote: > > > Hi Ferenc, > > Thanks for the proposal, using SQLvariables to exclude > environment-specific configuration from code sounds like a good idea. > > I'm new to Flink SQL and I'm curious if these variables can be > calculated from statements or expression [1]? In FLIP, it seems that > the values are in the form of StringLiteral. > > > [1] > https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-aux-set-variable.html > > Jim Hughes jhug...@confluent.io.invalid 于2024年3月28日周四 04:54写道: > > > Hi Ferenc, > > > > Looks like a good idea. > > > > I'd prefer sticking to the SQL standard if possible. Would it be possible > > / sensible to allow for each syntax, perhaps managed by a config setting? > > > > Cheers, > > > > Jim > > > > On Tue, Mar 26, 2024 at 6:59 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > > > Hello devs, > > > > > > I would like to start a discussion about FLIP-XXX: Introduce Flink SQL > > > variables [1]. > > > > > > The main motivation behing this change is to be able to abstract Flink SQL > > > from > > > environment-specific configuration and provide a way to carry jobs between > > > environments (e.g. dev-stage-prod) without the need to make changes in the > > > code. > > > It can also be a way to decouple sensitive information from the job code, > > > or help > > > with redundant literals. > > > > > > The main decision regarding the proposed solution is to handle the > > > variable resolution > > > as early as possible on the given string statement, so the whole operation > > > is an easy and > > > lightweight string replace. But this approach introduces some limitations > > > as well: > > > > > > - The executed SQL will always be the unresolved, raw string, so in case > > > of secrets > > > a DESC operation would show them. > > > - Changing the value of a variable can break code that uses that variable. > > > > > > For more details, please check the FLIP [1]. There is also a stale Jira > > > about this [2]. > > > > > > Looking forward to any comments and opinions! > > > > > > Thanks, > > > Ferenc > > > > > > [1] > > > https://docs.google.com/document/d/1-eUz-PBCdqNggG_irDT0X7fdL61ysuHOaWnrkZHb5Hc/edit?usp=sharing > > > [2] https://issues.apache.org/jira/browse/FLINK-17377 > > > > > -- > Best, > Yanfei
[DISCUSS] FLIP-XXX: Introduce Flink SQL variables
Hello devs, I would like to start a discussion about FLIP-XXX: Introduce Flink SQL variables [1]. The main motivation behing this change is to be able to abstract Flink SQL from environment-specific configuration and provide a way to carry jobs between environments (e.g. dev-stage-prod) without the need to make changes in the code. It can also be a way to decouple sensitive information from the job code, or help with redundant literals. The main decision regarding the proposed solution is to handle the variable resolution as early as possible on the given string statement, so the whole operation is an easy and lightweight string replace. But this approach introduces some limitations as well: - The executed SQL will always be the unresolved, raw string, so in case of secrets a DESC operation would show them. - Changing the value of a variable can break code that uses that variable. For more details, please check the FLIP [1]. There is also a stale Jira about this [2]. Looking forward to any comments and opinions! Thanks, Ferenc [1] https://docs.google.com/document/d/1-eUz-PBCdqNggG_irDT0X7fdL61ysuHOaWnrkZHb5Hc/edit?usp=sharing [2] https://issues.apache.org/jira/browse/FLINK-17377
Re: [DISCUSS] Flink Website Menu Adjustment
Suggested changes makes sense, +1 for the proposed menus and order. Best, Ferenc On Monday, March 25th, 2024 at 14:50, Gyula Fóra wrote: > > > +1 for the proposal > > Gyula > > On Mon, Mar 25, 2024 at 12:49 PM Leonard Xu xbjt...@gmail.com wrote: > > > Thanks Zhongqiang for starting this discussion, updating documentation > > menus according to sub-projects' activities makes sense to me. > > > > +1 for the proposed menus: > > > > > After: > > > > > > With Flink > > > With Flink Kubernetes Operator > > > With Flink CDC > > > With Flink ML > > > With Flink Stateful Functions > > > Training Course > > > > Best, > > Leonard > > > > > 2024年3月25日 下午3:48,gongzhongqiang gongzhongqi...@apache.org 写道: > > > > > > Hi everyone, > > > > > > I'd like to start a discussion on adjusting the Flink website [1] menu to > > > improve accuracy and usability.While migrating Flink CDC documentation > > > to the website, I found outdated links, need to review and update menus > > > for the most relevant information for our users. > > > > > > Proposal: > > > > > > - Remove Paimon [2] from the "Getting Started" and "Documentation" menus: > > > Paimon [2] is now an independent top project of ASF. CC: jingsong lees > > > > > > - Sort the projects in the subdirectory by the activity of the projects. > > > Here I list the number of releases for each project in the past year. > > > > > > Flink Kubernetes Operator : 7 > > > Flink CDC : 5 > > > Flink ML : 2 > > > Flink Stateful Functions : 1 > > > > > > Expected Outcome : > > > > > > - Menu "Getting Started" > > > > > > Before: > > > > > > With Flink > > > > > > With Flink Stateful Functions > > > > > > With Flink ML > > > > > > With Flink Kubernetes Operator > > > > > > With Paimon(incubating) (formerly Flink Table Store) > > > > > > With Flink CDC > > > > > > Training Course > > > > > > After: > > > > > > With Flink > > > With Flink Kubernetes Operator > > > > > > With Flink CDC > > > > > > With Flink ML > > > > > > With Flink Stateful Functions > > > > > > Training Course > > > > > > - Menu "Documentation" will same with "Getting Started" > > > > > > I look forward to hearing your thoughts and suggestions on this proposal. > > > > > > [1] https://flink.apache.org/ > > > [2] https://github.com/apache/incubator-paimon > > > [3] https://github.com/apache/flink-statefun > > > > > > Best regards, > > > > > > Zhongqiang Gong
[jira] [Created] (FLINK-34931) Update Kudu connector DataStream Source/Sink implementation
Ferenc Csaky created FLINK-34931: Summary: Update Kudu connector DataStream Source/Sink implementation Key: FLINK-34931 URL: https://issues.apache.org/jira/browse/FLINK-34931 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky Update the DataSource API classes to use the current interfaces. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34930) Move existing Kudu connector code from Bahir repo to dedicated repo
Ferenc Csaky created FLINK-34930: Summary: Move existing Kudu connector code from Bahir repo to dedicated repo Key: FLINK-34930 URL: https://issues.apache.org/jira/browse/FLINK-34930 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34929) Create "flink-connector-kudu" repository
Ferenc Csaky created FLINK-34929: Summary: Create "flink-connector-kudu" repository Key: FLINK-34929 URL: https://issues.apache.org/jira/browse/FLINK-34929 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky We should create a "flink-connector-kudu" repositry under the "apache" GitHub organization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34928) FLIP-439: Externalize Kudu Connector from Bahir
Ferenc Csaky created FLINK-34928: Summary: FLIP-439: Externalize Kudu Connector from Bahir Key: FLINK-34928 URL: https://issues.apache.org/jira/browse/FLINK-34928 Project: Flink Issue Type: Improvement Reporter: Ferenc Csaky Umbrella issue for: https://cwiki.apache.org/confluence/display/FLINK/FLIP-439%3A+Externalize+Kudu+Connector+from+Bahir -- This message was sent by Atlassian Jira (v8.20.10#820010)
[RESULT][VOTE] FLIP-439: Externalize Kudu Connector from Bahir
Hi everyone, I'm happy to announce that FLIP-439: Externalize Kudu Connector from Bahir [1] has been accepted with 12 approving votes, 6 of which are binding: [2]: - Mate Czagany (non-binding) - Gyula Fóra (binding) - Gabor Somogyi (binding) - Mátyás Őrhidi (binding) - Hang Ruan (non-binding) - Samrat Deb (non-binding) - Gong Zhongqiang(non-binding) - Martijn Visser (binding) - Leonard Xu Jin (binding) - Marton Balassi (binding) - Jeyhun Karimov (non-binding) - Yuepeng Pan (non-binding) There are no disapproving votes. Thanks to everyone who participated in the discussion and voting. Best, Ferenc [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-439%3A+Externalize+Kudu+Connector+from+Bahir [2] https://lists.apache.org/thread/cjv0lq7x3m98dbhjk4p2wjps5rk2l9kj
[VOTE] FLIP-439: Externalize Kudu Connector from Bahir
Hello devs, I would like to start a vote about FLIP-439 [1]. The FLIP is about to externalize the Kudu connector from the recently retired Apache Bahir project [2] to keep it maintainable and make it up to date as well. Discussion thread [3]. The vote will be open for at least 72 hours (until 2024 March 23 14:03 UTC) unless there are any objections or insufficient votes. Thanks, Ferenc [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-439%3A+Externalize+Kudu+Connector+from+Bahir [2] https://attic.apache.org/projects/bahir.html [3] https://lists.apache.org/thread/oydhcfkco2kqp4hdd1glzy5vkw131rkz
Re: [VOTE] FLIP-402: Extend ZooKeeper Curator configurations
+1 (non-binding), thanks for driving this! Best, Ferenc On Wednesday, March 20th, 2024 at 10:57, Yang Wang wrote: > > > +1 (binding) since ZK HA is still widely used. > > > Best, > Yang > > On Thu, Mar 14, 2024 at 6:27 PM Matthias Pohl > matthias.p...@aiven.io.invalid wrote: > > > Nothing to add from my side. Thanks, Alex. > > > > +1 (binding) > > > > On Thu, Mar 7, 2024 at 4:09 PM Alex Nitavsky alexnitav...@gmail.com > > wrote: > > > > > Hi everyone, > > > > > > I'd like to start a vote on FLIP-402 [1]. It introduces new configuration > > > options for Apache Flink's ZooKeeper integration for high availability by > > > reflecting existing Apache Curator configuration options. It has been > > > discussed in this thread [2]. > > > > > > I would like to start a vote. The vote will be open for at least 72 > > > hours > > > (until March 10th 18:00 GMT) unless there is an objection or > > > insufficient votes. > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-402%3A+Extend+ZooKeeper+Curator+configurations > > > > > [2] https://lists.apache.org/thread/gqgs2jlq6bmg211gqtgdn8q5hp5v9l1z > > > > > > Thanks > > > Alex
Re: [VOTE] FLIP-436: Introduce Catalog-related Syntax
+1 (non-binding). Best, Ferenc On Tuesday, March 19th, 2024 at 12:39, Jark Wu wrote: > > > +1 (binding) > > Best, > Jark > > On Tue, 19 Mar 2024 at 19:05, Yuepeng Pan panyuep...@apache.org wrote: > > > Hi, Yubin > > > > Thanks for driving it ! > > > > +1 non-binding. > > > > Best, > > Yuepeng Pan. > > > > At 2024-03-19 17:56:42, "Yubin Li" lyb5...@gmail.com wrote: > > > > > Hi everyone, > > > > > > Thanks for all the feedback, I'd like to start a vote on the FLIP-436: > > > Introduce Catalog-related Syntax [1]. The discussion thread is here > > > [2]. > > > > > > The vote will be open for at least 72 hours unless there is an > > > objection or insufficient votes. > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-436%3A+Introduce+Catalog-related+Syntax > > > [2] https://lists.apache.org/thread/10k1bjb4sngyjwhmfqfky28lyoo7sv0z > > > > > > Best regards, > > > Yubin
Re: [DISCUSS] FLIP Suggestion: Externalize Kudu Connector from Bahir
Hi, since there are no more comments for a while, if there are no more comments for another day, I will start a vote thread. Thanks, Ferenc On Thursday, March 14th, 2024 at 11:20, Ferenc Csaky wrote: > > > Hi, > > Gentle ping to see if there are any other concerns or things that seems > missing from the FLIP. > > Best, > Ferenc > > > > > On Monday, March 11th, 2024 at 11:11, Ferenc Csaky ferenc.cs...@pm.me.INVALID > wrote: > > > Hi Jing, > > > > Thank you for your comments! Updated the FLIP with reasoning on the > > proposed release versions and included them in the headline "Release" field. > > > > Best, > > Ferenc > > > > On Sunday, March 10th, 2024 at 16:59, Jing Ge j...@ververica.com.INVALID > > wrote: > > > > > Hi Ferenc, > > > > > > Thanks for the proposal! +1 for it! > > > > > > Similar to what Leonard mentioned. I would suggest: > > > 1. Use the "release" to define the release version of the Kudu connector > > > itself. > > > 2. Optionally, add one more row underneath to describe which Flink > > > versions > > > this release will be compatible with, e.g. 1.17, 1.18. I think it makes > > > sense to support at least two last Flink releases. An example could be > > > found at [1] > > > > > > Best regards, > > > Jing > > > > > > [1] https://lists.apache.org/thread/jcjfy3fgpg5cdnb9noslq2c77h0gtcwp > > > > > > On Sun, Mar 10, 2024 at 3:46 PM Yanquan Lv decq12y...@gmail.com wrote: > > > > > > > Hi Ferenc, +1 for this FLIP. > > > > > > > > Ferenc Csaky ferenc.cs...@pm.me.invalid 于2024年3月9日周六 01:49写道: > > > > > > > > > Thank you Jeyhun, Leonard, and Hang for your comments! Let me > > > > > address them from earliest to latest. > > > > > > > > > > > How do you plan the review process in this case (e.g. incremental > > > > > > over existing codebase or cumulative all at once) ? > > > > > > > > > > I think incremental would be less time consuming and complex for > > > > > reviewers so I would leaning towards that direction. I would > > > > > imagine multiple subtasks for migrating the existing code, and > > > > > updating the deprecated interfaces, so those should be separate PRs > > > > > and > > > > > the release can be initiated when everything is merged. > > > > > > > > > > > (1) About the release version, could you specify kudu connector > > > > > > version > > > > > > instead of flink version 1.18 as external connector version is > > > > > > different > > > > > > with flink? > > > > > > (2) About the connector config options, could you enumerate these > > > > > > options so that we can review they’re reasonable or not? > > > > > > > > > > I added these to the FLIP, copied the current configs options as is, > > > > > PTAL. > > > > > > > > > > > (3) Metrics is also key part of connector, could you add the > > > > > > supported > > > > > > connector metrics to public interface as well? > > > > > > > > > > The current Bahir conenctor code does not include any metrics and I > > > > > did > > > > > not plan to include them into the scope of this FLIP. > > > > > > > > > > > I think that how to state this code originally lived in Bahir may > > > > > > be in > > > > > > the > > > > > > FLIP. > > > > > > > > > > I might miss your point, but the FLIP contains this: "Migrating the > > > > > current code keeping the history and noting it explicitly it was > > > > > forked > > > > > from the Bahir repository [2]." Pls. share if you meant something > > > > > else. > > > > > > > > > > Best, > > > > > Ferenc > > > > > > > > > > On Friday, March 8th, 2024 at 10:42, Hang Ruan ruanhang1...@gmail.com > > > > > wrote: > > > > > > > > > > > Hi, Ferenc. > > > > > > > > > > > > Thanks for the FLIP discussion. +1 for the proposal. > > > > > > I think that how to
Re: [VOTE] Release 1.19.0, release candidate #2
+1 (non-binding) - Verified checksum and signature - Verified no binary in src - Built from src - Reviewed release note PR - Reviewed web PR - Tested a simple datagen query and insert to blackhole sink via SQL Gateway Best, Ferenc On Thursday, March 14th, 2024 at 12:14, Jane Chan wrote: > > > Hi Lincoln, > > Thank you for the prompt response and the effort to provide clarity on this > matter. > > Best, > Jane > > On Thu, Mar 14, 2024 at 6:02 PM Lincoln Lee lincoln.8...@gmail.com wrote: > > > Hi Jane, > > > > Thank you for raising this question. I saw the discussion in the Jira > > (include Matthias' point) > > and sought advice from several PMCs (including the previous RMs), the > > majority of people > > are in favor of merging the bugfix into the release branch even during the > > release candidate > > (RC) voting period, so we should accept all bugfixes (unless there is a > > specific community > > rule preventing it). > > > > Thanks again for contributing to the community! > > > > Best, > > Lincoln Lee > > > > Matthias Pohl matthias.p...@aiven.io.invalid 于2024年3月14日周四 17:50写道: > > > > > Update on FLINK-34227 [1] which I mentioned above: Chesnay helped > > > identify > > > a concurrency issue in the JobMaster shutdown logic which seems to be in > > > the code for quite some time. I created a PR fixing the issue hoping that > > > the test instability is resolved with it. > > > > > > The concurrency issue doesn't really explain why it only started to > > > appear > > > recently in a specific CI setup (GHA with AdaptiveScheduler). There is no > > > hint in the git history indicating that it's caused by some newly > > > introduced change. That is why I wouldn't make FLINK-34227 a reason to > > > cancel rc2. Instead, the fix can be provided in subsequent patch > > > releases. > > > > > > Matthias > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-34227 > > > > > > On Thu, Mar 14, 2024 at 8:49 AM Jane Chan qingyue@gmail.com wrote: > > > > > > > Hi Yun, Jing, Martijn and Lincoln, > > > > > > > > I'm seeking guidance on whether merging the bugfix[1][2] at this stage > > > > is > > > > appropriate. I want to ensure that the actions align with the current > > > > release process and do not disrupt the ongoing preparations. > > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-29114 > > > > [2] https://github.com/apache/flink/pull/24492 > > > > > > > > Best, > > > > Jane > > > > > > > > On Thu, Mar 14, 2024 at 1:33 PM Yun Tang myas...@live.com wrote: > > > > > > > > > +1 (non-binding) > > > > > > > > > > * > > > > > Verified the signature and checksum. > > > > > * > > > > > Reviewed the release note PR > > > > > * > > > > > Reviewed the web announcement PR > > > > > * > > > > > Start a standalone cluster to submit the state machine example, which > > > > > works well. > > > > > * > > > > > Checked the pre-built jars are generated via JDK8 > > > > > * > > > > > Verified the process profiler works well after setting > > > > > rest.profiling.enabled: true > > > > > > > > > > Best > > > > > Yun Tang > > > > > > > > > > > > > > > From: Qingsheng Ren re...@apache.org > > > > > Sent: Wednesday, March 13, 2024 12:45 > > > > > To: dev@flink.apache.org dev@flink.apache.org > > > > > Subject: Re: [VOTE] Release 1.19.0, release candidate #2 > > > > > > > > > > +1 (binding) > > > > > > > > > > - Verified signature and checksum > > > > > - Verified no binary in source > > > > > - Built from source > > > > > - Tested reading and writing Kafka with SQL client and Kafka > > > > > connector > > > > > 3.1.0 > > > > > - Verified source code tag > > > > > - Reviewed release note > > > > > - Reviewed web PR > > > > > > > > > > Thanks to all release managers and contributors for the awesome work! > > > > > > > > > > Best, > > > > > Qingsheng > > > > > > > > > > On Wed, Mar 13, 2024 at 1:23 AM Matthias Pohl > > > > > matthias.p...@aiven.io.invalid wrote: > > > > > > > > > > > I want to share an update on FLINK-34227 [1]: It's still not clear > > > > > > what's > > > > > > causing the test instability. So far, we agreed in today's release > > > > > > sync > > > > > > [2] > > > > > > that it's not considered a blocker because it is observed in 1.18 > > > > > > nightly > > > > > > builds and it only appears in the GitHub Actions workflow. But I > > > > > > still > > > > > > have > > > > > > a bit of a concern that this is something that was introduced in > > > > > > 1.19 > > > > > > and > > > > > > backported to 1.18 after the 1.18.1 release (because the test > > > > > > instability > > > > > > started to appear more regularly in March; with one occurrence in > > > > > > January). > > > > > > Additionally, I have no reason to believe, yet, that the > > > > > > instability > > > > > > is > > > > > > caused by some GHA-related infrastructure issue. > > > > > > > > > > > > So, if someone else has some capacity to help looking into it; that > > > >
Re: [DISCUSS] FLIP-436: Introduce "SHOW CREATE CATALOG" Syntax
Hi Yubin, Thank you for initiating this discussion! +1 for the proposal. I also think it makes sense to group the missing catalog related SQL syntaxes under this FLIP. Looking forward to these features! Best, Ferenc On Thursday, March 14th, 2024 at 08:31, Jane Chan wrote: > > > Hi Yubin, > > Thanks for leading the discussion. I'm +1 for the FLIP. > > As Jark said, it's a good opportunity to enhance the syntax for Catalog > from a more comprehensive perspective. So, I suggest expanding the scope of > this FLIP by focusing on the mechanism instead of one use case to enhance > the overall functionality. WDYT? > > Best, > Jane > > On Thu, Mar 14, 2024 at 11:38 AM Hang Ruan ruanhang1...@gmail.com wrote: > > > Hi, Yubin. > > > > Thanks for the FLIP. +1 for it. > > > > Best, > > Hang > > > > Yubin Li lyb5...@gmail.com 于2024年3月14日周四 10:15写道: > > > > > Hi Jingsong, Feng, and Jeyhun > > > > > > Thanks for your support and feedback! > > > > > > > However, could we add a new method `getCatalogDescriptor()` to > > > > CatalogManager instead of directly exposing CatalogStore? > > > > > > Good point, Besides the audit tracking issue, The proposed feature > > > only requires `getCatalogDescriptor()` function. Exposing components > > > with excessive functionality will bring unnecessary risks, I have made > > > modifications in the FLIP doc [1]. Thank Feng :) > > > > > > > Showing the SQL parser implementation in the FLIP for the SQL syntax > > > > might be a bit confusing. Also, the formal definition is missing for > > > > this SQL clause. > > > > > > Thank Jeyhun for pointing it out :) I have updated the doc [1] . > > > > > > [1] > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=296290756 > > > > > Best, > > > Yubin > > > > > > On Thu, Mar 14, 2024 at 2:18 AM Jeyhun Karimov je.kari...@gmail.com > > > wrote: > > > > > > > Hi Yubin, > > > > > > > > Thanks for the proposal. +1 for it. > > > > I have one comment: > > > > > > > > I would like to see the SQL syntax for the proposed statement. Showing > > > > the > > > > SQL parser implementation in the FLIP > > > > for the SQL syntax might be a bit confusing. Also, the formal > > > > definition > > > > is > > > > missing for this SQL clause. > > > > Maybe something like [1] might be useful. WDYT? > > > > > > > > Regards, > > > > Jeyhun > > > > > > > > [1] > > > > https://github.com/apache/flink/blob/0da60ca1a4754f858cf7c52dd4f0c97ae0e1b0cb/docs/content/docs/dev/table/sql/show.md?plain=1#L620-L632 > > > > > > On Wed, Mar 13, 2024 at 3:28 PM Feng Jin jinfeng1...@gmail.com > > > > wrote: > > > > > > > > > Hi Yubin > > > > > > > > > > Thank you for initiating this FLIP. > > > > > > > > > > I have just one minor question: > > > > > > > > > > I noticed that we added a new function `getCatalogStore` to expose > > > > > CatalogStore, and it seems fine. > > > > > However, could we add a new method `getCatalogDescriptor()` to > > > > > CatalogManager instead of directly exposing CatalogStore? > > > > > By only providing the `getCatalogDescriptor()` interface, it may be > > > > > easier > > > > > for us to implement audit tracking in CatalogManager in the future. > > > > > WDYT ? > > > > > Although we have only collected some modified events at the > > > > > moment.[1] > > > > > > > > > > [1]. > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-294%3A+Support+Customized+Catalog+Modification+Listener > > > > > > > Best, > > > > > Feng > > > > > > > > > > On Wed, Mar 13, 2024 at 5:31 PM Jingsong Li jingsongl...@gmail.com > > > > > wrote: > > > > > > > > > > > +1 for this. > > > > > > > > > > > > We are missing a series of catalog related syntaxes. > > > > > > Especially after the introduction of catalog store. [1] > > > > > > > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations > > > > > > > > Best, > > > > > > Jingsong > > > > > > > > > > > > On Wed, Mar 13, 2024 at 5:09 PM Yubin Li lyb5...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > I'd like to start a discussion about FLIP-436: Introduce "SHOW > > > > > > > CREATE > > > > > > > CATALOG" Syntax [1]. > > > > > > > > > > > > > > At present, the `SHOW CREATE TABLE` statement provides strong > > > > > > > support > > > > > > > for > > > > > > > users to easily > > > > > > > reuse created tables. However, despite the increasing importance > > > > > > > of the > > > > > > > `Catalog` in user's > > > > > > > business, there is no similar statement for users to use. > > > > > > > > > > > > > > According to the online discussion in FLINK-24939 [2] with Jark > > > > > > > Wu > > > > > > > and > > > > > > > Feng > > > > > > > Jin, since `CatalogStore` > > > > > > > has been introduced in FLIP-295 [3], we could use this component > > > > > > > to > > > > > > > implement such a long-awaited > > > > > >
Re: [DISCUSS] FLIP Suggestion: Externalize Kudu Connector from Bahir
Hi, Gentle ping to see if there are any other concerns or things that seems missing from the FLIP. Best, Ferenc On Monday, March 11th, 2024 at 11:11, Ferenc Csaky wrote: > > > Hi Jing, > > Thank you for your comments! Updated the FLIP with reasoning on the proposed > release versions and included them in the headline "Release" field. > > Best, > Ferenc > > > > > On Sunday, March 10th, 2024 at 16:59, Jing Ge j...@ververica.com.INVALID > wrote: > > > Hi Ferenc, > > > > Thanks for the proposal! +1 for it! > > > > Similar to what Leonard mentioned. I would suggest: > > 1. Use the "release" to define the release version of the Kudu connector > > itself. > > 2. Optionally, add one more row underneath to describe which Flink versions > > this release will be compatible with, e.g. 1.17, 1.18. I think it makes > > sense to support at least two last Flink releases. An example could be > > found at [1] > > > > Best regards, > > Jing > > > > [1] https://lists.apache.org/thread/jcjfy3fgpg5cdnb9noslq2c77h0gtcwp > > > > On Sun, Mar 10, 2024 at 3:46 PM Yanquan Lv decq12y...@gmail.com wrote: > > > > > Hi Ferenc, +1 for this FLIP. > > > > > > Ferenc Csaky ferenc.cs...@pm.me.invalid 于2024年3月9日周六 01:49写道: > > > > > > > Thank you Jeyhun, Leonard, and Hang for your comments! Let me > > > > address them from earliest to latest. > > > > > > > > > How do you plan the review process in this case (e.g. incremental > > > > > over existing codebase or cumulative all at once) ? > > > > > > > > I think incremental would be less time consuming and complex for > > > > reviewers so I would leaning towards that direction. I would > > > > imagine multiple subtasks for migrating the existing code, and > > > > updating the deprecated interfaces, so those should be separate PRs and > > > > the release can be initiated when everything is merged. > > > > > > > > > (1) About the release version, could you specify kudu connector > > > > > version > > > > > instead of flink version 1.18 as external connector version is > > > > > different > > > > > with flink? > > > > > (2) About the connector config options, could you enumerate these > > > > > options so that we can review they’re reasonable or not? > > > > > > > > I added these to the FLIP, copied the current configs options as is, > > > > PTAL. > > > > > > > > > (3) Metrics is also key part of connector, could you add the supported > > > > > connector metrics to public interface as well? > > > > > > > > The current Bahir conenctor code does not include any metrics and I did > > > > not plan to include them into the scope of this FLIP. > > > > > > > > > I think that how to state this code originally lived in Bahir may be > > > > > in > > > > > the > > > > > FLIP. > > > > > > > > I might miss your point, but the FLIP contains this: "Migrating the > > > > current code keeping the history and noting it explicitly it was forked > > > > from the Bahir repository [2]." Pls. share if you meant something else. > > > > > > > > Best, > > > > Ferenc > > > > > > > > On Friday, March 8th, 2024 at 10:42, Hang Ruan ruanhang1...@gmail.com > > > > wrote: > > > > > > > > > Hi, Ferenc. > > > > > > > > > > Thanks for the FLIP discussion. +1 for the proposal. > > > > > I think that how to state this code originally lived in Bahir may be > > > > > in > > > > > the > > > > > FLIP. > > > > > > > > > > Best, > > > > > Hang > > > > > > > > > > Leonard Xu xbjt...@gmail.com 于2024年3月7日周四 14:14写道: > > > > > > > > > > > Thanks Ferenc for kicking off this discussion, I left some comments > > > > > > here: > > > > > > > > > > > > (1) About the release version, could you specify kudu connector > > > > > > version > > > > > > instead of flink version 1.18 as external connector version is > > > > > > different > > > > > > w
Re: [DISCUSS] FLIP-399: Flink Connector Doris
Hi, Thanks for driving this, +1 for the FLIP. Best, Ferenc On Monday, March 11th, 2024 at 15:17, Ahmed Hamdy wrote: > > > Hello, > Thanks for the proposal, +1 for the FLIP. > > Best Regards > Ahmed Hamdy > > > On Mon, 11 Mar 2024 at 15:12, wudi 676366...@qq.com.invalid wrote: > > > Hi, Leonard > > Thank you for your suggestion. > > I referred to other Connectors[1], modified the naming and types of > > relevant parameters[2], and also updated FLIP. > > > > [1] > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/overview/ > > [1] > > https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/main/java/org/apache/doris/flink/table/DorisConfigOptions.java > > > > Brs, > > di.wu > > > > > 2024年3月7日 14:33,Leonard Xu xbjt...@gmail.com 写道: > > > > > > Thanks wudi for the updating, the FLIP generally looks good to me, I > > > only left two minor suggestions: > > > > > > (1) The suffix `.s` in configoption doris.request.query.timeout.s looks > > > strange to me, could we change all time interval related option value type > > > to Duration ? > > > > > > (2) Could you check and improve all config options like > > > `doris.exec.mem.limit` to make them to follow flink config option naming > > > and value type? > > > > > > Best, > > > Leonard > > > > > > > > 2024年3月6日 06:12,Jing Ge j...@ververica.com.INVALID 写道: > > > > > > > > > > Hi Di, > > > > > > > > > > Thanks for your proposal. +1 for the contribution. I'd like to know > > > > > your > > > > > thoughts about the following questions: > > > > > > > > > > 1. According to your clarification of the exactly-once, thanks for it > > > > > BTW, > > > > > no PreCommitTopology is required. Does it make sense to let > > > > > DorisSink[1] > > > > > implement SupportsCommitter, since the TwoPhaseCommittingSink is > > > > > deprecated[2] before turning the Doris connector into a Flink > > > > > connector? > > > > > 2. OLAP engines are commonly used as the tail/downstream of a data > > > > > pipeline > > > > > to support further e.g. ad-hoc query or cube with feasible > > > > > pre-aggregation. > > > > > Just out of curiosity, would you like to share some real use cases > > > > > that > > > > > will use OLAP engines as the source of a streaming data pipeline? Or > > > > > it > > > > > will only be used as the source for the batch? > > > > > 3. The E2E test only covered sink[3], if I am not mistaken. Would you > > > > > like > > > > > to test the source in E2E too? > > > > > > > > > > [1] > > > > https://github.com/apache/doris-flink-connector/blob/43e0e5cf9b832854ea228fb093077872e3a311b6/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/DorisSink.java#L55 > > > > > > > [2] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Enhance+and+synchronize+Sink+API+to+match+the+Source+API > > > > > > > [3] > > > > https://github.com/apache/doris-flink-connector/blob/43e0e5cf9b832854ea228fb093077872e3a311b6/flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/MySQLDorisE2ECase.java#L96 > > > > > > > Best regards, > > > > > Jing > > > > > > > > > > On Tue, Mar 5, 2024 at 11:18 AM wudi 676366...@qq.com.invalid wrote: > > > > > > > > > > > Hi, Jeyhun Karimov. > > > > > > Thanks for your question. > > > > > > > > > > > > - How to ensure Exactly-Once? > > > > > > 1. When the Checkpoint Barrier arrives, DorisSink will trigger the > > > > > > precommit api of StreamLoad to complete the persistence of data in > > > > > > Doris > > > > > > (the data will not be visible at this time), and will also pass this > > > > > > TxnID > > > > > > to the Committer. > > > > > > 2. When this Checkpoint of the entire Job is completed, the > > > > > > Committer > > > > > > will > > > > > > call the commit api of StreamLoad and commit TxnID to complete the > > > > > > visibility of the transaction. > > > > > > 3. When the task is restarted, the Txn with successful precommit and > > > > > > failed commit will be aborted based on the label-prefix, and Doris' > > > > > > abort > > > > > > API will be called. (At the same time, Doris will also abort > > > > > > transactions > > > > > > that have not been committed for a long time) > > > > > > > > > > > > ps: At the same time, this part of the content has been updated in > > > > > > FLIP > > > > > > > > > > > > - Because the default table model in Doris is Duplicate ( > > > > > > https://doris.apache.org/docs/data-table/data-model/), which does > > > > > > not > > > > > > have a primary key, batch writing may cause data duplication, but > > > > > > UNIQ The > > > > > > model has a primary key, which ensures the idempotence of writing, > > > > > > thus > > > > > > achieving Exactly-Once > > > > > > > > > > > > Brs, > > > > > > di.wu > > > > > > > > > > > > > 2024年3月2日 17:50,Jeyhun Karimov je.kari...@gmail.com 写道: > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Thanks for the proposal. +1 for the FLIP. > >
Re: [DISCUSS] FLIP Suggestion: Externalize Kudu Connector from Bahir
Hi Jing, Thank you for your comments! Updated the FLIP with reasoning on the proposed release versions and included them in the headline "Release" field. Best, Ferenc On Sunday, March 10th, 2024 at 16:59, Jing Ge wrote: > > > Hi Ferenc, > > Thanks for the proposal! +1 for it! > > Similar to what Leonard mentioned. I would suggest: > 1. Use the "release" to define the release version of the Kudu connector > itself. > 2. Optionally, add one more row underneath to describe which Flink versions > this release will be compatible with, e.g. 1.17, 1.18. I think it makes > sense to support at least two last Flink releases. An example could be > found at [1] > > Best regards, > Jing > > [1] https://lists.apache.org/thread/jcjfy3fgpg5cdnb9noslq2c77h0gtcwp > > On Sun, Mar 10, 2024 at 3:46 PM Yanquan Lv decq12y...@gmail.com wrote: > > > Hi Ferenc, +1 for this FLIP. > > > > Ferenc Csaky ferenc.cs...@pm.me.invalid 于2024年3月9日周六 01:49写道: > > > > > Thank you Jeyhun, Leonard, and Hang for your comments! Let me > > > address them from earliest to latest. > > > > > > > How do you plan the review process in this case (e.g. incremental > > > > over existing codebase or cumulative all at once) ? > > > > > > I think incremental would be less time consuming and complex for > > > reviewers so I would leaning towards that direction. I would > > > imagine multiple subtasks for migrating the existing code, and > > > updating the deprecated interfaces, so those should be separate PRs and > > > the release can be initiated when everything is merged. > > > > > > > (1) About the release version, could you specify kudu connector version > > > > instead of flink version 1.18 as external connector version is different > > > > with flink? > > > > (2) About the connector config options, could you enumerate these > > > > options so that we can review they’re reasonable or not? > > > > > > I added these to the FLIP, copied the current configs options as is, > > > PTAL. > > > > > > > (3) Metrics is also key part of connector, could you add the supported > > > > connector metrics to public interface as well? > > > > > > The current Bahir conenctor code does not include any metrics and I did > > > not plan to include them into the scope of this FLIP. > > > > > > > I think that how to state this code originally lived in Bahir may be in > > > > the > > > > FLIP. > > > > > > I might miss your point, but the FLIP contains this: "Migrating the > > > current code keeping the history and noting it explicitly it was forked > > > from the Bahir repository [2]." Pls. share if you meant something else. > > > > > > Best, > > > Ferenc > > > > > > On Friday, March 8th, 2024 at 10:42, Hang Ruan ruanhang1...@gmail.com > > > wrote: > > > > > > > Hi, Ferenc. > > > > > > > > Thanks for the FLIP discussion. +1 for the proposal. > > > > I think that how to state this code originally lived in Bahir may be in > > > > the > > > > FLIP. > > > > > > > > Best, > > > > Hang > > > > > > > > Leonard Xu xbjt...@gmail.com 于2024年3月7日周四 14:14写道: > > > > > > > > > Thanks Ferenc for kicking off this discussion, I left some comments > > > > > here: > > > > > > > > > > (1) About the release version, could you specify kudu connector > > > > > version > > > > > instead of flink version 1.18 as external connector version is > > > > > different > > > > > with flink ? > > > > > > > > > > (2) About the connector config options, could you enumerate these > > > > > options > > > > > so that we can review they’re reasonable or not? > > > > > > > > > > (3) Metrics is also key part of connector, could you add the > > > > > supported > > > > > connector metrics to public interface as well? > > > > > > > > > > Best, > > > > > Leonard > > > > > > > > > > > 2024年3月6日 下午11:23,Ferenc Csaky ferenc.cs...@pm.me.INVALID 写道: > > > > > > > > > > > > Hello devs, > > > > > > > > > > > > Opening this thread to discuss a FLIP [1] about externalizing the > > > > > > Kudu > > > > > > connector, as recently > > > > > > the Apache Bahir project were moved to the attic [2]. Some details > > > > > > were > > > > > > discussed already > > > > > > in another thread [3]. I am proposing to externalize this connector > > > > > > and > > > > > > keep it maintainable, > > > > > > and up to date. > > > > > > > > > > > > Best regards, > > > > > > Ferenc > > > > > > > > > > > > [1] > > > > https://docs.google.com/document/d/1vHF_uVe0FTYCb6PRVStovqDeqb_C_FKjt2P5xXa7uhE > > > > > > > > [2] https://bahir.apache.org/ > > > > > > [3] > > > > > > https://lists.apache.org/thread/2nb8dxxfznkyl4hlhdm3vkomm8rk4oyq
Re: [DISCUSS] FLIP Suggestion: Externalize Kudu Connector from Bahir
Thank you Jeyhun, Leonard, and Hang for your comments! Let me address them from earliest to latest. > How do you plan the review process in this case (e.g. incremental over existing codebase or cumulative all at once) ? I think incremental would be less time consuming and complex for reviewers so I would leaning towards that direction. I would imagine multiple subtasks for migrating the existing code, and updating the deprecated interfaces, so those should be separate PRs and the release can be initiated when everything is merged. > (1) About the release version, could you specify kudu connector version > instead of flink version 1.18 as external connector version is different with > flink? > (2) About the connector config options, could you enumerate these options so > that we can review they’re reasonable or not? I added these to the FLIP, copied the current configs options as is, PTAL. > (3) Metrics is also key part of connector, could you add the supported > connector metrics to public interface as well? The current Bahir conenctor code does not include any metrics and I did not plan to include them into the scope of this FLIP. > I think that how to state this code originally lived in Bahir may be in the FLIP. I might miss your point, but the FLIP contains this: "Migrating the current code keeping the history and noting it explicitly it was forked from the Bahir repository [2]." Pls. share if you meant something else. Best, Ferenc On Friday, March 8th, 2024 at 10:42, Hang Ruan wrote: > > > Hi, Ferenc. > > Thanks for the FLIP discussion. +1 for the proposal. > I think that how to state this code originally lived in Bahir may be in the > FLIP. > > Best, > Hang > > Leonard Xu xbjt...@gmail.com 于2024年3月7日周四 14:14写道: > > > Thanks Ferenc for kicking off this discussion, I left some comments here: > > > > (1) About the release version, could you specify kudu connector version > > instead of flink version 1.18 as external connector version is different > > with flink ? > > > > (2) About the connector config options, could you enumerate these options > > so that we can review they’re reasonable or not? > > > > (3) Metrics is also key part of connector, could you add the supported > > connector metrics to public interface as well? > > > > Best, > > Leonard > > > > > 2024年3月6日 下午11:23,Ferenc Csaky ferenc.cs...@pm.me.INVALID 写道: > > > > > > Hello devs, > > > > > > Opening this thread to discuss a FLIP [1] about externalizing the Kudu > > > connector, as recently > > > the Apache Bahir project were moved to the attic [2]. Some details were > > > discussed already > > > in another thread [3]. I am proposing to externalize this connector and > > > keep it maintainable, > > > and up to date. > > > > > > Best regards, > > > Ferenc > > > > > > [1] > > > https://docs.google.com/document/d/1vHF_uVe0FTYCb6PRVStovqDeqb_C_FKjt2P5xXa7uhE > > > [2] https://bahir.apache.org/ > > > [3] https://lists.apache.org/thread/2nb8dxxfznkyl4hlhdm3vkomm8rk4oyq
[DISCUSS] FLIP Suggestion: Externalize Kudu Connector from Bahir
Hello devs, Opening this thread to discuss a FLIP [1] about externalizing the Kudu connector, as recently the Apache Bahir project were moved to the attic [2]. Some details were discussed already in another thread [3]. I am proposing to externalize this connector and keep it maintainable, and up to date. Best regards, Ferenc [1] https://docs.google.com/document/d/1vHF_uVe0FTYCb6PRVStovqDeqb_C_FKjt2P5xXa7uhE [2] https://bahir.apache.org/ [3] https://lists.apache.org/thread/2nb8dxxfznkyl4hlhdm3vkomm8rk4oyq
Re: [DISCUSS] Apache Bahir retired
Hi Jing, Thank you, I will create a new discussion thread. Best, Ferenc On Wednesday, March 6th, 2024 at 10:43, Jing Ge wrote: > > > Hi Ferenc, > > +1 for it! The proposal looks good, thanks! I would suggest starting a new > discuss thread following the new FLIP process created by Martijn[1]. > > [1] https://issues.apache.org/jira/browse/FLINK-34515 > > Best regards, > Jing > > On Fri, Mar 1, 2024 at 5:25 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi, > > > > According to the current standards I created a Google doc for the FLIP > > [1]. I pointed it to the this discussion but if a dedicated discussion > > should be started for it, pls. let me know. PTAL. > > > > Regards, > > Ferenc > > > > [1] > > https://docs.google.com/document/d/1vHF_uVe0FTYCb6PRVStovqDeqb_C_FKjt2P5xXa7uhE/edit > > > > On Thursday, February 29th, 2024 at 09:48, Ferenc Csaky > > ferenc.cs...@pm.me.INVALID wrote: > > > > > Thank you Marton and Martijn, I will proceed with the FLIP then. > > > > > > Best, > > > Ferenc > > > > > > On Wednesday, February 28th, 2024 at 21:15, Martijn Visser > > > martijnvis...@apache.org wrote: > > > > > > > Hi all, > > > > > > > > +1 to have a connector FLIP to propose a Kudu connector. I'm +0 overall > > > > because I don't see a lot of activity happening in newly proposed > > > > connectors, but if there's demand for it and people want to volunteer > > > > with > > > > contributions, there's no reason to block it. > > > > > > > > Best regards, > > > > > > > > Martijn > > > > > > > > On Wed, Feb 28, 2024 at 4:31 PM Márton Balassi > > > > balassi.mar...@gmail.com > > > > > > > > wrote: > > > > > > > > > Hi team, > > > > > > > > > > Thanks for bringing this up, Feri. I am +1 for maintaining the Kudu > > > > > connector as an external Flink connector. > > > > > > > > > > As per the legal/trademark questions this is actually fair game > > > > > because one > > > > > does not donate code to a specific Apache project, technically it is > > > > > donated to the Apache Software foundation. Consequently moving > > > > > between ASF > > > > > projects is fine, I would add a line to the NOTICE file stating that > > > > > this > > > > > code originally lived in Bahir once we forked it. > > > > > > > > > > Although I did not find an easy to link precedent this is also > > > > > implied in > > > > > the Attic Bahir site [1] ("notify us if you fork outside Apache") > > > > > and in > > > > > this [2] Apache community dev list chat. We should notify the Attic > > > > > team in > > > > > any case. :-) > > > > > > > > > > [1] https://attic.apache.org/projects/bahir.html > > > > > [2] https://lists.apache.org/thread/p31mz4x4dcvd43f026d5p05rpglzfyrt > > > > > > > > > > On Tue, Feb 27, 2024 at 10:09 AM Ferenc Csaky > > > > > ferenc.cs...@pm.me.invalid > > > > > wrote: > > > > > > > > > > > Thank you Leonard for sharing your thoughts on this topic. > > > > > > > > > > > > I agree that complying with the Flink community connector > > > > > > development process would be a must, if there are no legal or > > > > > > copyright issues, I would be happy to take that task for this > > > > > > particular case. > > > > > > > > > > > > I am no legal/copyright expert myslef, but Bahir uses the Apache > > > > > > 2.0 license as well, so I believe it should be possible without > > > > > > too many > > > > > > complications, but I try to look for help on that front. > > > > > > > > > > > > FYI we are using and supporting a downstream fork of the Kudu > > > > > > connector > > > > > > on > > > > > > top of Flink 1.18 without any major modifications, so it is pretty > > > > > > up to > > > > > > date upstream as well. > > > > > > > > > > > > Regards, > > > > > > Ferenc
[jira] [Created] (FLINK-34580) Job run via REST erases "pipeline.classpaths" config
Ferenc Csaky created FLINK-34580: Summary: Job run via REST erases "pipeline.classpaths" config Key: FLINK-34580 URL: https://issues.apache.org/jira/browse/FLINK-34580 Project: Flink Issue Type: Bug Components: Runtime / REST Affects Versions: 1.18.1, 1.17.2, 1.19.0 Reporter: Ferenc Csaky Fix For: 1.20.0 The [{{JarHandlerContext#applyToConfiguration}}|https://github.com/apache/flink/blob/e0b6c121eaf7aeb2974a45d199e452b022f07d29/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/utils/JarHandlerUtils.java#L134] creates a {{PackagedProgram}} and then overwrites the {{pipeline.jars}} and {{pipeline.classpaths}} values according to that newly created {{{}PackagedProgram{}}}. Although that [{{PackagedProgram}} init|https://github.com/apache/flink/blob/e0b6c121eaf7aeb2974a45d199e452b022f07d29/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/utils/JarHandlerUtils.java#L185] does not set {{classpaths}} at all, so it will always overwrites the effective configuration with an empty value, even if it had something previously. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Apache Bahir retired
Hi, According to the current standards I created a Google doc for the FLIP [1]. I pointed it to the this discussion but if a dedicated discussion should be started for it, pls. let me know. PTAL. Regards, Ferenc [1] https://docs.google.com/document/d/1vHF_uVe0FTYCb6PRVStovqDeqb_C_FKjt2P5xXa7uhE/edit On Thursday, February 29th, 2024 at 09:48, Ferenc Csaky wrote: > > > Thank you Marton and Martijn, I will proceed with the FLIP then. > > Best, > Ferenc > > > > > On Wednesday, February 28th, 2024 at 21:15, Martijn Visser > martijnvis...@apache.org wrote: > > > Hi all, > > > > +1 to have a connector FLIP to propose a Kudu connector. I'm +0 overall > > because I don't see a lot of activity happening in newly proposed > > connectors, but if there's demand for it and people want to volunteer with > > contributions, there's no reason to block it. > > > > Best regards, > > > > Martijn > > > > On Wed, Feb 28, 2024 at 4:31 PM Márton Balassi balassi.mar...@gmail.com > > > > wrote: > > > > > Hi team, > > > > > > Thanks for bringing this up, Feri. I am +1 for maintaining the Kudu > > > connector as an external Flink connector. > > > > > > As per the legal/trademark questions this is actually fair game because > > > one > > > does not donate code to a specific Apache project, technically it is > > > donated to the Apache Software foundation. Consequently moving between ASF > > > projects is fine, I would add a line to the NOTICE file stating that this > > > code originally lived in Bahir once we forked it. > > > > > > Although I did not find an easy to link precedent this is also implied in > > > the Attic Bahir site [1] ("notify us if you fork outside Apache") and in > > > this [2] Apache community dev list chat. We should notify the Attic team > > > in > > > any case. :-) > > > > > > [1] https://attic.apache.org/projects/bahir.html > > > [2] https://lists.apache.org/thread/p31mz4x4dcvd43f026d5p05rpglzfyrt > > > > > > On Tue, Feb 27, 2024 at 10:09 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > > wrote: > > > > > > > Thank you Leonard for sharing your thoughts on this topic. > > > > > > > > I agree that complying with the Flink community connector > > > > development process would be a must, if there are no legal or > > > > copyright issues, I would be happy to take that task for this > > > > particular case. > > > > > > > > I am no legal/copyright expert myslef, but Bahir uses the Apache > > > > 2.0 license as well, so I believe it should be possible without too many > > > > complications, but I try to look for help on that front. > > > > > > > > FYI we are using and supporting a downstream fork of the Kudu connector > > > > on > > > > top of Flink 1.18 without any major modifications, so it is pretty up to > > > > date upstream as well. > > > > > > > > Regards, > > > > Ferenc > > > > > > > > On Monday, February 26th, 2024 at 10:29, Leonard Xu xbjt...@gmail.com > > > > wrote: > > > > > > > > > Hey, Ferenc > > > > > > > > > > Thanks for initiating this discussion. Apache Bahir is a great project > > > > > that provided significant assistance to many Apache Flink/Spark users. > > > > > It's > > > > > pity news that it has been retired. > > > > > > > > > > I believe that connectivity is crucial for building the ecosystem of > > > > > the > > > > > Flink such a computing engine. The community, or at least I, would > > > > > actively > > > > > support the introduction and maintenance of new connectors. Therefore, > > > > > adding a Kudu connector or other connectors from Bahir makes sense to > > > > > me, > > > > > as long as we adhere to the development process for connectors in the > > > > > Flink > > > > > community[1]. > > > > > I recently visited the Bahir Flink repository. Although the last > > > > > release > > > > > of Bahir Flink was in August ’22[2] which is compatible with Flink > > > > > 1.14, > > > > > its latest code is compatible with Flink 1.17[3]. So, based on the > > > > > existing > > > > > codeba
Re: [DISCUSS] Apache Bahir retired
Thank you Marton and Martijn, I will proceed with the FLIP then. Best, Ferenc On Wednesday, February 28th, 2024 at 21:15, Martijn Visser wrote: > > > Hi all, > > +1 to have a connector FLIP to propose a Kudu connector. I'm +0 overall > because I don't see a lot of activity happening in newly proposed > connectors, but if there's demand for it and people want to volunteer with > contributions, there's no reason to block it. > > Best regards, > > Martijn > > On Wed, Feb 28, 2024 at 4:31 PM Márton Balassi balassi.mar...@gmail.com > > wrote: > > > Hi team, > > > > Thanks for bringing this up, Feri. I am +1 for maintaining the Kudu > > connector as an external Flink connector. > > > > As per the legal/trademark questions this is actually fair game because one > > does not donate code to a specific Apache project, technically it is > > donated to the Apache Software foundation. Consequently moving between ASF > > projects is fine, I would add a line to the NOTICE file stating that this > > code originally lived in Bahir once we forked it. > > > > Although I did not find an easy to link precedent this is also implied in > > the Attic Bahir site [1] ("notify us if you fork outside Apache") and in > > this [2] Apache community dev list chat. We should notify the Attic team in > > any case. :-) > > > > [1] https://attic.apache.org/projects/bahir.html > > [2] https://lists.apache.org/thread/p31mz4x4dcvd43f026d5p05rpglzfyrt > > > > On Tue, Feb 27, 2024 at 10:09 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > > > Thank you Leonard for sharing your thoughts on this topic. > > > > > > I agree that complying with the Flink community connector > > > development process would be a must, if there are no legal or > > > copyright issues, I would be happy to take that task for this > > > particular case. > > > > > > I am no legal/copyright expert myslef, but Bahir uses the Apache > > > 2.0 license as well, so I believe it should be possible without too many > > > complications, but I try to look for help on that front. > > > > > > FYI we are using and supporting a downstream fork of the Kudu connector > > > on > > > top of Flink 1.18 without any major modifications, so it is pretty up to > > > date upstream as well. > > > > > > Regards, > > > Ferenc > > > > > > On Monday, February 26th, 2024 at 10:29, Leonard Xu xbjt...@gmail.com > > > wrote: > > > > > > > Hey, Ferenc > > > > > > > > Thanks for initiating this discussion. Apache Bahir is a great project > > > > that provided significant assistance to many Apache Flink/Spark users. > > > > It's > > > > pity news that it has been retired. > > > > > > > > I believe that connectivity is crucial for building the ecosystem of > > > > the > > > > Flink such a computing engine. The community, or at least I, would > > > > actively > > > > support the introduction and maintenance of new connectors. Therefore, > > > > adding a Kudu connector or other connectors from Bahir makes sense to > > > > me, > > > > as long as we adhere to the development process for connectors in the > > > > Flink > > > > community[1]. > > > > I recently visited the Bahir Flink repository. Although the last > > > > release > > > > of Bahir Flink was in August ’22[2] which is compatible with Flink 1.14, > > > > its latest code is compatible with Flink 1.17[3]. So, based on the > > > > existing > > > > codebase, developing an official Apache Flink connector for Kudu or > > > > other > > > > connectors should be manageable. One point to consider is that if we're > > > > not > > > > developing a connector entirely from scratch but based on an existing > > > > repository, we must ensure that there are no copyright issues. Here, "no > > > > issues" means satisfying both Apache Bahir's and Apache Flink's > > > > copyright > > > > requirements. Honestly, I'm not an expert in copyright or legal matters. > > > > If > > > > you're interested in contributing to the Kudu connector, it might be > > > > necessary to attract other experienced community members to participate > > > > in > > > > this aspect. > > > > > > > > Best, &g
Re: [DISCUSS] Apache Bahir retired
Thank you Leonard for sharing your thoughts on this topic. I agree that complying with the Flink community connector development process would be a must, if there are no legal or copyright issues, I would be happy to take that task for this particular case. I am no legal/copyright expert myslef, but Bahir uses the Apache 2.0 license as well, so I believe it should be possible without too many complications, but I try to look for help on that front. FYI we are using and supporting a downstream fork of the Kudu connector on top of Flink 1.18 without any major modifications, so it is pretty up to date upstream as well. Regards, Ferenc On Monday, February 26th, 2024 at 10:29, Leonard Xu wrote: > > > Hey, Ferenc > > Thanks for initiating this discussion. Apache Bahir is a great project that > provided significant assistance to many Apache Flink/Spark users. It's pity > news that it has been retired. > > I believe that connectivity is crucial for building the ecosystem of the > Flink such a computing engine. The community, or at least I, would actively > support the introduction and maintenance of new connectors. Therefore, adding > a Kudu connector or other connectors from Bahir makes sense to me, as long as > we adhere to the development process for connectors in the Flink community[1]. > I recently visited the Bahir Flink repository. Although the last release of > Bahir Flink was in August ’22[2] which is compatible with Flink 1.14, its > latest code is compatible with Flink 1.17[3]. So, based on the existing > codebase, developing an official Apache Flink connector for Kudu or other > connectors should be manageable. One point to consider is that if we're not > developing a connector entirely from scratch but based on an existing > repository, we must ensure that there are no copyright issues. Here, "no > issues" means satisfying both Apache Bahir's and Apache Flink's copyright > requirements. Honestly, I'm not an expert in copyright or legal matters. If > you're interested in contributing to the Kudu connector, it might be > necessary to attract other experienced community members to participate in > this aspect. > > Best, > Leonard > > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template > [2] https://github.com/apache/bahir-flink/releases/tag/v1.1.0 > [3] https://github.com/apache/bahir-flink/blob/master/pom.xml#L116 > > > > > 2024年2月22日 下午6:37,Ferenc Csaky ferenc.cs...@pm.me.INVALID 写道: > > > > Hello devs, > > > > Just saw that the Bahir project is retired [1]. Any plans on what's > > happening with the Flink connectors that were part of this project? We > > specifically use the Kudu connector and integrate it to our platform at > > Cloudera, so we would be okay to maintain it. Would it be possible to carry > > it over as separate connector repu under the Apache umbrella similarly as > > it happened with the external connectors previously? > > > > Thanks, > > Ferenc
Re: Flink's treatment to "hadoop" and "yarn" configuration overrides seems unintuitive
Thanks for the more shared details Venkata, I did not used spark widely myself, but if there are more examples like follows that approach it makes sense to comply and be consistent. Regarding the planned release schedule on the 2.0 wiki page [1] the expected releases are 1.19 -> 1.20 -> 2.0. I am not sure about how realistic is that or there are any chance there will be a 1.21, but even if not, deprecating the current behavior for even 1.20 would not hurt IMO. WDYT? Looking for other opinios as well of course. Regards, Ferenc [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release On Tuesday, February 27th, 2024 at 07:08, Venkatakrishnan Sowrirajan wrote: > > > Thanks for sharing your thoughts, Ferenc. > > Just my 2 cents, coming from Spark background that uses "spark.hadoop." > prefix to handle all Hadoop configs, I prefer the "flink.hadoop." prefix > and "flink.yarn." prefix. Generally, users use both these systems and I > prefer to be consistent that way. Having said that, Flink doesn't need to > be consistent with Spark so I am fine with the other approach as well. > > I believe in order to make backwards incompatible changes, Flink needs the > change to be in deprecated status for at least 2 minor versions which means > we will already have 2.0, therefore this can probably go in 3.0 only. > > It is still good to deprecate the current behavior and fix with the right > behavior and get rid of this in 3.0 totally. > > Looking for more thoughts from others in the community to make sure that I > don't miss anything. Once the discussion settles, I can start a FLIP with > the new proposal. > > Thanks > Venkat > > > On Mon, Feb 26, 2024, 1:09 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi Venkata krishnan, > > > > Thanks for starting a discussion on this topic. I completely > > agree with you on that, this behavior can create confusion and > > cause debugging sessions that could be spared with aligning how Flink > > parses external properties. > > > > Personally, I find the Yarn props prefixing more intuitive, but > > I do not have strong opinions other than prefixing configs for > > external systems should follow the same semantics and behavior. > > > > It would make sense to align these in Flink 2.0 IMO, but I would > > be curious about other opinions. > > > > On Saturday, February 24th, 2024 at 07:36, Venkatakrishnan Sowrirajan < > > vsowr...@asu.edu> wrote: > > > > > Gentle ping on the ^^ question to surface this back up again. Any > > > thoughts? > > > > > > Regards > > > Venkata krishnan > > > > > > On Fri, Feb 16, 2024 at 7:32 PM Venkatakrishnan Sowrirajan > > > vsowr...@asu.edu > > > > > > wrote: > > > > > > > Hi Flink devs, > > > > > > > > Flink supports overriding "hadoop" and "yarn" configuration. As part of > > > > the override mechanism, users have to prefix `hadoop` configs with " > > > > flink.hadoop." and the prefix will be removed, while with `yarn` > > > > configs > > > > users have to prefix it with "flink.yarn." but "flink." only is > > > > removed, > > > > not "flink.yarn.". > > > > > > > > Following is an example: > > > > > > > > 1. "Hadoop" config > > > > > > > > Hadoop config key = hadoop.tmp.dir => Flink config = > > > > flink.hadoop.hadoop.tmp.dir => Hadoop's configuration object would have > > > > hadoop.tmp.dir*.* > > > > > > > > 2. "YARN" config > > > > > > > > YARN config key = yarn.application.classpath => Flink config = > > > > flink.yarn.yarn.application.classpath => YARN's configuration object > > > > would have yarn.yarn.application.classpath*.* > > > > > > > > Although this is documented > > > > https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/*flink-yarn-__;Iw!!IKRxdwAv5BmarQ!ewtlUgGysGDiWgKPr9D1bsGDp-jLagZqppUvXAtqvbO5lNMg7QTr4y5L4OL-hTFPO1qTR1nvh4ALBEQtm0RnE6X1WTEkp0Sb$ > > > > > > > > properly, it feels unintuitive and it tripped me, took quite a while to > > > > understand why the above YARN configuration override was not working as > > > > expected. Is this something that should be fixed? The problem with > > > > fixing > > > > it is, it will become backwards incompatible. Therefore, can this be > > > > addressed as part of Flink-2.0? > > > > > > > > Any thoughts? > > > > > > > > Regards > > > > Venkata krishnan
Re: [DISCUSS] Support the Ozone Filesystem
Hi, gentle reminder on this thread, any opinions or thoughts? Regards, Ferenc On Thursday, February 8th, 2024 at 18:02, Ferenc Csaky wrote: > > > Hello devs, > > I would like to start a discussion regarding Apache Ozone FS support. The > jira [1] is stale for quite a while, but supporting it with some limitations > could > be done with minimal effort. > > Ozone do not have truncate() impl, so it falls to the same category as > Hadoop < 2.7 [2], on Datastream API it requires the usage of > OnCheckpointRollingPolicy when checkpointing enabled to make sure > the FileSink will not use truncate(). > > Table API is a bit trickier, because checkpointing policy cannot be ocnfigured > explicitly (why?), it behaves differently regarding the write mode [3]. Bulk > mode > is covered, but for fow format, auto-compaction has to be set. > > Even with the mentioned limitations, I think it would worth to add support > for OFS, > it would require 1 small change to enable "ofs" [4] and documenting the > limitations. > > WDYT? > > Regards, > Ferenc > > [1] https://issues.apache.org/jira/browse/FLINK-28231 > [2] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/filesystem/#general > [3] > https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java#L226 > [4] > https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java#L62
Re: Flink's treatment to "hadoop" and "yarn" configuration overrides seems unintuitive
Hi Venkata krishnan, Thanks for starting a discussion on this topic. I completely agree with you on that, this behavior can create confusion and cause debugging sessions that could be spared with aligning how Flink parses external properties. Personally, I find the Yarn props prefixing more intuitive, but I do not have strong opinions other than prefixing configs for external systems should follow the same semantics and behavior. It would make sense to align these in Flink 2.0 IMO, but I would be curious about other opinions. On Saturday, February 24th, 2024 at 07:36, Venkatakrishnan Sowrirajan wrote: > > > Gentle ping on the ^^ question to surface this back up again. Any thoughts? > > Regards > Venkata krishnan > > > On Fri, Feb 16, 2024 at 7:32 PM Venkatakrishnan Sowrirajan vsowr...@asu.edu > > wrote: > > > Hi Flink devs, > > > > Flink supports overriding "hadoop" and "yarn" configuration. As part of > > the override mechanism, users have to prefix `hadoop` configs with " > > flink.hadoop." and the prefix will be removed, while with `yarn` configs > > users have to prefix it with "flink.yarn." but "flink." only is removed, > > not "flink.yarn.". > > > > Following is an example: > > > > 1. "Hadoop" config > > > > Hadoop config key = hadoop.tmp.dir => Flink config = > > flink.hadoop.hadoop.tmp.dir => Hadoop's configuration object would have > > hadoop.tmp.dir*.* > > > > 2. "YARN" config > > > > YARN config key = yarn.application.classpath => Flink config = > > flink.yarn.yarn.application.classpath => YARN's configuration object > > would have yarn.yarn.application.classpath*.* > > > > Although this is documented > > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/#flink-yarn- > > properly, it feels unintuitive and it tripped me, took quite a while to > > understand why the above YARN configuration override was not working as > > expected. Is this something that should be fixed? The problem with fixing > > it is, it will become backwards incompatible. Therefore, can this be > > addressed as part of Flink-2.0? > > > > Any thoughts? > > > > Regards > > Venkata krishnan
[jira] [Created] (FLINK-34506) Do not copy "file://" schemed artifact in standalone application modes
Ferenc Csaky created FLINK-34506: Summary: Do not copy "file://" schemed artifact in standalone application modes Key: FLINK-34506 URL: https://issues.apache.org/jira/browse/FLINK-34506 Project: Flink Issue Type: Bug Components: Client / Job Submission Affects Versions: 1.19.0 Reporter: Ferenc Csaky In standalone application mode, if an artifact is passed via a path witohut prefix, the file will be copied to `user.artifacts.base-dir`, although it should not be, as it can accessable locally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[DISCUSS] Apache Bahir retired
Hello devs, Just saw that the Bahir project is retired [1]. Any plans on what's happening with the Flink connectors that were part of this project? We specifically use the Kudu connector and integrate it to our platform at Cloudera, so we would be okay to maintain it. Would it be possible to carry it over as separate connector repu under the Apache umbrella similarly as it happened with the external connectors previously? Thanks, Ferenc
[DISCUSS] Support the Ozone Filesystem
Hello devs, I would like to start a discussion regarding Apache Ozone FS support. The jira [1] is stale for quite a while, but supporting it with some limitations could be done with minimal effort. Ozone do not have truncate() impl, so it falls to the same category as Hadoop < 2.7 [2], on Datastream API it requires the usage of OnCheckpointRollingPolicy when checkpointing enabled to make sure the FileSink will not use truncate(). Table API is a bit trickier, because checkpointing policy cannot be ocnfigured explicitly (why?), it behaves differently regarding the write mode [3]. Bulk mode is covered, but for fow format, auto-compaction has to be set. Even with the mentioned limitations, I think it would worth to add support for OFS, it would require 1 small change to enable "ofs" [4] and documenting the limitations. WDYT? Regards, Ferenc [1] https://issues.apache.org/jira/browse/FLINK-28231 [2] https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/filesystem/#general [3] https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-connectors/flink-connector-files/src/main/java/org/apache/flink/connector/file/table/FileSystemTableSink.java#L226 [4] https://github.com/apache/flink/blob/a33a0576364ac3d9c0c038c74362f1faac8d47b8/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopRecoverableWriter.java#L62
Re: [VOTE] Release flink-connector-hbase v3.0.1, release candidate #2
+1 (non-binding) - Validated checksum - Verified signature - Verified no binaries in src archive - Built with Maven 3.8.6 + JDK11 - Verified web PR BR, Ferenc On Saturday, January 13th, 2024 at 04:59, Hang Ruan wrote: > > > +1 (non-binding) > > - Validated checksum hash > - Verified signature > - Verified that no binaries exist in the source archive > - Build the source with Maven and jdk11 > - Verified web PR > > Best, > Hang > > Martijn Visser martijnvis...@apache.org 于2024年1月12日周五 20:30写道: > > > Hi everyone, > > Please review and vote on the release candidate #2 for the > > flink-connector-hbase version > > 3.0.1, as follows: > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > This version is compatible with Flink 1.16.x, 1.17.x and 1.18.x > > > > The complete staging area is available for your review, which includes: > > * JIRA release notes [1], > > * the official Apache source release to be deployed to dist.apache.org > > [2], which are signed with the key with fingerprint > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3], > > * all artifacts to be deployed to the Maven Central Repository [4], > > * source code tag v3.0.1-rc1 [5], > > * website pull request listing the new release [6]. > > > > The vote will be open for at least 72 hours. It is adopted by majority > > approval, with at least 3 PMC affirmative votes. > > > > Thanks, > > Release Manager > > > > [1] https://issues.apache.org/jira/projects/FLINK/versions/12353603 > > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-hbase-3.0.1-rc2 > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1696/ > > [5] > > https://github.com/apache/flink-connector-hbase/releases/tag/v3.0.1-rc2 > > [6] https://github.com/apache/flink-web/pull/708
[jira] [Created] (FLINK-34388) Release Testing: Verify FLINK-28915 Support artifact fetching in Standalone and native K8s application mode
Ferenc Csaky created FLINK-34388: Summary: Release Testing: Verify FLINK-28915 Support artifact fetching in Standalone and native K8s application mode Key: FLINK-34388 URL: https://issues.apache.org/jira/browse/FLINK-34388 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Affects Versions: 1.19.0 Reporter: Ferenc Csaky Fix For: 1.19.0 This ticket covers testing three related features: FLINK-33695, FLINK-33735 and FLINK-33696. Instructions: # Configure Flink to use [Slf4jTraceReporter|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/trace_reporters/#slf4j] and with enabled *INFO* level logging (can be to console or to a file, doesn't matter). # Start a streaming job with enabled checkpointing. # Let it run for a couple of checkpoints. # Verify presence of a single *JobInitialization* [1] trace logged just after job start up. # Verify presence of a couple of *Checkpoint* [1] traces logged after each successful or failed checkpoint. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/traces/#checkpointing-and-initialization -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Drop support for HBase v1
Hi Martijn, thanks for starting the discussion. Let me link the older discussion regarding the same topic [1]. My opinion did not change, so +1. BR, Ferenc [1] https://lists.apache.org/thread/x7l2gj8g93r4v6x6953cyt6jrs8c4r1b On Monday, January 29th, 2024 at 09:37, Martijn Visser wrote: > > > Hi all, > > While working on adding support for Flink 1.19 for HBase, we've run into a > dependency convergence issue because HBase v1 relies on a really old > version of Guava. > > HBase v2 has been made available since May 2018, and there have been no new > releases of HBase v1 since August 2022. > > I would like to propose that the Flink HBase connector drops support for > HBase v1, and will only continue HBase v2 in the future. I don't think this > requires a full FLIP and vote, but I do want to start a discussion thread > for this. > > Best regards, > > Martijn
Re: [VOTE] FLIP-393: Make QueryOperations SQL serializable
+1 (non-binding) Lookgin forward to this! Best, Ferenc On Tuesday, November 21st, 2023 at 12:21, Martijn Visser wrote: > > > +1 (binding) > > Thanks for driving this. > > Best regards, > > Martijn > > On Tue, Nov 21, 2023 at 12:18 PM Benchao Li libenc...@apache.org wrote: > > > +1 (binding) > > > > Dawid Wysakowicz wysakowicz.da...@gmail.com 于2023年11月21日周二 18:56写道: > > > > > Hi everyone, > > > > > > Thank you to everyone for the feedback on FLIP-393: Make QueryOperations > > > SQL serializable[1] > > > which has been discussed in this thread [2]. > > > > > > I would like to start a vote for it. The vote will be open for at least 72 > > > hours unless there is an objection or not enough votes. > > > > > > [1] https://cwiki.apache.org/confluence/x/vQ2ZE > > > [2] https://lists.apache.org/thread/ztyk68brsbmwwo66o1nvk3f6fqqhdxgk > > > > -- > > > > Best, > > Benchao Li
Re: [DISCUSS] FLIP-316: Introduce SQL Driver
Hello devs, Is any active work happening on this FLIP? As far as I see there are blockers that needs to happen first to implement regarding artifact distribution. Is this work in halt completetly or some efforts are going into resolve the blockers first or something? Our platform would benefit this feature a lot, we have a kind of working custom implementation at the moment, but it is uniquely adapted to our app and platform. I could help out to move this forward. Best, Ferenc On Friday, June 30th, 2023 at 04:53, Paul Lam wrote: > > > Hi Jing, > > Thanks for your input! > > > Would you like to add > > one section to describe(better with script/code example) how to use it in > > these two scenarios from users' perspective? > > > OK. I’ll update the FLIP with the code snippet after I get the POC branch > done. > > > NIT: the pictures have transparent background when readers click on it. It > > would be great if you can replace them with pictures with white background. > > > Fixed. Thanks for pointing that out :) > > Best, > Paul Lam > > > 2023年6月27日 06:51,Jing Ge j...@ververica.com.INVALID 写道: > > > > Hi Paul, > > > > Thanks for driving it and thank you all for the informative discussion! The > > FLIP is in good shape now. As described in the FLIP, SQL Driver will be > > mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway in > > application mode and 2. external system integration. Would you like to add > > one section to describe(better with script/code example) how to use it in > > these two scenarios from users' perspective? > > > > NIT: the pictures have transparent background when readers click on it. It > > would be great if you can replace them with pictures with white background. > > > > Best regards, > > Jing > > > > On Mon, Jun 26, 2023 at 1:31 PM Paul Lam > mailto:paullin3...@gmail.com> wrote: > > > > > Hi Shengkai, > > > > > > > * How can we ship the json plan to the JobManager? > > > > > > The Flink K8s module should be responsible for file distribution. We could > > > introduce > > > an option like `kubernetes.storage.dir`. For each flink cluster, there > > > would be a > > > dedicated subdirectory, with the pattern like > > > `${kubernetes.storage.dir}/${cluster-id}`. > > > > > > All resources-related options (e.g. pipeline jars, json plans) that are > > > configured with > > > scheme `file://` > would be > > > uploaded to the resource directory > > > and downloaded to the > > > jobmanager, before SQL Driver accesses the files with the original > > > filenames. > > > > > > > * Classloading strategy > > > > > > We could directly specify the SQL Gateway jar as the jar file in > > > PackagedProgram. > > > It would be treated like a normal user jar and the SQL Driver is loaded > > > into the user > > > classloader. WDYT? > > > > > > > * Option `$internal.sql-gateway.driver.sql-config` is string type > > > > I think it's better to use Map type here > > > > > > By Map type configuration, do you mean a nested map that contains all > > > configurations? > > > > > > I hope I've explained myself well, it’s a file that contains the extra SQL > > > configurations, which would be shipped to the jobmanager. > > > > > > > * PoC branch > > > > > > Sure. I’ll let you know once I get the job done. > > > > > > Best, > > > Paul Lam > > > > > > > 2023年6月26日 14:27,Shengkai Fang > > > mailto:fskm...@gmail.com> 写道: > > > > > > > > Hi, Paul. > > > > > > > > Thanks for your update. I have a few questions about the new design: > > > > > > > > * How can we ship the json plan to the JobManager? > > > > > > > > The current design only exposes an option about the URL of the json > > > > plan. It seems the gateway is responsible to upload to an external > > > > stroage. > > > > Can we reuse the PipelineOptions.JARS to ship to the remote filesystem? > > > > > > > > * Classloading strategy > > > > > > > > Currently, the Driver is in the sql-gateway package. It means the Driver > > > > is not in the JM's classpath directly. Because the sql-gateway jar is > > > > now > > > > in the opt directory rather than lib directory. It may need to add the > > > > external dependencies as Python does[1]. BTW, I think it's better to > > > > move > > > > the Driver into the flink-table-runtime package, which is much easier to > > > > find(Sorry for the wrong opinion before). > > > > > > > > * Option `$internal.sql-gateway.driver.sql-config` is string type > > > > > > > > I think it's better to use Map type here > > > > > > > > * PoC branch > > > > > > > > Because this FLIP involves many modules, do you have a PoC branch to > > > > verify it does work? > > > > > > > > Best, > > > > Shengkai > > > > > > > > [1] > > > > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L940 > > > > > > > >
[jira] [Created] (FLINK-33542) Update HBase connector tests to JUnit5
Ferenc Csaky created FLINK-33542: Summary: Update HBase connector tests to JUnit5 Key: FLINK-33542 URL: https://issues.apache.org/jira/browse/FLINK-33542 Project: Flink Issue Type: Improvement Components: Connectors / HBase Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [RESULT][VOTE] Release flink-connector-hbase v3.0.0, release candidate #2
Hi Martijn! Is this work in progress? Thanks, Ferenc --- Original Message --- On Tuesday, September 12th, 2023 at 10:47, Martijn Visser wrote: > > > I'm happy to announce that we have unanimously approved this release. > > There are 7 approving votes, 3 of which are binding: > * Ahmed (non-binding) > * Sergey (non-binding) > * Samrat (non-binding) > * Ferenc (non-binding) > * Danny (binding) > * Leonard (binding) > * Dong (binding) > > There are no disapproving votes. > > I'll work on completing the release. Thanks all! > > Best regards, > > Martijn
[jira] [Created] (FLINK-33440) Bump flink version on flink-connectors-hbase
Ferenc Csaky created FLINK-33440: Summary: Bump flink version on flink-connectors-hbase Key: FLINK-33440 URL: https://issues.apache.org/jira/browse/FLINK-33440 Project: Flink Issue Type: Improvement Components: Connectors / HBase Reporter: Ferenc Csaky Follow-up the 1.18 release in the connector repo as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33353) SQL fails because "TimestampType.kind" is not serialized
Ferenc Csaky created FLINK-33353: Summary: SQL fails because "TimestampType.kind" is not serialized Key: FLINK-33353 URL: https://issues.apache.org/jira/browse/FLINK-33353 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.18.0 Reporter: Ferenc Csaky We have a custom persistent catalog store, which stores tables, views etc. in a DB. In our application, it is required to utilize the serialized formats of entities, but the same applies to the Hive, as it functions as a persistent catalog. Take the following example SQL: {code:sql} CREATE TABLE IF NOT EXISTS `txn_gen` ( `txn_id` INT, `amount` INT, `ts` TIMESTAMP(3), WATERMARK FOR `ts` AS `ts` - INTERVAL '1' SECOND ) WITH ( 'connector' = 'datagen', 'fields.txn_id.min' = '1', 'fields.txn_id.max' = '5', 'rows-per-second' = '1' ); CREATE VIEW IF NOT EXISTS aggr_ten_sec AS SELECT txn_id, TUMBLE_ROWTIME(`ts`, INTERVAL '10' SECOND) AS w_row_time, COUNT(txn_id) AS txn_count FROM txn_gen GROUP BY txn_id, TUMBLE(`ts`, INTERVAL '10' SECOND); SELECT txn_id, SUM(txn_count), TUMBLE_START(w_row_time, INTERVAL '20' SECOND) AS total_txn_count FROM aggr_ten_sec GROUP BY txn_id, TUMBLE(w_row_time, INTERVAL '20' SECOND); {code} This will work without any problems when we simply execute it in a {{TableEnvironment}}, but it fails with the below error when we try to execute the query based on the serialized table metadata. {code} org.apache.flink.table.api.TableException: Window aggregate can only be defined over a time attribute column, but TIMESTAMP(3) encountered. {code} If there is a view which would require to use ROWTIME, it will be lost and we cannot recreate the same query from the serialized entites. Currently in {{TimestampType}} the "kind" field is deliberatly annotated as {{@Internal}} and is not serialized, although it breaks this functionality. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] Release 1.18.0, release candidate #1
Thanks everyone for the efforts! Checked the following: - Downloaded artifacts - Built Flink from source - Verified checksums/signatures - Verified NOTICE, LICENSE files - Deployed dummy SELECT job via SQL gateway on standalone cluster, things seemed fine according to the log files +1 (non-binding) Best, Ferenc --- Original Message --- On Friday, September 29th, 2023 at 22:12, Gabor Somogyi wrote: > > > Thanks for the efforts! > > +1 (non-binding) > > * Verified versions in the poms > * Built from source > * Verified checksums and signatures > * Started basic workloads with kubernetes operator > * Verified NOTICE and LICENSE files > > G > > On Fri, Sep 29, 2023, 18:16 Matthias Pohl matthias.p...@aiven.io.invalid > > wrote: > > > Thanks for creating RC1. I did the following checks: > > > > * Downloaded artifacts > > * Built Flink from sources > > * Verified SHA512 checksums GPG signatures > > * Compared checkout with provided sources > > * Verified pom file versions > > * Went over NOTICE file/pom files changes without finding anything > > suspicious > > * Deployed standalone session cluster and ran WordCount example in batch > > and streaming: Nothing suspicious in log files found > > > > +1 (binding) > > > > On Fri, Sep 29, 2023 at 10:34 AM Etienne Chauchot echauc...@apache.org > > wrote: > > > > > Hi all, > > > > > > Thanks to the team for this RC. > > > > > > I did a quick check of this RC against user pipelines (1) coded with > > > DataSet (even if deprecated and soon removed), DataStream and SQL APIs > > > > > > based on the small scope of this test, LGTM > > > > > > +1 (non-binding) > > > > > > [1] https://github.com/echauchot/tpcds-benchmark-flink > > > > > > Best > > > Etienne > > > > > > Le 28/09/2023 à 19:35, Jing Ge a écrit : > > > > > > > Hi everyone, > > > > > > > > The RC1 for Apache Flink 1.18.0 has been created. The related voting > > > > process will be triggered once the announcement is ready. The RC1 has > > > > all > > > > the artifacts that we would typically have for a release, except for > > > > the > > > > release note and the website pull request for the release announcement. > > > > > > > > The following contents are available for your review: > > > > > > > > - Confirmation of no benchmarks regression at the thread[1]. > > > > - The preview source release and binary convenience releases [2], which > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 [3]. > > > > - all artifacts that would normally be deployed to the Maven > > > > Central Repository [4]. > > > > - source code tag "release-1.18.0-rc1" [5] > > > > > > > > Your help testing the release will be greatly appreciated! And we'll > > > > create the rc1 release and the voting thread as soon as all the efforts > > > > are > > > > finished. > > > > > > > > [1]https://lists.apache.org/thread/yxyphglwwvq57wcqlfrnk3qo9t3sr2ro > > > > [2]https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc1/ > > > > [3]https://dist.apache.org/repos/dist/release/flink/KEYS > > > > [4] > > > > https://repository.apache.org/content/repositories/orgapacheflink-1657 > > > > [5]https://github.com/apache/flink/releases/tag/release-1.18.0-rc1 > > > > > > > > Best regards, > > > > Qingsheng, Sergei, Konstantin and Jing
Re: Future of classical remoting in Pekko
That is a fair point. It not fixes a bug per se, but it would mitigate security vulnerabilities (Netty 3.x CVEs), so my thought was it might qualify it for addressing in a patch release. IMO handling security vulnerabilities is a gray area, if it only requires to bump some deps that are only used internally, it can be considered as a patch compatible change. I am not sure at this point what changes would be required to use the updated Pekko version, so it is possible that we can only introduce it in 1.19, just wanted to clarify my thought process. --- Original Message --- On Wednesday, September 20th, 2023 at 14:26, Martijn Visser wrote: > > > Just chipping in that I don't think we should add Pekko changes in a > patch release, because I think the Pekko related changes don't fix a > bug. > > On Tue, Sep 19, 2023 at 9:06 PM Ferenc Csaky ferenc.cs...@pm.me.invalid wrote: > > > I think that is totally fine, because any Pekko related changes can only be > > added to the first patch release of 1.18 at this point, as there is an RC0 > > [1] already so the release process will be initiated soon. > > > > I am glad the mentioned PR got merged, did not have the chance to review. > > > > [1] https://lists.apache.org/thread/5x28rp3zct4p603hm4zdwx6kfr101w38 > > > > --- Original Message --- > > On Monday, September 18th, 2023 at 14:20, Matthew de Detrich > > matthew.dedetr...@aiven.io.INVALID wrote: > > > > > I think that the end of September is too soon for a Pekko 1.1.x, there are > > > still more things > > > that we would like to merge before making a release. > > > > > > Good news is that the PR to migrate to netty4 for classic remoting has > > > been > > > merged > > > (see https://github.com/apache/incubator-pekko/pull/643). Improvements are > > > also > > > still be done, so the next minor version release of Pekko (1.1.0) will > > > contain these > > > changes. > > > > > > On Wed, Sep 13, 2023 at 11:22 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > > > > > wrote: > > > > > > > The target release date for 1.18 is the end of Sept [1], but I'm not > > > > sure > > > > everything will come together by then. Maybe it will pushed by a couple > > > > days. > > > > > > > > I'm happy to help out, even making the Flink related changes when we're > > > > at > > > > that point. > > > > > > > > [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release > > > > > > > > --- Original Message --- > > > > On Tuesday, September 12th, 2023 at 17:43, He Pin he...@apache.org > > > > wrote: > > > > > > > > > Hi Ferenc: > > > > > What's the ETA of the Flink 1.18? I think we should beable to > > > > > collaborate on this,and at work we are using Flink too. > > > > > > > > > > On 2023/09/12 15:16:11 Ferenc Csaky wrote: > > > > > > > > > > > Hi Matthew, > > > > > > > > > > > > Thanks for bringing this up! Cca half a year ago I started to work > > > > > > on > > > > > > an Akka Artery migration, there is a draft PR for that 1. It might > > > > > > be an > > > > > > option to revive that work and point it against Pekko instead. > > > > > > Although I > > > > > > would highlight FLINK-29281 2 which will replace the whole RPC > > > > > > implementation in Flink to a gRPC-based one when it is done. > > > > > > > > > > > > I am not sure about the progess on the gRPC work, it looks hanging > > > > > > for > > > > > > a while now, so I think if there is a chance to replace Netty3 with > > > > > > Netty4 > > > > > > in Pekko in the short term it would benefit Flink and then we can > > > > > > decide if > > > > > > it would worth to upgrade to Artery, or how fast the gRPC solution > > > > > > can be > > > > > > done and then it will not be necessary. > > > > > > > > > > > > All in all, in the short term I think Flink would benefit to have > > > > > > that > > > > > > mentioned PR 3 merged, then the updated Pekko version could be > > > > > > included in > > > > > > the first 1.18 patch proba
Re: Future of classical remoting in Pekko
I think that is totally fine, because any Pekko related changes can only be added to the first patch release of 1.18 at this point, as there is an RC0 [1] already so the release process will be initiated soon. I am glad the mentioned PR got merged, did not have the chance to review. [1] https://lists.apache.org/thread/5x28rp3zct4p603hm4zdwx6kfr101w38 --- Original Message --- On Monday, September 18th, 2023 at 14:20, Matthew de Detrich wrote: > > > I think that the end of September is too soon for a Pekko 1.1.x, there are > still more things > that we would like to merge before making a release. > > Good news is that the PR to migrate to netty4 for classic remoting has been > merged > (see https://github.com/apache/incubator-pekko/pull/643). Improvements are > also > still be done, so the next minor version release of Pekko (1.1.0) will > contain these > changes. > > On Wed, Sep 13, 2023 at 11:22 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > The target release date for 1.18 is the end of Sept [1], but I'm not sure > > everything will come together by then. Maybe it will pushed by a couple > > days. > > > > I'm happy to help out, even making the Flink related changes when we're at > > that point. > > > > [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release > > > > --- Original Message --- > > On Tuesday, September 12th, 2023 at 17:43, He Pin he...@apache.org > > wrote: > > > > > Hi Ferenc: > > > What's the ETA of the Flink 1.18? I think we should beable to > > > collaborate on this,and at work we are using Flink too. > > > > > > On 2023/09/12 15:16:11 Ferenc Csaky wrote: > > > > > > > Hi Matthew, > > > > > > > > Thanks for bringing this up! Cca half a year ago I started to work on > > > > an Akka Artery migration, there is a draft PR for that 1. It might be an > > > > option to revive that work and point it against Pekko instead. Although > > > > I > > > > would highlight FLINK-29281 2 which will replace the whole RPC > > > > implementation in Flink to a gRPC-based one when it is done. > > > > > > > > I am not sure about the progess on the gRPC work, it looks hanging for > > > > a while now, so I think if there is a chance to replace Netty3 with > > > > Netty4 > > > > in Pekko in the short term it would benefit Flink and then we can > > > > decide if > > > > it would worth to upgrade to Artery, or how fast the gRPC solution can > > > > be > > > > done and then it will not be necessary. > > > > > > > > All in all, in the short term I think Flink would benefit to have that > > > > mentioned PR 3 merged, then the updated Pekko version could be included > > > > in > > > > the first 1.18 patch probably to mitigate those pesky Netty3 CVEs that > > > > are > > > > carried for a while ASAP. > > > > > > > > Cheers, > > > > Ferenc > > > > > > > > 1 https://github.com/apache/flink/pull/22271 > > > > 2 https://issues.apache.org/jira/browse/FLINK-29281 > > > > 3 https://github.com/apache/incubator-pekko/pull/643 > > > > > > > > --- Original Message --- > > > > On Tuesday, September 12th, 2023 at 10:29, Matthew de Detrich > > > > matthew.dedetr...@aiven.io.INVALID wrote: > > > > > > > > > It's come to my attention that Flink is using Pekko's classical > > > > > remoting, > > > > > if this is the case then I would recommend making a response at > > > > > https://lists.apache.org/thread/19h2wrs2om91g5vhnftv583fo0ddfshm . > > > > > > > > > > Quick summary of what is being discussed is what to do with Pekko's > > > > > classical remoting. Classic remoting is considered deprecated since > > > > > 2019, > > > > > an artifact that we inherited from Akka1. Ontop of this classical > > > > > remoting happens to be using netty3 which has known CVE's2, these > > > > > CVE's > > > > > were never fixed in the netty3 series. > > > > > > > > > > The question is what should be done given this, i.e. some people in > > > > > the > > > > > Pekko community are wanting to drop classical remoting as quickly as > > > > > possible (i.e. even sooner then what semver allows but this is being >
Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and support the Standalone Autoscaler
Looking forward to this! +1 (non-binding) Cheers, Ferenc --- Original Message --- On Wednesday, September 13th, 2023 at 12:33, Maximilian Michels wrote: > > > +1 (binding) > > On Wed, Sep 13, 2023 at 12:28 PM Gyula Fóra gyula.f...@gmail.com wrote: > > > +1 (binding) > > > > Gyula > > > > On Wed, 13 Sep 2023 at 09:33, Matt Wang wang...@163.com wrote: > > > > > Thank you for driving this FLIP, > > > > > > +1 (non-binding) > > > > > > -- > > > > > > Best, > > > Matt Wang > > > > > > Replied Message > > > | From | conradjamjam.gz...@gmail.com | > > > | Date | 09/13/2023 15:28 | > > > | To | dev@flink.apache.org | > > > | Subject | Re: [VOTE] FLIP-334: Decoupling autoscaler and kubernetes and > > > support the Standalone Autoscaler | > > > best idea > > > +1 (non-binding) > > > > > > Ahmed Hamdy hamdy10...@gmail.com 于2023年9月13日周三 15:23写道: > > > > > > Hi Rui, > > > I have gone through the thread. > > > +1 (non-binding) > > > > > > Best Regards > > > Ahmed Hamdy > > > > > > On Wed, 13 Sept 2023 at 03:53, Rui Fan 1996fan...@gmail.com wrote: > > > > > > Hi all, > > > > > > Thanks for all the feedback about the FLIP-334: > > > Decoupling autoscaler and kubernetes and > > > support the Standalone Autoscaler[1]. > > > This FLIP was discussed in [2]. > > > > > > I'd like to start a vote for it. The vote will be open for at least 72 > > > hours (until Sep 16th 11:00 UTC+8) unless there is an objection or > > > insufficient votes. > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-334+%3A+Decoupling+autoscaler+and+kubernetes+and+support+the+Standalone+Autoscaler > > > [2] https://lists.apache.org/thread/kmm03gls1vw4x6vk1ypr9ny9q9522495 > > > > > > Best, > > > Rui > > > > > > -- > > > Best > > > > > > ConradJam
Re: Future of classical remoting in Pekko
The target release date for 1.18 is the end of Sept [1], but I'm not sure everything will come together by then. Maybe it will pushed by a couple days. I'm happy to help out, even making the Flink related changes when we're at that point. [1] https://cwiki.apache.org/confluence/display/FLINK/1.18+Release --- Original Message --- On Tuesday, September 12th, 2023 at 17:43, He Pin wrote: > > > Hi Ferenc: > What's the ETA of the Flink 1.18? I think we should beable to collaborate on > this,and at work we are using Flink too. > > On 2023/09/12 15:16:11 Ferenc Csaky wrote: > > > Hi Matthew, > > > > Thanks for bringing this up! Cca half a year ago I started to work on an > > Akka Artery migration, there is a draft PR for that 1. It might be an > > option to revive that work and point it against Pekko instead. Although I > > would highlight FLINK-29281 2 which will replace the whole RPC > > implementation in Flink to a gRPC-based one when it is done. > > > > I am not sure about the progess on the gRPC work, it looks hanging for a > > while now, so I think if there is a chance to replace Netty3 with Netty4 in > > Pekko in the short term it would benefit Flink and then we can decide if it > > would worth to upgrade to Artery, or how fast the gRPC solution can be done > > and then it will not be necessary. > > > > All in all, in the short term I think Flink would benefit to have that > > mentioned PR 3 merged, then the updated Pekko version could be included in > > the first 1.18 patch probably to mitigate those pesky Netty3 CVEs that are > > carried for a while ASAP. > > > > Cheers, > > Ferenc > > > > 1 https://github.com/apache/flink/pull/22271 > > 2 https://issues.apache.org/jira/browse/FLINK-29281 > > 3 https://github.com/apache/incubator-pekko/pull/643 > > > > --- Original Message --- > > On Tuesday, September 12th, 2023 at 10:29, Matthew de Detrich > > matthew.dedetr...@aiven.io.INVALID wrote: > > > > > It's come to my attention that Flink is using Pekko's classical remoting, > > > if this is the case then I would recommend making a response at > > > https://lists.apache.org/thread/19h2wrs2om91g5vhnftv583fo0ddfshm . > > > > > > Quick summary of what is being discussed is what to do with Pekko's > > > classical remoting. Classic remoting is considered deprecated since 2019, > > > an artifact that we inherited from Akka1. Ontop of this classical > > > remoting happens to be using netty3 which has known CVE's2, these CVE's > > > were never fixed in the netty3 series. > > > > > > The question is what should be done given this, i.e. some people in the > > > Pekko community are wanting to drop classical remoting as quickly as > > > possible (i.e. even sooner then what semver allows but this is being > > > discussed) and others are wanting to leave it as it is (even with the > > > CVE's) since we don't want to incentivize and/or create impression that we > > > are officially supporting it. There is also a currently open PR3 which > > > upgrades Pekko's classical remoting's from netty3 to netty4 with the > > > primary motivator being removing said CVE's. > > > > > > My personal position on the matter is that Pekko shouldn't drop classical > > > remoting until 2.0.x (to satisfy semver) while also updating Pekko's > > > classical remoting netty dependency to netty4 so that we are not shipping > > > Pekko with known CVE's (if this gets approved such a change would likely > > > land in Pekko 1.1.0). As is customary, such a decision should be agreed > > > upon broadly in the Pekko community. > > > > > > Note that regardless of this change, it's recommended that a plan should > > > be > > > made at some point by Flink to move from classical remoting to artery4 > > > although the decision that Pekko ultimately makes may influence the > > > timeline (hence the reason for this thread). > > > > > > -- > > > > > > Matthew de Detrich > > > > > > Aiven Deutschland GmbH > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > m: +491603708037 > > > > > > w: aiven.io e: matthew.dedetr...@aiven.io
Re: Future of classical remoting in Pekko
Hi Matthew, Thanks for bringing this up! Cca half a year ago I started to work on an Akka Artery migration, there is a draft PR for that [1]. It might be an option to revive that work and point it against Pekko instead. Although I would highlight FLINK-29281 [2] which will replace the whole RPC implementation in Flink to a gRPC-based one when it is done. I am not sure about the progess on the gRPC work, it looks hanging for a while now, so I think if there is a chance to replace Netty3 with Netty4 in Pekko in the short term it would benefit Flink and then we can decide if it would worth to upgrade to Artery, or how fast the gRPC solution can be done and then it will not be necessary. All in all, in the short term I think Flink would benefit to have that mentioned PR [3] merged, then the updated Pekko version could be included in the first 1.18 patch probably to mitigate those pesky Netty3 CVEs that are carried for a while ASAP. Cheers, Ferenc [1] https://github.com/apache/flink/pull/22271 [2] https://issues.apache.org/jira/browse/FLINK-29281 [3] https://github.com/apache/incubator-pekko/pull/643 --- Original Message --- On Tuesday, September 12th, 2023 at 10:29, Matthew de Detrich wrote: > > > It's come to my attention that Flink is using Pekko's classical remoting, > if this is the case then I would recommend making a response at > https://lists.apache.org/thread/19h2wrs2om91g5vhnftv583fo0ddfshm . > > Quick summary of what is being discussed is what to do with Pekko's > classical remoting. Classic remoting is considered deprecated since 2019, > an artifact that we inherited from Akka[1]. Ontop of this classical > remoting happens to be using netty3 which has known CVE's[2], these CVE's > were never fixed in the netty3 series. > > The question is what should be done given this, i.e. some people in the > Pekko community are wanting to drop classical remoting as quickly as > possible (i.e. even sooner then what semver allows but this is being > discussed) and others are wanting to leave it as it is (even with the > CVE's) since we don't want to incentivize and/or create impression that we > are officially supporting it. There is also a currently open PR[3] which > upgrades Pekko's classical remoting's from netty3 to netty4 with the > primary motivator being removing said CVE's. > > My personal position on the matter is that Pekko shouldn't drop classical > remoting until 2.0.x (to satisfy semver) while also updating Pekko's > classical remoting netty dependency to netty4 so that we are not shipping > Pekko with known CVE's (if this gets approved such a change would likely > land in Pekko 1.1.0). As is customary, such a decision should be agreed > upon broadly in the Pekko community. > > Note that regardless of this change, it's recommended that a plan should be > made at some point by Flink to move from classical remoting to artery[4] > although the decision that Pekko ultimately makes may influence the > timeline (hence the reason for this thread). > > [1]: https://github.com/akka/akka/issues/31764 > [2]: https://mvnrepository.com/artifact/io.netty/netty/3.10.6.Final > [3]: https://github.com/apache/incubator-pekko/pull/643 > [4]: https://pekko.apache.org/docs/pekko/current/remoting-artery.html > > -- > > Matthew de Detrich > > Aiven Deutschland GmbH > > Immanuelkirchstraße 26, 10405 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > m: +491603708037 > > w: aiven.io e: matthew.dedetr...@aiven.io
Re: [VOTE] Release flink-connector-hbase v3.0.0, release candidate 2
Hi, Thanks Martijn for initiating the release! +1 (non-binding) - checked signatures and checksums - checked source has no binaries - checked LICENSE and NOTICE files - approved web PR Cheers, Ferenc --- Original Message --- On Monday, September 4th, 2023 at 12:54, Samrat Deb wrote: > > > Hi, > > +1 (non-binding) > > Verified NOTICE files > Verified CheckSum and signatures > Glanced through PR[1] , Looks good to me > > Bests, > Samrat > > [1]https://github.com/apache/flink-web/pull/591 > > > > On 04-Sep-2023, at 2:22 PM, Ahmed Hamdy hamdy10...@gmail.com wrote: > > > > Hi Martijn, > > +1 (non-binding) > > > > - verified Checksums and signatures > > - no binaries in source > > - Checked NOTICE files contains migrated artifacts > > - tag is correct > > - Approved Web PR > > > > Best Regards > > Ahmed Hamdy > > > > On Fri, 1 Sept 2023 at 15:35, Martijn Visser martijnvis...@apache.org > > wrote: > > > > > Hi everyone, > > > > > > Please review and vote on the release candidate #2 for the version 3.0.0, > > > as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source release to be deployed to dist.apache.org > > > [2], > > > which are signed with the key with fingerprint > > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tag v3.0.0-rc2 [5], > > > * website pull request listing the new release [6]. > > > > > > This replaces the old, cancelled vote of RC1 [7]. This version is the > > > externalized version which is compatible with Flink 1.16 and 1.17. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Release Manager > > > > > > [1] > > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352578 > > > [2] > > > > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-hbase-3.0.0-rc2 > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] https://repository.apache.org/content/repositories/orgapacheflink-1650 > > > [5] > > > https://github.com/apache/flink-connector-hbase/releases/tag/v3.0.0-rc2 > > > [6] https://github.com/apache/flink-web/pull/591 > > > [7] https://lists.apache.org/thread/wbl6sc86q9s5mmz5slx4z09svh91cpr0
[jira] [Created] (FLINK-32811) Add port range support for taskmanager.data.bind-port
Ferenc Csaky created FLINK-32811: Summary: Add port range support for taskmanager.data.bind-port Key: FLINK-32811 URL: https://issues.apache.org/jira/browse/FLINK-32811 Project: Flink Issue Type: Improvement Components: Runtime / Configuration, Runtime / Coordination Reporter: Ferenc Csaky Fix For: 1.19.0 Adding this feature could be helpful for installation in a restrictive network setup. The "port range" support is already available for some other port config options anyway. Right now, it is possible to specify a {{taskmanager.data.port}} and {{taskmanager.data.bind-port}} to be able to support NAT-like setups, although {{taskmanager.data.port}} is not bound to anything itself, so supporting a port range there is not an option according to my understanding. Although, supporting a port range only for {{taskmanager.data.bind-port}} can be still helpful for anyone who does not require a NAT capability, because if {{taskmanager.data.bind-port}} is set and {{taskmanager.data.port}} is set to *0*, then the bound port will be used everywhere. This change should keep the already possible setups working as is. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] Flink 1.18 feature freeze
Hi, I would like to ask if releasing the external HBase connector and remove those modules from the core Flink repo before Flink 1.18 sounds feasible at this point or not? The connector source is ready for quite a while, I externalized the E2E tests a couple weeks before and CI is passing so in theory there is no remaining task to release the connector. Martijn manages the tasks regarding connector externalization, but he is on holiday at the moment, so if this is something that can be done, is there anyone who can help with the connector release? I volunteer to help. Thanks, Ferenc --- Original Message --- On Wednesday, August 2nd, 2023 at 03:27, Jing Ge wrote: > > > Hi, > > Thanks for keeping us in the loop. +1 for merging these PRs. > > @Alex Please feel free to contact us for CR and PR merges. > > @Matthias the Pekko migration is an important task, since it is related to > CVE issues. Thanks for bringing it to our attention. > > Best regards, > Jing > > > > On Tue, Aug 1, 2023 at 5:23 PM Qingsheng Ren re...@apache.org wrote: > > > Thanks for letting us know, Matthias! > > > > As discussed in the release sync, +1 for merging these PRs. > > > > Best, > > Qingsheng > > > > On Tue, Aug 1, 2023 at 5:17 PM Matthias Pohl matthias.p...@aiven.io > > wrote: > > > > > I'm requesting to merge in FLINK-32098 [1]. It's a minor change that > > > reduces the amount of exists calls to S3 while submitting a job (which > > > can > > > be an expensive operation if the object actually doesn't exist but the > > > corresponding bucket itself contains a lot of objects). The PR is > > > reviewed > > > and ready to be merged. The change itself is minor and covered by > > > existing > > > tests. > > > > > > Additionally, I want to mention the two already merged (after > > > feature-freeze) changes: > > > - FLINK-32583 [2] which is a minor bugfix. I didn't explicitly mention it > > > initially because of the fact that it fixes a bug (which admittedly was > > > already present in older versions of Flink). I'm happy to revert that one > > > if the release managers have concerns. It's fixing a scenario where the > > > RestClient becomes unresponsive when submitting a request in rare cases. > > > - Migration from Akka to Pekko (FLINK-32468 [3], FLINK-32683 [4]): This > > > was agreed on in last week's 1.18 release sync. I just forgot to make it > > > public on the ML. The Pekko change will be "release-tested" as part of > > > FLINK-32678 stress test. > > > > > > Matthias > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-32098 > > > [2] https://issues.apache.org/jira/browse/FLINK-32583 > > > [3] https://issues.apache.org/jira/browse/FLINK-32468 > > > [4] https://issues.apache.org/jira/browse/FLINK-32683 > > > [5] https://issues.apache.org/jira/browse/FLINK-32678 > > > > > > On Tue, Aug 1, 2023 at 10:58 AM Qingsheng Ren re...@apache.org wrote: > > > > > > > Thanks for letting us know, Alexander and Leonard! > > > > > > > > I checked these PRs and the changes are trivial. +1 for merging them. > > > > > > > > Best, > > > > Qingsheng > > > > > > > > On Tue, Aug 1, 2023 at 12:14 AM Leonard Xu xbjt...@gmail.com wrote: > > > > > > > > > Thank all Release Managers for driving the 1.18 release work! > > > > > > > > > > > - Deprecation works for 2.0 > > > > > > > > > > > > As discussed in another thread [3], we will not give extra > > > > > > extensions > > > > > > to > > > > > > deprecation works considering the overhead and potential side > > > > > > effects > > > > > > to > > > > > > the timeline of 1.18. We can accept tiny changes that only add > > > > > > annotations > > > > > > and JavaDocs, but please let us know before you are going to do > > > > > > that. > > > > > > > > > > Alexander and I ready to deprecate SourceFunction APIs as above > > > > > discussion > > > > > thread[1][2], now we apply the permission to merge following three > > > > > PRs : > > > > > > > > > > The first two PRs [3][4] only contains @Deprecated annotations and > > > > > JavaDocs, the PR[5] contains @Deprecated annotations, JavaDocs, and > > > > > necessary tiny changes for example code as some examples with strict > > > > > deprecation compiler checks to SourceFunction API, it should be okay > > > > > as > > > > > it > > > > > only changed example code, you can check the tiny change in this > > > > > commit[5]. > > > > > > > > > > Best, > > > > > Alexander and Leonard > > > > > > > > > > [1]https://lists.apache.org/thread/yyw52k45x2sp1jszldtdx7hc98n72w7k > > > > > [2]https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v > > > > > [3]https://github.com/apache/flink/pull/23106 > > > > > [4]https://github.com/apache/flink/pull/23079 > > > > > [5]https://github.com/apache/flink/pull/23105 > > > > > [6] > > > > https://github.com/apache/flink/pull/23079/commits/f5ea3c073d36f21fb4fe47e83c717ac080995509
[jira] [Created] (FLINK-32660) Support external file systems in FileCatalogStore
Ferenc Csaky created FLINK-32660: Summary: Support external file systems in FileCatalogStore Key: FLINK-32660 URL: https://issues.apache.org/jira/browse/FLINK-32660 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky Fix For: 1.18.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] Persistent SQL Gateway
Hi Shammon, Thank you for your answer and explanation, my latest experiment was a SELECT query and my assumptions were based on that, INSERT works as described. Regarding the state of FLIP-295, I just checked out the recently created jiras [1] and if I can help out with any part, please let me know. Cheers, F [1] https://issues.apache.org/jira/browse/FLINK-32427 --- Original Message --- On Tuesday, June 27th, 2023 at 13:39, Shammon FY wrote: > > > Hi Ferenc, > > If I understand correctly, there will be two types of jobs in sql-gateway: > `SELECT` and `NON-SELECT` such as `INSERT`. > > 1. `SELECT` jobs need to collect results from Flink cluster in a > corresponding session of sql gateway, and when the session is closed, the > job should be canceled. These jobs are generally short queries similar to > OLAP and I think it may be acceptable. > > 2. `NON-SELECT` jobs may be batch or streaming jobs, and when the jobs are > submitted successfully, they won't be killed or canceled even if the > session or sql-gateway is closed. After these assignments are successfully > submitted, the lifecycle is no longer managed by SQL gateway. > > I don't know if it covers your usage scenario. Could you describe yours for > us to test and confirm? > > Best, > Shammon FY > > > On Tue, Jun 27, 2023 at 6:43 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi Jark, > > > > In the current implementation, any job submitted via the SQL Gateway has > > to be done through a session, cause all the operations are grouped under > > sessions. > > > > Starting from there, if I close a session, that will close the > > "SessionContext", which closes the "OperationManager" [1], and the > > "OperationManager" closes all submitted operations tied to that session > > [2], which results closing all the jobs executed in the session. > > > > Maybe I am missing something, but my experience is that the jobs I submit > > via the SQL Gateway are getting cleaned up on gateway session close. > > > > WDYT? > > > > Cheers, > > F > > > > [1] > > https://github.com/apache/flink/blob/149a5e34c1ed8d8943c901a98c65c70693915811/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/context/SessionContext.java#L204 > > [2] > > https://github.com/apache/flink/blob/149a5e34c1ed8d8943c901a98c65c70693915811/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/operation/OperationManager.java#L194 > > > > --- Original Message --- > > On Tuesday, June 27th, 2023 at 04:37, Jark Wu imj...@gmail.com wrote: > > > > > Hi Ferenc, > > > > > > But the job lifecycle doesn't tie to the SQL Gateway session. > > > Even if the session is closed, all the running jobs are not affected. > > > > > > Best, > > > Jark > > > > > > On Tue, 27 Jun 2023 at 04:14, Ferenc Csaky ferenc.cs...@pm.me.invalid > > > > > > wrote: > > > > > > > Hi Jark, > > > > > > > > Thank you for pointing out FLIP-295 abouth catalog persistence, I was > > > > not > > > > aware the current state. Although as far as I see, that persistent > > > > catalogs > > > > are necessary, but not sufficient achieving a "persistent gateway". > > > > > > > > The current implementation ties the job lifecycle to the SQL gateway > > > > session, so if it gets closed, it will cancel all the jobs. So that > > > > would > > > > be the next step I think. Any work or thought regarding this aspect? > > > > We are > > > > definitely willing to help out on this front. > > > > > > > > Cheers, > > > > F > > > > > > > > --- Original Message --- > > > > On Sunday, June 25th, 2023 at 06:23, Jark Wu imj...@gmail.com wrote: > > > > > > > > > Hi Ferenc, > > > > > > > > > > Making SQL Gateway to be an easy-to-use platform infrastructure of > > > > > Flink > > > > > SQL > > > > > is one of the important roadmaps 1. > > > > > > > > > > The persistence ability of the SQL Gateway is a major work in 1.18 > > > > > release. > > > > > One of the persistence demand is that the registered catalogs are > > > > > currently > > > > > kept in memory and lost when Gateway restarts. There is an accepted > >
Re: [DISCUSS] Persistent SQL Gateway
Hi Jark, In the current implementation, any job submitted via the SQL Gateway has to be done through a session, cause all the operations are grouped under sessions. Starting from there, if I close a session, that will close the "SessionContext", which closes the "OperationManager" [1], and the "OperationManager" closes all submitted operations tied to that session [2], which results closing all the jobs executed in the session. Maybe I am missing something, but my experience is that the jobs I submit via the SQL Gateway are getting cleaned up on gateway session close. WDYT? Cheers, F [1] https://github.com/apache/flink/blob/149a5e34c1ed8d8943c901a98c65c70693915811/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/context/SessionContext.java#L204 [2] https://github.com/apache/flink/blob/149a5e34c1ed8d8943c901a98c65c70693915811/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/operation/OperationManager.java#L194 --- Original Message --- On Tuesday, June 27th, 2023 at 04:37, Jark Wu wrote: > > > Hi Ferenc, > > But the job lifecycle doesn't tie to the SQL Gateway session. > Even if the session is closed, all the running jobs are not affected. > > Best, > Jark > > > > > On Tue, 27 Jun 2023 at 04:14, Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi Jark, > > > > Thank you for pointing out FLIP-295 abouth catalog persistence, I was not > > aware the current state. Although as far as I see, that persistent catalogs > > are necessary, but not sufficient achieving a "persistent gateway". > > > > The current implementation ties the job lifecycle to the SQL gateway > > session, so if it gets closed, it will cancel all the jobs. So that would > > be the next step I think. Any work or thought regarding this aspect? We are > > definitely willing to help out on this front. > > > > Cheers, > > F > > > > --- Original Message --- > > On Sunday, June 25th, 2023 at 06:23, Jark Wu imj...@gmail.com wrote: > > > > > Hi Ferenc, > > > > > > Making SQL Gateway to be an easy-to-use platform infrastructure of Flink > > > SQL > > > is one of the important roadmaps 1. > > > > > > The persistence ability of the SQL Gateway is a major work in 1.18 > > > release. > > > One of the persistence demand is that the registered catalogs are > > > currently > > > kept in memory and lost when Gateway restarts. There is an accepted FLIP > > > (FLIP-295)[2] target to resolve this issue and make Gateway can persist > > > the > > > registered catalogs information into files or databases. > > > > > > I'm not sure whether this is something you are looking for? > > > > > > Best, > > > Jark > > > > > > [2]: > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations > > > > > On Fri, 23 Jun 2023 at 00:25, Ferenc Csaky ferenc.cs...@pm.me.invalid > > > > > > wrote: > > > > > > > Hello devs, > > > > > > > > I would like to open a discussion about persistence possibilitis for > > > > the > > > > SQL Gateway. At Cloudera, we are happy to see the work already done on > > > > this > > > > project and looking for ways to utilize it on our platform as well, but > > > > currently it lacks some features that would be essential in our case, > > > > where > > > > we could help out. > > > > > > > > I am not sure if any thought went into gateway persistence specifics > > > > already, and this feature could be implemented in fundamentally > > > > differnt > > > > ways, so I think the frist step could be to agree on the basics. > > > > > > > > First, in my opinion, persistence should be an optional feature of the > > > > gateway, that can be enabled if desired. There can be a lot of > > > > implementation details, but there can be some major directions to > > > > follow: > > > > > > > > - Utilize Hive catalog: The Hive catalog can already be used to have > > > > persistenct meta-objects, so the crucial thing that would be missing in > > > > this case is other catalogs. Personally, I would not pursue this > > > > option, > > > > because in my opinion it would limit the usability of this feature to
Re: [DISCUSS] Persistent SQL Gateway
Hi Jark, Thank you for pointing out FLIP-295 abouth catalog persistence, I was not aware the current state. Although as far as I see, that persistent catalogs are necessary, but not sufficient achieving a "persistent gateway". The current implementation ties the job lifecycle to the SQL gateway session, so if it gets closed, it will cancel all the jobs. So that would be the next step I think. Any work or thought regarding this aspect? We are definitely willing to help out on this front. Cheers, F --- Original Message --- On Sunday, June 25th, 2023 at 06:23, Jark Wu wrote: > > > Hi Ferenc, > > Making SQL Gateway to be an easy-to-use platform infrastructure of Flink > SQL > is one of the important roadmaps [1]. > > The persistence ability of the SQL Gateway is a major work in 1.18 release. > One of the persistence demand is that the registered catalogs are currently > kept in memory and lost when Gateway restarts. There is an accepted FLIP > (FLIP-295)[2] target to resolve this issue and make Gateway can persist the > registered catalogs information into files or databases. > > I'm not sure whether this is something you are looking for? > > Best, > Jark > > > [1]: https://flink.apache.org/roadmap/#a-unified-sql-platform > [2]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-295%3A+Support+lazy+initialization+of+catalogs+and+persistence+of+catalog+configurations > > On Fri, 23 Jun 2023 at 00:25, Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hello devs, > > > > I would like to open a discussion about persistence possibilitis for the > > SQL Gateway. At Cloudera, we are happy to see the work already done on this > > project and looking for ways to utilize it on our platform as well, but > > currently it lacks some features that would be essential in our case, where > > we could help out. > > > > I am not sure if any thought went into gateway persistence specifics > > already, and this feature could be implemented in fundamentally differnt > > ways, so I think the frist step could be to agree on the basics. > > > > First, in my opinion, persistence should be an optional feature of the > > gateway, that can be enabled if desired. There can be a lot of > > implementation details, but there can be some major directions to follow: > > > > - Utilize Hive catalog: The Hive catalog can already be used to have > > persistenct meta-objects, so the crucial thing that would be missing in > > this case is other catalogs. Personally, I would not pursue this option, > > because in my opinion it would limit the usability of this feature too much. > > - Serialize the session as is: Saving the whole session (or its context) > > [1] as is to durable storage, so it can be kept and picked up again. > > - Serialize the required elements (catalogs, tables, functions, etc.), not > > necessarily as a whole: The main point here would be to serialize a > > different object, so the persistent data will not be that sensitive to > > changes of the session (or its context). There can be numerous factors > > here, like try to keep the model close to the session itself, so the > > boilerplate required for the mapping can be kept to minimal, or focus on > > saving what is actually necessary, making the persistent storage more > > portable. > > > > WDYT? > > > > Cheers, > > F > > > > [1] > > https://github.com/apache/flink/blob/master/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/session/Session.java
[DISCUSS] Persistent SQL Gateway
Hello devs, I would like to open a discussion about persistence possibilitis for the SQL Gateway. At Cloudera, we are happy to see the work already done on this project and looking for ways to utilize it on our platform as well, but currently it lacks some features that would be essential in our case, where we could help out. I am not sure if any thought went into gateway persistence specifics already, and this feature could be implemented in fundamentally differnt ways, so I think the frist step could be to agree on the basics. First, in my opinion, persistence should be an optional feature of the gateway, that can be enabled if desired. There can be a lot of implementation details, but there can be some major directions to follow: - Utilize Hive catalog: The Hive catalog can already be used to have persistenct meta-objects, so the crucial thing that would be missing in this case is other catalogs. Personally, I would not pursue this option, because in my opinion it would limit the usability of this feature too much. - Serialize the session as is: Saving the whole session (or its context) [1] as is to durable storage, so it can be kept and picked up again. - Serialize the required elements (catalogs, tables, functions, etc.), not necessarily as a whole: The main point here would be to serialize a different object, so the persistent data will not be that sensitive to changes of the session (or its context). There can be numerous factors here, like try to keep the model close to the session itself, so the boilerplate required for the mapping can be kept to minimal, or focus on saving what is actually necessary, making the persistent storage more portable. WDYT? Cheers, F [1] https://github.com/apache/flink/blob/master/flink-table/flink-sql-gateway/src/main/java/org/apache/flink/table/gateway/service/session/Session.java
[jira] [Created] (FLINK-32174) Update Cloudera product and link in doc page
Ferenc Csaky created FLINK-32174: Summary: Update Cloudera product and link in doc page Key: FLINK-32174 URL: https://issues.apache.org/jira/browse/FLINK-32174 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31085) Add schema option to confluent registry avro formats
Ferenc Csaky created FLINK-31085: Summary: Add schema option to confluent registry avro formats Key: FLINK-31085 URL: https://issues.apache.org/jira/browse/FLINK-31085 Project: Flink Issue Type: Improvement Reporter: Ferenc Csaky Fix For: 1.17.0 When using {{avro-confluent}} and {{debezium-avro-confluent}} formats with schemas already defined in the Confluent Schema Registry, serialization fails, because Flink uses a default name `record` when converting row types to avro schema. So if the predefined schema has a different name, the serialization schema will be incompatible with the registered schema due to name mismatch. Check [this|https://lists.apache.org/thread/5xppmnqjqwfzxqo4gvd3lzz8wzs566zp] thread about reproducing the issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Release flink-connector-hbase v3.0.0, release candidate #1
Hi! According to the last comment on the e2e test PR [1], it would be okay to release the connector in its current format and deliver a new/rewritten test later. Every other issue should be resolved. [1] https://github.com/apache/flink-connector-hbase/pull/5 --- Original Message --- On Friday, December 9th, 2022 at 08:13, Martijn Visser wrote: > > > Thanks all for the check. This RC is cancelled and I'll create a new one > when the fixes are done. > > On Thu, Dec 8, 2022 at 1:22 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Heck, thank you Chesnay for catching this. I spent some time on removing > > the 1.4 hbase-client from flink-connector-hbase-2.2, cause excluding it > > from flink-connector-hbase-base seemed not working. I missed removing that > > exclusion from the 2.2 sql connector pom. Opened a PR [1] with the fix. > > > > [1] https://github.com/apache/flink-connector-hbase/pull/4 > > > > --- Original Message --- > > On Thursday, December 8th, 2022 at 11:15, Chesnay Schepler < > > ches...@apache.org> wrote: > > > > > -1 > > > > > > The packaging of the sql-connector-connector-hbase-2.2 module has > > > changed significantly, no longer bundling hbase-client (it now only > > > bundles Flink classes). > > > > > > On 05/12/2022 13:24, Ferenc Csaky wrote: > > > > > > > Hi Martijn, > > > > > > > > +1 (non-binding) > > > > > > > > - Verified hashes/signatures > > > > - Maven repo content LGTM > > > > - No binaries in the source archive > > > > - Built source/tests pass > > > > - Tag exists in GH > > > > - Reviewed web PR > > > > > > > > Thanks, > > > > F > > > > > > > > --- Original Message --- > > > > On Friday, December 2nd, 2022 at 14:04, Martijn Visser > > > > martijnvis...@apache.org wrote: > > > > > > > > > Hi everyone, > > > > > Please review and vote on the release candidate #1 for the > > > > > flink-connector-hbase version v3.0.0, as follows: > > > > > [ ] +1, Approve the release > > > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > > > > > Note: This is the first externalized version of the HBase connector. > > > > > > > > > > The complete staging area is available for your review, which > > > > > includes: > > > > > * JIRA release notes [1], > > > > > * the official Apache source release to be deployed to > > > > > dist.apache.org [2], > > > > > which are signed with the key with fingerprint > > > > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3], > > > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > > > * source code tag v3.0.0-rc1 [5], > > > > > * website pull request listing the new release [6]. > > > > > > > > > > The vote will be open for at least 72 hours. It is adopted by > > > > > majority > > > > > approval, with at least 3 PMC affirmative votes. > > > > > > > > > > Thanks, > > > > > Martijn > > > > > > > > > > https://twitter.com/MartijnVisser82 > > > > > https://github.com/MartijnVisser > > > > > > > > > > [1] > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352578 > > > > > > > [2] > > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-hbase-3.0.0-rc1/ > > > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > > > [4] > > > > > https://repository.apache.org/content/repositories/orgapacheflink-1555/ > > > > > [5] > > > > > https://github.com/apache/flink-connector-hbase/releases/tag/v3.0.0-rc1 > > > > > [6] https://github.com/apache/flink-web/pull/591
Re: [VOTE] Release flink-connector-hbase v3.0.0, release candidate #1
Heck, thank you Chesnay for catching this. I spent some time on removing the 1.4 hbase-client from flink-connector-hbase-2.2, cause excluding it from flink-connector-hbase-base seemed not working. I missed removing that exclusion from the 2.2 sql connector pom. Opened a PR [1] with the fix. [1] https://github.com/apache/flink-connector-hbase/pull/4 --- Original Message --- On Thursday, December 8th, 2022 at 11:15, Chesnay Schepler wrote: > > > -1 > > The packaging of the sql-connector-connector-hbase-2.2 module has > changed significantly, no longer bundling hbase-client (it now only > bundles Flink classes). > > On 05/12/2022 13:24, Ferenc Csaky wrote: > > > Hi Martijn, > > > > +1 (non-binding) > > > > - Verified hashes/signatures > > - Maven repo content LGTM > > - No binaries in the source archive > > - Built source/tests pass > > - Tag exists in GH > > - Reviewed web PR > > > > Thanks, > > F > > > > --- Original Message --- > > On Friday, December 2nd, 2022 at 14:04, Martijn Visser > > martijnvis...@apache.org wrote: > > > > > Hi everyone, > > > Please review and vote on the release candidate #1 for the > > > flink-connector-hbase version v3.0.0, as follows: > > > [ ] +1, Approve the release > > > [ ] -1, Do not approve the release (please provide specific comments) > > > > > > Note: This is the first externalized version of the HBase connector. > > > > > > The complete staging area is available for your review, which includes: > > > * JIRA release notes [1], > > > * the official Apache source release to be deployed to dist.apache.org > > > [2], > > > which are signed with the key with fingerprint > > > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3], > > > * all artifacts to be deployed to the Maven Central Repository [4], > > > * source code tag v3.0.0-rc1 [5], > > > * website pull request listing the new release [6]. > > > > > > The vote will be open for at least 72 hours. It is adopted by majority > > > approval, with at least 3 PMC affirmative votes. > > > > > > Thanks, > > > Martijn > > > > > > https://twitter.com/MartijnVisser82 > > > https://github.com/MartijnVisser > > > > > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352578 > > > [2] > > > https://dist.apache.org/repos/dist/dev/flink/flink-connector-hbase-3.0.0-rc1/ > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > > [4] > > > https://repository.apache.org/content/repositories/orgapacheflink-1555/ > > > [5] > > > https://github.com/apache/flink-connector-hbase/releases/tag/v3.0.0-rc1 > > > [6] https://github.com/apache/flink-web/pull/591 > >
Re: [VOTE] Release flink-connector-hbase v3.0.0, release candidate #1
Hi Martijn, +1 (non-binding) - Verified hashes/signatures - Maven repo content LGTM - No binaries in the source archive - Built source/tests pass - Tag exists in GH - Reviewed web PR Thanks, F --- Original Message --- On Friday, December 2nd, 2022 at 14:04, Martijn Visser wrote: > > > Hi everyone, > Please review and vote on the release candidate #1 for the > flink-connector-hbase version v3.0.0, as follows: > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > Note: This is the first externalized version of the HBase connector. > > The complete staging area is available for your review, which includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to dist.apache.org [2], > which are signed with the key with fingerprint > A5F3BCE4CBE993573EC5966A65321B8382B219AF [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag v3.0.0-rc1 [5], > * website pull request listing the new release [6]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Martijn > > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser > > [1] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352578 > [2] > https://dist.apache.org/repos/dist/dev/flink/flink-connector-hbase-3.0.0-rc1/ > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] https://repository.apache.org/content/repositories/orgapacheflink-1555/ > [5] https://github.com/apache/flink-connector-hbase/releases/tag/v3.0.0-rc1 > [6] https://github.com/apache/flink-web/pull/591
Re: [DISCUSS] assign SQL Table properties from environment variables
Hello devs, I'd like to revive this discussion. There is also a ticket about this effort for some time [1] and this thing also affects us as well. Right now we have a custom solution that is similar to "environment variables", but it only can be used in parts of our downstream product. The main thing for us to achieve would be to be able to use variables in DDLs (not necessarily for hiding sensitive props). I think it would be really handy to have the ability to reuse values in multiple tables. With that said, comes the temptation to hit two birds with one stone, although a sensitive property requires much more care than a regular one, so I think these two things should be handled separately. At least in the beginning. The tricky part of the "environment variables" are their scope, and if they are not coming from an external system, it will probably be necessary to persist them. Or keep them in memory, but that may be insufficient according to what is the scope of the "environment variables". Considering the sensitive props, I think a small step forward could be to hide the values in case of a "SHOW CREATE TABLE" op. For a varible to be used in a DDL I'd imagine it could apply for a whole catalog as starters. As long as the catalog is present, those variables would be valid. I did not check implementation details yet, so it is possible I'm missing something important or wrong in some places, but I wanted to get some feedback about the idea. WDYT? [1] https://issues.apache.org/jira/browse/FLINK-28028 Best, F --- Original Message --- On Monday, April 4th, 2022 at 09:53, Timo Walther wrote: > > > Hi Fred, > > thanks for starting this discussion. I totally agree that this an issue > that the community should solve. It popped up before and is still > unsolved today. Great that you offer your help here. So let's clarify > the implementation details. > > 1) Global vs. Local solution > > Is this a DDL-only problem? If yes, it would be easier to solve it in > the `FactoryUtil` that all Flink connectors and formats use. > > 2) Configruation vs. enviornment variables > > I agree with Qingsheng that environment variable are not always > straightforward to identify if you have a "pre-flight phase" and a > "cluster phase". > In the DynamicTableFactory, one has access to Flink configuration and > could resolve `${...}` variables. > > > What do you think? > > Regards, > Timo > > > Am 01.04.22 um 12:26 schrieb Qingsheng Ren: > > > Hi Fred, > > > > Thanks for raising the discussion! I think the definition of “environment > > variable” varies under different context. Under Flink on K8s it means the > > environment variable for a container, and if you are a SQL client user it > > could refer to environment variable of SQL client, or even the system > > properties on JVM. So using “environment variable” is a bit vague under > > different environments. > > > > A more generic solution in my mind is that we can take advantage of > > configurations in Flink, to pass table options dynamically by adding > > configs to TableConfig or even flink-conf.yaml. For example option > > “table.dynamic.options.my_catalog.my_db_.my_table.accessId = foo” means > > adding table option “accessId = foo” to table “my_catalog.my_db.my_table”. > > By this way we could de-couple DDL statement with table options containing > > secret credentials. What do you think? > > > > Best regards, > > > > Qingsheng > > > > > On Mar 30, 2022, at 16:25, Teunissen, F.G.J. (Fred) > > > fred.teunis...@ing.com.INVALID wrote: > > > > > > Hi devs, > > > > > > Some SQL Table properties contain sensitive data, like passwords that we > > > do not want to expose in the VVP ui to other users. Also, having them > > > clear text in a SQL statement is not secure. For example, > > > > > > CREATE TABLE Orders ( > > > `user` BIGINT, > > > product STRING, > > > order_time TIMESTAMP(3) > > > ) WITH ( > > > 'connector' = 'kafka', > > > > > > 'properties.bootstrap.servers' = 'kafka-host-1:9093,kafka-host-2:9093', > > > 'properties.security.protocol' = 'SSL', > > > 'properties.ssl.key.password' = 'should-be-a-secret', > > > 'properties.ssl.keystore.location' = '/tmp/secrets/my-keystore.jks', > > > 'properties.ssl.keystore.password' = 'should-also-be-a-secret', > > > 'properties.ssl.truststore.location' = '/tmp/secrets/my-truststore.jks', > > > 'properties.ssl.truststore.password' = 'should-again-be-a-secret', > > > 'scan.startup.mode' = 'earliest-offset' > > > ); > > > > > > I would like to bring up for a discussion a proposal to provide these > > > secrets values via environment variables since these can be populated > > > from a K8s configMap or secrets. > > > > > > For implementing the SQL Table properties, the ConfigOption class is > > > used in connectors and formatters. This class could be extended that it > > > checks whether the config-value contains certain tokens, like > > > ‘${env-var-name}’. If
Re: [DISCUSS] Retroactively externalize some connectors for 1.16
Hi! I think this would be a good idea. I was wondering that could we include the hbase connector to this group as well? The externalization PR [1] should be in a good shape now and Dec 9th as a release date sounds doable. WDYT? [1] https://github.com/apache/flink-connector-hbase/pull/2 Best, F --- Original Message --- On Thursday, December 1st, 2022 at 16:01, Chesnay Schepler wrote: > > > Hello, > > let me clarify the title first. > > In the original proposal for the connector externalization we said that > an externalized connector has to exist in parallel with the version > shipped in the main Flink release for 1 cycle. > > For example, 1.16.0 shipped with the elasticsearch connector, but at the > same time there's the externalized variant as a drop-in replacement, and > the 1.17.0 release will not include a ES connector. > > The rational was to give users some window to update their projects. > > > We are now about to externalize a few more connectors (cassandra, > pulsar, jdbc), targeting 1.16 within the next week. > The 1.16.0 release has now been about a month ago; so it hasn't been a > lot of time since then. > I'm now wondering if we could/should treat these connectors as > externalized for 1.16, meaning that we would remove them from the master > branch now, not ship them in 1.17 and move all further development into > the connector repos. > > The main benefit is that we won't have to bother with syncing changes > across repos all the time. > > We would of course need some sort-of cutoff date for this (December > 9th?), to ensure there's still some reasonably large gap left for users > to migrate. > > Let me know what you think. > > Regards, > Chesnay
Re: [VOTE] FLIP-271: Autoscaling
+1 (non-binding) --- Original Message --- On Tuesday, November 29th, 2022 at 15:39, Márton Balassi wrote: > > > +1 (binding) > > On Tue, Nov 29, 2022 at 6:13 AM Chenya Zhang chenyazhangche...@gmail.com > > wrote: > > > +1 (non-binding) > > > > On Sun, Nov 27, 2022 at 5:49 PM Jiangang Liu liujiangangp...@gmail.com > > wrote: > > > > > +1 (non-binding) > > > > > > Best, > > > Jiangang Liu > > > > > > Thomas Weise t...@apache.org 于2022年11月28日周一 06:23写道: > > > > > > > +1 (binding) > > > > > > > > On Sat, Nov 26, 2022 at 8:11 AM Zheng Yu Chen jam.gz...@gmail.com > > > > wrote: > > > > > > > > > +1(no-binding) > > > > > > > > > > Maximilian Michels m...@apache.org 于 2022年11月24日周四 上午12:25写道: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > I'd like to start a vote for FLIP-271 [1] which we previously > > > > > > discussed > > > > > > on > > > > > > the dev mailing list [2]. > > > > > > > > > > > > I'm planning to keep the vote open for at least until Tuesday, Nov > > > > > > 29. > > > > > > > > > > > > -Max > > > > > > > > > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling > > > > > > > > [2] > > > > > > https://lists.apache.org/thread/pvfb3fw99mj8r1x8zzyxgvk4dcppwssz
Re: [VOTE] FLIP-272: Generalized delegation token support
+1 (non-binding) Best, F --- Original Message --- On Friday, November 18th, 2022 at 16:11, Márton Balassi wrote: > > > +1 (binding) > > On Thu, Nov 17, 2022 at 9:06 AM Gabor Somogyi gabor.g.somo...@gmail.com > > wrote: > > > Hi All, > > > > I'm hereby opening a vote for FLIP-272 Generalized delegation token > > support. > > The related documents can be found here: > > - FLIP on wiki: [1] > > - Discussion thread: [2] > > > > Voting will be open for at least 72 hours (since weekend is involved EOB > > Monday is planned). > > > > BR, > > G > > > > [1] > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support > > [2] https://lists.apache.org/thread/vgg5hbf5jljcxopfhb32w3l0wjoyko4o
Re: [ANNOUNCE] New Apache Flink Committer - Matyas Orhidi
Congrats Matyas! Best, F --- Original Message --- On Monday, November 21st, 2022 at 15:17, Márton Balassi wrote: > > > Hi everyone, > > On behalf of the PMC, I'm very happy to announce Matyas Orhidi as a new > Flink > committer. > > Matyas has over a decade of experience of the Big Data ecosystem and has > been working with Flink full time for the past 3 years. In the open source > community he is one of the key driving members of the Kubernetes Operator > subproject. He implemented multiple key features in the operator including > the metrics system and the ability to dynamically configure watched > namespaces. He enjoys spreading the word about Flink and regularly does so > via authoring blogposts and giving talks or interviews representing the > community. > > Please join me in congratulating Matyas for becoming a Flink committer! > > Best, > Marton
Re: [DISCUSS] Updating Flink HBase Connectors
Hi! Sure, thank you both for the input. If the repo is ready, I'll start the work. Best, F --- Original Message --- On Thursday, November 17th, 2022 at 09:39, Gabor Somogyi wrote: > > > Hi All, > > +1 to go. Since we are refurbishing the HBase area maybe we can move the > token provider into HBase base project. > This would fit into the high level effort to extract everything into > external connectors. If you do this and facing any issues just ping me :) > > BR, > G > > > On Thu, Nov 17, 2022 at 9:29 AM Martijn Visser martijnvis...@apache.org > > wrote: > > > Hi Ferenc, > > > > I think you're good to go, since no comments were there. Do let us know if > > you need any help :) > > > > Thanks, > > > > Martijn > > > > On Mon, Oct 24, 2022 at 7:47 PM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > > > Hi, > > > > > > just pinging this thread in case someone missed it and has any opinion > > > about the discussed actions. > > > > > > Best, > > > F > > > > > > --- Original Message --- > > > On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky > > > ferenc.cs...@pm.me.INVALID wrote: > > > > > > > Hi Martijn, > > > > > > > > Thank you for your comment. About HBase 2.x, correct, that is my > > > > thought > > > > process, but it has to be tested and verified. > > > > > > > > +1 from my side about merging these updates with the connector > > > > externalization. > > > > > > > > Best, > > > > F > > > > > > > > --- Original Message --- > > > > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser > > > > martijnvis...@apache.org wrote: > > > > > > > > > Hi Ferenc, > > > > > > > > > > Thanks for opening the discussion on this topic! > > > > > > > > > > +1 for dropping HBase 1.x. > > > > > > > > > > Regarding HBase 2.x, if I understand correctly it should be possible > > > > > to > > > > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't > > > > > it > > > > > make > > > > > more sense to always support the latest available version, so > > > > > basically 2.5 > > > > > at the moment? We could always include a test to check that > > > > > implementation > > > > > against an older HBase version. > > > > > > > > > > I also have a follow-up question already: if there's an agreement on > > > > > this > > > > > topic, does it make sense to directly build a new HBase connector in > > > > > its > > > > > own external connector repo, since I believe the current connector > > > > > uses the > > > > > old source/sink interfaces. We could then directly drop the ones in > > > > > the > > > > > Flink repo and replace it with new implementations? > > > > > > > > > > Best regards, > > > > > > > > > > Martijn > > > > > > > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Now that the connector externalization effort ig going on, I think > > > > > > it is > > > > > > definitely work to revisit the currently supported HBase versions > > > > > > for the > > > > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 > > > > > > connector > > > > > > versions, although both of those versions are kind of outdated. > > > > > > > > > > > > From the HBase point of view the following can be considered [1]: > > > > > > > > > > > > - HBase 1.x is dead, so on the way forward it should be safe to > > > > > > drop > > > > > > it. > > > > > > - HBase 2.2 is EoL, but still used actively, we are also supporting > > > > > > it ins > > > > > > some of our still active releases as Cloudera. > > > > > > - HBase 2.4 is the main thing now and probably will be supported > > > > > > for > > > > > > a > > > > > > while (by us, definitely). > > > > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it > > > > > > is > > > > > > possible it won't live long. > > > > > > - HBase 3 is in alpha, but shooting for that probably would be > > > > > > early > > > > > > in > > > > > > the near future. > > > > > > > > > > > > In addition, if we are only using the standard HBase 2.x client > > > > > > APIs, then > > > > > > it should be possible to be compile it with any Hbase 2.x version. > > > > > > Also, > > > > > > any HBase 2.x cluster should be backwards compatible with all > > > > > > earlier HBase > > > > > > 2.x client libraries. I did not check this part thorougly but I > > > > > > think this > > > > > > should be true, so ideally it would be enough to have an HBase 2.4 > > > > > > connector. [2] > > > > > > > > > > > > Looking forward to your opinions about this topic. > > > > > > > > > > > > Best, > > > > > > F > > > > > > > > > > > > [1] https://hbase.apache.org/downloads.html > > > > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10 > > > > > > (Client > > > > > > API compatibility) > > > > > > > > > > -- > > > > > Martijn > > > > > https://twitter.com/MartijnVisser82 > > > > > https://github.com/MartijnVisser
Re: [DISCUSS] Updating Flink HBase Connectors
Hi, just pinging this thread in case someone missed it and has any opinion about the discussed actions. Best, F --- Original Message --- On Tuesday, October 11th, 2022 at 23:29, Ferenc Csaky wrote: > > > Hi Martijn, > > Thank you for your comment. About HBase 2.x, correct, that is my thought > process, but it has to be tested and verified. > > +1 from my side about merging these updates with the connector > externalization. > > Best, > F > > > --- Original Message --- > On Tuesday, October 11th, 2022 at 16:30, Martijn Visser > martijnvis...@apache.org wrote: > > > > > Hi Ferenc, > > > > Thanks for opening the discussion on this topic! > > > > +1 for dropping HBase 1.x. > > > > Regarding HBase 2.x, if I understand correctly it should be possible to > > connect to any 2.x cluster if you're using the 2.x client. Wouldn't it make > > more sense to always support the latest available version, so basically 2.5 > > at the moment? We could always include a test to check that implementation > > against an older HBase version. > > > > I also have a follow-up question already: if there's an agreement on this > > topic, does it make sense to directly build a new HBase connector in its > > own external connector repo, since I believe the current connector uses the > > old source/sink interfaces. We could then directly drop the ones in the > > Flink repo and replace it with new implementations? > > > > Best regards, > > > > Martijn > > > > Op ma 10 okt. 2022 om 16:24 schreef Ferenc Csaky > > > > Hi everyone, > > > > > > Now that the connector externalization effort ig going on, I think it is > > > definitely work to revisit the currently supported HBase versions for the > > > Flink connector. Currently, ther is an HBase 1.4 and HBase 2.2 connector > > > versions, although both of those versions are kind of outdated. > > > > > > From the HBase point of view the following can be considered [1]: > > > > > > - HBase 1.x is dead, so on the way forward it should be safe to drop it. > > > - HBase 2.2 is EoL, but still used actively, we are also supporting it ins > > > some of our still active releases as Cloudera. > > > - HBase 2.4 is the main thing now and probably will be supported for a > > > while (by us, definitely). > > > - HBase 2.5 just came out, but 2.6 is expected pretty soon, so it is > > > possible it won't live long. > > > - HBase 3 is in alpha, but shooting for that probably would be early in > > > the near future. > > > > > > In addition, if we are only using the standard HBase 2.x client APIs, then > > > it should be possible to be compile it with any Hbase 2.x version. Also, > > > any HBase 2.x cluster should be backwards compatible with all earlier > > > HBase > > > 2.x client libraries. I did not check this part thorougly but I think this > > > should be true, so ideally it would be enough to have an HBase 2.4 > > > connector. [2] > > > > > > Looking forward to your opinions about this topic. > > > > > > Best, > > > F > > > > > > [1] https://hbase.apache.org/downloads.html > > > [2] https://hbase.apache.org/book.html#hbase.versioning.post10 (Client > > > API compatibility) > > > > -- > > Martijn > > https://twitter.com/MartijnVisser82 > > https://github.com/MartijnVisser
[jira] [Created] (FLINK-29707) Fix possible comparator violation for "flink list"
Ferenc Csaky created FLINK-29707: Summary: Fix possible comparator violation for "flink list" Key: FLINK-29707 URL: https://issues.apache.org/jira/browse/FLINK-29707 Project: Flink Issue Type: Bug Components: Command Line Client Affects Versions: 1.16.0 Reporter: Ferenc Csaky For the {{list}} CLI option, the code that prints the jobs, there is a {{startTimeComparator}} definition, which orders the jobs and it is done this way: {code:java} Comparator startTimeComparator = (o1, o2) -> (int) (o1.getStartTime() - o2.getStartTime()); {code} In some rare situation this can lead to this: {code:java} 2022-10-19 09:58:11,690 ERROR org.apache.flink.client.cli.CliFrontend [] - Error while running the command. java.lang.IllegalArgumentException: Comparison method violates its general contract! at java.util.TimSort.mergeLo(TimSort.java:777) ~[?:1.8.0_312] at java.util.TimSort.mergeAt(TimSort.java:514) ~[?:1.8.0_312] at java.util.TimSort.mergeForceCollapse(TimSort.java:457) ~[?:1.8.0_312] at java.util.TimSort.sort(TimSort.java:254) ~[?:1.8.0_312] at java.util.Arrays.sort(Arrays.java:1512) ~[?:1.8.0_312] at java.util.ArrayList.sort(ArrayList.java:1464) ~[?:1.8.0_312] at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:392) ~[?:1.8.0_312] at java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[?:1.8.0_312] at java.util.stream.Sink$ChainedReference.end(Sink.java:258) ~[?:1.8.0_312] at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:363) ~[?:1.8.0_312] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483) ~[?:1.8.0_312] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[?:1.8.0_312] at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:1.8.0_312] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:1.8.0_312] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_312] at java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:490) ~[?:1.8.0_312] at org.apache.flink.client.cli.CliFrontend.printJobStatusMessages(CliFrontend.java:574) {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] Externalized connector release details
+1 from my side (non-binding) Best, F --- Original Message --- On Wednesday, October 12th, 2022 at 15:47, Martijn Visser wrote: > > > +1 (binding), I am indeed assuming that Chesnay meant the last two minor > versions as supported. > > Op wo 12 okt. 2022 om 20:18 schreef Danny Cranmer dannycran...@apache.org > > > Thanks for the concise summary Chesnay. > > > > +1 from me (binding) > > > > Just one clarification, for "3.1) The Flink versions supported by the > > project (last 2 major Flink versions) must be supported.". Do we actually > > mean major here, as in Flink 1.x.x and 2.x.x? Right now we would only > > support Flink 1.15.x and not 1.14.x? I would be inclined to support the > > latest 2 minor Flink versions (major.minor.patch) given that we only have 1 > > active major Flink version. > > > > Danny > > > > On Wed, Oct 12, 2022 at 2:12 PM Chesnay Schepler ches...@apache.org > > wrote: > > > > > Since the discussion > > > (https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo) has > > > stalled a bit but we need a conclusion to move forward I'm opening a > > > vote. > > > > > > Proposal summary: > > > > > > 1) Branch model > > > 1.1) The default branch is called "main" and used for the next major > > > iteration. > > > 1.2) Remaining branches are called "vmajor.minor". (e.g., v3.2) > > > 1.3) Branches are not specific to a Flink version. (i.e., no v3.2-1.15) > > > > > > 2) Versioning > > > 2.1) Source releases: major.minor.patch > > > 2.2) Jar artifacts: major.minor.match-flink-major.flink-minor > > > (This may imply releasing the exact same connector jar multiple times > > > under different versions) > > > > > > 3) Flink compatibility > > > 3.1) The Flink versions supported by the project (last 2 major Flink > > > versions) must be supported. > > > 3.2) How this is achived is left to the connector, as long as it > > > conforms to the rest of the proposal. > > > > > > 4) Support > > > 4.1) The last 2 major connector releases are supported with only the > > > latter receiving additional features, with the following exceptions: > > > 4.1.a) If the older major connector version does not support any > > > currently supported Flink version, then it is no longer supported. > > > 4.1.b) If the last 2 major versions do not cover all supported Flink > > > versions, then the latest connector version that supports the older > > > Flink version /additionally /gets patch support. > > > 4.2) For a given major connector version only the latest minor version > > > is supported. > > > (This means if 1.1.x is released there will be no more 1.0.x release) > > > > > > I'd like to clarify that these won't be set in stone for eternity. > > > We should re-evaluate how well this model works over time and adjust it > > > accordingly, consistently across all connectors. > > > I do believe that as is this strikes a good balance between > > > maintainability for us and clarity to users. > > > > > > Voting schema: > > > > > > Consensus, committers have binding votes, open for at least 72 hours. > > -- > Martijn > https://twitter.com/MartijnVisser82 > https://github.com/MartijnVisser