Re: [Discuss] FLIP-362: Support minimum resource limitation

Thanks for driving this @Xiangyu. This is a feature that many users have requested for a long time. +1 for the overall proposal. Best, Yangze Guo On Tue, Sep 19, 2023 at 11:48 AM xiangyu feng wrote: > > Hi Devs, > > I'm opening this thread to discuss FLIP-362: Support minimum resource >

Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

+1, thanks everyone who discussed here. Best, Rui On Tue, 19 Sep 2023 at 11:41, Chen Zhanghao wrote: > Hi Jing, > > Thanks for the clarification, I now see the point. +1 for using endpoint > now. @fan...@apache.org WDYT? > > Best, > Zhanghao Chen > -- > *发件人:*

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

Thanks for the reply, @Gyula. I would like to first provide more context on OLAP scenarios. In OLAP scenarios, users typically submit multiple short batch jobs that have execution times typically measured in seconds or even sub-seconds. Additionally, due to the lightweight nature of these jobs,

[jira] [Created] (FLINK-33112) Support placement constraint

Junfan Zhang created FLINK-33112: Summary: Support placement constraint Key: FLINK-33112 URL: https://issues.apache.org/jira/browse/FLINK-33112 Project: Flink Issue Type: New Feature

[Discuss] FLIP-362: Support minimum resource limitation

Hi Devs, I'm opening this thread to discuss FLIP-362: Support minimum resource limitation. The design doc can be found at: FLIP-362: Support minimum resource limitation Currently, the Flink

回复: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

Hi Jing, Thanks for the clarification, I now see the point. +1 for using endpoint now. @fan...@apache.org WDYT? Best, Zhanghao Chen 发件人: Yangze Guo 发送时间: 2023年9月19日 11:18 收件人: dev@flink.apache.org 主题: Re: [DISCUSS] FLIP-363: Unify the

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

Hi Jark, Thanks for the reply. Please see my comments inline. On Tue, Sep 19, 2023 at 10:12 AM Jark Wu wrote: > Hi Dong, > > Sorry for the late reply. > > > The rationale is that if there is any strategy that is triggered and says > > backlog=true, then job's backlog should be true. Otherwise,

Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

Thanks for the clarification, Jing. I agree that using another term like endpoint can help us to distinguish it from the existing concept of "location". +1 for using the term endpoint and introducing TaskManagerLocation.getEndpoint(). Best, Yangze Guo On Mon, Sep 18, 2023 at 11:52 PM Jing Ge

[jira] [Created] (FLINK-33111) Flink Pulsar Connector to Pulsar Client Version Mismatch

Jason Kania created FLINK-33111: --- Summary: Flink Pulsar Connector to Pulsar Client Version Mismatch Key: FLINK-33111 URL: https://issues.apache.org/jira/browse/FLINK-33111 Project: Flink Issue

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

Hi Dong, Sorry for the late reply. > The rationale is that if there is any strategy that is triggered and says > backlog=true, then job's backlog should be true. Otherwise, the job's > backlog status is false. I'm quite confused about this. Does that mean, if the source is in the changelog

Re: [DISCUSS] Implementing SQL remote functions

Hi, Alan Thanks for driving this proposal. It sounds interesting. Regarding implementing the Remote Function, can you go into more detail about your idea, how we should support it, and how users should use it, from API design to semantic explanation？and how does the remote function help to solve

Re: [DISCUSS] FLIP-328: Allow source operators to determine isProcessingBacklog based on watermark lag

Hi Jark, Do you have time to comment on whether the current design looks good? I plan to start voting in 3 days if there is no follow-up comment. Thanks, Dong On Fri, Sep 15, 2023 at 2:01 PM Jark Wu wrote: > Hi Dong, > > > Note that we can not simply enforce the semantics of "any invocation

[DISCUSS] Implementing SQL remote functions

Hello all, We want to implement a custom function that sends HTTP requests to a remote endpoint using Flink SQL. Even though the function will behave like a normal UDF, the runtime would issue calls asynchronously to achieve high throughput for these remote (potentially high latency) calls. What

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

Hi Danny this is a good idea. We can do it easily. I am amending the FLIP accordingly Regards Lorenzo On Mon, 18 Sept 2023 at 20:17, Danny Cranmer wrote: > Hello Lorenzo, > > > Please consider this is not an AWS-specific connector and will not depend > on flink-connector-aws-base > > This is

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

Hello Lorenzo, > Please consider this is not an AWS-specific connector and will not depend on flink-connector-aws-base This is my point, how can we introduce AWS specific functionality without coupling them. How about this. The Prometheus connector does not depend on aws-base but publishes a

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

Hi Danny Please consider this is not an AWS-specific connector and will not depend on flink-connector-aws-base. Prometheus remote-write specs deem authentication as out-of-scope [1]. Amazon Managed Prometheus uses request signing to authenticate remote-writes. To support AMP but also keep the

Re: [VOTE] FLIP-361: Improve GC Metrics

looks great, thanks! +1(binding) Best regards, Jing On Mon, Sep 18, 2023 at 8:11 AM Yuepeng Pan wrote: > +1 (non-binding). > > > Best, > Yuepeng. > > > > > > 在 2023-09-16 12:52:09，"Weihua Hu" 写道： > >+1 (binding) > > > >Best, > >Weihua > > > > > >On Fri, Sep 15, 2023 at 4:28 PM Yangze Guo

Re: [VOTE] FLIP-307: Flink Connector Redshift

+1 (non-binding) Best Regards Ahmed Hamdy On Mon, 18 Sept 2023 at 16:52, Jing Ge wrote: > +1(binding). Thanks! > > Best regards, > Jing > > On Mon, Sep 18, 2023 at 5:26 PM Danny Cranmer > wrote: > > > Thanks for driving this Samrat! > > > > +1 (binding) > > > > Thanks, > > Danny > > > > On

Re: [VOTE] FLIP-307: Flink Connector Redshift

+1(binding). Thanks! Best regards, Jing On Mon, Sep 18, 2023 at 5:26 PM Danny Cranmer wrote: > Thanks for driving this Samrat! > > +1 (binding) > > Thanks, > Danny > > On Mon, Sep 18, 2023 at 4:17 AM Samrat Deb wrote: > > > Hi All, > > > > Thanks for all the feedback on FLIP-307: Flink

Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

Hi Zhanghao, That is exactly the reason why location should not be used, because there is a clear definition of location in Flink, e.g. TaskManagerLocation which contains more information than hostname+port. If you think endpoint is too generic, how about locationEndpoint? But if we build that

[jira] [Created] (FLINK-33110) Array content gets replaced with last element duplicates

Roman Lupiichuk created FLINK-33110: --- Summary: Array content gets replaced with last element duplicates Key: FLINK-33110 URL: https://issues.apache.org/jira/browse/FLINK-33110 Project: Flink

Re: [VOTE] FLIP-307: Flink Connector Redshift

Thanks for driving this Samrat! +1 (binding) Thanks, Danny On Mon, Sep 18, 2023 at 4:17 AM Samrat Deb wrote: > Hi All, > > Thanks for all the feedback on FLIP-307: Flink Connector Redshift [1][2] > > I'd like to start a vote for FLIP-307. The vote will be open for at least > 72 > hours unless

Re: [DISCUSS] FLIP-307: Flink connector Redshift

Thanks Samrat, this makes sense to me and addresses connector coupling concerns. On Sat, Sep 16, 2023 at 4:30 PM Samrat Deb wrote: > Hi , > > I've made updates to the FLIP[1] by incorporating relevant changes to avoid > using the Flink connector JDBC. This decision was based on the following >

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

Thanks for the reply Lorenzo. > Static credentials are just for the sake of the example. The current prototype implementation already uses DefaultAWSCredentialsProviderChain that supports static and dynamic credentials. We can make the credential provider configurable. The point here was that

Re: [DISCUSS] FLIP-229: Prometheus Sink Connector

Hello I am also re-sending an old answer I sent on May 24th, that, for some reason, did not appear in the thread. -- Q1) The fact we are using the remote write feature is not covered beyond the code example. Can we add details on this to make it clear? Additionally would this

Re: [VOTE] Apache Flink Stateful Functions Release 3.3.0, release candidate #2

+1 (binding) - Contents of dist matches 3.2.0 - Checksum/sigs of dist and maven repo match - Source archive does not contain any binaries - Reviewed the Docker PR - Reviewed website PR - Release notes look good - Tag is present in Github - Maven build+tests complete successfully - Verfied NOTICE

Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

Hi Yangze, Thanks for your reply! For a subtask, the taskManagerLocation is the physical location. And the task id, operator id and subtask index may be the logical locations of a subtask. Actually, I don't think the taskManagerLocation is a good name here. Unlike task id, operator id, and

Re: [DISCUSS] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

Hi, Rui, I think the term "location" might be sufficiently clear in the specific context, e.g. SubtaskXXXInfo and TaskManagerXXXInfo. Could you elaborate more on what concept you think one might confuse it with? Best, Yangze Guo On Mon, Sep 18, 2023 at 12:07 PM Rui Fan <1996fan...@gmail.com>

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Thanks Shammon for the informations, the comment makes the lifecycle clearer. +1 Best, Leonard > On Sep 18, 2023, at 7:54 PM, Shammon FY wrote: > > Hi devs, > > After discussing with @Qingsheng, I fixed a minor issue of the lineage > lifecycle in `StreamExecutionEnvironment`. I have

[jira] [Created] (FLINK-33109) Watermark alignment not applied after recovery from checkpoint

Yordan Pavlov created FLINK-33109: - Summary: Watermark alignment not applied after recovery from checkpoint Key: FLINK-33109 URL: https://issues.apache.org/jira/browse/FLINK-33109 Project: Flink

Re: Future of classical remoting in Pekko

I think that the end of September is too soon for a Pekko 1.1.x, there are still more things that we would like to merge before making a release. Good news is that the PR to migrate to netty4 for classic remoting has been merged (see https://github.com/apache/incubator-pekko/pull/643).

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

Hi devs, After discussing with @Qingsheng, I fixed a minor issue of the lineage lifecycle in `StreamExecutionEnvironment`. I have added the comment to explain that the lineage information in `StreamExecutionEnvironment` will be consistent with that of transformations. When users clear the

[VOTE] FLIP-312: Prometheus Sink Connector

Hi All, Thanks for the feedback on FLIP-312: Prometheus Sink Connector [1]. We updated the FLIP accordingly [2]. I would like to open the vote on FLIP-312. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1]

Re: [VOTE] Apache Flink Stateful Functions Release 3.3.0, release candidate #2

+1 (binding) - Validated hashes - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven - Verified licenses - Verified web PRs Still looking for vote nr 3 :) On Thu, Sep 14, 2023 at 10:31 PM Robert Metzger wrote: > I did a shallow pass over

回复: [VOTE] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

Thanks for pointing that out. Let's give it a bit more time for reaching consensus on the naming issue and postpone the voting for now. Sorry for the inconvenience here. Will send another email once the voting restarts. Best, Zhanghao Chen 发件人: Rui Fan

[ANNOUNCE] Release 1.18.0, release candidate #0

Hi everyone, The RC0 for Apache Flink 1.18.0 has been created. This RC is currently for preview only to facilitate the integrated testing since the benchmark tests are not available yet[1] and the release announcement is still under review. The RC1 will be released after all benchmarks tests are

RE: [DISCUSS]clean up the savepoints compatibility table

Hi, I like the proposal from Martijn. Having an ‘archived’ area for the older material would make sense so users on the older versions can still access the content for upgrade, keeping maybe the latest 3 releases in active table, Kind regards, David. From: Martijn Visser Date: Friday, 15

[jira] [Created] (FLINK-33108) Error during error status handling

Gabor Somogyi created FLINK-33108: - Summary: Error during error status handling Key: FLINK-33108 URL: https://issues.apache.org/jira/browse/FLINK-33108 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-33107) Update stable-spec upgrade mode on reconciled-spec change

Gyula Fora created FLINK-33107: -- Summary: Update stable-spec upgrade mode on reconciled-spec change Key: FLINK-33107 URL: https://issues.apache.org/jira/browse/FLINK-33107 Project: Flink Issue

[jira] [Created] (FLINK-33106) pyflink脚本通过python命令运行正常，但部署至Flink（单机）运行时报错

chenwei created FLINK-33106: --- Summary: pyflink脚本通过python命令运行正常，但部署至Flink（单机）运行时报错 Key: FLINK-33106 URL: https://issues.apache.org/jira/browse/FLINK-33106 Project: Flink Issue Type: Bug

Re: [Discuss] CRD for flink sql gateway in the flink k8s operator

As I wrote in my previous answer, this could be done as a helm chart or as part of the operator easily. Both would work. My main concern for adding this into the operator is that the SQL Gateway itself is not properly integrated with the Operator Custom resources. Gyula On Mon, Sep 18, 2023 at

[jira] [Created] (FLINK-33105) Evaluate operator log entry: Exception while fetching cluster info

Gabor Somogyi created FLINK-33105: - Summary: Evaluate operator log entry: Exception while fetching cluster info Key: FLINK-33105 URL: https://issues.apache.org/jira/browse/FLINK-33105 Project: Flink

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

Thanks for addressing my comments. LGTM Best, Xintong On Mon, Sep 18, 2023 at 3:10 PM Dong Lin wrote: > Hi Xintong, > > Thank you for all the comments. Please see my reply inline. > > > On Mon, Sep 18, 2023 at 11:31 AM Xintong Song > wrote: > > > Thanks for addressing my comments, Dong. >

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

Hi Xintong, Thank you for all the comments. Please see my reply inline. On Mon, Sep 18, 2023 at 11:31 AM Xintong Song wrote: > Thanks for addressing my comments, Dong. > > The expected behavior of checkpointing and failover depends on whether > > there is any operator currently running in the

Re: Re: [DISCUSS] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

Hi, Dong, Thanks for your efforts. +1 to this proposal, I believe this will improve the performance in some mixture circumstances of bounded and unbounded workloads. Best, Yuxin Xintong Song 于2023年9月18日周一 10:56写道： > Thanks for addressing my comments, Dong. > > LGTM. > > Best, > > Xintong > >

[jira] [Created] (FLINK-33104) Nightly run for Flink Kafka connector fails

Martijn Visser created FLINK-33104: -- Summary: Nightly run for Flink Kafka connector fails Key: FLINK-33104 URL: https://issues.apache.org/jira/browse/FLINK-33104 Project: Flink Issue Type:

[jira] [Created] (FLINK-33103) Hybrid shuffle ITCase supports the new mode

Yuxin Tan created FLINK-33103: - Summary: Hybrid shuffle ITCase supports the new mode Key: FLINK-33103 URL: https://issues.apache.org/jira/browse/FLINK-33103 Project: Flink Issue Type:

Re: [VOTE] FLIP-361: Improve GC Metrics

+1 (non-binding). Best, Yuepeng. 在 2023-09-16 12:52:09，"Weihua Hu" 写道： >+1 (binding) > >Best, >Weihua > > >On Fri, Sep 15, 2023 at 4:28 PM Yangze Guo wrote: > >> +1 (binding) >> >> Best, >> Yangze Guo >> >> On Thu, Sep 14, 2023 at 5:47 PM Maximilian Michels wrote: >> > >> > +1 (binding)