Re: [DISCUSS] FLIP-469: Supports Adaptive Optimization of StreamGraph

One additional question: for a JobEvent, are all StreamGraphOptimizationStrategy traversed and executed once? Best, Ron Ron Liu 于2024年7月24日周三 13:59写道： > Hi, Junrui > > Thanks for the proposal, this design allows the Flink engine to become > smarter by doing more dynamic optimizations at runtime

Re: [DISCUSS] FLIP-470: Support Adaptive Broadcast Join

Hi, Xia This FLIP looks good to me, +1. I've two questions: 1. >> Accordingly, in terms of implementation, we will delay the codegen and creation of the join operators until runtime. How are you delaying codegen to runtime, the current runtime is not SQL planner aware. in other words, how do I

Re: [DISCUSS] FLIP-469: Supports Adaptive Optimization of StreamGraph

Hi, Junrui Thanks for the proposal, this design allows the Flink engine to become smarter by doing more dynamic optimizations at runtime. so +1 from my side. For your FLIP, I've one minor question. Regarding the StreamGraphOptimizationStrategy, you mentioned introducing the option `execution.bat

Re: [DISCUSS] FLIP-468: Introducing StreamGraph-Based Job Submission.

Hi, Junrui Thanks for your detailed reply. After reading the updated FLIP-468 & FLIP-470, I see that the design looks good. Best. Ron Junrui Lee 于2024年7月18日周四 14:26写道： > Hi all, > > I would like to follow up on my previous email regarding your feedback. > Below > is a concise summary of my m

Re: Discussion on Vertica-Flink Connector

Hi, Vikas. Vertica can be connected and accessed through JDBC API, I'm not sure if you connected through this API, Flink already provided a flink-connector-jdbc repo, and Supported a considerable number of databases through dialect, could we add a Vertica implementation on this? [1] https://github

Re: [DISCUSS] FLIP-XXX Amazon SQS Source Connector

Hi Saurabh, Thank you for sharing the FLIP for the SQS source connector. An SQS source connector will be a great addition to the Flink ecosystem, as there is a growing demand for SQS source/sink integration. I have a few queries: 1. AWS has announced JSON protocol support in SQS [1]. Can you she

Re: [DISCUSS] CLI action deprecation process

Sounds good to me. Thank you, Ferenc. Best, Xintong On Tue, Jul 23, 2024 at 9:42 PM Ferenc Csaky wrote: > > IIUC, you mean applying the strict mode for the programming APIs > > as well? > > Correct! The original thought that came to my mind that this could > be achieved with something simil

[jira] [Created] (FLINK-35888) Add e2e test for paimon DataSink

LvYanquan created FLINK-35888: - Summary: Add e2e test for paimon DataSink Key: FLINK-35888 URL: https://issues.apache.org/jira/browse/FLINK-35888 Project: Flink Issue Type: Technical Debt

[jira] [Created] (FLINK-35887) Null Pointer Exception in TypeExtractor.isRecord when trying to provide type info for interface

Jacob Jona Fahlenkamp created FLINK-35887: - Summary: Null Pointer Exception in TypeExtractor.isRecord when trying to provide type info for interface Key: FLINK-35887 URL: https://issues.apache.org/jira/bro

[DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

Hi All, A bit unusual FLIP [1], as this is a bug fix for a problem that I have recently discovered [2]. However I think FLIP is required, as properly fixing the issue requires changes to the public API. As this is a bug fix, I would propose to back-port this change to previous releases (1.19 and

[jira] [Created] (FLINK-35886) Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked

Piotr Nowojski created FLINK-35886: -- Summary: Incorrect watermark idleness timeout accounting when subtask is backpressured/blocked Key: FLINK-35886 URL: https://issues.apache.org/jira/browse/FLINK-35886

[FLINK-22748] Update for "Allow dynamic target topic selection in SQL Kafka sinks"

Hi there, I've spent some time picking up the existing work for FLINK-22748 , and created a PR, building on Nicholas Jiang's work: https://github.com/apache/flink-connector-kafka/pull/109 I saw the issue is marked stale, so I'm starting this mai

Force co-location of operators

Hi Team, I have 2 operators with `forceNonParallel`: - Lock generator - Lock remover I need them to be located on the same JVM. My testing shows that if I put them to the same slotSharingGroup (or leave both of them as "default"), then they will be running on the same task manager slot. This ensur

Re: [DISCUSS] CLI action deprecation process

> IIUC, you mean applying the strict mode for the programming APIs > as well? Correct! The original thought that came to my mind that this could be achieved with something similar to the "jdeprscan" Java tool, that checks for deprecated usage. In our case, the presence of @Deprecated and @Public o

[jira] [Created] (FLINK-35885) SlicingWindowOperator should not process watermark with proctime

Baozhu Zhao created FLINK-35885: --- Summary: SlicingWindowOperator should not process watermark with proctime Key: FLINK-35885 URL: https://issues.apache.org/jira/browse/FLINK-35885 Project: Flink

Re: [DISCUSS] FLIP-XXX: Aligning timeout logic in the AdaptiveScheduler's WaitingForResources and Executing states

Hi All! Thank you for the proposal, I think it will be great to simplify the current rescaling flow to make it more digestible :) I have 2 comments: 1. Related to what Matthias already pointed out, I think in production scenarios it may be a typical requirement to have a fairly short stabilizati

Discussion on Vertica-Flink Connector

Hello, We have developed a connector for Vertica that will work with Flink. I have created a Google page with all the details to carry the discussion on the connector. The connector will be maintained by our organization. I have created Google Page with the details of the connector that will b

[jira] [Created] (FLINK-35884) Support

JunboWang created FLINK-35884: - Summary: Support Key: FLINK-35884 URL: https://issues.apache.org/jira/browse/FLINK-35884 Project: Flink Issue Type: Improvement Components: Flink CDC

[jira] [Created] (FLINK-35883) Wildcard projection inserts column at wrong place

yux created FLINK-35883: --- Summary: Wildcard projection inserts column at wrong place Key: FLINK-35883 URL: https://issues.apache.org/jira/browse/FLINK-35883 Project: Flink Issue Type: Bug Com

[jira] [Created] (FLINK-35882) CLONE - [Release-1.20] Vote on the release candidate

Weijie Guo created FLINK-35882: -- Summary: CLONE - [Release-1.20] Vote on the release candidate Key: FLINK-35882 URL: https://issues.apache.org/jira/browse/FLINK-35882 Project: Flink Issue Type:

[jira] [Created] (FLINK-35880) CLONE - [Release-1.20] Stage source and binary releases on dist.apache.org

Weijie Guo created FLINK-35880: -- Summary: CLONE - [Release-1.20] Stage source and binary releases on dist.apache.org Key: FLINK-35880 URL: https://issues.apache.org/jira/browse/FLINK-35880 Project: Flink

[jira] [Created] (FLINK-35881) CLONE - [Release-1.20] Propose a pull request for website updates

Weijie Guo created FLINK-35881: -- Summary: CLONE - [Release-1.20] Propose a pull request for website updates Key: FLINK-35881 URL: https://issues.apache.org/jira/browse/FLINK-35881 Project: Flink

[jira] [Created] (FLINK-35879) CLONE - [Release-1.20] Build and stage Java and Python artifacts

Weijie Guo created FLINK-35879: -- Summary: CLONE - [Release-1.20] Build and stage Java and Python artifacts Key: FLINK-35879 URL: https://issues.apache.org/jira/browse/FLINK-35879 Project: Flink

[jira] [Created] (FLINK-35878) Build Release Candidate: 1.20.0-rc2

Weijie Guo created FLINK-35878: -- Summary: Build Release Candidate: 1.20.0-rc2 Key: FLINK-35878 URL: https://issues.apache.org/jira/browse/FLINK-35878 Project: Flink Issue Type: New Feature A

[jira] [Created] (FLINK-35877) Shade protobuf in flink

zhuanshenbsj1 created FLINK-35877: - Summary: Shade protobuf in flink Key: FLINK-35877 URL: https://issues.apache.org/jira/browse/FLINK-35877 Project: Flink Issue Type: Improvement C

[jira] [Created] (FLINK-35876) mysql cdc may change from snapshot mode to binlog mode failed when cpu load of mysql server is 100%

Youle created FLINK-35876: - Summary: mysql cdc may change from snapshot mode to binlog mode failed when cpu load of mysql server is 100% Key: FLINK-35876 URL: https://issues.apache.org/jira/browse/FLINK-35876