Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

Hi, xiangyu, Thanks for your attention as well. >1, About the waiting mechanism: Will the waiting mechanism happen only in >the second level 'assigning slots to TM'? IIUC, the first level 'assigning >Tasks to Slots' needs only the asynchronous slot result from slotpool. As described in the

[jira] [Created] (FLINK-33236) Remove the unused high-availability.zookeeper.path.running-registry option

Zhanghao Chen created FLINK-33236: - Summary: Remove the unused high-availability.zookeeper.path.running-registry option Key: FLINK-33236 URL: https://issues.apache.org/jira/browse/FLINK-33236

Re: Re: [DISCUSS] FLIP-373: Support Configuring Different State TTLs using SQL Hint

Hi Jane, Thank you for providing this explanation. Another small question, since there is no exception thrown when the STATE hint is set incorrectly, should we somehow show that the TTL setting has taken effect? For instance, exhibiting the TTL option within the operator's description? Best,

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

Hi, David, Thank you very much for your attention. >The problem you're trying to solve only exists in complex graphs with >different per-vertex parallelism. If the parallelism is set globally >(assuming the pipeline has roughly even data skew), the algorithm could >make things slightly worse by

[jira] [Created] (FLINK-33235) Quickstart guide for Flink OLAP should support building from master branch

xiangyu feng created FLINK-33235: Summary: Quickstart guide for Flink OLAP should support building from master branch Key: FLINK-33235 URL: https://issues.apache.org/jira/browse/FLINK-33235 Project:

Re: [VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

+1(non-binding) Thanks for driving this. Best, Zakelly On Wed, Oct 11, 2023 at 11:22 AM Weihua Hu wrote: > > +1(binding) > > Best, > Weihua > > > On Wed, Oct 11, 2023 at 10:56 AM xiangyu feng wrote: > > > +1(non-binding) > > > > Regards, > > Xiangyu > > > > Shammon FY 于2023年10月11日周三 10:30写道：

Re: FW: RE: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

Hi David, Thanks for your response. The exceptions thrown by state interfaces are NOT retriable. For example, there may be some elements sent to the wrong subtask due to a non-deterministic hashCode() algorithm and the key group is not matching. Or the rocksdb may fail to read a file if it has

Re: [VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

+1(binding) Best, Weihua On Wed, Oct 11, 2023 at 10:56 AM xiangyu feng wrote: > +1(non-binding) > > Regards, > Xiangyu > > Shammon FY 于2023年10月11日周三 10:30写道： > > > +1(binding), good job! > > > > Best, > > Shammon FY > > > > On Wed, Oct 11, 2023 at 10:18 AM Benchao Li > wrote: > > > > > +1

Re: [VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

+1(non-binding) Regards, Xiangyu Shammon FY 于2023年10月11日周三 10:30写道： > +1(binding), good job! > > Best, > Shammon FY > > On Wed, Oct 11, 2023 at 10:18 AM Benchao Li wrote: > > > +1 (binding) > > > > Rui Fan <1996fan...@gmail.com> 于2023年10月11日周三 10:17写道： > > > > > > +1(binding) > > > > > >

Re: [VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

+1(binding), good job! Best, Shammon FY On Wed, Oct 11, 2023 at 10:18 AM Benchao Li wrote: > +1 (binding) > > Rui Fan <1996fan...@gmail.com> 于2023年10月11日周三 10:17写道： > > > > +1(binding) > > > > Best, > > Rui > > > > On Wed, Oct 11, 2023 at 10:07 AM Yangze Guo wrote: > > > > > Hi everyone, > >

Re: [VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

+1 (binding) Rui Fan <1996fan...@gmail.com> 于2023年10月11日周三 10:17写道： > > +1(binding) > > Best, > Rui > > On Wed, Oct 11, 2023 at 10:07 AM Yangze Guo wrote: > > > Hi everyone, > > > > I'd like to start the vote of FLIP-374 [1]. This FLIP is discussed in > > the thread [2]. > > > > The vote will be

Re: [VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

+1(binding) Best, Rui On Wed, Oct 11, 2023 at 10:07 AM Yangze Guo wrote: > Hi everyone, > > I'd like to start the vote of FLIP-374 [1]. This FLIP is discussed in > the thread [2]. > > The vote will be open for at least 72 hours. Unless there is an > objection, I'll try to close it by October

[VOTE] FLIP-374: Adding a separate configuration for specifying Java Options of the SQL Gateway

Hi everyone, I'd like to start the vote of FLIP-374 [1]. This FLIP is discussed in the thread [2]. The vote will be open for at least 72 hours. Unless there is an objection, I'll try to close it by October 16, 2023 if we have received sufficient votes. [1]

[ANNOUNCE] Release 1.18.0, release candidate #2

Hi everyone, The RC2 for Apache Flink 1.18.0 has been created. The related voting process will be triggered once the announcement is ready. The RC2 has all the artifacts that we would typically have for a release, except for the release note and the website pull request for the release

[jira] [Created] (FLINK-33234) Bump used Guava version in Kafka E2E tests

Martijn Visser created FLINK-33234: -- Summary: Bump used Guava version in Kafka E2E tests Key: FLINK-33234 URL: https://issues.apache.org/jira/browse/FLINK-33234 Project: Flink Issue Type:

[jira] [Created] (FLINK-33233) Null point exception when non-native udf used in join condition

yunfan created FLINK-33233: -- Summary: Null point exception when non-native udf used in join condition Key: FLINK-33233 URL: https://issues.apache.org/jira/browse/FLINK-33233 Project: Flink Issue

[jira] [Created] (FLINK-33232) Kubernetive Operator Not Able to Take Other Python paramters While Submitting Job Deployment

Amarjeet Singh created FLINK-33232: -- Summary: Kubernetive Operator Not Able to Take Other Python paramters While Submitting Job Deployment Key: FLINK-33232 URL: https://issues.apache.org/jira/browse/FLINK-33232

[jira] [Created] (FLINK-33231) Memory leak in KafkaSourceReader if no data in consumed topic

Lauri Suurväli created FLINK-33231: -- Summary: Memory leak in KafkaSourceReader if no data in consumed topic Key: FLINK-33231 URL: https://issues.apache.org/jira/browse/FLINK-33231 Project: Flink

Re: Operator 1.6 to Olm

That would be great David, thank you! Gyula On Tue, 10 Oct 2023 at 14:13, David Radley wrote: > Hi, > I notice that the latest version in olm of the operator is 1.5. I plan to > run the scripts to publish the 1.6 Flink operator to olm, > Kind regards, David. > > Unless otherwise stated

Operator 1.6 to Olm

Hi, I notice that the latest version in olm of the operator is 1.5. I plan to run the scripts to publish the 1.6 Flink operator to olm, Kind regards, David. Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO

Re:Re: [DISCUSS] FLIP-373: Support Configuring Different State TTLs using SQL Hint

Hi, Jane. I think this syntax will be easier for users to set operator ttl. So big +1. I left some minor comments here. I notice that using STATE_TTL hints wrongly will not throw any exceptions. But it seems that in the current join hint scenario, if user uses an unknown table name as the

[jira] [Created] (FLINK-33230) Support Expanding ExecutionGraph to StreamGraph in Flink

Yu Chen created FLINK-33230: --- Summary: Support Expanding ExecutionGraph to StreamGraph in Flink Key: FLINK-33230 URL: https://issues.apache.org/jira/browse/FLINK-33230 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-373: Support Configuring Different State TTLs using SQL Hint

Hi Feng, Thank you for your valuable comments. The reason for not including the scenarios above is as follows: For <1>, the automatically inferred stateful operators are not easily expressible in SQL. This issue was discussed in FLIP-292, and besides ChangelogNormalize, SinkUpsertMateralizer

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

Hi Zhu, Thanks for your clarification! I misunderstood before, it's clear now. Best, Rui On Tue, Oct 10, 2023 at 6:17 PM Zhu Zhu wrote: > Hi Rui, > > Not sure if I understand your question correctly. The two modes are not > the same: > {taskmanager.load-balance.mode: Slots} =

FW: RE: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

Hi, I notice https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/state/ValueState.html is an external API. I am concerned that this change will break existing applications using the old interface, they are likely to have catches / throws around the

Re: [DISCUSS] FLIP-370 : Support Balanced Tasks Scheduling

Hi Rui, Not sure if I understand your question correctly. The two modes are not the same: {taskmanager.load-balance.mode: Slots} = {cluster.evenly-spread-out-slots: true, slot.sharing-strategy: LOCAL_INPUT_PREFERRED} {taskmanager.load-balance.mode: Tasks} = {cluster.evenly-spread-out-slots: true,

RE: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

Hi , The argument seems to be that the errors cannot be acted on so should be runtime exceptions. I want to confirm that none of these errors could / should be retriable. If there is a possibility that the state is available at some time later then I assume a checked retriable Exception would

[jira] [Created] (FLINK-33229) Moving Java FlinkRecomputeStatisticsProgram from scala package to java package

dalongliu created FLINK-33229: - Summary: Moving Java FlinkRecomputeStatisticsProgram from scala package to java package Key: FLINK-33229 URL: https://issues.apache.org/jira/browse/FLINK-33229 Project:

Re: [ANNOUNCE] Release 1.18.0, release candidate #1

Hi folks, In the release managers' sync-up meeting today, we have discussed all issues and came to a consensus that there are no blocking issues for 1.18 release. Since there were some new commits after rc1, I will create a rc2 now. Thanks again for your contributions. Best regards, Jing On

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

Hi Yun, 3% extra cost sounds good! It could be useful, if the reference could be added into the FLIP. Thanks for answering my questions. Best regards, Jing On Tue, Oct 10, 2023 at 9:15 AM Yun Tang wrote: > Hi Jing, > > First, developers would accept the little overhead when debugging the >

Re: Support AWS SDK V2 for Flink's S3 FileSystem

+1 for the s3 file consolidation. We already have many issues with internal communication and talking to customers. Different file schemas are not very user friendly, btw. Best regards, Jing On Mon, Oct 9, 2023 at 6:49 PM Matthias Pohl wrote: > I would agree with David's proposal as well. > >

[jira] [Created] (FLINK-33228) Fix the total current resource calculation in fulfill requirements

xiangyu feng created FLINK-33228: Summary: Fix the total current resource calculation in fulfill requirements Key: FLINK-33228 URL: https://issues.apache.org/jira/browse/FLINK-33228 Project: Flink

Re: Support AWS SDK V2 for Flink's S3 FileSystem

Looks like Maomao was missed from previous replies. Adding back @Maomao. Thanks everyone for your response. We are having some discussion within AWS EMR team. Will get back to you very soon. Regards, Kevin From: Matthias Pohl Date: Tuesday, October 10, 2023 at

Re: Support AWS SDK V2 for Flink's S3 FileSystem

Just to add a bit more context to the performance test question: What I had in mind was the exists call on a (non-existing) directories in a bucket with a lot of objects. A comment from one of the SDK contributors about that call was that it could be an expensive call in an object store if

Re: [DISCUSS] FLIP-375: Built-in cross-platform powerful java profiler on taskmanagers

Hi Jing, First, developers would accept the little overhead when debugging the performance issues. Secondly, according to the async-profiler's report, it should only have less than 3% extra cost. I also verified it with a CPU-intensive ETL Flink job with this profiler, it does not show any