Re: [VOTE] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

2023-12-19 Thread Yingjie Cao
+1 (binding) Best, Yingjie Xintong Song 于2023年12月18日周一 15:27写道: > +1 (binding) > > > Best, > > Xintong > > > > On Fri, Dec 15, 2023 at 5:15 PM weijie guo > wrote: > > > +1 (binding) > > > > Best regards, > > > > Weijie > > > > > > Wencong Liu 于2023年12月15日周五 09:15写道: > > > > > Hi dev, > > > >

[jira] [Created] (FLINK-33892) FLIP-383: Support Job Recovery for Batch Jobs

2023-12-19 Thread Lijie Wang (Jira)
Lijie Wang created FLINK-33892: -- Summary: FLIP-383: Support Job Recovery for Batch Jobs Key: FLINK-33892 URL: https://issues.apache.org/jira/browse/FLINK-33892 Project: Flink Issue Type: New

Re: [DISCUSS] FLIP-387: Support named parameters for functions and call procedures

2023-12-19 Thread Benchao Li
I didn't see your POC code, so I assumed that you'll need to add SqlStdOperatorTable#DEFAULT and SqlStdOperatorTable#ARGUMENT_ASSIGNMENT to FlinkSqlOperatorTable, am I right? If yes, this would enable many builtin functions to allow default and optional arguments, for example, `select

Re: [DISCUSS] FLIP-387: Support named parameters for functions and call procedures

2023-12-19 Thread Benchao Li
Thanks Feng for driving this, it's a very useful feature. In the FLIP, you mentioned that > During POC verification, bugs were discovered in Calcite that caused issues > during the validation phase. We need to modify the SqlValidatorImpl and > SqlToRelConverter to address these problems. Could

Re: [VOTE] Release flink-connector-pulsar 4.1.0, release candidate #1

2023-12-19 Thread tison
Hi Leonard, You are a PMC member also. Perhaps you can check the candidate and vote on what you do :D Best, tison. Leonard Xu 于2023年12月20日周三 11:35写道: > > Bubble up, I need more votes, especially from PMC members. > > Best, > Leonard > > > 2023年12月14日 下午11:03,Hang Ruan 写道: > > > > +1

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-12-19 Thread Jiabao Sun
Thanks Becket, The behavior description has been added to the Public Interfaces section. Best, Jiabao > 2023年12月20日 08:17,Becket Qin 写道: > > Hi Jiabao, > > Thanks for updating the FLIP. > Can you add the behavior of the policies that are only applicable to some > but not all of the

Re: [VOTE] Release flink-connector-pulsar 4.1.0, release candidate #1

2023-12-19 Thread Leonard Xu
Bubble up, I need more votes, especially from PMC members. Best, Leonard > 2023年12月14日 下午11:03,Hang Ruan 写道: > > +1 (non-binding) > > - Validated checksum hash > - Verified signature > - Verified that no binaries exist in the source archive > - Build the source with jdk8 > - Verified web PR >

Re: [VOTE] FLIP-372: Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

2023-12-19 Thread Yanquan Lv
+1 (non-binding) Merging multiple write results into one submission is a common scenario we encounter. But in the process of implementing the addPreCommitTopology, I feel that the logic of regenerating CommittableSummary objects is a bit hidden. Best, Yanquan Lv. Péter Váry 于2023年12月18日周一

[jira] [Created] (FLINK-33891) Remove the obsolete SingleJobGraphStore

2023-12-19 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-33891: - Summary: Remove the obsolete SingleJobGraphStore Key: FLINK-33891 URL: https://issues.apache.org/jira/browse/FLINK-33891 Project: Flink Issue Type:

Re: [VOTE] FLIP-364: Update the default value of backoff-multiplier from 1.2 to 1.5

2023-12-19 Thread Zhu Zhu
+1 (binding) Thanks, Zhu Rui Fan <1996fan...@gmail.com> 于2023年12月19日周二 17:17写道: > Hi everyone, > > Thank you to everyone for the feedback on FLIP-364: > Improve the restart-strategy[1] which has been voted in this thread[2]. > > After the vote on FLIP-364, there was some feedback on the user

Re: [DISCUSS] FLIP-389: Annotate SingleThreadFetcherManager and FutureCompletingBlockingQueue as PublicEvolving

2023-12-19 Thread Hongshun Wang
Hi Becket, It has been a long time since we last discussed. Are there any other problems with this Flip from your side? I am looking forward to hearing from you. Thanks, Hongshun Wang

Re: [DISCUSS] Should Configuration support getting value based on String key?

2023-12-19 Thread Xuannan Su
Hi Rui, I am fine with keeping the `String getString(String key, String defaultValue)` if more people favor it. However, we should let the user know that we encourage using ConfigOptions over the string-based configuration key, as Timo said, we should add the message to `String getString(String

[jira] [Created] (FLINK-33890) Determine the initial status before receiving the first RecordAttributes

2023-12-19 Thread Xuannan Su (Jira)
Xuannan Su created FLINK-33890: -- Summary: Determine the initial status before receiving the first RecordAttributes Key: FLINK-33890 URL: https://issues.apache.org/jira/browse/FLINK-33890 Project: Flink

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-19 Thread Alan Sheinberg
Thanks for the comments Timo. > Can you remove the necessary parts? Esp.: @Override > public Set getRequirements() { > return Collections.singleton(FunctionRequirement.ORDERED); > } I removed this section from the FLIP since presumably, there's no use in adding to the

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-12-19 Thread Becket Qin
Hi Jiabao, Thanks for updating the FLIP. Can you add the behavior of the policies that are only applicable to some but not all of the databases? This is a part of the intended behavior of the proposed configuration. So, we should include that in the FLIP. Otherwise, the FLIP looks good to me.

Re: [DISCUSS] Release Flink 1.18.1

2023-12-19 Thread Jing Ge
Hi everyone, Because of some new incoming blockers and other tasks, the rc1 has been created but the vote has been skipped. I just started the vote for 1.18.1-rc2 at [1]. It would be great if you might help check. Thanks! Best regards, Jing [1]

[VOTE] Release 1.18.1, release candidate #2

2023-12-19 Thread Jing Ge
Hi everyone, The release candidate #1 has been skipped. Please review and vote on the release candidate #2 for the version 1.18.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your

[jira] [Created] (FLINK-33886) CLONE - Build and stage Java and Python artifacts

2023-12-19 Thread Jing Ge (Jira)
Jing Ge created FLINK-33886: --- Summary: CLONE - Build and stage Java and Python artifacts Key: FLINK-33886 URL: https://issues.apache.org/jira/browse/FLINK-33886 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-33889) CLONE - Vote on the release candidate

2023-12-19 Thread Jing Ge (Jira)
Jing Ge created FLINK-33889: --- Summary: CLONE - Vote on the release candidate Key: FLINK-33889 URL: https://issues.apache.org/jira/browse/FLINK-33889 Project: Flink Issue Type: Sub-task Affects

[jira] [Created] (FLINK-33888) CLONE - Propose a pull request for website updates

2023-12-19 Thread Jing Ge (Jira)
Jing Ge created FLINK-33888: --- Summary: CLONE - Propose a pull request for website updates Key: FLINK-33888 URL: https://issues.apache.org/jira/browse/FLINK-33888 Project: Flink Issue Type:

[jira] [Created] (FLINK-33887) CLONE - Stage source and binary releases on dist.apache.org

2023-12-19 Thread Jing Ge (Jira)
Jing Ge created FLINK-33887: --- Summary: CLONE - Stage source and binary releases on dist.apache.org Key: FLINK-33887 URL: https://issues.apache.org/jira/browse/FLINK-33887 Project: Flink Issue

[jira] [Created] (FLINK-33885) Build Release Candidate: 1.18.1-rc2

2023-12-19 Thread Jing Ge (Jira)
Jing Ge created FLINK-33885: --- Summary: Build Release Candidate: 1.18.1-rc2 Key: FLINK-33885 URL: https://issues.apache.org/jira/browse/FLINK-33885 Project: Flink Issue Type: New Feature

status of FLINK-32476

2023-12-19 Thread Lu Niu
Hi, Flink dev What's the current status of FLINK-32476? https://issues.apache.org/jira/browse/FLINK-32476 . I see this feature is deprioritized. We are interested in this feature and willing to work with the community on this if no one is actively working on it. Best Lu

[jira] [Created] (FLINK-33884) Update Pulsar dependency to 3.0.2 in Pulsar Connector

2023-12-19 Thread David Christle (Jira)
David Christle created FLINK-33884: -- Summary: Update Pulsar dependency to 3.0.2 in Pulsar Connector Key: FLINK-33884 URL: https://issues.apache.org/jira/browse/FLINK-33884 Project: Flink

[DISCUSS] FLIP-404: Create a Redis HyperLogLog Connector for Flink

2023-12-19 Thread Jinsui Chen
Hi, there We would like to start a discussion thread on "FLIP-404: Create a Redis HyperLogLog Connector for Flink"[1]. There has also been a lot of discussion in the past about Redis connectors, including creating lookup and sink connectors for generic key-value get and put, and creating source

[jira] [Created] (FLINK-33883) Bump CI flink version on flink-connector-jdbc

2023-12-19 Thread Jiabao Sun (Jira)
Jiabao Sun created FLINK-33883: -- Summary: Bump CI flink version on flink-connector-jdbc Key: FLINK-33883 URL: https://issues.apache.org/jira/browse/FLINK-33883 Project: Flink Issue Type:

Re: [VOTE] FLIP-372: Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

2023-12-19 Thread Gabor Somogyi
+1 (binding) G On Mon, Dec 18, 2023 at 9:34 AM Péter Váry wrote: > Hi everyone, > > Since there were no further comments on the discussion thread [1], I would > like to start the vote for FLIP-372 [2]. > > The FLIP started as a small new feature, but in the discussion thread and > in a

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-12-19 Thread Jiabao Sun
Hi Becket, I share the same view as you regarding the prefix for this configuration option. For the JDBC connector, I prefer setting 'filter.handling.policy' = 'FOO' and throwing an exception when the database do not support that specific policy. Not using a prefix can reduce the learning

Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-19 Thread Timo Walther
Hi Xuyang, sorry I missed the ping. Sounds reasonable to me. One FLIP about changelog semantics, the other about SQL semantics. Regards, Timo On 19.12.23 02:39, Xuyang wrote: Hi, Timo. Sorry for this noise. What do you think about splitting the flip like this? -- Best! Xuyang

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-12-19 Thread Jiabao Sun
Hi David, Sorry, the last two comments at the bottom of the email list were replies from a long time ago. I didn't have a good understanding of the display order of the email list before, which caused inconvenience. You can refer to the comments above and the latest FLIP-377 document[1] for

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-19 Thread Timo Walther
> I would be totally fine with the first version only having ORDERED > mode. For a v2, we could attempt to do the next most conservative > thing Sounds good to me. I also cheked AsyncWaitOperator and could not find n access of StreamRecord's timestamp but only watermarks. But as we said, let's

[jira] [Created] (FLINK-33882) UT/IT for checkpointing statistics

2023-12-19 Thread Jing Ge (Jira)
Jing Ge created FLINK-33882: --- Summary: UT/IT for checkpointing statistics Key: FLINK-33882 URL: https://issues.apache.org/jira/browse/FLINK-33882 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-33881) [TtlListState]Avoid copy and update value when TtlListState don‘t have expired element

2023-12-19 Thread Jinzhong Li (Jira)
Jinzhong Li created FLINK-33881: --- Summary: [TtlListState]Avoid copy and update value when TtlListState don‘t have expired element Key: FLINK-33881 URL: https://issues.apache.org/jira/browse/FLINK-33881

[RESULT][VOTE] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-19 Thread Lijie Wang
Hi devs, I'm happy to announce that FLIP-383 [1] has been approved unanimously. According to the vote thread[2], there are 5 approving votes, 4 of which are binding: Xintong Song (binding) Zhu Zhu (binding) Weijie Guo (binding) Rui Fan (binding) Yuepeng Pan (non-binding) [1]

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-12-19 Thread David Radley
Hi, I had a 3 of comments: - the name of the config option is "scan.filter-push-down.enabled". This implies it is only for scan sources and not lookups. I suggest removing the scan. prefix. - there is a talk of having a numeric option, as the filter pushdown might result in a full table scan.

Re: [DISCUSS] Should Configuration support getting value based on String key?

2023-12-19 Thread Rui Fan
> > I noticed that Configuration is used in > > DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig > > to store some cacheFile meta-information. Their keys are > > temporary(key name with number) and it is not convenient > > to predefine ConfigOption. > > > True, this one requires a

Re: [DISCUSS] FLIP-377: Support configuration to disable filter push down for Table/SQL Sources

2023-12-19 Thread Becket Qin
Hi Jiabao, Thanks for updating the FLIP. One more question regarding the JDBC connector, since it is a connector shared by multiple databases, what if there is a filter handling policy that is only applicable to one of the databases, but not the others? In that case, how would the users specify

[jira] [Created] (FLINK-33880) Introducing Retry Mechanism for Listing TaskManager Pods to Prevent API Server Connection Failures

2023-12-19 Thread Yuan Huang (Jira)
Yuan Huang created FLINK-33880: --- Summary: Introducing Retry Mechanism for Listing TaskManager Pods to Prevent API Server Connection Failures Key: FLINK-33880 URL: https://issues.apache.org/jira/browse/FLINK-33880

[jira] [Created] (FLINK-33879) Hybrid Shuffle may hang during redistribution

2023-12-19 Thread Jiang Xin (Jira)
Jiang Xin created FLINK-33879: - Summary: Hybrid Shuffle may hang during redistribution Key: FLINK-33879 URL: https://issues.apache.org/jira/browse/FLINK-33879 Project: Flink Issue Type: Bug

[DISCUSS] FLIP-403: High Availability Services for OLAP Scenarios

2023-12-19 Thread Yangze Guo
Hi, there, We would like to start a discussion thread on "FLIP-403: High Availability Services for OLAP Scenarios"[1]. Currently, Flink's high availability service consists of two mechanisms: leader election/retrieval services for JobManager and persistent services for job metadata. However,

Re: FLIP-403: High Availability Services for OLAP Scenarios

2023-12-19 Thread Yangze Guo
I just found that I missed the "[DISCUSS]" prefix. Will start another thread for this FLIP. Best, Yangze Guo On Tue, Dec 19, 2023 at 5:31 PM Yangze Guo wrote: > > Thanks for the response, Xiangyu. > > As said in FLIP, this is pure refactoring and should not affect the > functionality of the

Re: FLIP-403: High Availability Services for OLAP Scenarios

2023-12-19 Thread Yangze Guo
Thanks for the response, Xiangyu. As said in FLIP, this is pure refactoring and should not affect the functionality of the current HA mechanism. The leader path should not be changed as well. There will not be any compatibility issue for existing HA. Best, Yangze Guo On Tue, Dec 19, 2023 at

Re: [DISCUSS] Change the default restart-strategy to exponential-delay

2023-12-19 Thread Rui Fan
Thanks everyone for the feedback! It doesn't have more feedback here, so I started the new vote[1] just now to update the default value of backoff-multiplier from 1.2 to 1.5. [1] https://lists.apache.org/thread/0b1dcwb49owpm6v1j8rhrg9h0fvs5nkt Best, Rui On Tue, Dec 12, 2023 at 7:14 PM

[VOTE] FLIP-364: Update the default value of backoff-multiplier from 1.2 to 1.5

2023-12-19 Thread Rui Fan
Hi everyone, Thank you to everyone for the feedback on FLIP-364: Improve the restart-strategy[1] which has been voted in this thread[2]. After the vote on FLIP-364, there was some feedback on the user mail list[3] suggesting changing the default value of backoff-multiplier from 1.2 to 1.5. I

Re: FLIP-403: High Availability Services for OLAP Scenarios

2023-12-19 Thread xiangyu feng
Hi Yangze, Thanks for driving this. I like the idea to separate the current HighAvailabilityServices into LeaderServices and PersistentServices. AFAIK, there is no need to bind JobGraphStore/JobResultStore/BlobStore/CheckpointStore modules with LeaderElection/LeaderRetrieval services. The formers

Re:[VOTE] FLIP-383: Support Job Recovery for Batch Jobs

2023-12-19 Thread Yuepeng Pan
+1 (non-binding) Best, Yuepeng Pan. At 2023-12-14 15:12:48, "Lijie Wang" wrote: >Hi devs, Thanks for all feedback about the FLIP-383: Support Job Recovery >for Batch Jobs[1]. This FLIP was discussed in [2]. > >I'd like to start a vote for it. The vote will be open for at least 72 >hours (until