Re: [VOTE] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-30 Thread Zakelly Lan
+1 (binding) Best, Zakelly On Wed, Jul 31, 2024 at 12:07 AM Piotr Nowojski wrote: > Hi all! > > I would like to open the vote for FLIP-471 [1]. It has been discussed here > [2]. > > The vote will remain open for at least 72 hours. > > Best, > Piotrek > > [1]

[DISCUSS] FLIP-455: Declare async state processing and checkpoint the in-flight requests

2024-07-30 Thread Zakelly Lan
Hi devs, I would like to initiate a discussion about FLIP-455: Declare async state processing and checkpoint the in-flight requests[1]. FLIP-423[2] and the related sub-FLIPs introduced the disaggregated state and async accessing model of state. However, the in-flight state requests (or records)

Re: [DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-29 Thread Zakelly Lan
the absolute > time. > > It > > > feels to me like it would be > > > confusing to potential users. From this point of view, I like adding a > > new > > > dedicated interface that actually > > > highlights this possibility. > > > > > &

[jira] [Created] (FLINK-35904) Test harness for async state processing operators

2024-07-26 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35904: --- Summary: Test harness for async state processing operators Key: FLINK-35904 URL: https://issues.apache.org/jira/browse/FLINK-35904 Project: Flink Issue Type

Re: [DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-26 Thread Zakelly Lan
Hi Piotr, I'd +1 for this idea. I partially share the same concern with Arvid. Could we just use `org.apache.flink.util.clock.Clock` instead of introducing a new one? Thus only a new implementation is required IIUC. My thinking is that clock is a general need for timing. How to generate time or

[jira] [Created] (FLINK-35858) Namespace support for async state

2024-07-17 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35858: --- Summary: Namespace support for async state Key: FLINK-35858 URL: https://issues.apache.org/jira/browse/FLINK-35858 Project: Flink Issue Type: Sub-task

Re: [VOTE] FLIP-460: Display source/sink I/O metrics on Flink Web UI

2024-07-16 Thread Zakelly Lan
+1 (binding) Best, Zakelly On Wed, Jul 17, 2024 at 9:48 AM Rui Fan <1996fan...@gmail.com> wrote: > +1 (binding) > > Best, > Rui > > On Wed, Jul 17, 2024 at 9:07 AM Xintong Song > wrote: > > > +1 (binding) > > > > Best, > > > > Xintong > > > > > > > > On Wed, Jul 17, 2024 at 5:58 AM Aleksandr

[jira] [Created] (FLINK-35843) Check cp directory empty may fail in file-merging testing

2024-07-15 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35843: --- Summary: Check cp directory empty may fail in file-merging testing Key: FLINK-35843 URL: https://issues.apache.org/jira/browse/FLINK-35843 Project: Flink

[jira] [Created] (FLINK-35812) Move tuple interfaces to flink-core-api

2024-07-11 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35812: --- Summary: Move tuple interfaces to flink-core-api Key: FLINK-35812 URL: https://issues.apache.org/jira/browse/FLINK-35812 Project: Flink Issue Type

[jira] [Created] (FLINK-35784) The cp file-merging directory not properly registered in SharedStateRegistry

2024-07-08 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35784: --- Summary: The cp file-merging directory not properly registered in SharedStateRegistry Key: FLINK-35784 URL: https://issues.apache.org/jira/browse/FLINK-35784 Project

[jira] [Created] (FLINK-35778) Escape URI reserved characters when creating file-merging directories

2024-07-07 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35778: --- Summary: Escape URI reserved characters when creating file-merging directories Key: FLINK-35778 URL: https://issues.apache.org/jira/browse/FLINK-35778 Project: Flink

Re: [DISCUSS] Release flink-shaded 19.0

2024-06-27 Thread Zakelly Lan
Hi Sergey, FYI we also meet the regression of stringRead or stringWrite when upgrading the JMH[1]. I guess it is caused by JIT. I'd suggest accepting it if JIT is so sensitive to 'unrelated' change in codebase, since this probably won't be a problem in production environment. I recommend we do

Re: [2.0] How to handle on-going feature development in Flink 2.0?

2024-06-26 Thread Zakelly Lan
+1 for a preview before the formal release. It would help us find issues in advance. Best, Zakelly On Wed, Jun 26, 2024 at 4:44 PM Jingsong Li wrote: > +1 to release a preview version. > > Best, > Jingsong > > On Wed, Jun 26, 2024 at 10:12 AM Jark Wu wrote: > > > > I also think this should

Re: [VOTE] FLIP-444: Native file copy support

2024-06-26 Thread Zakelly Lan
. > > > >Vote can be resumed. > > > >Best, > >Piotrek > > > >wt., 25 cze 2024 o 18:57 Piotr Nowojski > napisał(a): > > > >> Ops, I must have forgotten to update the FLIP as we discussed. I will > fix > >> it tomorrow and the

Re: [VOTE] FLIP-444: Native file copy support

2024-06-25 Thread Zakelly Lan
Hi Piotrek, I don't see any statement about removing or renaming the `DuplicatingFileSystem` in the FLIP, shall we do that as mentioned in the discussion thread? Best, Zakelly On Tue, Jun 25, 2024 at 4:58 PM Piotr Nowojski wrote: > Hi all, > > I would like to start a vote for the FLIP-444

[jira] [Created] (FLINK-35667) Implement Reducing Async State API for ForStStateBackend

2024-06-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35667: --- Summary: Implement Reducing Async State API for ForStStateBackend Key: FLINK-35667 URL: https://issues.apache.org/jira/browse/FLINK-35667 Project: Flink Issue

[jira] [Created] (FLINK-35666) Implement Aggregating Async State API for ForStStateBackend

2024-06-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35666: --- Summary: Implement Aggregating Async State API for ForStStateBackend Key: FLINK-35666 URL: https://issues.apache.org/jira/browse/FLINK-35666 Project: Flink

Re: [VOTE] FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing for the AdaptiveScheduler

2024-06-17 Thread Zakelly Lan
+1 (binding) Best, Zakelly On Mon, Jun 17, 2024 at 5:10 PM Rui Fan <1996fan...@gmail.com> wrote: > +1 (binding) > > Best, > Rui > > On Mon, Jun 17, 2024 at 4:58 PM David Morávek > wrote: > > > +1 (binding) > > > > On Mon, Jun 17, 2024 at 10:24 AM Matthias Pohl > wrote: > > > > > Hi everyone,

Re: [ANNOUNCE] New Apache Flink Committer - Hang Ruan

2024-06-16 Thread Zakelly Lan
Congratulations, Hang! Best, Zakelly On Mon, Jun 17, 2024 at 12:07 PM Yanquan Lv wrote: > Congratulations, Hang! > > Samrat Deb 于2024年6月17日周一 11:32写道: > > > Congratulations Hang Ruan ! > > > > Bests, > > Samrat > > > > On Mon, Jun 17, 2024 at 8:47 AM Leonard Xu wrote: > > > > > Hi everyone,

Re: [ANNOUNCE] New Apache Flink Committer - Zhongqiang Gong

2024-06-16 Thread Zakelly Lan
Congratulations, Zhongqiang! Best, Zakelly On Mon, Jun 17, 2024 at 12:05 PM Shawn Huang wrote: > Congratulations ! > > Best, > Shawn Huang > > > Yuepeng Pan 于2024年6月17日周一 12:03写道: > > > Congratulations ! Best regards Yuepeng Pan > > > > > > > > > > > > At 2024-06-17 11:20:30, "Leonard Xu"

[jira] [Created] (FLINK-35624) Release Testing: Verify FLIP-306 Unified File Merging Mechanism for Checkpoints

2024-06-16 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35624: --- Summary: Release Testing: Verify FLIP-306 Unified File Merging Mechanism for Checkpoints Key: FLINK-35624 URL: https://issues.apache.org/jira/browse/FLINK-35624

Re: [ANNOUNCE] Flink 1.20 feature freeze

2024-06-15 Thread Zakelly Lan
Hi Robert, Rui, Ufuk and Weijie, Thanks for the update! FYI: This PR[1] fixes & cleanup the left-over checkpoint directories for file-merging on TM exit. And the second commit fixes the wrong state handle usage. We encountered several unexpected CI fails, so we missed the feature freeze time. It

[jira] [Created] (FLINK-35617) Support object reuse in async state execution

2024-06-14 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35617: --- Summary: Support object reuse in async state execution Key: FLINK-35617 URL: https://issues.apache.org/jira/browse/FLINK-35617 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-35590) Cleanup deprecated options usage in docs about state and checkpoint

2024-06-13 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35590: --- Summary: Cleanup deprecated options usage in docs about state and checkpoint Key: FLINK-35590 URL: https://issues.apache.org/jira/browse/FLINK-35590 Project: Flink

Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"

2024-06-12 Thread Zakelly Lan
+1 (binding) Thanks for driving this! Best, Zakelly On Thu, Jun 13, 2024 at 10:05 AM Junrui Lee wrote: > +1 (non-binding) > > Best, > Junrui > > Biao Geng 于2024年6月13日周四 09:54写道: > > > Thanks for driving this. > > +1 (non-binding) > > > > Best, > > Biao Geng > > > > > > weijie guo

[jira] [Created] (FLINK-35570) Consider PlaceholderStreamStateHandle in checkpoint file merging

2024-06-11 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35570: --- Summary: Consider PlaceholderStreamStateHandle in checkpoint file merging Key: FLINK-35570 URL: https://issues.apache.org/jira/browse/FLINK-35570 Project: Flink

Re: [DISCUSS] The performance of serializerHeavyString starts regress since April 3

2024-06-11 Thread Zakelly Lan
Thanks Sam for your investigation. I revisited the logs and confirmed that the JDK has never changed. 'java -version' get: > openjdk version "11.0.19" 2023-04-18 LTS > OpenJDK Runtime Environment (Red_Hat-11.0.19.0.7-2) (build 11.0.19+7-LTS) > OpenJDK 64-Bit Server VM (Red_Hat-11.0.19.0.7-2)

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

2024-06-07 Thread Zakelly Lan
pares the cluster > for rescaling (eg. by replicating state to new slots / pre-splitting the > db, ...), to make things faster. > > Best, > D. > > On Wed, Jun 5, 2024 at 4:34 PM Matthias Pohl wrote: > > > Hi Zakelly, > > thanks for your reply. See my inlined response

[jira] [Created] (FLINK-35537) Error parsing list of enum in legacy yaml configuration

2024-06-06 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35537: --- Summary: Error parsing list of enum in legacy yaml configuration Key: FLINK-35537 URL: https://issues.apache.org/jira/browse/FLINK-35537 Project: Flink Issue

[jira] [Created] (FLINK-35535) Enable benchmark profiling in daily run

2024-06-05 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35535: --- Summary: Enable benchmark profiling in daily run Key: FLINK-35535 URL: https://issues.apache.org/jira/browse/FLINK-35535 Project: Flink Issue Type

[jira] [Created] (FLINK-35534) Support profiler for benchmarks

2024-06-05 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35534: --- Summary: Support profiler for benchmarks Key: FLINK-35534 URL: https://issues.apache.org/jira/browse/FLINK-35534 Project: Flink Issue Type: Improvement

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

2024-06-05 Thread Zakelly Lan
Congratulations, Rui! Best, Zakelly On Wed, Jun 5, 2024 at 6:02 PM Piotr Nowojski wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce another new Apache Flink > PMC Member - Fan Rui. > > Rui has been active in the community since August 2019. During this time he > has

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

2024-06-05 Thread Zakelly Lan
Hi Matthias, Thanks for your proposal! I have a few questions: 1. Is it possible a change event observed right after a complete checkpoint (or within a specific short time after a checkpoint) that triggers a rescale immediately? Sometimes the checkpoint interval is huge and it is better to

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

2024-06-04 Thread Zakelly Lan
Congratulations, Weijie! Best, Zakelly On Tue, Jun 4, 2024 at 7:49 PM Sergey Nuyanzin wrote: > Congratulations Weijio Guo! > > On Tue, Jun 4, 2024, 13:45 Jark Wu wrote: > > > Congratulations, Weijie! > > > > Best, > > Jark > > > > On Tue, 4 Jun 2024 at 19:10, spoon_lz wrote: > > > > >

Re: [VOTE] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Zakelly Lan
+1 (binding) Best, Zakelly On Thu, May 23, 2024 at 8:21 PM Piotr Nowojski wrote: > Hi all, > > After reaching what looks like a consensus in the discussion thread [1], I > would like to put FLIP-443 [2] to the vote. > > The vote will be open for at least 72 hours unless there is an objection

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Zakelly Lan
ng operators (and > interruptible mails), so I will just add `MailOptions` parameters only to > them. If necessary, we can add more in the future. > > I have updated the FLIP. If it looks good to you, I would start a voting > thread today/tomorrow. > > Best, > Piotrek > > czw.

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Zakelly Lan
a builder? > > > mailboxExecutor.execute(myThrowingRunnable).setInterriptuble().description("bla > %d").arg(42).submit(); > > It could be done in both in the future, if we would ever need to add even > more methods, or I could do it now. WDYT? > > Best, > Piotrek > > ś

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-22 Thread Zakelly Lan
iotrek > > > > > > [1] http://flink-speed.xyz/timeline/?ben=fireProcessingTimers=3 > > > > > > > > > > > > wt., 7 maj 2024 o 09:47 Rui Fan <1996fan...@gmail.com> napisał(a): > > > > > > > Hi Piotr, > > > >

Re: [DISCUSS] The performance of serializerHeavyString starts regress since April 3

2024-05-21 Thread Zakelly Lan
Hi Rui and RMs of Flink 1.20, Thanks for driving this! Available information indicates this issue is environment- and JDK-specific, and I also failed to reproduce it in my Mac. Thus I guess it is caused by JIT behavior, which is unpredictable and vulnerable to disturbance of the codebase.

[jira] [Created] (FLINK-35412) Batch execution of async state request callback

2024-05-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35412: --- Summary: Batch execution of async state request callback Key: FLINK-35412 URL: https://issues.apache.org/jira/browse/FLINK-35412 Project: Flink Issue Type

[jira] [Created] (FLINK-35411) Optimize wait logic in draining of async state requests

2024-05-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35411: --- Summary: Optimize wait logic in draining of async state requests Key: FLINK-35411 URL: https://issues.apache.org/jira/browse/FLINK-35411 Project: Flink Issue

[jira] [Created] (FLINK-35410) Avoid sync waiting in coordinator thread of ForSt executor

2024-05-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35410: --- Summary: Avoid sync waiting in coordinator thread of ForSt executor Key: FLINK-35410 URL: https://issues.apache.org/jira/browse/FLINK-35410 Project: Flink

[jira] [Created] (FLINK-35400) Rebuild FileMergingSnapshotManager in failover

2024-05-20 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35400: --- Summary: Rebuild FileMergingSnapshotManager in failover Key: FLINK-35400 URL: https://issues.apache.org/jira/browse/FLINK-35400 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-35379) File merging manager is not properly notified about checkpoint

2024-05-16 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35379: --- Summary: File merging manager is not properly notified about checkpoint Key: FLINK-35379 URL: https://issues.apache.org/jira/browse/FLINK-35379 Project: Flink

[jira] [Created] (FLINK-35356) Async reducing state

2024-05-14 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35356: --- Summary: Async reducing state Key: FLINK-35356 URL: https://issues.apache.org/jira/browse/FLINK-35356 Project: Flink Issue Type: Sub-task Components

[jira] [Created] (FLINK-35355) Async aggregating state

2024-05-14 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35355: --- Summary: Async aggregating state Key: FLINK-35355 URL: https://issues.apache.org/jira/browse/FLINK-35355 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35307) Add Compile CI check on jdk17

2024-05-07 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35307: --- Summary: Add Compile CI check on jdk17 Key: FLINK-35307 URL: https://issues.apache.org/jira/browse/FLINK-35307 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-06 Thread Zakelly Lan
l file from a remote location > > Maybe indeed that's a better solution vs having two separate interfaces for > copying and duplicating? > > > 3. Will the interface extracting introduce a break change? > > No. The signature of the existing abstract `FileSystem` class would remain >

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-06 Thread Zakelly Lan
th something? If an incoming async state access > response comes, it could just request to interrupt any currently ongoing > computations, regardless the currently executed mail is or is not > interruptible. > > Best, > Piotrek > > pon., 6 maj 2024 o 06:33 Zakelly Lan napisa

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-05 Thread Zakelly Lan
Hi Piotr, Thanks for the improvement, overall +1 for this. I'd leave a minor comment: 1. I'd suggest also providing `isInterruptable()` in `Mail`, and the continuation mail will return true. The FLIP-425 will leverage this queue to execute some state requests, and when the cp arrives, the

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-05 Thread Zakelly Lan
+1 (binding) Thanks for driving this! Best, Zakelly On Mon, May 6, 2024 at 10:54 AM yue ma wrote: > Hi everyone, > > Thanks for all the feedback, I'd like to start a vote on the FLIP-447: > Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. The discussion thread is here > [2]. > > The vote will be

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-05 Thread Zakelly Lan
Hi Piotr, Thanks for the proposal. It's meaningful to speed up the state download. I get into some questions: 1. What is the semantic of `canCopyPath`? Should it be associated with a specific destination path? e.g. It can be copied to local, but not to the remote FS. 2. Is the existing interface

Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-04-23 Thread Zakelly Lan
Hi Yue, Thanks for this proposal! Given the great improvement we could have, the slight regression in write performance is a worthwhile trade-off, particularly as the mem-table operations contribute only a minor part to the overall overhead. So +1 for this. Best, Zakelly On Tue, Apr 23, 2024

[jira] [Created] (FLINK-35186) Create State V2 from new StateDescriptor

2024-04-22 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35186: --- Summary: Create State V2 from new StateDescriptor Key: FLINK-35186 URL: https://issues.apache.org/jira/browse/FLINK-35186 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35168) Basic State Iterator for async processing

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35168: --- Summary: Basic State Iterator for async processing Key: FLINK-35168 URL: https://issues.apache.org/jira/browse/FLINK-35168 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35156) Wire new operators for async state with DataStream V2

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35156: --- Summary: Wire new operators for async state with DataStream V2 Key: FLINK-35156 URL: https://issues.apache.org/jira/browse/FLINK-35156 Project: Flink Issue

[jira] [Created] (FLINK-35153) Internal Async State Implementation and StateDescriptor for Map/List State

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35153: --- Summary: Internal Async State Implementation and StateDescriptor for Map/List State Key: FLINK-35153 URL: https://issues.apache.org/jira/browse/FLINK-35153 Project

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

2024-04-17 Thread Zakelly Lan
+1 binding Best, Zakelly On Wed, Apr 17, 2024 at 2:05 PM Rui Fan <1996fan...@gmail.com> wrote: > +1(binding) > > Best, > Rui > > On Wed, Apr 17, 2024 at 1:02 PM Xuannan Su wrote: > > > Hi everyone, > > > > Thanks for all the feedback about the FLIP-442: General Improvement to > >

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-16 Thread Zakelly Lan
, April 17, 2024 9:59 > To: dev@flink.apache.org > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan > > Congratulations, Zakelly! > > Best, > Shawn Huang > > > Feng Jin 于2024年4月17日周三 00:16写道: > > > Congratulations! > > > &

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-14 Thread Zakelly Lan
Congratulations, Jing! Best, Zakelly On Sat, Apr 13, 2024 at 12:47 AM Ferenc Csaky wrote: > Congratulations, Jing! > > Best, > Ferenc > > > > On Friday, April 12th, 2024 at 13:54, Ron liu wrote: > > > > > > > Congratulations, Jing! > > > > Best, > > Ron > > > > Junrui Lee jrlee@gmail.com

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-14 Thread Zakelly Lan
Congratulations, Lincoln! Best, Zakelly On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky wrote: > Congratulations, Lincoln! > > Best, > Ferenc > > > > > On Friday, April 12th, 2024 at 15:54, lorenzo.affe...@ververica.com.INVALID > wrote: > > > > > > > Huge congrats! Well done! > > On Apr 12,

Re: [VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

2024-04-11 Thread Zakelly Lan
+1 non-binding Best, Zakelly On Fri, Apr 12, 2024 at 11:05 AM Yuepeng Pan wrote: > Hi Rui, > > Thanks for driving it! > +1 (non-binding) > Best, > Yuepeng Pan > > At 2024-04-12 10:31:19, "Yanfei Lei" wrote: > >Hi Rui, > > > >Thanks for driving it! > > > >+1 (binding) > > > >Hangxiang Yu

Re: [DISCUSSION] FLIP-442: FLIP-442: General Improvement to Configuration for Flink 2.0

2024-04-09 Thread Zakelly Lan
Thanks Xuannan for driving this! +1 for cleaning these up. And minor comments: It seems the StateBackendOptions is already annotated with @PublicEvolving. Best, Zakelly On Tue, Apr 9, 2024 at 4:21 PM Xuannan Su wrote: > Hi all, > > I'd like to start a discussion on FLIP-442: General

Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

2024-04-09 Thread Zakelly Lan
Thanks Rui for driving this! +1 for this idea. Best, Zakelly On Mon, Apr 8, 2024 at 7:17 PM Ahmed Hamdy wrote: > Acknowledged, +1 to start a vote. > Best Regards > Ahmed Hamdy > > > On Mon, 8 Apr 2024 at 12:04, Rui Fan <1996fan...@gmail.com> wrote: > > > Sorry, it's a typo. It should be

[jira] [Created] (FLINK-35060) Provide compatibility of old CheckpointMode for connector testing framework

2024-04-09 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35060: --- Summary: Provide compatibility of old CheckpointMode for connector testing framework Key: FLINK-35060 URL: https://issues.apache.org/jira/browse/FLINK-35060 Project

[jira] [Created] (FLINK-34986) Basic framework for async execution of state

2024-04-01 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34986: --- Summary: Basic framework for async execution of state Key: FLINK-34986 URL: https://issues.apache.org/jira/browse/FLINK-34986 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-34978) Introduce Asynchronous State APIs

2024-03-31 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34978: --- Summary: Introduce Asynchronous State APIs Key: FLINK-34978 URL: https://issues.apache.org/jira/browse/FLINK-34978 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34979) Implement State Future and related utilities

2024-03-31 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34979: --- Summary: Implement State Future and related utilities Key: FLINK-34979 URL: https://issues.apache.org/jira/browse/FLINK-34979 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-34974) FLIP-424: Asynchronous State APIs

2024-03-31 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34974: --- Summary: FLIP-424: Asynchronous State APIs Key: FLINK-34974 URL: https://issues.apache.org/jira/browse/FLINK-34974 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-34973) FLIP-423: Disaggregated State Storage and Management

2024-03-31 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34973: --- Summary: FLIP-423: Disaggregated State Storage and Management Key: FLINK-34973 URL: https://issues.apache.org/jira/browse/FLINK-34973 Project: Flink Issue

[RESULT][VOTE] FLIP-424: Asynchronous State APIs

2024-03-31 Thread Zakelly Lan
Hi devs, I'm happy to announce that FLIP-424: Asynchronous State APIs [1] has been accepted with 11 approving votes, 7 of which are binding [2]: - Xuannan Su - Yuan Mei (binding) - Piotr Nowojski (binding) - Jing Ge (binding) - Feifan Wang - Rui Fan (binding) - Yunfeng Zhou - Yuepeng Pan -

Re: [VOTE] FLIP-424: Asynchronous State APIs

2024-03-31 Thread Zakelly Lan
g Pan > > > > > > > > > On 2024/03/29 03:03:53 Yunfeng Zhou wrote: > > > > +1 (non-binding) > > > > > > > > Best, > > > > Yunfeng > > > > > > > > On Wed, Mar 27, 2024 at 6:23 PM Zakelly Lan > > > wrote: &

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-29 Thread Zakelly Lan
te APIs > > Hi Zakelly, > > Thanks for your clarification. I'm +1 for using `onNext`. > > Best, > Jane > > On Tue, Mar 19, 2024 at 6:38 PM Zakelly Lan wrote: > > > Hi Jane, > > > > Thanks for your comments! > > > > I guess there is no problem with

Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

2024-03-28 Thread Zakelly Lan
Congratulations! Best, Zakelly On Thu, Mar 28, 2024 at 10:13 PM Jing Ge wrote: > Congrats! > > Best regards, > Jing > > On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang wrote: > > > Congratulations!—— > > > > Best regards, > > > > Feifan Wang > > > > > > > > > > At 2024-03-28

[VOTE] FLIP-424: Asynchronous State APIs

2024-03-27 Thread Zakelly Lan
Hi devs, I'd like to start a vote on the FLIP-424: Asynchronous State APIs [1]. The discussion thread is here [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1] https://cwiki.apache.org/confluence/x/SYp3EQ [2]

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-27 Thread Zakelly Lan
parallel in a follow up FLIP, we could discuss the > exact shape of the declarative async API that would be actually > checkpointable. My main concern here, is to prevent a situation where we > end up with duplicate code base of the operators: > - the current set of operators that a

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-26 Thread Zakelly Lan
Hi devs, It seems there is no more concern or suggestion for a while. We'd like to start voting recently. Best, Zakelly On Wed, Mar 27, 2024 at 11:46 AM Zakelly Lan wrote: > Hi everyone, > > Piotr and I had a long discussion about the checkpoint duration issue. We > think tha

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-26 Thread Zakelly Lan
another FLIP(s) describing the whole optimized checkpointing process, and in the meantime, we can proceed on current FLIPs. The new FLIP(s) are built on top of current ones and we can achieve this incrementally. Best, Zakelly On Thu, Mar 21, 2024 at 12:31 AM Zakelly Lan wrote: > Hi Piot

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-24 Thread Zakelly Lan
Hi Lorenzo, Thanks for your comments! I think you got my question, and I did not realize that is not even allowed > to modify some externally scoped variable in a lambda. > I guess the point is that it is possible, but the user would really need > to be willing to do it and "shoot him/herself in

Re: [DISCUSS] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

2024-03-22 Thread Zakelly Lan
Hi Yue, Thanks for bringing this up! The CURRENT FILE is the special one, which should be snapshot during the sync phase (temporary load into memory). Thus we can solve this. Best, Zakelly On Fri, Mar 22, 2024 at 4:55 PM yue ma wrote: > Hi jinzhong, > Thanks for you reply. I still have some

Re: [VOTE] FLIP-433: State Access on DataStream API V2

2024-03-21 Thread Zakelly Lan
+1 non-binding Best, Zakelly On Thu, Mar 21, 2024 at 5:34 PM Gyula Fóra wrote: > +1 (binding) > > Gyula > > On Thu, Mar 21, 2024 at 3:33 AM Rui Fan <1996fan...@gmail.com> wrote: > > > +1(binding) > > > > Thanks to Weijie for driving this proposal, which solves the problem > that I > > raised

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Zakelly Lan
Congratulations! Best, Zakelly On Thu, Mar 21, 2024 at 12:05 PM weijie guo wrote: > Congratulations! Well done. > > > Best regards, > > Weijie > > > Feng Jin 于2024年3月21日周四 11:40写道: > >> Congratulations! >> >> >> Best, >> Feng >> >> >> On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: >> >> >

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-19 Thread Zakelly Lan
nc: This name clearly states that the function is an asynchronous > > version of the typical hasNext method found in synchronous iterators. > > nextExists: This name suggests the method checks for the existence of a > > next item, without the potential confusion of event handle

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-19 Thread Zakelly Lan
:35 PM Zakelly Lan wrote: > Hi Yue, > > Thanks for your comments! > > 1. Is it possible for all `FutureUtils` in Flink to reuse the same util >> class? > > Actually, the `FutureUtils` here is a new util class that will share the > same package path with the `StateFu

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-19 Thread Zakelly Lan
Hi everyone, Thanks for your valuable feedback! Our discussions have been going on for a while and are nearing a consensus. So I would like to start a vote after 72 hours. Please let me know if you have any concerns, thanks! Best, Zakelly On Tue, Mar 19, 2024 at 3:37 PM Zakelly Lan wrote

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-19 Thread Zakelly Lan
3日周三 20:00写道: > > > indeed! I missed that part. Thanks for the hint! > > > > Best regards, > > Jing > > > > On Wed, Mar 13, 2024 at 6:02 AM Zakelly Lan > wrote: > > > > > Hi Jing, > > > > > > The deprecation and rem

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-19 Thread Zakelly Lan
ck response. Sounds good to me. > > > > Best, > > > > Xintong > > > > > > > > On Tue, Mar 19, 2024 at 1:03 PM Zakelly Lan > wrote: > > > > > Hi Xintong, > > > > > > Thanks for sharing your thoughts! > >

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-18 Thread Zakelly Lan
he.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864FLIP-425 > . > > > > NIT fix: > > > > FLIP-424: > https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864 > > > > FLIP-425: > https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h > &g

Re: [ANNOUNCE] Apache Flink 1.19.0 released

2024-03-18 Thread Zakelly Lan
Congratulations! Thanks Lincoln, Yun, Martijn and Jing for driving this release. Thanks everyone involved. Best, Zakelly On Mon, Mar 18, 2024 at 5:05 PM weijie guo wrote: > Congratulations! > > Thanks release managers and all the contributors involved. > > Best regards, > > Weijie > > >

Re: Unaligned checkpoint blocked by long Async operation

2024-03-17 Thread Zakelly Lan
> > Gyula > > On Fri, Mar 15, 2024 at 3:49 AM Zakelly Lan wrote: > >> Hi Gyula, >> >> Processing checkpoint halfway through `processElement` is problematic. >> The current element will not be included in the input in-flight data, and >> we cannot assum

[jira] [Created] (FLINK-34704) Process checkpoint barrier in AsyncWaitOperator when the element queue is full

2024-03-17 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34704: --- Summary: Process checkpoint barrier in AsyncWaitOperator when the element queue is full Key: FLINK-34704 URL: https://issues.apache.org/jira/browse/FLINK-34704 Project

[jira] [Created] (FLINK-34668) Report State handle of file merging directory to JM

2024-03-14 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34668: --- Summary: Report State handle of file merging directory to JM Key: FLINK-34668 URL: https://issues.apache.org/jira/browse/FLINK-34668 Project: Flink Issue Type

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-12 Thread Zakelly Lan
y, > > Thanks for your responses. I agree with it that we can keep the design > as it is for now and see if others have any better ideas for these > questions. > > Best, > Yunfeng > > On Tue, Mar 12, 2024 at 5:23 PM Zakelly Lan wrote: > > > > Hi Xuannan, &

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

2024-03-12 Thread Zakelly Lan
may use which could be > >>> explained by the caller, right ? > >>> > >>> On Mon, Mar 11, 2024 at 10:57 PM Maximilian Michels > >>> wrote: > >>> > >>> > The FLIP mentions: "The contents described in this FLIP are all n

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-12 Thread Zakelly Lan
d in > future, so I still suggest we improve the package name now if > possible. But given the existing practice of sink v2 and > AbstractStreamOperatorV2, the current package name would be acceptable > to me if other reviewers of this FLIP agrees on it. > > Best, > Yunfeng > > On Mon, Mar 11, 2024

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
sses and > their methods. Yes, this will be added in implementation, I just omitted them for easy reading. Thanks & Best, Zakelly On Mon, Mar 11, 2024 at 5:25 PM Zakelly Lan wrote: > Hi Yunfeng, > > Thanks for your comments! > > +1 for JingGe's suggestion to introduce an

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
from both sides comply with a unified naming > style. If we reach an agreement on the first comment, my personal idea > is that we can place the AsyncState interfaces to > "org.apache.flink.api.common.state.async", and the existing state APIs > to "org.apache.flink.api.common.state&qu

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
package by mistake etc.) and > ease the job development with state? > > Best regards, > Jing > > > [1] > > https://github.com/apache/flink/blob/d6a4eb966fbc47277e07b79e7c64939a62eb1d54/flink-architecture-tests/flink-architecture-tests-production/src/main/java/org/apache/f

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

2024-03-07 Thread Zakelly Lan
> >> > > > >> We actually define three types RedistributionMode instead of two > > because > > > >> we don't want to think of IDENTICAL as a redistribution strategy, > it's > > > just > > > >> an invariant: the State of that type

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-07 Thread Zakelly Lan
写道: > > > > > > > > > > Hi Zakelly, > > > > > > > > > > > 5. I'm not very sure ... revisiting this later since it is not > > > > important. > > > > > > > > > > It seems that we still have some details

  1   2   3   >