Re: [VOTE] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Zakelly Lan
+1 (binding) Best, Zakelly On Thu, May 23, 2024 at 8:21 PM Piotr Nowojski wrote: > Hi all, > > After reaching what looks like a consensus in the discussion thread [1], I > would like to put FLIP-443 [2] to the vote. > > The vote will be open for at least 72 hours unless there is an objection

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Zakelly Lan
ng operators (and > interruptible mails), so I will just add `MailOptions` parameters only to > them. If necessary, we can add more in the future. > > I have updated the FLIP. If it looks good to you, I would start a voting > thread today/tomorrow. > > Best, > Piotrek > > czw.

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-23 Thread Zakelly Lan
a builder? > > > mailboxExecutor.execute(myThrowingRunnable).setInterriptuble().description("bla > %d").arg(42).submit(); > > It could be done in both in the future, if we would ever need to add even > more methods, or I could do it now. WDYT? > > Best, > Piotrek > > ś

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-22 Thread Zakelly Lan
iotrek > > > > > > [1] http://flink-speed.xyz/timeline/?ben=fireProcessingTimers=3 > > > > > > > > > > > > wt., 7 maj 2024 o 09:47 Rui Fan <1996fan...@gmail.com> napisał(a): > > > > > > > Hi Piotr, > > > >

Re: [DISCUSS] The performance of serializerHeavyString starts regress since April 3

2024-05-21 Thread Zakelly Lan
Hi Rui and RMs of Flink 1.20, Thanks for driving this! Available information indicates this issue is environment- and JDK-specific, and I also failed to reproduce it in my Mac. Thus I guess it is caused by JIT behavior, which is unpredictable and vulnerable to disturbance of the codebase.

[jira] [Created] (FLINK-35412) Batch execution of async state request callback

2024-05-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35412: --- Summary: Batch execution of async state request callback Key: FLINK-35412 URL: https://issues.apache.org/jira/browse/FLINK-35412 Project: Flink Issue Type

[jira] [Created] (FLINK-35411) Optimize wait logic in draining of async state requests

2024-05-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35411: --- Summary: Optimize wait logic in draining of async state requests Key: FLINK-35411 URL: https://issues.apache.org/jira/browse/FLINK-35411 Project: Flink Issue

[jira] [Created] (FLINK-35410) Avoid sync waiting in coordinator thread of ForSt executor

2024-05-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35410: --- Summary: Avoid sync waiting in coordinator thread of ForSt executor Key: FLINK-35410 URL: https://issues.apache.org/jira/browse/FLINK-35410 Project: Flink

[jira] [Created] (FLINK-35400) Rebuild FileMergingSnapshotManager in failover

2024-05-20 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35400: --- Summary: Rebuild FileMergingSnapshotManager in failover Key: FLINK-35400 URL: https://issues.apache.org/jira/browse/FLINK-35400 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-35379) File merging manager is not properly notified about checkpoint

2024-05-16 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35379: --- Summary: File merging manager is not properly notified about checkpoint Key: FLINK-35379 URL: https://issues.apache.org/jira/browse/FLINK-35379 Project: Flink

[jira] [Created] (FLINK-35356) Async reducing state

2024-05-14 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35356: --- Summary: Async reducing state Key: FLINK-35356 URL: https://issues.apache.org/jira/browse/FLINK-35356 Project: Flink Issue Type: Sub-task Components

[jira] [Created] (FLINK-35355) Async aggregating state

2024-05-14 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35355: --- Summary: Async aggregating state Key: FLINK-35355 URL: https://issues.apache.org/jira/browse/FLINK-35355 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35307) Add Compile CI check on jdk17

2024-05-07 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35307: --- Summary: Add Compile CI check on jdk17 Key: FLINK-35307 URL: https://issues.apache.org/jira/browse/FLINK-35307 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-06 Thread Zakelly Lan
l file from a remote location > > Maybe indeed that's a better solution vs having two separate interfaces for > copying and duplicating? > > > 3. Will the interface extracting introduce a break change? > > No. The signature of the existing abstract `FileSystem` class would remain >

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-06 Thread Zakelly Lan
th something? If an incoming async state access > response comes, it could just request to interrupt any currently ongoing > computations, regardless the currently executed mail is or is not > interruptible. > > Best, > Piotrek > > pon., 6 maj 2024 o 06:33 Zakelly Lan napisa

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-05 Thread Zakelly Lan
Hi Piotr, Thanks for the improvement, overall +1 for this. I'd leave a minor comment: 1. I'd suggest also providing `isInterruptable()` in `Mail`, and the continuation mail will return true. The FLIP-425 will leverage this queue to execute some state requests, and when the cp arrives, the

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-05-05 Thread Zakelly Lan
+1 (binding) Thanks for driving this! Best, Zakelly On Mon, May 6, 2024 at 10:54 AM yue ma wrote: > Hi everyone, > > Thanks for all the feedback, I'd like to start a vote on the FLIP-447: > Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. The discussion thread is here > [2]. > > The vote will be

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-05 Thread Zakelly Lan
Hi Piotr, Thanks for the proposal. It's meaningful to speed up the state download. I get into some questions: 1. What is the semantic of `canCopyPath`? Should it be associated with a specific destination path? e.g. It can be copied to local, but not to the remote FS. 2. Is the existing interface

Re: [DISCUSS] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

2024-04-23 Thread Zakelly Lan
Hi Yue, Thanks for this proposal! Given the great improvement we could have, the slight regression in write performance is a worthwhile trade-off, particularly as the mem-table operations contribute only a minor part to the overall overhead. So +1 for this. Best, Zakelly On Tue, Apr 23, 2024

[jira] [Created] (FLINK-35186) Create State V2 from new StateDescriptor

2024-04-22 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35186: --- Summary: Create State V2 from new StateDescriptor Key: FLINK-35186 URL: https://issues.apache.org/jira/browse/FLINK-35186 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35168) Basic State Iterator for async processing

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35168: --- Summary: Basic State Iterator for async processing Key: FLINK-35168 URL: https://issues.apache.org/jira/browse/FLINK-35168 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-35156) Wire new operators for async state with DataStream V2

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35156: --- Summary: Wire new operators for async state with DataStream V2 Key: FLINK-35156 URL: https://issues.apache.org/jira/browse/FLINK-35156 Project: Flink Issue

[jira] [Created] (FLINK-35153) Internal Async State Implementation and StateDescriptor for Map/List State

2024-04-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35153: --- Summary: Internal Async State Implementation and StateDescriptor for Map/List State Key: FLINK-35153 URL: https://issues.apache.org/jira/browse/FLINK-35153 Project

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

2024-04-17 Thread Zakelly Lan
+1 binding Best, Zakelly On Wed, Apr 17, 2024 at 2:05 PM Rui Fan <1996fan...@gmail.com> wrote: > +1(binding) > > Best, > Rui > > On Wed, Apr 17, 2024 at 1:02 PM Xuannan Su wrote: > > > Hi everyone, > > > > Thanks for all the feedback about the FLIP-442: General Improvement to > >

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

2024-04-16 Thread Zakelly Lan
, April 17, 2024 9:59 > To: dev@flink.apache.org > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan > > Congratulations, Zakelly! > > Best, > Shawn Huang > > > Feng Jin 于2024年4月17日周三 00:16写道: > > > Congratulations! > > > &

Re: [ANNOUNCE] New Apache Flink PMC Member - Jing Ge

2024-04-14 Thread Zakelly Lan
Congratulations, Jing! Best, Zakelly On Sat, Apr 13, 2024 at 12:47 AM Ferenc Csaky wrote: > Congratulations, Jing! > > Best, > Ferenc > > > > On Friday, April 12th, 2024 at 13:54, Ron liu wrote: > > > > > > > Congratulations, Jing! > > > > Best, > > Ron > > > > Junrui Lee jrlee@gmail.com

Re: [ANNOUNCE] New Apache Flink PMC Member - Lincoln Lee

2024-04-14 Thread Zakelly Lan
Congratulations, Lincoln! Best, Zakelly On Sat, Apr 13, 2024 at 12:48 AM Ferenc Csaky wrote: > Congratulations, Lincoln! > > Best, > Ferenc > > > > > On Friday, April 12th, 2024 at 15:54, lorenzo.affe...@ververica.com.INVALID > wrote: > > > > > > > Huge congrats! Well done! > > On Apr 12,

Re: [VOTE] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

2024-04-11 Thread Zakelly Lan
+1 non-binding Best, Zakelly On Fri, Apr 12, 2024 at 11:05 AM Yuepeng Pan wrote: > Hi Rui, > > Thanks for driving it! > +1 (non-binding) > Best, > Yuepeng Pan > > At 2024-04-12 10:31:19, "Yanfei Lei" wrote: > >Hi Rui, > > > >Thanks for driving it! > > > >+1 (binding) > > > >Hangxiang Yu

Re: [DISCUSSION] FLIP-442: FLIP-442: General Improvement to Configuration for Flink 2.0

2024-04-09 Thread Zakelly Lan
Thanks Xuannan for driving this! +1 for cleaning these up. And minor comments: It seems the StateBackendOptions is already annotated with @PublicEvolving. Best, Zakelly On Tue, Apr 9, 2024 at 4:21 PM Xuannan Su wrote: > Hi all, > > I'd like to start a discussion on FLIP-442: General

Re: [DISCUSS] FLIP-441: Show the JobType and remove Execution Mode on Flink WebUI

2024-04-09 Thread Zakelly Lan
Thanks Rui for driving this! +1 for this idea. Best, Zakelly On Mon, Apr 8, 2024 at 7:17 PM Ahmed Hamdy wrote: > Acknowledged, +1 to start a vote. > Best Regards > Ahmed Hamdy > > > On Mon, 8 Apr 2024 at 12:04, Rui Fan <1996fan...@gmail.com> wrote: > > > Sorry, it's a typo. It should be

[jira] [Created] (FLINK-35060) Provide compatibility of old CheckpointMode for connector testing framework

2024-04-09 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35060: --- Summary: Provide compatibility of old CheckpointMode for connector testing framework Key: FLINK-35060 URL: https://issues.apache.org/jira/browse/FLINK-35060 Project

[jira] [Created] (FLINK-34986) Basic framework for async execution of state

2024-04-01 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34986: --- Summary: Basic framework for async execution of state Key: FLINK-34986 URL: https://issues.apache.org/jira/browse/FLINK-34986 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-34978) Introduce Asynchronous State APIs

2024-03-31 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34978: --- Summary: Introduce Asynchronous State APIs Key: FLINK-34978 URL: https://issues.apache.org/jira/browse/FLINK-34978 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34979) Implement State Future and related utilities

2024-03-31 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34979: --- Summary: Implement State Future and related utilities Key: FLINK-34979 URL: https://issues.apache.org/jira/browse/FLINK-34979 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-34974) FLIP-424: Asynchronous State APIs

2024-03-31 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34974: --- Summary: FLIP-424: Asynchronous State APIs Key: FLINK-34974 URL: https://issues.apache.org/jira/browse/FLINK-34974 Project: Flink Issue Type: New Feature

[jira] [Created] (FLINK-34973) FLIP-423: Disaggregated State Storage and Management

2024-03-31 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34973: --- Summary: FLIP-423: Disaggregated State Storage and Management Key: FLINK-34973 URL: https://issues.apache.org/jira/browse/FLINK-34973 Project: Flink Issue

[RESULT][VOTE] FLIP-424: Asynchronous State APIs

2024-03-31 Thread Zakelly Lan
Hi devs, I'm happy to announce that FLIP-424: Asynchronous State APIs [1] has been accepted with 11 approving votes, 7 of which are binding [2]: - Xuannan Su - Yuan Mei (binding) - Piotr Nowojski (binding) - Jing Ge (binding) - Feifan Wang - Rui Fan (binding) - Yunfeng Zhou - Yuepeng Pan -

Re: [VOTE] FLIP-424: Asynchronous State APIs

2024-03-31 Thread Zakelly Lan
g Pan > > > > > > > > > On 2024/03/29 03:03:53 Yunfeng Zhou wrote: > > > > +1 (non-binding) > > > > > > > > Best, > > > > Yunfeng > > > > > > > > On Wed, Mar 27, 2024 at 6:23 PM Zakelly Lan > > > wrote: &

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-29 Thread Zakelly Lan
te APIs > > Hi Zakelly, > > Thanks for your clarification. I'm +1 for using `onNext`. > > Best, > Jane > > On Tue, Mar 19, 2024 at 6:38 PM Zakelly Lan wrote: > > > Hi Jane, > > > > Thanks for your comments! > > > > I guess there is no problem with

Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

2024-03-28 Thread Zakelly Lan
Congratulations! Best, Zakelly On Thu, Mar 28, 2024 at 10:13 PM Jing Ge wrote: > Congrats! > > Best regards, > Jing > > On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang wrote: > > > Congratulations!—— > > > > Best regards, > > > > Feifan Wang > > > > > > > > > > At 2024-03-28

[VOTE] FLIP-424: Asynchronous State APIs

2024-03-27 Thread Zakelly Lan
Hi devs, I'd like to start a vote on the FLIP-424: Asynchronous State APIs [1]. The discussion thread is here [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1] https://cwiki.apache.org/confluence/x/SYp3EQ [2]

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-27 Thread Zakelly Lan
parallel in a follow up FLIP, we could discuss the > exact shape of the declarative async API that would be actually > checkpointable. My main concern here, is to prevent a situation where we > end up with duplicate code base of the operators: > - the current set of operators that a

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-26 Thread Zakelly Lan
Hi devs, It seems there is no more concern or suggestion for a while. We'd like to start voting recently. Best, Zakelly On Wed, Mar 27, 2024 at 11:46 AM Zakelly Lan wrote: > Hi everyone, > > Piotr and I had a long discussion about the checkpoint duration issue. We > think tha

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-26 Thread Zakelly Lan
another FLIP(s) describing the whole optimized checkpointing process, and in the meantime, we can proceed on current FLIPs. The new FLIP(s) are built on top of current ones and we can achieve this incrementally. Best, Zakelly On Thu, Mar 21, 2024 at 12:31 AM Zakelly Lan wrote: > Hi Piot

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

2024-03-24 Thread Zakelly Lan
Hi Lorenzo, Thanks for your comments! I think you got my question, and I did not realize that is not even allowed > to modify some externally scoped variable in a lambda. > I guess the point is that it is possible, but the user would really need > to be willing to do it and "shoot him/herself in

Re: [DISCUSS] FLIP-428: Fault Tolerance/Rescale Integration for Disaggregated State

2024-03-22 Thread Zakelly Lan
Hi Yue, Thanks for bringing this up! The CURRENT FILE is the special one, which should be snapshot during the sync phase (temporary load into memory). Thus we can solve this. Best, Zakelly On Fri, Mar 22, 2024 at 4:55 PM yue ma wrote: > Hi jinzhong, > Thanks for you reply. I still have some

Re: [VOTE] FLIP-433: State Access on DataStream API V2

2024-03-21 Thread Zakelly Lan
+1 non-binding Best, Zakelly On Thu, Mar 21, 2024 at 5:34 PM Gyula Fóra wrote: > +1 (binding) > > Gyula > > On Thu, Mar 21, 2024 at 3:33 AM Rui Fan <1996fan...@gmail.com> wrote: > > > +1(binding) > > > > Thanks to Weijie for driving this proposal, which solves the problem > that I > > raised

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Zakelly Lan
Congratulations! Best, Zakelly On Thu, Mar 21, 2024 at 12:05 PM weijie guo wrote: > Congratulations! Well done. > > > Best regards, > > Weijie > > > Feng Jin 于2024年3月21日周四 11:40写道: > >> Congratulations! >> >> >> Best, >> Feng >> >> >> On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: >> >> >

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-19 Thread Zakelly Lan
nc: This name clearly states that the function is an asynchronous > > version of the typical hasNext method found in synchronous iterators. > > nextExists: This name suggests the method checks for the existence of a > > next item, without the potential confusion of event handle

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-19 Thread Zakelly Lan
:35 PM Zakelly Lan wrote: > Hi Yue, > > Thanks for your comments! > > 1. Is it possible for all `FutureUtils` in Flink to reuse the same util >> class? > > Actually, the `FutureUtils` here is a new util class that will share the > same package path with the `StateFu

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-19 Thread Zakelly Lan
Hi everyone, Thanks for your valuable feedback! Our discussions have been going on for a while and are nearing a consensus. So I would like to start a vote after 72 hours. Please let me know if you have any concerns, thanks! Best, Zakelly On Tue, Mar 19, 2024 at 3:37 PM Zakelly Lan wrote

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-19 Thread Zakelly Lan
3日周三 20:00写道: > > > indeed! I missed that part. Thanks for the hint! > > > > Best regards, > > Jing > > > > On Wed, Mar 13, 2024 at 6:02 AM Zakelly Lan > wrote: > > > > > Hi Jing, > > > > > > The deprecation and rem

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-19 Thread Zakelly Lan
ck response. Sounds good to me. > > > > Best, > > > > Xintong > > > > > > > > On Tue, Mar 19, 2024 at 1:03 PM Zakelly Lan > wrote: > > > > > Hi Xintong, > > > > > > Thanks for sharing your thoughts! > >

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-18 Thread Zakelly Lan
he.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864FLIP-425 > . > > > > NIT fix: > > > > FLIP-424: > https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864 > > > > FLIP-425: > https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h > &g

Re: [ANNOUNCE] Apache Flink 1.19.0 released

2024-03-18 Thread Zakelly Lan
Congratulations! Thanks Lincoln, Yun, Martijn and Jing for driving this release. Thanks everyone involved. Best, Zakelly On Mon, Mar 18, 2024 at 5:05 PM weijie guo wrote: > Congratulations! > > Thanks release managers and all the contributors involved. > > Best regards, > > Weijie > > >

Re: Unaligned checkpoint blocked by long Async operation

2024-03-17 Thread Zakelly Lan
> > Gyula > > On Fri, Mar 15, 2024 at 3:49 AM Zakelly Lan wrote: > >> Hi Gyula, >> >> Processing checkpoint halfway through `processElement` is problematic. >> The current element will not be included in the input in-flight data, and >> we cannot assum

[jira] [Created] (FLINK-34704) Process checkpoint barrier in AsyncWaitOperator when the element queue is full

2024-03-17 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34704: --- Summary: Process checkpoint barrier in AsyncWaitOperator when the element queue is full Key: FLINK-34704 URL: https://issues.apache.org/jira/browse/FLINK-34704 Project

[jira] [Created] (FLINK-34668) Report State handle of file merging directory to JM

2024-03-14 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34668: --- Summary: Report State handle of file merging directory to JM Key: FLINK-34668 URL: https://issues.apache.org/jira/browse/FLINK-34668 Project: Flink Issue Type

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-12 Thread Zakelly Lan
y, > > Thanks for your responses. I agree with it that we can keep the design > as it is for now and see if others have any better ideas for these > questions. > > Best, > Yunfeng > > On Tue, Mar 12, 2024 at 5:23 PM Zakelly Lan wrote: > > > > Hi Xuannan, &

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

2024-03-12 Thread Zakelly Lan
may use which could be > >>> explained by the caller, right ? > >>> > >>> On Mon, Mar 11, 2024 at 10:57 PM Maximilian Michels > >>> wrote: > >>> > >>> > The FLIP mentions: "The contents described in this FLIP are all n

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-12 Thread Zakelly Lan
d in > future, so I still suggest we improve the package name now if > possible. But given the existing practice of sink v2 and > AbstractStreamOperatorV2, the current package name would be acceptable > to me if other reviewers of this FLIP agrees on it. > > Best, > Yunfeng > > On Mon, Mar 11, 2024

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
sses and > their methods. Yes, this will be added in implementation, I just omitted them for easy reading. Thanks & Best, Zakelly On Mon, Mar 11, 2024 at 5:25 PM Zakelly Lan wrote: > Hi Yunfeng, > > Thanks for your comments! > > +1 for JingGe's suggestion to introduce an

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
from both sides comply with a unified naming > style. If we reach an agreement on the first comment, my personal idea > is that we can place the AsyncState interfaces to > "org.apache.flink.api.common.state.async", and the existing state APIs > to "org.apache.flink.api.common.state&qu

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-11 Thread Zakelly Lan
package by mistake etc.) and > ease the job development with state? > > Best regards, > Jing > > > [1] > > https://github.com/apache/flink/blob/d6a4eb966fbc47277e07b79e7c64939a62eb1d54/flink-architecture-tests/flink-architecture-tests-production/src/main/java/org/apache/f

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

2024-03-07 Thread Zakelly Lan
> >> > > > >> We actually define three types RedistributionMode instead of two > > because > > > >> we don't want to think of IDENTICAL as a redistribution strategy, > it's > > > just > > > >> an invariant: the State of that type

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-07 Thread Zakelly Lan
写道: > > > > > > > > > > Hi Zakelly, > > > > > > > > > > > 5. I'm not very sure ... revisiting this later since it is not > > > > important. > > > > > > > > > > It seems that we still have some details

[DISCUSS] FLIP-424: Asynchronous State APIs

2024-03-07 Thread Zakelly Lan
Hi devs, I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated State Storage and Management[1], which is a joint work of Yuan Mei, Zakelly Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang: - FLIP-424: Asynchronous State APIs [2] This FLIP introduces new APIs

Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

2024-03-06 Thread Zakelly Lan
+1 non-binding Thanks for proposing this. Best, Zakelly On Thu, Mar 7, 2024 at 10:13 AM Yanfei Lei wrote: > +1(binding) for this vote. > > Hangxiang Yu 于2024年3月7日周四 09:54写道: > > > > +1 (binding) > > > > On Thu, Mar 7, 2024 at 9:34 AM Yun Tang wrote: > > > > > > +1 for this FLIP. > > > Sorry

Re: [DISCUSS] FLIP-433: State Access on DataStream API V2

2024-03-06 Thread Zakelly Lan
Hi Weijie, Thanks for proposing this! Unifying and optimizing state definitions is a very good thing. I like the idea of 'definition goes before using', so overall +1 for this proposal. However, I think the current definition is somewhat unclear. From a user's point of view, I believe that

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-04 Thread Zakelly Lan
s FLIP introduce metrics that measure the time a Flink > job is back-pressured by State IOs? Under the current design, this > metric could measure the time when the blocking buffer is full and > yield() cannot get callbacks to process, which means the operator is > fully waiting for

Re: [DISCUSS] Move CheckpointingMode to flink-core

2024-03-04 Thread Zakelly Lan
4 at 6:16 PM Junrui Lee wrote: > > > Hi Zakelly, > > > > +1 for option 1. I prefer to minimize unnecessary additional development > > and discussions due to internal code relocations and to avoid imposing > > migration costs on users. > > > > Best regards, &

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-03 Thread Zakelly Lan
onceals many underlying details, so I guess the file consistency of different DFS is not a big thing in our implementation. Of course, there may be some optimization when dealing with different DFS, we may achieve this later. Thanks again & Best, Zakelly On Mon, Mar 4, 2024 at 12:33 PM Zakelly L

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-03 Thread Zakelly Lan
, watermark > strategy, etc) > > - DFS consistency guarantees. The proposal in FLIP-427 is DFS-agnostic. > However, different cloud providers have different storage consistency > models. > How do we want to deal with them? > > Regards, > Jeyhun > > > > > On Fri

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-03-01 Thread Zakelly Lan
alysis. > > 6. Why do we need to record the hashcode of a record in its > RecordContext? It seems not used. > > 7. In "timers can be stored on the JVM heap or RocksDB", the link > points to a document in flink-1.15. It might be better to verify the > referenced

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-02-29 Thread Zakelly Lan
as > "available" for sync access if it's in local disks (not RAM) is probably > good enough (comparable to RocksDB). > > Best, > Piotrek > > czw., 29 lut 2024 o 07:17 Yuan Mei napisał(a): > > > Hi Devs, > > > > This is a joint work of Yuan Mei

Re: [DISCUSS] Move CheckpointingMode to flink-core

2024-02-27 Thread Zakelly Lan
this class to flink-core makes sense to me which could make the > > code > > >> path and configs clearer. > > >> It's marked as @Public from 1.0 and 1.20 should be the next long-term > > >> version, so 1.19 should have been a suitable version to do it. > &

[DISCUSS] Move CheckpointingMode to flink-core

2024-02-26 Thread Zakelly Lan
Hi devs, When working on the FLIP-406[1], I realized that moving all options of ExecutionCheckpointingOptions(flink-streaming-java) to CheckpointingOptions(flink-core) depends on relocating the enum CheckpointingMode(flink-streaming-java) to flink-core module. However, the CheckpointingMode is

[jira] [Created] (FLINK-34516) Move CheckpointingMode to flink-core

2024-02-26 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34516: --- Summary: Move CheckpointingMode to flink-core Key: FLINK-34516 URL: https://issues.apache.org/jira/browse/FLINK-34516 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34511) Flink 2.0: Deprecate legacy State options

2024-02-25 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34511: --- Summary: Flink 2.0: Deprecate legacy State options Key: FLINK-34511 URL: https://issues.apache.org/jira/browse/FLINK-34511 Project: Flink Issue Type: Sub

[jira] [Created] (FLINK-34510) Flink 2.0: Rename RestoreMode to RecoveryClaimMode

2024-02-25 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34510: --- Summary: Flink 2.0: Rename RestoreMode to RecoveryClaimMode Key: FLINK-34510 URL: https://issues.apache.org/jira/browse/FLINK-34510 Project: Flink Issue Type

[jira] [Created] (FLINK-34484) Split 'state.backend.local-recovery' into two options for checkpointing and recovery

2024-02-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34484: --- Summary: Split 'state.backend.local-recovery' into two options for checkpointing and recovery Key: FLINK-34484 URL: https://issues.apache.org/jira/browse/FLINK-34484

[jira] [Created] (FLINK-34483) Merge 'state.checkpoints.dir' and 'state.checkpoints-storage' into one

2024-02-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34483: --- Summary: Merge 'state.checkpoints.dir' and 'state.checkpoints-storage' into one Key: FLINK-34483 URL: https://issues.apache.org/jira/browse/FLINK-34483 Project: Flink

[jira] [Created] (FLINK-34482) Rename options for checkpointing

2024-02-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34482: --- Summary: Rename options for checkpointing Key: FLINK-34482 URL: https://issues.apache.org/jira/browse/FLINK-34482 Project: Flink Issue Type: Sub-task

Re: [ANNOUNCE] New Apache Flink Committer - Jiabao Sun

2024-02-19 Thread Zakelly Lan
Congratulations, Jiabao! Best, Zakelly On Mon, Feb 19, 2024 at 9:40 PM Jing Ge wrote: > Congrats! Jiabao! > > Best regards, > Jing > > On Mon, Feb 19, 2024 at 2:38 PM Jeyhun Karimov > wrote: > > > Congratulations, Jiabao! > > Well deserved! > > > > On Mon, Feb 19, 2024 at 2:26 PM

[jira] [Created] (FLINK-34458) Rename options for Generalized incremental checkpoints (changelog)

2024-02-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34458: --- Summary: Rename options for Generalized incremental checkpoints (changelog) Key: FLINK-34458 URL: https://issues.apache.org/jira/browse/FLINK-34458 Project: Flink

[jira] [Created] (FLINK-34457) Rename options for latency tracking

2024-02-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34457: --- Summary: Rename options for latency tracking Key: FLINK-34457 URL: https://issues.apache.org/jira/browse/FLINK-34457 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34456) Move all checkpoint-related options into CheckpointingOptions

2024-02-18 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34456: --- Summary: Move all checkpoint-related options into CheckpointingOptions Key: FLINK-34456 URL: https://issues.apache.org/jira/browse/FLINK-34456 Project: Flink

[jira] [Created] (FLINK-34455) Move RestoreMode from flink-runtime to flink-core

2024-02-17 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34455: --- Summary: Move RestoreMode from flink-runtime to flink-core Key: FLINK-34455 URL: https://issues.apache.org/jira/browse/FLINK-34455 Project: Flink Issue Type

[jira] [Created] (FLINK-34454) Introduce RecoveryOptions in flink-core

2024-02-17 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34454: --- Summary: Introduce RecoveryOptions in flink-core Key: FLINK-34454 URL: https://issues.apache.org/jira/browse/FLINK-34454 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34453) [Benchmark] Add warmup in SchedulerBenchmarkExecutor

2024-02-17 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34453: --- Summary: [Benchmark] Add warmup in SchedulerBenchmarkExecutor Key: FLINK-34453 URL: https://issues.apache.org/jira/browse/FLINK-34453 Project: Flink Issue

[jira] [Created] (FLINK-34255) FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-28 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34255: --- Summary: FLIP-406: Reorganize State & Checkpointing & Recovery Configuration Key: FLINK-34255 URL: https://issues.apache.org/jira/browse/FLINK-34255 Projec

[RESULT][VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-28 Thread Zakelly Lan
Hi devs, I'm glad to announce that the FLIP-406[1] has been accepted. The voting thread is here[2]. The proposal received 6 approving votes, 5 of which are binding: - Rui Fan (binding) - Hangxiang Yu (binding) - Yanfei Lei (binding) - Lijie Wang (binding) - Xuannan Su (non-binding) - Yuan

Re: [VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-28 Thread Zakelly Lan
> +1 (binding) > > > > > > > > Hangxiang Yu 于2024年1月25日周四 10:00写道: > > > > > > > > > > +1 (binding) > > > > > > > > > > On Thu, Jan 25, 2024 at 8:49 AM Rui Fan <1996fan...@gmail.com> > > wrote: >

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

2024-01-24 Thread Zakelly Lan
Hi Jinzhong, Thanks for driving this! +1 for fixing the lack of annotation. I'm wondering if we really need to annotate *RocksDBStateUploader* and *RocksDBStateDownloader *with @Internal, as they seem to be ordinary classes without interacting with other modules. Also, I have reservations about

[VOTE] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-24 Thread Zakelly Lan
Hi everyone, I'd like to start a vote on the FLIP-406: Reorganize State & Checkpointing & Recovery Configuration [1]. The discussion thread is here [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1]

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-23 Thread Zakelly Lan
/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process Best, Zakelly On Mon, Jan 22, 2024 at 6:31 PM Zakelly Lan wrote: > Hi everyone, > > It has been 6 days since the last call for discussion. I'd like to start a > vote after another 2 days. > > Please let me kn

Re: [DISCUSS] FLIP-406: Reorganize State & Checkpointing & Recovery Configuration

2024-01-22 Thread Zakelly Lan
Hi everyone, It has been 6 days since the last call for discussion. I'd like to start a vote after another 2 days. Please let me know if you have any concerns. Thanks! Best, Zakelly On Tue, Jan 16, 2024 at 2:54 PM Zakelly Lan wrote: > Thanks for the suggestion Rui! The type is ad

[jira] [Created] (FLINK-34191) Remove RestoreMode#LEGACY

2024-01-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34191: --- Summary: Remove RestoreMode#LEGACY Key: FLINK-34191 URL: https://issues.apache.org/jira/browse/FLINK-34191 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34190) Deprecate RestoreMode#LEGACY

2024-01-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34190: --- Summary: Deprecate RestoreMode#LEGACY Key: FLINK-34190 URL: https://issues.apache.org/jira/browse/FLINK-34190 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34189) FLIP-416: Deprecate and remove the RestoreMode#LEGACY

2024-01-21 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-34189: --- Summary: FLIP-416: Deprecate and remove the RestoreMode#LEGACY Key: FLINK-34189 URL: https://issues.apache.org/jira/browse/FLINK-34189 Project: Flink Issue

  1   2   3   >