Hi Devs,
This is a joint work of Yuan Mei, Zakelly Lan, Jinzhong Li, Hangxiang Yu,
Yanfei Lei and Feng Wang. We'd like to start a discussion about introducing
Disaggregated State Storage and Management in Flink 2.0.
The past decade has witnessed a dramatic shift in Flink's deployment mode,
worklo
Hey Zakelly!
Sorry for the late reply. I still have concerns about the proposed
solution, with my main concerns coming from
the implications of the asynchronous state access API on the checkpointing
and responsiveness of Flink.
>> What also worries me a lot in this fine grained model is the effec
Hi everyone,
Piotr and I had a long discussion about the checkpoint duration issue. We
think that the lambda serialization approach I proposed in last mail may
bring in more issues, the most serious one is that users may not be able to
modify their code in serialized lambda to perform a bug fix.
Hi devs,
It seems there is no more concern or suggestion for a while. We'd like to
start voting recently.
Best,
Zakelly
On Wed, Mar 27, 2024 at 11:46 AM Zakelly Lan wrote:
> Hi everyone,
>
> Piotr and I had a long discussion about the checkpoint duration issue. We
> think that the lambda seri
Hi!
Yes, after some long offline discussions we agreed to proceed as planned
here, but we should treat the current API as experimental. The issues are
that either we can not checkpoint lambdas as they are currently defined,
leading to problems caused by in-flight records draining under backpressur
Thank Piotrek for your valuable input!
I will prepare the following FLIPs about faster checkpointing in the
current async execution model and the new APIs. And I have added some brief
description of this part in FLIP-423/424/425.
Regarding your concern:
> My main concern here, is to prevent a s
Thanks Zakelly for driving the entire discussion and making endless efforts
to improve the proposal. Thanks everyone for giving invaluable input and
suggestions.
Asycn execution is fundamental to disaggregate CPU utils and IO/network
utils. It is also widely adopted in many modern systems to impr
Hi!
Thanks for this proposal. It looks like it will be a great improvement!
I've only started reading the FLIP's, but I already have some questions
about the FLIP-425, the async execution.
What's the motivation behind splitting execution of a single element into
multiple independent steps in indi
Hi Piotr,
Thanks for your thoughtful feedback!
In fact, we had a lot of internal debates on this topic :). The proposal
you mentioned is one of several possible alternatives we've considered. It
works well for simple use cases of ValueState or ListState, where users
typically read state by partit
Thanks for your answers Zakelly. I get your points.
> (...) it may not be suitable for scenarios where
> - A state is read by condition.
> - MapState with the user key cannot be determined in advance.
I guess it depends how we would like to treat the local disks. I've always
thought about them t
Hi,
Thanks for proposing this design! I just read FLIP-424 and FLIP-425
and have some questions about the proposed changes.
For Async API (FLIP-424)
1. Why do we need a close() method on StateIterator? This method seems
unused in the usage example codes.
2. In FutureUtils.combineAll()'s JavaDoc
Thanks Piotr for sharing your thoughts!
I guess it depends how we would like to treat the local disks. I've always
> thought about them that almost always eventually all state from the DFS
> should end up cached in the local disks.
OK I got it. In our proposal we treat local disk as an optional
Hi,
Thanks for the great proposals. I have a few comments comments:
- Backpressure Handling. Flink's original backpressure handling is quite
robust and the semantics is quite "simple" (simple is beautiful).
This mechanism has proven to perform better/robust than the other open
source streaming sy
Hi Yunfeng,
Thanks for your detailed comments!
1. Why do we need a close() method on StateIterator? This method seems
> unused in the usage example codes.
The `close()` is introduced to release internal resources, but it does not
seem to require the user to call it. I removed this.
2. In Futur
Hi Jeyhun,
Thanks for your reply!
- Backpressure Handling.
I agree that Flink's original back pressure handling is simple and robust,
and in the current proposal we try our best to make the minimal change to
Mailbox for callback execution. When jumping out of the detailed execution
logic of sta
Hi Zakelly,
Thanks for the responses!
> 1. I will discuss this with some expert SQL developers. ... mode for
> StreamRecord processing.
In DataStream API there should also be use cases when the order of
output is strictly required. I agree with it that SQL experts may help
provide more concrete
Hi Yunfeng,
For 1:
I had a discussion with Lincoln Lee, and I realize it is a common case the
same-key record should be blocked before the `processElement`. It is easier
for users to understand. Thus I will introduce a strict mode for this and
make it default. My rough idea is just like yours, by
Hi Zakelly,
> 5. I'm not very sure ... revisiting this later since it is not important.
It seems that we still have some details to confirm about this
question. Let's postpone this to after the critical parts of the
design are settled.
> 8. Yes, we had considered ... metrics should be like after
Hi team,
Thanks for your discussion. Regarding FLIP-425, we have supplemented
several updates to answer high-frequency questions:
1. We captured a flame graph of the Hashmap state backend in
"Synchronous execution with asynchronous APIs"[1], which reveals that
the framework overhead (including re
Hey all!
This is a massive improvement / work. I just started going through the
Flips and have a more or less meta comment.
While it's good to keep the overall architecture discussion here, I think
we should still have separate discussions for each FLIP where we can
discuss interface details etc.
Hi,
+ 1 for the suggestion.
Maybe we can the discussion with the FLIPs with minimum dependencies (from
the other new/proposed FLIPs).
Based on our discussion on a particular FLIP, the subsequent (or its
dependent) FLIP(s) can be updated accordingly?
Regards,
Jeyhun
On Wed, Mar 6, 2024 at 5:34 PM
I have been a bit busy these few weeks and sorry for responding late.
The original thinking of keeping discussion within one thread is for easier
tracking and avoid for repeated discussion in different threads.
For details, It might be good to start in different threads if needed.
We will think
Hi everyone,
Thank you all for a lively discussion here, and it is a good time to move
forward to more detailed discussions. Thus we open several threads for
sub-FLIPs:
FLIP-424: https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864
FLIP-425: https://lists.apache.org/thread/wxn1j848fnf
+1 for Gyula's suggestion. I just finished FLIP-423 which introduced the
intention of the big change and high level architecture. Great content btw!
The only public interface change for this FLIP is one new config to use
ForSt. It makes sense to have one dedicated discussion thread for each
concret
Sorry for joining the discussion late.
I have two questions about FLIP-425.
1. Regarding Record-ordered and State-ordered of processElement.
I understand that while State-ordered likely provides better performance,
Record-ordered is sometimes required for correctness. The question is how
should
Hi Xintong,
Thanks for sharing your thoughts!
1. Regarding Record-ordered and State-ordered of processElement.
>
> I understand that while State-ordered likely provides better performance,
> Record-ordered is sometimes required for correctness. The question is how
> should a user choose between t
Thanks for the quick response. Sounds good to me.
Best,
Xintong
On Tue, Mar 19, 2024 at 1:03 PM Zakelly Lan wrote:
> Hi Xintong,
>
> Thanks for sharing your thoughts!
>
> 1. Regarding Record-ordered and State-ordered of processElement.
> >
> > I understand that while State-ordered likely pro
Hi Xintong and Zakelly,
> 2. Regarding Strictly-ordered and Out-of-order of Watermarks
I agree with it that watermarks can use only out-of-order mode for
now, because there is still not a concrete example showing the
correctness risk about it. However, the strictly-ordered mode should
still be sup
Hi Yunfeng,
Thanks for the suggestion!
I will reorganize the FLIP-425 accordingly.
Best,
Zakelly
On Tue, Mar 19, 2024 at 3:20 PM Yunfeng Zhou
wrote:
> Hi Xintong and Zakelly,
>
> > 2. Regarding Strictly-ordered and Out-of-order of Watermarks
> I agree with it that watermarks can use only out
Hi everyone,
Thanks for your valuable feedback!
Our discussions have been going on for a while and are nearing a
consensus. So I would like to start a vote after 72 hours.
Please let me know if you have any concerns, thanks!
Best,
Zakelly
On Tue, Mar 19, 2024 at 3:37 PM Zakelly Lan wrote:
>
30 matches
Mail list logo
