Thanks Chesnay & Zhu.
I will start a new vote thread soon.
Best,
Lijie
Chesnay Schepler 于2022年6月15日周三 15:49写道：
> To expand a bit for transparency, Zhu Zhu and I had long discussion
> (literally spanning days) about this FLIP and it's relation to
> speculative execution.
>
> The gist of it is
To expand a bit for transparency, Zhu Zhu and I had long discussion
(literally spanning days) about this FLIP and it's relation to
speculative execution.
The gist of it is that speculative execution doesn't strictly need the
block list; just _some_ mechanism to select/request slots from other
Hi everyone,
Thank you all for the feedback!
We receive concerns that the blocklist feature is not strongly
required except for the needs of speculative execution. So as the
first step, we will limit the scope of FLIP-224 to only support
speculative execution, and therefore, not add public
1) With the declarative slot allocation protocol, it's not easy to
keep the slow slots which may have satisfy the slot request already,
but ask for more slots from the resource manager.
And we are also concerned to use detected slow slots, which may add
load to the slow nodes and further slow down
1)
It's true that if we handle this entirely in the scheduler we may get a
bunch of slow slots from the RM.
My point is that this isn't necessarily a problem; it depends on how you
handle those slots.
We anyway need to account for the possibility that we're not getting any
new fast slots
Thanks Chesnay for the feedback in the vote thread.
(https://lists.apache.org/thread/opc7jg3rpxnwotkb0fcn4wnm02m4397o)
I'd like to continue the discussion in this thread so that the
discussions can be better tracked.
Regarding your questions, here are my thoughts:
1. The current locality
Hi Chesnay,
Would you please take a look at the FLIP and discussion to see if all
your concerns have been addressed?
Thanks,
Zhu
Zhu Zhu 于2022年5月28日周六 13:26写道：
>
> Regarding the concern of the SlotManager, my two cents:
> 1. it is necessary for the SlotManager to host blocked slots, in 2
Regarding the concern of the SlotManager, my two cents:
1. it is necessary for the SlotManager to host blocked slots, in 2 cases:
a. In standalone mode, a taskmanager may be temporarily added to
the blocklist. We do not want the TM to get disconnected and shut down.
So we need to keep its
Hi everyone,
I've updated the FLIP according to Chesnay's feedback, changes as follows:
1. Change the GET result to a map.
2. Only left *endTimestamp* in ADD operation, and change the rest (from
POST) to PUT
3. Introduce a new slot pool implementation(BlocklistSlotPool) to
encapsulate blocklist
Hi Chesnay,
Thanks for feedback.
1. Regarding the TM/Node id. Do you mean special characters may appear in
the rest URL? Actually, I don't think so. The task manager id in REST API
should be the *ResourceID* of taskmanager in Flink, there should be no
special characters, and some existing REST
I have a number of concerns:
Is the id used for deleting an item the same sent in the initial request
(and not one returned by Flink)?
I'm very concerned that the tm/node id can contain special characters.
The GET query should return a map, not a list of items. This makes it
easier to work
Hi everyone,
I have started a vote for this FLIP [1]. Please cast your vote there or ask
additional questions here. [1]
https://lists.apache.org/thread/3416vks1j35co9608gkmsoplvcjjz7bg
Best, Lijie
Lijie Wang 于2022年5月19日周四 17:34写道：
> Hi Konstantin,
>
> We found that Flink REST URL does not
Hi Konstantin,
We found that Flink REST URL does not support the format ":merge" , which
will be recognized as a parameter in the URL(due to start with a colon).
We will keep the previous way, i.e.
POST: http://{jm_rest_address:port}/blocklist/taskmanagers
and the "id" and "merge" flag are put
Hi Weihua,
thanks for feedback.
1. Yes, only *Manually* is supported in this FLIP, but it's the first step
towards auto-detection.
2. We wii print the blocked nodes in logs. Maybe also put it into the
exception of insufficient resources.
3. No. This FLIP won't change the WebUI. The blocklist
Hi,
Thanks for creating this FLIP.
We have implemented an automatic blocklist detection mechanism internally,
which is indeed very effective for handling node failures.
Due to the large number of nodes, although SREs already support automatic
offline failure nodes, the detection is not 100%
Hi Konstantin,
Maybe change it to the following:
1. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}
Merge is not allowed. If the {id} already exists, return error. Otherwise,
create a new item.
2. POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge
Merge is
Hi Lijie,
hm, maybe the following is more appropriate in that case
POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge
Best,
Konstantin
Am Mo., 16. Mai 2022 um 07:05 Uhr schrieb Lijie Wang <
wangdachui9...@gmail.com>:
> Hi Konstantin,
> thanks for your feedback.
>
> From
Hi Konstantin,
thanks for your feedback.
>From what I understand, PUT should be idempotent. However, we have a
*timeout* field in the request. This means that initiating the same request
at two different times will lead to different resource status (timestamps
of the items to be removed will be
Hi Lijie,
wouldn't the REST API-idiomatic way for an update/replace be a PUT on the
resource?
PUT: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}
Best,
Konstantin
Am Fr., 13. Mai 2022 um 11:01 Uhr schrieb Lijie Wang <
wangdachui9...@gmail.com>:
> Hi everyone,
>
> I've had an
Hi everyone,
I've had an offline discussion with Becket Qin and Zhu Zhu, and made the
following changes on REST API:
1. To avoid ambiguity, *timeout* and *endTimestamp* can only choose one. If
both are specified, will return error.
2. If the specified item is already there, the *ADD* operation
Hi Becket Qin,
Thanks for your suggestions. I have moved the description of
configurations, metrics and REST API into "Public Interface" section, and
made a few updates according to your suggestion. And in this FLIP, there
no public java Interfaces or pluggables that users need to implement by
Hi Lijie,
Thanks for updating the FLIP. It looks like the public interface section
did not fully reflect all the user sensible behavior and API. Can you put
everything that users may be aware of there? That would include the REST
API, metrics, configurations, public java Interfaces or pluggables
Hi everyone,
Based on the discussion in the mailing list, I updated the FLIP doc, the
changes include:
1. Changed the description of the motivation section to more clearly
describe the problem this FLIP is trying to solve.
2. Only *Manually* is supported.
3. Adopted some suggestions, such as
Hi Lijie!
*a) “Probably storing inside Zookeeper/Configmap might be helpfulhere.”
Can you explain it in detail? I don't fully understand that. In myopinion,
non-active and active are the same, and no special treatment isrequired.*
Sorry this was a misunderstanding from my side. I thought we
Thanks Lijie and ZhuZhu for the explanation.
I just overlooked the "MARK_BLOCKLISTED". For tasks level, it is indeed
some functionalities the external tools(e.g. kubectl taint) could not
support.
Best,
Yang
Lijie Wang 于2022年5月6日周五 22:18写道：
> Thanks for your feedback, Jiangang and Martijn.
>
Thanks for your feedback, Jiangang and Martijn.
@Jiangang
> For auto-detecting, I wonder how to make the strategy and mark a node
blocked?
In fact, we currently plan to not support auto-detection in this FLIP. The
part about auto-detection may be continued in a separate FLIP in the
future.
> If we only support to block nodes manually, then I could not see
the obvious advantages compared with current SRE's approach(via *yarn
rmadmin or kubectl taint*).
I agree with Yang Wang on this.
> To me this sounds yet again like one of those magical mechanisms that
will rarely work just
Thanks for the valuable design. The auto-detecting can decrease great work
for us. We have implemented the similar feature in our inner flink version.
Below is something that I care about:
1. For auto-detecting, I wonder how to make the strategy and mark a node
blocked? Sometimes the
Thank you for all your feedback!
Besides the answers from Lijie, I'd like to share some of my thoughts:
1. Whether to enable automatical blocklist
Generally speaking, it is not a goal of FLIP-224.
The automatical way should be something built upon the blocklist
mechanism and well decoupled. It
Hi everyone,
Thanks for your feedback.
There's one detail that I'd like to re-emphasize here because it can affect
the value and design of the blocklist mechanism (perhaps I should highlight
it in the FLIP). We propose two actions in FLIP:
1) MARK_BLOCKLISTED: Just mark the task manager or
Thanks Lijie and Zhu for creating the proposal.
I want to share some thoughts about Flink cluster operations.
In the production environment, the SRE(aka Site Reliability Engineer)
already has many tools to detect the unstable nodes, which could take the
system logs/metrics into consideration.
Thanks for the proposal, Lijie.
This is an interesting feature and discussion, and somewhat related to the
design principle about how people should operate Flink.
I think there are three things involved in this FLIP.
a) Detect and report the unstable node.
b) Collect the information of
Thanks for good initiative, Lijie and Zhu!
If it's possible I'd like to participate in development.
I agree with 3rd point of Konstantin's reply - we should consider to move
somehow the information of blocklisted nodes/TMs from active
ResourceManager to non-active ones. Probably storing inside
I do share the concern between blurring the lines a bit.
That said, I'd prefer to not have any auto-detection and only have an
opt-in mechanism
to manually block processes/nodes. To me this sounds yet again like one
of those
magical mechanisms that will rarely work just right.
An external
Hi everyone,
Thanks for creating this FLIP. I can understand the problem and I see value
in the automatic detection and blocklisting. I do have some concerns with
the ability to manually specify to be blocked resources. I have two
concerns;
* Most organizations explicitly have a separation of
Thanks for driving this, Zhu and Lijie.
+1 for the overall proposal. Just share some cents here:
- Why do we need to expose
cluster.resource-blacklist.item.timeout-check-interval to the user?
I think the semantics of `cluster.resource-blacklist.item.timeout` is
sufficient for the user. How to
Hi Konstantin,
Thanks for your feedback. I will response your 4 remarks:
1) Thanks for reminding me of the controversy. I think “BlockList” is good
enough, and I will change it in FLIP.
2) Your suggestion for the REST API is a good idea. Based on the above, I
would change REST API as
Hi Lijie,
I think, this makes sense and +1 to only support manually blocking
taskmanagers and nodes. Maybe the different strategies can also be
maintained outside of Apache Flink.
A few remarks:
1) Can we use another term than "bla.cklist" due to the controversy around
the term? [1] There was
Hi all,
Flink job failures may happen due to cluster node issues (insufficient disk
space, bad hardware, network abnormalities). Flink will take care of the
failures and redeploy the tasks. However, due to data locality and limited
resources, the new tasks are very likely to be redeployed to the
39 matches
Mail list logo
