FYI, I updated the FLIP accordingly. To sum up, Flink will throw an
exception and tell user to configure an internal
"fine-grained.shuffle-mode.all-blocking" to be true in this scenario.
Best,
Yangze Guo
On Tue, Jun 22, 2021 at 2:20 PM Yangze Guo wrote:
>
> Thanks for the comment, Xintong.
>
> I
Thanks for the comment, Xintong.
I used to wonder if it was reasonable or worthwhile to introduce a
configuration like "table.exec.shuffle-mode" for DataStream API.
Narrow down the scope of effect sounds good to me.
Best,
Yangze Guo
On Tue, Jun 22, 2021 at 2:08 PM Xintong Song wrote:
>
> I seco
I second Zhu and Till's opinion.
Failing with an exception that also includes how to resolve the problem
sounds better, in terms of making it explicit to users that pipelined edges
are replaced with blocking edges.
Concerning absence of knobs tuning the edge types, we can introduce a
configuratio
Thanks for the feedback, Till!
Actually, we cannot give user any resolution for this issue as there
is no API for DataStream users to influence the edge types at the
moment. The edge types are currently fixed based on the jobs' mode
(batch or streaming).
a) I think it might not confuse the user a
I would be more in favor of what Zhu Zhu proposed to throw an exception
with a meaningful and understandable explanation that also includes how to
resolve this problem. I do understand the reasoning behind automatically
switching the edge types in order to make things easier to use but a) this
can
Thanks, I append it to the known limitations of this FLIP.
Best,
Yangze Guo
On Mon, Jun 21, 2021 at 3:20 PM Zhu Zhu wrote:
>
> Thanks for the quick response Yangze.
> The proposal sounds good to me.
>
> Thanks,
> Zhu
>
> Yangze Guo 于2021年6月21日周一 下午3:01写道：
>>
>> Thanks for the comments, Zhu!
>>
Thanks for the quick response Yangze.
The proposal sounds good to me.
Thanks,
Zhu
Yangze Guo 于2021年6月21日周一 下午3:01写道：
> Thanks for the comments, Zhu!
>
> Yes, it is a known limitation for fine-grained resource management. We
> also have filed this issue in FLINK-20865 when we proposed FLIP-156.
Thanks for the comments, Zhu!
Yes, it is a known limitation for fine-grained resource management. We
also have filed this issue in FLINK-20865 when we proposed FLIP-156.
As a first step, I agree that we can mark batch jobs with PIPELINED
edges as an invalid case for this feature. However, just th
Thanks for proposing this @Yangze Guo and sorry for
joining the discussion so late.
The proposal generally looks good to me. But I find one problem that batch
job with PIPELINED edges might hang if enabling fine-grained resources. see
"Resource Deadlocks could still happen in certain Cases" sectio
Thanks for the supplement, Arvid and Yun. I've annotated these two
points in the FLIP.
The vote is now started in [1].
[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-169-DataStream-API-for-Fine-Grained-Resource-Requirements-td51381.html
Best,
Yangze Guo
On Fri, Jun
Hi,
Very thanks @Yangze for bringing up this discuss. Overall +1 for
exposing the fine-grained resource requirements in the DataStream API.
One similar issue as Arvid has pointed out is that users may also creating
different SlotSharingGroup objects, with different names but with different
reso
Hi Yangze,
Thanks for incorporating the ideas and sorry for missing the builder part.
My main idea is that SlotSharingGroup is immutable, such that the user
doesn't do:
ssg = new SlotSharingGroup();
ssg.setCpus(2);
operator1.slotSharingGroup(ssg);
ssg.setCpus(4);
operator2.slotSharingGroup(ssg);
Thanks all for the discussion. I've updated the FLIP accordingly, the
key changes are:
- Introduce SlotSharingGroup instead of ResourceSpec which contains
the resource spec of slot sharing group
- Introduce two interfaces for specifying the SlotSharingGroup:
#slotSharingGroup(SlotSharingGroup) and
Thanks for the valuable suggestion, Arvid.
1) Yes, we can add a new SlotSharingGroup which includes the name and
its resource. After that, we have two interfaces for configuring the
slot sharing group of an operator:
- #slotSharingGroup(String name)// the resource of it can be
configured throu
Hi Yangze,
I like the general approach to bind requirements to slotsharing groups. I
think the current approach is also flexible enough that a user could simply
use ParameterTool or similar to use config values and wire that with their
slotgroups, such that different requirements can be tested wit
@Yang
In short, the external resources will participate in resource
deduction and be logically ensured, but requesting an external
resource must still be done through config options with the current
default resource allocation strategy.
In FLIP-56, we abstract the logic of resource allocation to th
Thanks @Yangze for preparing this FLIP.
I think this is a good start point for the community users to have a taste
on the fine-grained
resource management, which we all believe it could improve the Flink job
stability and
cluster utilization.
I have a simple question about the extended resources.
@Xintong
> introduce a general approach for overwriting such job specifics without
> re-compiling the job
I think that would be a good direction. Just share some cents on this
topic. I'd divide the job-level specifics into two categories:
- Specifics which affect how Flink executes the job, e.g.
"
I think being able to specify fine grained resource requirements without
having to change the codes and recompile the job is indeed a good idea. It
definitely improves the usability.
However, this requires more careful designs, which probably deserves a
separate thread. I'd be good to have that di
@Wenlong
After another consideration, the config option approach I mentioned
above might not be appropriate. The resource requirements for SSG
should be a job level configuration and should no be set in the
flink-conf.
I think we can define a JSON format, which would be the ResourceSpecs
mapped by
Thanks for the feedbacks, Xintong and Wenlong!
@Wenlong
I think that is a good idea, adjust the resource without re-compiling
the job will facilitate the tuning process.
We can define a pattern "slot-sharing-group.resource.{ssg name}"
(welcome any proposal for the prefix naming) for the resource s
Thanks Yangze for the flip, it is great for users to be able to declare the
fine-grained resource requirements for the job.
I have one minor suggestion: can we support setting resource requirements
by configuration? Currently most of the config options in execution config
can be configured by conf
Thanks Yangze for preparing the FLIP.
The proposed changes look good to me.
As you've mentioned in the implementation plan, I believe one of the most
important tasks of this FLIP is to have the feature well documented. It
would be really nice if we can keep that in mind and start drafting the
doc
23 matches
Mail list logo
