@Shephan
Do you mean Minicluster? Yes, it makes sense to share the GPU Manager
in such scenario.
If that's what you worry about, I'm +1 for holding
GPUManager(ExternalResourceManagers) in TaskExecutor instead of
TaskManagerServices.
Regarding the RuntimeContext/FunctionContext, it just holds the GPU
info instead of the GPU Manager. AFAIK, it's the only place we could
pass GPU info to the RichFunction/UserDefinedFunction.
Best,
Yangze Guo
On Sat, Mar 14, 2020 at 4:06 AM Isaac Godfried <is...@paddlesoft.net> wrote:
>
>
>
>
>
> ---- On Fri, 13 Mar 2020 15:58:20 +0000 se...@apache.org wrote ----
>
> > > Can we somehow keep this out of the TaskManager services
> > I fear that we could not. IMO, the GPUManager(or
> > ExternalServicesManagers in future) is conceptually one of the task
> > manager services, just like MemoryManager before 1.10.
> > - It maintains/holds the GPU resource at TM level and all of the
> > operators allocate the GPU resources from it. So, it should be
> > exclusive to a single TaskExecutor.
> > - We could add a collection called ExternalResourceManagers to hold
> > all managers of other external resources in the future.
> >
>
> Can you help me understand why this needs the addition in TaskMagerServices
> or in the RuntimeContext?
> Are you worried about the case when multiple Task Executors run in the same
> JVM? That's not common, but wouldn't it actually be good in that case to
> share the GPU Manager, given that the GPU is shared?
>
> Thanks,
> Stephan
>
> ---------------------------
>
>
> > What parts need information about this?
> > In this FLIP, operators need the information. Thus, we expose GPU
> > information to the RuntimeContext/FunctionContext. The slot profile is
> > not aware of GPU resources as GPU is TM level resource now.
> >
> > > Can the GPU Manager be a "self contained" thing that simply takes the
> > configuration, and then abstracts everything internally?
> > Yes, we just pass the path/args of the discover script and how many
> > GPUs per TM to it. It takes the responsibility to get the GPU
> > information and expose them to the RuntimeContext/FunctionContext of
> > Operators. Meanwhile, we'd better not allow operators to directly
> > access GPUManager, it should get what they want from Context. We could
> > then decouple the interface/implementation of GPUManager and Public
> > API.
> >
> > Best,
> > Yangze Guo
> >
> > On Fri, Mar 13, 2020 at 7:26 PM Stephan Ewen <se...@apache.org> wrote:
> > >
> > > It sounds fine to initially start with GPU specific support and think
> > about
> > > generalizing this once we better understand the space.
> > >
> > > About the implementation suggested in FLIP-108:
> > > - Can we somehow keep this out of the TaskManager services? Anything we
> > > have to pull through all layers of the TM makes the TM components yet
> > more
> > > complex and harder to maintain.
> > >
> > > - What parts need information about this?
> > > -> do the slot profiles need information about the GPU?
> > > -> Can the GPU Manager be a "self contained" thing that simply takes
> > > the configuration, and then abstracts everything internally? Operators
> > can
> > > access it via "GPUManager.get()" or so?
> > >
> > >
> > >
> > > On Wed, Mar 4, 2020 at 4:19 AM Yangze Guo <karma...@gmail.com> wrote:
> > >
> > > > Thanks for all the feedbacks.
> > > >
> > > > @Becket
> > > > Regarding the WebUI and GPUInfo, you're right, I'll add them to the
> > > > Public API section.
> > > >
> > > >
> > > > @Stephan @Becket
> > > > Regarding the general extended resource mechanism, I second Xintong's
> > > > suggestion.
> > > > - It's better to leverage ResourceProfile and ResourceSpec after we
> > > > supporting fine-grained GPU scheduling. As a first step proposal, I
> > > > prefer to not include it in the scope of this FLIP.
> > > > - Regarding the "Extended Resource Manager", if I understand
> > > > correctly, it just a code refactoring atm, we could extract the
> > > > open/close/allocateExtendResources of GPUManager to that interface. If
> > > > that is the case, +1 to do it during implementation.
> > > >
> > > > @Xingbo
> > > > As Xintong said, we looked into how Spark supports a general "Custom
> > > > Resource Scheduling" before and decided to introduce a common resource
> > > > configuration
> > > > schema(taskmanager.resource.{resourceName}.amount/discovery-script)
> > > > to make it more extensible. I think the "resource" is a proper level
> > > > to contain all the configs of extended resources.
> > > >
> > > > Best,
> > > > Yangze Guo
> > > >
> > > > On Wed, Mar 4, 2020 at 10:48 AM Xingbo Huang <hxbks...@gmail.com>
> > wrote:
> > > > >
> > > > > Thanks a lot for the FLIP, Yangze.
> > > > >
> > > > > There is no doubt that GPU resource management support will greatly
> > > > > facilitate the development of AI-related applications by PyFlink
> > users.
> > > > >
> > > > > I have only one comment about this wiki:
> > > > >
> > > > > Regarding the names of several GPU configurations, I think it is
> > better
> > > > to
> > > > > delete the resource field makes it consistent with the names of other
> > > > > resource-related configurations in TaskManagerOption.
> > > > >
> > > > > e.g. taskmanager.resource.gpu.discovery-script.path ->
> > > > > taskmanager.gpu.discovery-script.path
> > > > >
> > > > > Best,
> > > > >
> > > > > Xingbo
> > > > >
> > > > >
> > > > > Xintong Song <tonysong...@gmail.com> 于2020年3月4日周三 上午10:39写道：
> > > > >
> > > > > > @Stephan, @Becket,
> > > > > >
> > > > > > Actually, Yangze, Yang and I also had an offline discussion about
> > > > making
> > > > > > the "GPU Support" as some general "Extended Resource Support". We
> > > > believe
> > > > > > supporting extended resources in a general mechanism is definitely
> > a
> > > > good
> > > > > > and extensible way. The reason we propose this FLIP narrowing its
> > scope
> > > > > > down to GPU alone, is mainly for the concern on extra efforts and
> > > > review
> > > > > > capacity needed for a general mechanism.
> > > > > >
> > > > > > To come up with a well design on a general extended resource
> > management
> > > > > > mechanism, we would need to investigate more on how people use
> > > > different
> > > > > > kind of resources in practice. For GPU, we learnt such knowledge
> > from
> > > > the
> > > > > > experts, Becket and his team members. But for FPGA, or other
> > potential
> > > > > > extended resources, we don't have such convenient information
> > sources,
> > > > > > making the investigation requires more efforts, which I tend to
> > think
> > > > is
> > > > > > not necessary atm.
> > > > > >
> > > > > > On the other hand, we also looked into how Spark supports a general
> > > > "Custom
> > > > > > Resource Scheduling". Assuming we want to have a similar general
> > > > extended
> > > > > > resource mechanism in the future, we believe that the current GPU
> > > > support
> > > > > > design can be easily extended, in an incremental way without too
> > many
> > > > > > reworks.
> > > > > >
> > > > > > - The most important part is probably user interfaces. Spark
> > offers
> > > > > > configuration options to define the amount, discovery script and
> > > > vendor
> > > > > > (on
> > > > > > k8s) in a per resource type bias [1], which is very similar to
> > what
> > > > we
> > > > > > proposed in this FLIP. I think it's not necessary to expose
> > config
> > > > > > options
> > > > > > in the general way atm, since we do not have supports for other
> > > > resource
> > > > > > types now. If later we decided to have per resource type config
> > > > > > options, we
> > > > > > can have backwards compatibility on the current proposed options
> > > > with
> > > > > > simple key mapping.
> > > > > > - For the GPU Manager, if later needed we can change it to a
> > > > "Extended
> > > > > > Resource Manager" (or whatever it is called). That should be a
> > pure
> > > > > > component-internal refactoring.
> > > > > > - For ResourceProfile and ResourceSpec, there are already
> > fields for
> > > > > > general extended resource. We can of course leverage them when
> > > > > > supporting
> > > > > > fine grained GPU scheduling. That is also not in the scope of
> > this
> > > > first
> > > > > > step proposal, and would require FLIP-56 to be finished first.
> > > > > >
> > > > > > To summary up, I agree with Becket that have a separate FLIP for
> > the
> > > > > > general extended resource mechanism, and keep it in mind when
> > > > discussing
> > > > > > and implementing the current one.
> > > > > >
> > > > > > Thank you~
> > > > > >
> > > > > > Xintong Song
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > >
> > https://spark.apache.org/docs/3.0.0-preview/configuration.html#custom-resource-scheduling-and-configuration-overview
> > > > > >
> > > > > > On Wed, Mar 4, 2020 at 9:18 AM Becket Qin <becket....@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > That's a good point, Stephan. It makes total sense to generalize
> > the
> > > > > > > resource management to support custom resources. Having that
> > allows
> > > > users
> > > > > > > to add new resources by themselves. The general resource
> > management
> > > > may
> > > > > > > involve two different aspects:
> > > > > > >
> > > > > > > 1. The custom resource type definition. It is supported by the
> > > > extended
> > > > > > > resources in ResourceProfile and ResourceSpec. This will likely
> > cover
> > > > > > > majority of the cases.
> > > > > > >
> > > > > > > 2. The custom resource allocation logic, i.e. how to assign the
> > > > resources
> > > > > > > to different tasks, operators, and so on. This may require two
> > > > levels /
> > > > > > > steps:
> > > > > > > a. Subtask level - make sure the subtasks are put into
> > suitable
> > > > > > slots.
> > > > > > > It is done by the global RM and is not customizable right now.
> > > > > > > b. Operator level - map the exact resource to the operators
> > in
> > > > TM.
> > > > > > e.g.
> > > > > > > GPU 1 for operator A, GPU 2 for operator B. This step is needed
> > > > assuming
> > > > > > > the global RM does not distinguish individual resources of the
> > same
> > > > type.
> > > > > > > It is true for memory, but not for GPU.
> > > > > > >
> > > > > > > The GPU manager is designed to do 2.b here. So it should
> > discover the
> > > > > > > physical GPU information and bind/match them to each operators.
> > > > Making
> > > > > > this
> > > > > > > general will fill in the missing piece to support custom resource
> > > > type
> > > > > > > definition. But I'd avoid calling it a "External Resource
> > Manager" to
> > > > > > avoid
> > > > > > > confusion with RM, maybe something like "Operator Resource
> > Assigner"
> > > > > > would
> > > > > > > be more accurate. So for each resource type users can have an
> > > > optional
> > > > > > > "Operator Resource Assigner" in the TM. For memory, users don't
> > need
> > > > > > this,
> > > > > > > but for other extended resources, users may need that.
> > > > > > >
> > > > > > > Personally I think a pluggable "Operator Resource Assigner" is
> > > > achievable
> > > > > > > in this FLIP. But I am also OK with having that in a separate
> > FLIP
> > > > > > because
> > > > > > > the interface between the "Operator Resource Assigner" and
> > operator
> > > > may
> > > > > > > take a while to settle down if we want to make it generic. But I
> > > > think
> > > > > > our
> > > > > > > implementation should take this future work into consideration so
> > > > that we
> > > > > > > don't need to break backwards compatibility once we have that.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Wed, Mar 4, 2020 at 12:27 AM Stephan Ewen <se...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > Thank you for writing this FLIP.
> > > > > > > >
> > > > > > > > I cannot really give much input into the mechanics of GPU-aware
> > > > > > > scheduling
> > > > > > > > and GPU allocation, as I have no experience with that.
> > > > > > > >
> > > > > > > > One thought I had when reading the proposal is if it makes
> > sense to
> > > > > > look
> > > > > > > at
> > > > > > > > the "GPU Manager" as an "External Resource Manager", and GPU
> > is one
> > > > > > such
> > > > > > > > resource.
> > > > > > > > The way I understand the ResourceProfile and ResourceSpec,
> > that is
> > > > how
> > > > > > it
> > > > > > > > is done there.
> > > > > > > > It has the advantage that it looks more extensible. Maybe
> > there is
> > > > a
> > > > > > GPU
> > > > > > > > Resource, a specialized NVIDIA GPU Resource, and FPGA
> > Resource, a
> > > > > > Alibaba
> > > > > > > > TPU Resource, etc.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stephan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 3, 2020 at 7:57 AM Becket Qin <
> > becket....@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks for the FLIP Yangze. GPU resource management support
> > is a
> > > > > > > > must-have
> > > > > > > > > for machine learning use cases. Actually it is one of the
> > mostly
> > > > > > asked
> > > > > > > > > question from the users who are interested in using Flink
> > for ML.
> > > > > > > > >
> > > > > > > > > Some quick comments / questions to the wiki.
> > > > > > > > > 1. The WebUI / REST API should probably also be mentioned in
> > the
> > > > > > public
> > > > > > > > > interface section.
> > > > > > > > > 2. Is the data structure that holds GPU info also a public
> > API?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Tue, Mar 3, 2020 at 10:15 AM Xintong Song <
> > > > tonysong...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for drafting the FLIP and kicking off the
> > discussion,
> > > > > > Yangze.
> > > > > > > > > >
> > > > > > > > > > Big +1 for this feature. Supporting using of GPU in Flink
> > is
> > > > > > > > significant,
> > > > > > > > > > especially for the ML scenarios.
> > > > > > > > > > I've reviewed the FLIP wiki doc and it looks good to me. I
> > > > think
> > > > > > > it's a
> > > > > > > > > > very good first step for Flink's GPU supports.
> > > > > > > > > >
> > > > > > > > > > Thank you~
> > > > > > > > > >
> > > > > > > > > > Xintong Song
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 2, 2020 at 12:06 PM Yangze Guo <
> > karma...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > We would like to start a discussion thread on "FLIP-108:
> > Add
> > > > GPU
> > > > > > > > > > > support in Flink"[1].
> > > > > > > > > > >
> > > > > > > > > > > This FLIP mainly discusses the following issues:
> > > > > > > > > > >
> > > > > > > > > > > - Enable user to configure how many GPUs in a task
> > executor
> > > > and
> > > > > > > > > > > forward such requirements to the external resource
> > managers
> > > > (for
> > > > > > > > > > > Kubernetes/Yarn/Mesos setups).
> > > > > > > > > > > - Provide information of available GPU resources to
> > > > operators.
> > > > > > > > > > >
> > > > > > > > > > > Key changes proposed in the FLIP are as follows:
> > > > > > > > > > >
> > > > > > > > > > > - Forward GPU resource requirements to Yarn/Kubernetes.
> > > > > > > > > > > - Introduce GPUManager as one of the task manager
> > services to
> > > > > > > > discover
> > > > > > > > > > > and expose GPU resource information to the context of
> > > > functions.
> > > > > > > > > > > - Introduce the default script for GPU discovery, in
> > which we
> > > > > > > provide
> > > > > > > > > > > the privilege mode to help user to achieve worker-level
> > > > isolation
> > > > > > > in
> > > > > > > > > > > standalone mode.
> > > > > > > > > > >
> > > > > > > > > > > Please find more details in the FLIP wiki document [1].
> > > > Looking
> > > > > > > > forward
> > > > > > > > > > to
> > > > > > > > > > > your feedbacks.
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-108%3A+Add+GPU+support+in+Flink
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Yangze Guo
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
>
>
